diff --git a/README.md b/README.md
index 5074df07a..d824a9a2e 100644
--- a/README.md
+++ b/README.md
@@ -1,8 +1,9 @@
# Amazon SageMaker Debugger
- [Overview](#overview)
-- [Examples](#sagemaker-example)
+- [Examples](#examples)
- [How It Works](#how-it-works)
+- [Docs](#docs)
## Overview
Amazon SageMaker Debugger is an offering from AWS which help you automate the debugging of machine learning training jobs.
@@ -15,6 +16,7 @@ It supports TensorFlow, PyTorch, MXNet, and XGBoost on Python 3.6+.
- Real-time training job monitoring through Rules
- Automated anomaly detection and state assertions
- Interactive exploration of saved tensors
+- Actions on your training jobs based on the status of Rules
- Distributed training support
- TensorBoard support
@@ -51,6 +53,12 @@ sagemaker_simple_estimator = sm.tensorflow.TensorFlow(
)
sagemaker_simple_estimator.fit()
+tensors_path = sagemaker_simple_estimator.latest_job_debugger_artifacts_path()
+
+import smdebug as smd
+trial = smd.trials.create_trial(out_dir=tensors_path)
+print(f"Saved these tensors: {trial.tensor_names()}")
+print(f"Loss values during evaluation were {trial.tensor('CrossEntropyLoss:0').values(mode=smd.modes.EVAL)}")
```
That's it! Amazon SageMaker will automatically monitor your training job for you with the Rules specified and create a CloudWatch
@@ -101,12 +109,15 @@ Amazon SageMaker Debugger can be used inside or outside of SageMaker. There are
The reason for different setups is that SageMaker Zero-Script-Change (via Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost to save tensors automatically.
These framework forks are not available in custom containers or non-SM environments, so you must modify your training script in these environments.
-See the [SageMaker page](docs/sagemaker.md) for details on SageMaker Zero-Code-Change and Bring-Your-Own-Container (BYOC) experience.\
-See the frameworks pages for details on modifying the training script:
-- [TensorFlow](docs/tensorflow.md)
-- [PyTorch](docs/pytorch.md)
-- [MXNet](docs/mxnet.md)
-- [XGBoost](docs/xgboost.md)
+## Docs
+
+| Section | Description |
+| --- | --- |
+| [SageMaker Training](docs/sagemaker.md) | SageMaker users, we recommend you start with this page on how to run SageMaker training jobs with SageMaker Debugger |
+| Frameworks
[TensorFlow](docs/tensorflow.md)
[PyTorch](docs/pytorch.md)
[MXNet](docs/mxnet.md)
[XGBoost](docs/xgboost.md)
| See the frameworks pages for details on what's supported and how to modify your training script if applicable |
+| [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by our APIs which allows you to perform interactive exploration of tensors saved as well as to write your own Rules monitoring your training jobs. |
+| [APIs](docs/api.md) | Full description of our APIs |
+
## License
This library is licensed under the Apache 2.0 License.
diff --git a/docs/sagemaker.md b/docs/sagemaker.md
index 094f9bdd5..aa5bce967 100644
--- a/docs/sagemaker.md
+++ b/docs/sagemaker.md
@@ -29,7 +29,7 @@ Here's a list of frameworks and versions which support this experience.
| [TensorFlow](tensorflow.md) | 1.15 |
| [MXNet](mxnet.md) | 1.6 |
| [PyTorch](pytorch.md) | 1.3 |
-| [XGBoost](xgboost.md) | |
+| [XGBoost](xgboost.md) | >=0.90-2 [As Built-in algorithm](xgboost.md#use-xgboost-as-a-built-in-algorithm)|
More details for the deep learning frameworks on which containers these are can be found here: [SageMaker Framework Containers](https://docs.aws.amazon.com/sagemaker/latest/dg/pre-built-containers-frameworks-deep-learning.html) and [AWS Deep Learning Containers](https://aws.amazon.com/machine-learning/containers/). You do not have to specify any training container image if you want to use them on SageMaker. You only need to specify the version above to use these containers.
@@ -43,7 +43,7 @@ This library `smdebug` itself supports versions other than the ones listed above
| Keras (with TensorFlow backend) | 2.3 |
| [MXNet](mxnet.md) | 1.4, 1.5, 1.6 |
| [PyTorch](pytorch.md) | 1.2, 1.3 |
-| [XGBoost](xgboost.md) | |
+| [XGBoost](xgboost.md) | [As Framework](xgboost.md#use-xgboost-as-a-framework) |
#### Setting up SageMaker Debugger with your script on your container
@@ -189,7 +189,7 @@ The Built-in Rules, or SageMaker Rules, are described in detail on [this page](h
Scope of Validity | Rules |
|---|---|
| Generic Deep Learning models (TensorFlow, Apache MXNet, and PyTorch) |
|
diff --git a/docs/xgboost.md b/docs/xgboost.md
index 35c63ba03..14a8220bc 100644
--- a/docs/xgboost.md
+++ b/docs/xgboost.md
@@ -10,9 +10,9 @@
### Use XGBoost as a built-in algorithm
The XGBoost algorithm can be used 1) as a built-in algorithm, or 2) as a framework such as MXNet, PyTorch, or Tensorflow.
-If SageMaker XGBoost is used as a built-in algorithm in container verision `0.90-2` or later, Amazon SageMaker Debugger will be available by default (i.e., zero code change experience).
+If SageMaker XGBoost is used as a built-in algorithm in container version `0.90-2` or later, Amazon SageMaker Debugger will be available by default (i.e., zero code change experience).
See [XGBoost Algorithm AWS docmentation](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html) for more information on how to use XGBoost as a built-in algorithm.
-See [Amazon SageMaker Debugger examples](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger) for sample notebooks that demonstrate debugging and monitoring capabilities of Aamazon SageMaker Debugger.
+See [Amazon SageMaker Debugger examples](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger) for sample notebooks that demonstrate debugging and monitoring capabilities of Amazon SageMaker Debugger.
See [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/) for more information on how to configure the Amazon SageMaker Debugger from the Python SDK.
### Use XGBoost as a framework
diff --git a/examples/mxnet/README.md b/examples/mxnet/README.md
new file mode 100644
index 000000000..9818d371a
--- /dev/null
+++ b/examples/mxnet/README.md
@@ -0,0 +1,2 @@
+## Example Notebooks
+Please refer to the example notebooks in [Amazon SageMaker Examples repository](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger)
diff --git a/examples/mxnet/notebooks/MNISTSimpleInteractiveAnalysis.ipynb b/examples/mxnet/notebooks/MNISTSimpleInteractiveAnalysis.ipynb
deleted file mode 100644
index 67a818ebe..000000000
--- a/examples/mxnet/notebooks/MNISTSimpleInteractiveAnalysis.ipynb
+++ /dev/null
@@ -1,652 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Simple Interactive Analysis in Tornasole\n",
- "This notebook will demonstrate the simplest kind of interactive analysis that can be run in smdebug. It will focus on the [vanishing/exploding gradient](https://medium.com/learn-love-ai/the-curious-case-of-the-vanishing-exploding-gradient-bf58ec6822eb) problems on a simple MNIST digit recognition."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Some basic setup that's always helpful"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Make sure that MXNet is accessible! If you are on the EC2 Deep Learning AMI, you will probably want\n",
- "to activate the right MXNet environment\n",
- "```\n",
- "sh> source activate mxnet_p36\n",
- "```\n",
- "You'll probably have to restart this notebook after doing this"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's import some basic libraries for ML"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import numpy as np\n",
- "import mxnet as mx\n",
- "from mxnet import gluon, autograd\n",
- "from mxnet.gluon import nn\n",
- "import matplotlib.pyplot as plt"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's copy the Tornasole libraries to this instance, this step has to be executed only once. \n",
- "Please make sure that the AWS account you are using can access the `tornasole-external-preview-use1` bucket.\n",
- "\n",
- "To do so you'll need the appropriate AWS credentials. There are several ways of doing this:\n",
- "- inject temporary credentials \n",
- "- if running on EC2, use [EC2 roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html) that can access all S3 buckets\n",
- "- (preferred) run this notebook on a [SageMaker notebook instance](https://docs.aws.amazon.com/sagemaker/latest/dg/nbi.html)\n",
- "\n",
- "The code below downloads the necessary `.whl` files and installs them in the current environment. Only run the first time!\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "#WARNING - uncomment this code only if you haven't done this before\n",
- "#!aws s3 sync s3://tornasole-external-preview-use1/sdk/ts-binaries/tornasole_mxnet/py3/latest/ tornasole_mxnet/\n",
- "#!pip install tornasole_mxnet/*\n",
- "\n",
- "# If you run into a version conflict with boto, run the following\n",
- "#!pip uninstall -y botocore boto3 aioboto3 aiobotocore && pip install botocore==1.12.91 boto3==1.9.91 aiobotocore==0.10.2 aioboto3==6.4.1"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "## Model Training and Gradient Analysis\n",
- "At this point we have all the ingredients installed on our machine. We can now start training.\n",
- "\n",
- "The goal of this notebook is to show how to detect the Vanishing Gradient problem. We will first do it manually and then automatic."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from smdebug.mxnet import SessionHook, SaveConfig\n",
- "from smdebug.trials import LocalTrial"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can change the logging level if appropriate "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "#import logging\n",
- "#logging.getLogger(\"tornasole\").setLevel(logging.WARNING)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can define a simple network - it doesn't really matter what it is.\n",
- "Importantly - we **add the Tornasole Hook**. This hook will be run at every batch and will save selected tensors (in this case, all of them) to the desired directory (in this case, `'{base_loc}/{run_id}'`.\n",
- "\n",
- "`{base_loc}` can be either a path on a local file system (for instance, `./ts_output/`) or an S3 bucket/object (`s3://mybucket/myprefix/`).\n",
- "\n",
- "See the documentation for more details."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import smdebug.mxnet as smd\n",
- "def create_net( tornasole_save_interval, base_loc, run_id ):\n",
- " net = nn.Sequential(prefix='sequential_')\n",
- " with net.name_scope():\n",
- " net.add(nn.Dense(128, activation='relu'))\n",
- " net.add(nn.Dense(64, activation='relu'))\n",
- " net.add(nn.Dense(10))\n",
- "\n",
- " # Create and add the hook. Arguments:\n",
- " # - save data in './{base_loc}/{run_id} - Note: s3 is also supported\n",
- " # - save every 100 batches\n",
- " # - save every tensor: inputs/outputs to each layer, as well as gradients\n",
- " trial_dir = base_loc + run_id\n",
- " hook = SessionHook(out_dir=trial_dir,\n",
- " save_config=SaveConfig(save_interval=100), \n",
- " save_all=True)\n",
- " hook.register_hook(net)\n",
- " return net"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And we create a simple training script. No Tornasole-specific code here, this is a slightly modified version of the [digit recognition](https://github.com/apache/incubator-mxnet/blob/master/example/gluon/mnist/mnist.py) example on the MXNet website."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def transformer(data, label):\n",
- " data = data.reshape((-1,)).astype(np.float32)/255\n",
- " return data, label\n",
- "\n",
- "def test(ctx, val_data):\n",
- " metric = mx.metric.Accuracy()\n",
- " for data, label in val_data:\n",
- " data = data.as_in_context(ctx)\n",
- " label = label.as_in_context(ctx)\n",
- " output = net(data)\n",
- " metric.update([label], [output])\n",
- " return metric.get()\n",
- "\n",
- "def train(net, epochs, ctx, learning_rate, momentum):\n",
- " train_data = gluon.data.DataLoader(\n",
- " gluon.data.vision.MNIST('./data', train=True, transform=transformer),\n",
- " batch_size=100, shuffle=True, last_batch='discard')\n",
- "\n",
- " val_data = gluon.data.DataLoader(\n",
- " gluon.data.vision.MNIST('./data', train=False, transform=transformer),\n",
- " batch_size=100, shuffle=False)\n",
- " \n",
- " # Collect all parameters from net and its children, then initialize them.\n",
- " net.initialize(mx.init.Xavier(magnitude=2.24), ctx=ctx)\n",
- " # Trainer is for updating parameters with gradient.\n",
- " trainer = gluon.Trainer(net.collect_params(), 'sgd',\n",
- " {'learning_rate': learning_rate, 'momentum': momentum})\n",
- " metric = mx.metric.Accuracy()\n",
- " loss = gluon.loss.SoftmaxCrossEntropyLoss()\n",
- "\n",
- " for epoch in range(epochs):\n",
- " # reset data iterator and metric at begining of epoch.\n",
- " metric.reset()\n",
- " for i, (data, label) in enumerate(train_data):\n",
- " # Copy data to ctx if necessary\n",
- " data = data.as_in_context(ctx)\n",
- " label = label.as_in_context(ctx)\n",
- " # Start recording computation graph with record() section.\n",
- " # Recorded graphs can then be differentiated with backward.\n",
- " with autograd.record():\n",
- " output = net(data)\n",
- " L = loss(output, label)\n",
- " L.backward()\n",
- " # take a gradient step with batch_size equal to data.shape[0]\n",
- " trainer.step(data.shape[0])\n",
- " # update metric at last.\n",
- " metric.update([label], [output])\n",
- "\n",
- " if i % 100 == 0 and i > 0:\n",
- " name, acc = metric.get()\n",
- " print('[Epoch %d Batch %d] Training: %s=%f'%(epoch, i, name, acc))\n",
- "\n",
- " name, acc = metric.get()\n",
- " print('[Epoch %d] Training: %s=%f'%(epoch, name, acc))\n",
- "\n",
- " name, val_acc = test(ctx, val_data)\n",
- " print('[Epoch %d] Validation: %s=%f'%(epoch, name, val_acc))\n",
- "\n",
- " net.save_parameters('mnist.params')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Clear up from previous runs, we remove old data (warning - we assume that we have set `ts_output` as the directory into which we send data)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "!rm -rf ./ts_output/"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "At this point we are ready to train. We will train this simple model.\n",
- "\n",
- "For the purposes of this example, we will name this run as `'good'` because we know it will converge to a good solution. If you have a GPU on your machine, you can change `ctx=mx.gpu(0)`.\n",
- "\n",
- "Behind the scenes, the `SessionHook` is saving the data requested."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "net = create_net( tornasole_save_interval=100, base_loc='./ts_output/', run_id='good')\n",
- "train(net=net, epochs=4, ctx=mx.cpu(), learning_rate=0.1, momentum=0.9)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Data Analysis - Manual\n",
- "Now that we have trained the system we can analyze the data. Notice that this notebook focuses on after-the-fact analysis. Tornasole also provides a collection of tools to do automatic analysis as the training run is progressing, which will be covered in a different notebook."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We import a basic analysis library, which defines a concept of `Trial`. A `Trial` is a single training run, which is depositing values in a local directory (`LocalTrial`) or S3 (`S3Trial`). In this case we are using a `LocalTrial` - if you wish, you can change the output from `./ts_output` to `s3://mybucket/myprefix` and use `S3Trial` instead of `LocalTrial`."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And we read the data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "good_trial = LocalTrial( 'myrun', './ts_output/good/')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can list all the tensors we know something about. Each one of these names is the name of a tensor - the name is a combination of the layer name (which, in these cases, is auto-assigned by MXNet) and whether it's an input/output/gradient."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "good_trial.tensor_names()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Another interesting feature is *collections*. Collection represents a set of tensors that are grouped together by a condition. See [more docs on Collections](https://github.com/awslabs/tornasole_core/blob/alpha/docs/mxnet/api.md#collection). For example, here we can inspect which tensors got into collection named '*gradients*'"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "good_trial.tensors_in_collection('gradients')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For each tensor we can ask for which steps we have data - in this case, every 100 steps"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "good_trial.tensor('gradient/sequential_dense0_weight').steps()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can obtain each tensor at each step as a `numpy` array"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "type(good_trial.tensor('gradient/sequential_dense0_weight').value(300))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Gradient Analysis"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can also create a simple function that prints the `np.mean` of the `np.abs` of each gradient. We expect each gradient to get smaller over time, as the system converges to a good solution. Now, remember that this is an interactive analysis - we are showing these tensors to give an idea of the data. \n",
- "\n",
- "Later on in this notebook we will run an automated analysis."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Define a function that, for the given tensor name, walks through all \n",
- "# the batches for which we have data and computes mean(abs(tensor)).\n",
- "# Returns the set of steps and the values\n",
- "\n",
- "def get_data(trial, tname):\n",
- " tensor = trial.tensor(tname)\n",
- " steps = tensor.steps()\n",
- " vals = []\n",
- " for s in steps:\n",
- " val = tensor.value(s)\n",
- " val = np.mean(np.abs(val))\n",
- " vals.append(val)\n",
- " return steps, vals"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def plot_gradients( lt ):\n",
- " for tname in lt.tensor_names():\n",
- " if not 'gradient' in tname: continue\n",
- " steps, data = get_data(lt, tname)\n",
- " plt.plot( steps, data, label=tname)\n",
- " plt.legend()\n",
- " plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can plot these gradiends. Notice how they are (mostly!) decreasing. We should investigate the spikes!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "plot_gradients(good_trial)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can also print inputs and outputs from the model. For instance, let's print the 83th sample of the 2700th batch, as seen by the network. \n",
- "\n",
- "Notice that we have to reshape the input data from a (784,) array to a (28,28) array and multiply by 255 - the exact inverse of the transformation we did above."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# The raw tensor\n",
- "raw_t = good_trial.tensor('sequential_input_0').value(2700)[83]\n",
- "# We have to undo the transformations in 'transformer' above. First of all, multiply by 255\n",
- "raw_t = raw_t * 255\n",
- "# Then reshape from a 784-long vector to a 28x28 square.\n",
- "input_image = raw_t.reshape(28,28)\n",
- "plt.imshow(input_image, cmap=plt.get_cmap('gray'))\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can also plot the relative values emitted by the network. Notice that the last layer is of type `Dense(10)`: it will emit 10 separate confidences, one for each 0-9 digit. The one with the highest output is the predicted value.\n",
- "\n",
- "We can capture and plot the network output for the same sample."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "plt.plot(good_trial.tensor('sequential_output0').value(2700)[83], 'bo')\n",
- "plt.show()\n",
- "print( 'The network predicted the value: {}'.format(np.argmax(good_trial.tensor('sequential_output0').value(2700)[83])))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Vanishing Gradient"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We have now worked through some of the basics. Let's pretend we are debugging a real problem: the [Vanishing Gradient](https://en.wikipedia.org/wiki/Vanishing_gradient_problem). When training a network, if the `learning_rate` is too high we will end up with a Vanishing Gradient. Let's set `learning_rate=1`.\n",
- "\n",
- "Notice how the accuracy remains at around ~10% - no better than random."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "net = create_net( tornasole_save_interval=100, base_loc='./ts_output/', run_id='bad')\n",
- "train(net=net, epochs=4, ctx=mx.cpu(), learning_rate=1, momentum=0.9)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "bad_trial = LocalTrial( 'myrun', './ts_output/bad/')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can plot the gradients - notice how every single one of them (apart from one) goes to zero and stays there!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "plot_gradients(bad_trial)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The `VanishingGradient` rule provided by Tornasole alerts for this automatically."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "input_image = (bad_trial.tensor('sequential_input_0').value(2700)[83]*255).reshape(28,28)\n",
- "plt.imshow(input_image, cmap=plt.get_cmap('gray'))\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "plt.plot(bad_trial.tensor('sequential_output0').value(2700)[83], 'bo')\n",
- "plt.show()\n",
- "print( 'The network predicted the value: {}'.format(np.argmax(bad_trial.tensor('sequential_output0').value(2700)[83])))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Data Analysis - Automatic\n",
- "So far we have conducted a human analysis, but the real power of Tornasole comes from having automatic monitoring of training runs. To do so we will build a SageMaker-based system that monitors existing runs in real time. Data traces deposited in S3 are the exchange mechanism: \n",
- "- the training system deposits data into s3://mybucket/myrun/\n",
- "- the monitoring system watches and reads data from s3://mybucket/myrun/\n",
- "\n",
- "In this example we will simulate that situation. The only difference from SageMaker-based system is that data traces have been stored locally, not in S3, so we will use previously created `LocalTrial` objects and run rule on them."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from smdebug.rules.generic import VanishingGradient\n",
- "from smdebug.rules.rule_invoker import invoke_rule"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "vr = VanishingGradient(base_trial=good_trial, threshold=0.0001)\n",
- "invoke_rule(vr, end_step=2700)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "vr_bad = VanishingGradient(base_trial=bad_trial, threshold=0.0001)\n",
- "invoke_rule(vr_bad, end_step=2700)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This concludes this notebook. For more information see the documentation at \n",
- "- https://github.com/awslabs/tornasole_core\n"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "conda_mxnet_p36",
- "language": "python",
- "name": "conda_mxnet_p36"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.5"
- },
- "pycharm": {
- "stem_cell": {
- "cell_type": "raw",
- "metadata": {
- "collapsed": false
- },
- "source": []
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/examples/mxnet/notebooks/mxnet-realtime-analysis.ipynb b/examples/mxnet/notebooks/mxnet-realtime-analysis.ipynb
deleted file mode 100644
index 417b27592..000000000
--- a/examples/mxnet/notebooks/mxnet-realtime-analysis.ipynb
+++ /dev/null
@@ -1,453 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Debugging SageMaker Training Jobs In Real Time with Tornasole"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Overview"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Tornasole is a new capability of Amazon SageMaker that allows debugging machine learning training. \n",
- "It lets you go beyond just looking at scalars like losses and accuracies during training and gives \n",
- "you full visibility into all tensors 'flowing through the graph' during training. Tornasole helps you to monitor your training in near real time using rules and would provide you alerts, once it has detected inconsistency in training flow.\n",
- "\n",
- "Using Tornasole is a two step process: Saving tensors and Analysis. Let's look at each one of them closely.\n",
- "\n",
- "### Saving tensors\n",
- "\n",
- "Tensors define the state of the training job at any particular instant in its lifecycle. Tornasole exposes a library which allows you to capture these tensors and save them for analysis.\n",
- "\n",
- "### Analysis\n",
- "\n",
- "There are two ways to get to tensors and run analysis on them. One way is to use concept called ***Rules***. Please refer to [DeveloperGuide_Rules.md](../../../../rules/DeveloperGuide_Rules.md) for more details about rules based approach to analysis. Focus of this notebook is on another way of analysis: **Manual**.\n",
- "\n",
- "Manual analysis is what you use when there are no rules available to detect type of an issue you are running into and you need to get to raw tensors in order to understand what data is travelling through your model duing training and, hopefully, root cause a problem or two with your training job.\n",
- "\n",
- "Manual analysis is powered by Tornasole API - a framework that allows to retrieve tensors and scalas (e.g. debugging data) saved during training job via few lines of code. One of the most powerful features provided by it is real time access to data - you can get tensors and scalars ***while your training job is running***.\n",
- "\n",
- "This example guides you through installation of the required components for emitting tensors in a \n",
- "SageMaker training job and using Tornasole API to access those tensors while training is running. We will use small gluon CNN model and train it on FashionMNIST dataset. While job is running we will retrieve activations of first convolutional layer from each 100 batches and visualize them. Also we will visualize weights of that level after the job is done."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup\n",
- "\n",
- "As a first step, we'll do the installation of required tools which will allow emission of tensors (saving tensors) and provide access to Tornasole API to retrieve them."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "!aws s3 sync s3://tornasole-external-preview-use1/ ./tornasole\n",
- "!pip install ./smdebug/sdk/ts-binaries/tornasole_mxnet/py3/latest/tornasole-0.3.4-py2.py3-none-any.whl --user\n",
- "!pip -q install ./smdebug/sdk/sagemaker-tornasole-latest.tar.gz\n",
- "!aws configure add-model --service-model file:///home/ec2-user/SageMaker/smdebug/sdk/sagemaker-smdebug.json --service-name sagemaker"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Training MXNet models in SageMaker with Tornasole\n",
- "\n",
- "We'll be training a small mxnet CNN model with FashonMNIST dataset in this notebook with Tornasole enabled. This will be done using SageMaker MXNet 1.4.1 Container with Script Mode. Note that Tornasole currently only works with python3, so be sure to set `py_version='py3'` when creating SageMaker Estimator.\n",
- "\n",
- "Let us first train with a simple training script mnist_gluon_realtime_visualize_demo.py with Tornasole enabled in SageMaker using the SageMaker Estimator API. In this example, for simplicity sake, Tornasole will capture all tensors as specified in its configuration every 100 steps (1 step is 1 batch). While training job is running we will use Tornasole API to access saved tensors in real time and visualize them. We will rely on Tornasole to take care of downloading fresh set of tensors every time we query for them.\n",
- "\n",
- "## Enable Tornasole in the training script\n",
- "\n",
- "Integrating Tornasole into the training job can be accomplished by following steps below.\n",
- "\n",
- "### Import the hook package\n",
- "Import the SessionHook class along with other helper classes in your training script as shown below\n",
- "\n",
- "```\n",
- "from smdebug.mxnet import SessionHook\n",
- "from smdebug import SaveConfig, modes\n",
- "```\n",
- "\n",
- "### Instantiate and initialize hook\n",
- "\n",
- "```\n",
- " # Create SaveConfig object that instructs engine to log graph tensors every 100 steps (1 step == 1 batch).\n",
- " save_config = SaveConfig(save_interval=100)\n",
- " # Create a hook that logs ***all*** tensors while training the model.\n",
- " hook = SessionHook(save_config=save_config, save_all=True)\n",
- "```\n",
- "\n",
- "### Register Tornasole hook to the model before starting of the training.\n",
- "\n",
- "*NOTE: The hook can only be registered to Gluon Non-hybrid models.\n",
- "*\n",
- "\n",
- "After creating or loading the desired model, you can register Tornasole hook with the model as shown below.\n",
- "\n",
- "```\n",
- "# Create a Gluon Model.\n",
- "net = create_gluon_model()\n",
- "\n",
- "# Create a hook for logging all tensors.\n",
- "hook = create_hook()\n",
- "\n",
- "# Apply hook to the model (e.g. instruct engine to recognize hook configuration\n",
- "# and enable mode in which engine will log graph tensors\n",
- "hook.register_hook(net)\n",
- "```\n",
- "\n",
- "#### Set the mode\n",
- "Tornasole has the concept of modes (TRAIN, EVAL, PREDICT) to separate out different modes of the jobs.\n",
- "Set the mode you are running in your job. Every time the mode changes in your job, please set the current mode. This helps you group steps by mode, for easier analysis. Setting the mode is optional but recommended. If you do not specify this, Tornasole saves all steps under a `GLOBAL` mode. \n",
- "```\n",
- "hook.set_mode(smd.modes.TRAIN)\n",
- "```\n",
- "\n",
- "Refer [DeveloperGuide_MXNet.md](../../DeveloperGuide_MXNet.md) for more details on the APIs Tornasole provides to help you save tensors.\n",
- "\n",
- "### Docker Images with Tornasole\n",
- "\n",
- "We have built SageMaker MXNet containers with smdebug. You can use them from ECR from SageMaker. Here are the links to the images. Please use the image from the appropriate region in which you want your jobs to run."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2\n",
- "import sagemaker\n",
- "import boto3\n",
- "import os\n",
- "from sagemaker.mxnet import MXNet\n",
- "from smdebug.mxnet import modes\n",
- "\n",
- "# Below changes the region to be one where this notebook is running\n",
- "TAG='latest'\n",
- "REGION = boto3.Session().region_name\n",
- "os.environ['AWS_REGION'] = REGION\n",
- "\n",
- "cpu_docker_image_name= '072677473360.dkr.ecr.{}.amazonaws.com/tornasole-preprod-mxnet-1.4.1-cpu:{}'.format(REGION, TAG)\n",
- "#gpu_docker_image_name= '072677473360.dkr.ecr.{}.amazonaws.com/tornasole-preprod-mxnet-1.4.1-gpu:{}'.format(REGION, TAG)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Configuring the inputs for the training job\n",
- "\n",
- "Now we'll call the Sagemaker MXNet Estimator to kick off a training job along with enabling Tornasole functionality.\n",
- "\n",
- "The *entry_point_script* points to the MXNet training script that has the SessionHook integrated.\n",
- "\n",
- "The *hyperparameters* are the parameters that will be passed to the training script."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "entry_point_script = '../scripts/mnist_gluon_realtime_visualize_demo.py'\n",
- "hyperparameters = {'batch-size': 256, 'learning_rate': 0.1, 'epochs': 10}\n",
- "base_job_name = 'mxnet-TS-realtime-analysis'"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "sagemaker_simple_estimator = MXNet(role=sagemaker.get_execution_role(),\n",
- " base_job_name=base_job_name,\n",
- " train_instance_count=1,\n",
- " train_instance_type='ml.m4.xlarge',\n",
- " image_name=cpu_docker_image_name,\n",
- " entry_point=entry_point_script,\n",
- " hyperparameters=hyperparameters,\n",
- " framework_version='1.4.1',\n",
- " py_version='py3',\n",
- " # following parameter is necesary to instruct SageMaker \n",
- " # that debugging data generated by Tornasole needs to be \n",
- " # uploaded to S3 bucket in your account. This way we can \n",
- " # access it while training job is running.\n",
- " debug=True)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# This is a fire and forget event. By setting wait=False, we just submit the job to run in the background.\n",
- "# SageMaker will spin off one training job and release control to next cells in the notebook.\n",
- "# Please follow this notebook to see status of the training job.\n",
- "sagemaker_simple_estimator.fit(wait=False)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "### Result\n",
- "\n",
- "As a result of the above command, SageMaker will spin off 1 training job for you and it will produce the tensors to be analyzed. This job will run in a background without you having to wait for it to complete in order to continue with the rest of the notebook. Because of this async nature of training job we will need to monitor its status so that we don't start to request debugging tensors too early. Tensors are only produced during training phase of SageMaker training job hence let's wait until that begins.\n",
- "\n",
- "### Checking on the training job status\n",
- "\n",
- "We can check the status of the training job by running the following code. It will check on a status of SageMaker training job every five seconds. Once job has started its traning cycle control is released to next cells in the notebook."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# some helper method first, to render status status updates\n",
- "import time\n",
- "import sys\n",
- "from time import gmtime, strftime\n",
- "\n",
- "def print_same_line(s):\n",
- " sys.stdout.write('\\r{}: {}'.format(strftime('%X', gmtime()), s))\n",
- " sys.stdout.flush()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Below command will give the status of training job\n",
- "# Note: In the output of below command you will see DebugConfig parameter \n",
- "import time\n",
- "\n",
- "job_name = sagemaker_simple_estimator.latest_training_job.name\n",
- "print('Training job name: ' + job_name)\n",
- "\n",
- "client = sagemaker_simple_estimator.sagemaker_session.sagemaker_client\n",
- "\n",
- "description = client.describe_training_job(TrainingJobName=job_name)\n",
- "\n",
- "if description['TrainingJobStatus'] != 'Completed':\n",
- " while description['SecondaryStatus'] not in {'Training', 'Completed'}:\n",
- " description = client.describe_training_job(TrainingJobName=job_name)\n",
- " primary_status = description['TrainingJobStatus']\n",
- " secondary_status = description['SecondaryStatus']\n",
- " print_same_line('Current job status: [PrimaryStatus: {}, SecondaryStatus: {}]'.format(primary_status, secondary_status))\n",
- " time.sleep(5)\n",
- "\n",
- "# uncomment next line to see full details of training job \n",
- "# client.describe_training_job(TrainingJobName=job_name)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Retrieving and Analyzing tensors\n",
- "\n",
- "Before getting to analysis, here are some notes on concepts being used in Tornasole that help with analysis.\n",
- "- ***Trial*** - object that is a center piece of Tornasole API when it comes to getting access to tensors. It is a top level abstract that represents a single run of a training job. All tensors emitted by training job are associated with its Trial.\n",
- "- ***Step*** - object that represents next level of abstraction. In Tornasole step is a representation of a single batch of a training job. Each trial has multiple steps. Each tensor is associated with multiple steps - having a particular value at each of the steps.\n",
- "- ***Tensor*** - object that represent actual tensor saved during training job. *Note* - it could be a scalar as well.\n",
- "- ***Mode*** - each DL engine does forward and backward passes during training job. During each of those passes tensors generated by the model are saved. However, in addition to training itself DL engine also uses forward passes for validation phase, and tensors generated during such forward passes are also saved. Tornasole introduces concept of Training Mode in order to allow to differentiate between tensors of each of the phases.\n",
- "\n",
- "For more details on aforementioned concepts as well as on Tornasole API in general (including examples) please refer to [Rules API](../../docs/rules/readme.md)\n",
- "\n",
- "Below, you can find several methods to help with retrieving and plotting tensors. In *get_data* we use concepts described above to retrieve data. We expect to get steps_range that will have 1 or more steps (batches) for which we want to get tensors for. Please note that we are going to retrieve only tensors saved by main training loop and will excluse tensors saved during validation (`mode=modes.TRAIN` will help with that). Two other methods are helpers to plot tensors."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "\n",
- "def get_data(trial, tname, batch_index, steps_range):\n",
- " tensor = trial.tensor(tname)\n",
- " vals = []\n",
- " for s in steps_range:\n",
- " val = tensor.value(step_num=s, mode=modes.TRAIN)[batch_index][0]\n",
- " vals.append(val)\n",
- " return vals\n",
- "\n",
- "def create_plots(steps_range):\n",
- " fig, axs = plt.subplots(nrows=1, ncols=len(steps_range), constrained_layout=True, figsize=(2*len(steps_range), 2),\n",
- " subplot_kw={'xticks': [], 'yticks': []})\n",
- " return fig, axs\n",
- "\n",
- "def plot_tensors(trial, layer, batch_index, steps_range):\n",
- " if len(steps_range) > 0: \n",
- " fig, axs = create_plots(steps_range)\n",
- " vals = get_data(trial, layer, batch_index, steps_range)\n",
- "\n",
- " for ax, image, step in zip(axs.flat if isinstance(axs, np.ndarray) else np.array([axs]), vals, steps_range):\n",
- " ax.imshow(image, cmap='gray')\n",
- " ax.set_title(str(step))\n",
- " plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now that we are prepared with methods to get data and plot it, let's get to it. The goal of the next block is to instantiate a ***Trial***, a central access point for all Tornasole API calls to get tensors. We will do that by inspecting currently running training job and extract necessary params from its debug config to instruct Tornasole where the data we are looking for is located. Couple notes here:\n",
- "- Tensors are being stored in your own S3 bucket to which you can navigate and manually inspect its content if desired.\n",
- "- You might notice a slight delay before trial object is created (last line in the cell). It is normal as Tornasole will monitor corresponding bucket with tensors and wait until tensors appear in it. The delay is introduced by less then instantenous upload of tensors from training container to your S3 bucket. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "from urllib.parse import urlparse\n",
- "import smdebug.trials\n",
- "from smdebug.trials import S3Trial\n",
- "import logging\n",
- "\n",
- "description = client.describe_training_job(TrainingJobName=job_name)\n",
- "s3_output_path = description[\"DebugConfig\"][\"DebugHookConfig\"][\"S3OutputPath\"]\n",
- "parse_result = urlparse(s3_output_path)\n",
- "bucket_name = parse_result.netloc\n",
- "prefix_name = parse_result.path.strip('/')\n",
- "\n",
- "logging.getLogger(\"tornasole\").setLevel(logging.INFO)\n",
- "\n",
- "# this is where we create a Trial object that allows access to saved tensors\n",
- "trial = S3Trial(base_job_name, bucket_name, prefix_name)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# feel free to inspect all tensors logged by uncommenting below line\n",
- "# trial.tensor_names()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Visualize tensors of a running training job\n",
- "Now to the final part of our example. Below we will wait until Tornasole has downloaded initial chunk of tensors for us to look at. Once that first chunk is ready - we will keep getting new chunks every 5 seconds and plot their tensors correspondingly one under another."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Below we select the very first tensor from every batch.\n",
- "# Feel free to modify this and select another tensor from the batch.\n",
- "batch_index = 0\n",
- "\n",
- "# This is a name of a tensor to retrieve data of.\n",
- "# Variable is called `layer` as this tensor happens to be output of first convolutional layer.\n",
- "layer = 'conv0_output0'\n",
- "\n",
- "steps = 0\n",
- "while steps == 0:\n",
- " # trial.steps return all steps that have been downloaded by Tornasole to date.\n",
- " # It doesn't represent all steps that are to be available once training job is complete -\n",
- " # it is a snapshot of a current state of the system. If you call it after training job is done\n",
- " # you will get all tensors available.\n",
- " steps = trial.steps(mode=modes.TRAIN)\n",
- " print_same_line('Waiting for tensors to become available...')\n",
- " time.sleep(3)\n",
- "print('\\nDone')\n",
- "\n",
- "print('Getting tensors and plotting...')\n",
- "rendered_steps = []\n",
- "# trial.training_ended is a way to keep monitoring for a state of a training job as seen by smdebug.\n",
- "# When SageMaker completes training job, trial becomes aware of it.\n",
- "while not trial.training_ended():\n",
- " steps = trial.steps(mode=modes.TRAIN)\n",
- " # quick way to get diff between two lists\n",
- " steps_to_render = list(set(steps).symmetric_difference(set(rendered_steps)))\n",
- " # plot only tensors from newer chunk\n",
- " plot_tensors(trial, layer, batch_index, steps_to_render)\n",
- " rendered_steps.extend(steps_to_render)\n",
- " time.sleep(5)\n",
- "print('\\nDone')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Additional visualizations\n",
- "\n",
- "Now that we completed plotting of tensors during training job run, let's plot some more tensors. This time we will get all of them at once as training job has finished and Tornasole is aware of all tensors emitted by it. Let's visualize tensors representing weights of first convolutional layer (e.g. its kernels). By inspecting each row of plotted tensors from left to right you can notice progression in how each kernel was \"learning\" its values."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Let's visualize weights of the first convolutional layer.\n",
- "layer = 'conv0_weight'\n",
- "\n",
- "for i in range(0, trial.tensor(layer).value(step_num=trial.tensor(layer).steps(mode=modes.TRAIN)[0], mode=modes.TRAIN).shape[0]):\n",
- " plot_tensors(trial, layer, i, trial.tensor(layer).steps(mode=modes.TRAIN))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "conda_mxnet_p36",
- "language": "python",
- "name": "conda_mxnet_p36"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.5"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/examples/mxnet/sagemaker-notebooks/mxnet.ipynb b/examples/mxnet/sagemaker-notebooks/mxnet.ipynb
deleted file mode 100644
index b4ce8c674..000000000
--- a/examples/mxnet/sagemaker-notebooks/mxnet.ipynb
+++ /dev/null
@@ -1,711 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Debugging SageMaker Training Jobs with Tornasole"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Overview"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Tornasole is a new capability of Amazon SageMaker that allows debugging machine learning training. \n",
- "It lets you go beyond just looking at scalars like losses and accuracies during training and gives \n",
- "you full visibility into all tensors 'flowing through the graph' during training. Tornasole helps you to monitor your training in near real time using rules and would provide you alerts, once it has detected inconsistency in training flow.\n",
- "\n",
- "Using Tornasole is a two step process: Saving tensors and Analysis. Let's look at each one of them closely.\n",
- "\n",
- "### Saving tensors\n",
- "\n",
- "Tensors define the state of the training job at any particular instant in its lifecycle. Tornasole exposes a library which allows you to capture these tensors and save them for analysis\n",
- "\n",
- "### Analysis\n",
- "\n",
- "Analysis of the tensors emitted is captured by the Tornasole concept called ***Rules***. On a very broad level, \n",
- "a rule is a python code used to detect certain conditions during training. Some of the conditions that a data scientist training a deep learning model may care about are monitoring for gradients getting too large or too small, detecting overfitting, and so on.\n",
- "Tornasole will come pre-packaged with certain rules. Users can write their own rules using the Tornasole APIs.\n",
- "You can also analyze raw tensor data outside of the Rules construct in say, a Sagemaker notebook, using Tornasole's full set of APIs. \n",
- "Please refer [DeveloperGuide_Rules.md](../../../../rules/DeveloperGuide_Rules.md) for more details about analysis.\n",
- "\n",
- "This example guides you through installation of the required components for emitting tensors in a \n",
- "SageMaker training job and applying a rule over the tensors to monitor the live status of the job."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup\n",
- "\n",
- "As a first step, we'll do the installation of required tools which will allow emission of tensors (saving tensors) and application of rules to analyze them"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "download: s3://tornasole-external-preview-use1/sdk/sagemaker-1.35.2.dev0.tar.gz to ./sagemaker-1.35.2.dev0.tar.gz\n",
- "Processing ./sagemaker-1.35.2.dev0.tar.gz\n",
- "Requirement already satisfied: boto3>=1.9.169 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from sagemaker==1.35.2.dev0) (1.9.213)\n",
- "Requirement already satisfied: numpy>=1.9.0 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from sagemaker==1.35.2.dev0) (1.14.5)\n",
- "Requirement already satisfied: protobuf>=3.1 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from sagemaker==1.35.2.dev0) (3.5.2)\n",
- "Requirement already satisfied: scipy>=0.19.0 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from sagemaker==1.35.2.dev0) (1.1.0)\n",
- "Requirement already satisfied: urllib3<1.25,>=1.21 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from sagemaker==1.35.2.dev0) (1.23)\n",
- "Requirement already satisfied: protobuf3-to-dict>=0.1.5 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from sagemaker==1.35.2.dev0) (0.1.5)\n",
- "Requirement already satisfied: docker-compose>=1.23.0 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from sagemaker==1.35.2.dev0) (1.24.1)\n",
- "Requirement already satisfied: requests<2.21,>=2.20.0 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from sagemaker==1.35.2.dev0) (2.20.0)\n",
- "Requirement already satisfied: enum34>=1.1.6 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from sagemaker==1.35.2.dev0) (1.1.6)\n",
- "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from boto3>=1.9.169->sagemaker==1.35.2.dev0) (0.9.4)\n",
- "Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from boto3>=1.9.169->sagemaker==1.35.2.dev0) (0.2.1)\n",
- "Requirement already satisfied: botocore<1.13.0,>=1.12.213 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from boto3>=1.9.169->sagemaker==1.35.2.dev0) (1.12.213)\n",
- "Requirement already satisfied: six>=1.9 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from protobuf>=3.1->sagemaker==1.35.2.dev0) (1.11.0)\n",
- "Requirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from protobuf>=3.1->sagemaker==1.35.2.dev0) (39.1.0)\n",
- "Requirement already satisfied: docker[ssh]<4.0,>=3.7.0 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (3.7.3)\n",
- "Requirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < \"3.5\" in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (3.5.0.1)\n",
- "Requirement already satisfied: PyYAML<4.3,>=3.10 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (3.12)\n",
- "Requirement already satisfied: texttable<0.10,>=0.9.0 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (0.9.1)\n",
- "Requirement already satisfied: dockerpty<0.5,>=0.4.1 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (0.4.1)\n",
- "Requirement already satisfied: ipaddress>=1.0.16; python_version < \"3.3\" in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (1.0.22)\n",
- "Requirement already satisfied: websocket-client<1.0,>=0.32.0 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (0.56.0)\n",
- "Requirement already satisfied: docopt<0.7,>=0.6.1 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (0.6.2)\n",
- "Requirement already satisfied: jsonschema<3,>=2.5.1 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (2.6.0)\n",
- "Requirement already satisfied: cached-property<2,>=1.2.0 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (1.5.1)\n",
- "Requirement already satisfied: idna<2.8,>=2.5 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from requests<2.21,>=2.20.0->sagemaker==1.35.2.dev0) (2.6)\n",
- "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from requests<2.21,>=2.20.0->sagemaker==1.35.2.dev0) (3.0.4)\n",
- "Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from requests<2.21,>=2.20.0->sagemaker==1.35.2.dev0) (2019.6.16)\n",
- "Requirement already satisfied: futures<4.0.0,>=2.2.0; python_version == \"2.6\" or python_version == \"2.7\" in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from s3transfer<0.3.0,>=0.2.0->boto3>=1.9.169->sagemaker==1.35.2.dev0) (3.2.0)\n",
- "Requirement already satisfied: docutils<0.16,>=0.10 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from botocore<1.13.0,>=1.12.213->boto3>=1.9.169->sagemaker==1.35.2.dev0) (0.14)\n",
- "Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= \"2.7\" in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from botocore<1.13.0,>=1.12.213->boto3>=1.9.169->sagemaker==1.35.2.dev0) (2.7.3)\n",
- "Requirement already satisfied: docker-pycreds>=0.4.0 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker[ssh]<4.0,>=3.7.0->docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (0.4.0)\n",
- "Requirement already satisfied: paramiko>=2.4.2; extra == \"ssh\" in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from docker[ssh]<4.0,>=3.7.0->docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (2.6.0)\n",
- "Requirement already satisfied: functools32 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from jsonschema<3,>=2.5.1->docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (3.2.3.post2)\n",
- "Requirement already satisfied: pynacl>=1.0.1 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from paramiko>=2.4.2; extra == \"ssh\"->docker[ssh]<4.0,>=3.7.0->docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (1.3.0)\n",
- "Requirement already satisfied: cryptography>=2.5 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from paramiko>=2.4.2; extra == \"ssh\"->docker[ssh]<4.0,>=3.7.0->docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (2.5)\n",
- "Requirement already satisfied: bcrypt>=3.1.3 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from paramiko>=2.4.2; extra == \"ssh\"->docker[ssh]<4.0,>=3.7.0->docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (3.1.7)\n",
- "Requirement already satisfied: cffi>=1.4.1 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from pynacl>=1.0.1->paramiko>=2.4.2; extra == \"ssh\"->docker[ssh]<4.0,>=3.7.0->docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (1.11.5)\n",
- "Requirement already satisfied: asn1crypto>=0.21.0 in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from cryptography>=2.5->paramiko>=2.4.2; extra == \"ssh\"->docker[ssh]<4.0,>=3.7.0->docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (0.24.0)\n",
- "Requirement already satisfied: pycparser in /home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages (from cffi>=1.4.1->pynacl>=1.0.1->paramiko>=2.4.2; extra == \"ssh\"->docker[ssh]<4.0,>=3.7.0->docker-compose>=1.23.0->sagemaker==1.35.2.dev0) (2.18)\n",
- "Building wheels for collected packages: sagemaker\n",
- " Running setup.py bdist_wheel for sagemaker ... \u001b[?25ldone\n",
- "\u001b[?25h Stored in directory: /home/ec2-user/.cache/pip/wheels/93/13/04/5fee9d22051b05a9789db3ce41ac6f7c50de67b47419007761\n",
- "Successfully built sagemaker\n",
- "\u001b[31mtyping-extensions 3.7.4 has requirement typing>=3.7.4, but you'll have typing 3.6.4 which is incompatible.\u001b[0m\n",
- "Installing collected packages: sagemaker\n",
- " Found existing installation: sagemaker 1.35.2.dev0\n",
- " Uninstalling sagemaker-1.35.2.dev0:\n",
- " Successfully uninstalled sagemaker-1.35.2.dev0\n",
- "Successfully installed sagemaker-1.35.2.dev0\n",
- "\u001b[33mYou are using pip version 10.0.1, however version 19.2.3 is available.\n",
- "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
- "download: s3://tornasole-external-preview-use1/sdk/sagemaker-smdebug.json to ./sagemaker-smdebug.json\n",
- "\n",
- "No JSON object could be decoded\n"
- ]
- }
- ],
- "source": [
- "!aws s3 cp s3://tornasole-external-preview-use1/sdk/sagemaker-1.35.2.dev0.tar.gz .\n",
- "!pip install sagemaker-1.35.2.dev0.tar.gz\n",
- "!aws s3 cp s3://tornasole-external-preview-use1/sdk/sagemaker-smdebug.json .\n",
- "!aws configure add-model --service-model sagemaker-smdebug.json --service-name sagemaker"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Training MXNet models in SageMaker with Tornasole\n",
- "\n",
- "We'll be training a mxnet gluon model for FashonMNIST dataset in this notebook with Tornasole enabled and monitor the training jobs with Tornasole's Rules. This will be done using SageMaker MXNet 1.4.1 Container with Script Mode. Note that Tornasole currently only works with python3, so be sure to set `py_version='py3'` when creating SageMaker Estimator.\n",
- "\n",
- "\n",
- "Let us first train a simple example training script mnist_gluon_vg_demo.py with Tornasole enabled in SageMaker using the SageMaker Estimator API, along with a VanishingGradient Rule to monitor the training job in realtime. A Tornasole Rule is essentially python code which analyses tensors saved by tornasole and validates some condition. VanishingGradient rule is a first party (1P) rule provided by smdebug. During training, Tornasole will capture tensors as specified in its configuration and VanishingGradient Rule job will monitor whether any gradient tensor has reached 0. The rule will emit a cloudwatch event if it finds an vanishing gradient tensor during training.\n",
- "\n",
- "## Enable Tornasole in the training script\n",
- "\n",
- "Integrating Tornasole into the training job can be accomplished by following steps below.\n",
- "\n",
- "### Import the hook package\n",
- "Import the SessionHook class along with other helper classes in your training script as shown below\n",
- "\n",
- "```\n",
- "from smdebug.mxnet.hook import SessionHook\n",
- "from smdebug.mxnet import SaveConfig, Collection\n",
- "```\n",
- "\n",
- "### Instantiate and initialize hook\n",
- "\n",
- "```\n",
- " # Create SaveConfig that instructs engine to log graph tensors every 10 steps.\n",
- " save_config = SaveConfig(save_interval=10)\n",
- " # Create a hook that logs tensors of weights, biases and gradients while training the model.\n",
- " hook = SessionHook(save_config=save_config)\n",
- "```\n",
- "\n",
- "### Register Tornasole hook to the model before starting of the training.\n",
- "\n",
- "### NOTE: The hook can only be registered to Gluon Non-hybrid models.\n",
- "\n",
- "After creating or loading the desired model, users can register the hook with the model as shown below.\n",
- "\n",
- "```\n",
- "net = create_gluon_model()\n",
- " # Apply hook to the model (e.g. instruct engine to recognize hook configuration\n",
- " # and enable mode in which engine will log graph tensors\n",
- "hook.register_hook(net)\n",
- "```\n",
- "\n",
- "#### Set the mode\n",
- "Tornasole has the concept of modes (TRAIN, EVAL, PREDICT) to separate out different modes of the jobs.\n",
- "Set the mode you are running in your job. Every time the mode changes in your job, please set the current mode. This helps you group steps by mode, for easier analysis. Setting the mode is optional but recommended. If you do not specify this, Tornasole saves all steps under a `GLOBAL` mode. \n",
- "```\n",
- "hook.set_mode(smd.modes.TRAIN)\n",
- "```\n",
- "\n",
- "Refer [DeveloperGuide_MXNet.md](../../DeveloperGuide_MXNet.md) for more details on the APIs Tornasole provides to help you save tensors.\n",
- "\n",
- "\n",
- "### Docker Images with Tornasole\n",
- "\n",
- "We have built SageMaker MXNet containers with smdebug. You can use them from ECR from SageMaker. Here are the links to the images. Please use the image from the appropriate region in which you want your jobs to run.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "import sagemaker\n",
- "import boto3\n",
- "from sagemaker.mxnet import MXNet\n",
- "\n",
- "# Below changes the region to be one where this notebook is running\n",
- "REGION = boto3.Session().region_name\n",
- "TAG='latest'\n",
- "\n",
- "cpu_docker_image_name= '072677473360.dkr.ecr.{}.amazonaws.com/tornasole-preprod-mxnet-1.4.1-cpu:{}'.format(REGION, TAG)\n",
- "gpu_docker_image_name= '072677473360.dkr.ecr.{}.amazonaws.com/tornasole-preprod-mxnet-1.4.1-gpu:{}'.format(REGION, TAG)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Configuring the inputs for the training job\n",
- "\n",
- "Now we'll call the Sagemaker MXNet Estimator to kick off a training job along with the VanishingGradient 1P rule to monitor the job.\n",
- "\n",
- "The 'entry_point_script' points to the MXNet training script that has the SessionHook integrated.\n",
- "\n",
- "The 'hyperparameters' are the parameters that will be passed to the training script.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "entry_point_script = '../scripts/mnist_gluon_vg_demo.py'\n",
- "bad_hyperparameters = {'random_seed' : True, 'num_steps': 33, 'save_frequency' : 30}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "sagemaker_simple_estimator = MXNet(role=sagemaker.get_execution_role(),\n",
- " base_job_name='mxnet-tornasole-simple-demo',\n",
- " train_instance_count=1,\n",
- " train_instance_type='ml.m4.xlarge',\n",
- " image_name=cpu_docker_image_name,\n",
- " entry_point=entry_point_script,\n",
- " hyperparameters=bad_hyperparameters,\n",
- " framework_version='1.4.1',\n",
- " debug=True,\n",
- " py_version='py3',\n",
- " # These are Tornasole specific parameters, \n",
- " # debug= True means rule specified in rules_specification \n",
- " # will run as rule job. \n",
- " # Below, we specify to run the first party rule VanishingGradient\n",
- " # on a ml.c5.4xlarge instance\n",
- " rules_specification=[\n",
- " {\n",
- " \"RuleName\": \"VanishingGradient\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"end-step\": \"33\"\n",
- " }\n",
- " }\n",
- " ])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "sagemaker_simple_estimator.fit(wait=False)\n",
- "# This is a fire and forget event. By setting wait=False, we just submit the job to run in the background.\n",
- "# In the background SageMaker will spin off 1 training job and 1 rule job for you.\n",
- "# Please follow this notebook to see status of the training job and the rule job"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "### Result\n",
- "\n",
- "As a result of the above command, SageMaker will spin off 1 training job and 1 rule job for you - the first one being the job which produces the tensors to be analyzed and the second one, which analyzes the tensors to check if there are any tensors that demonstrate the vanishing gradient issue during training. You will see that the VanishingGradient rule is triggered. Please note that the rule triggering is represented as custom Tornasole Python exception: whenever rule condition is evaluated to True - custom Python exception is thrown to signify that.\n",
- "\n",
- "### Describing the training job\n",
- "\n",
- "We can check the status of the training job by running the following command:\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Below command will give the status of training job\n",
- "# Note: In the output of below command you will see DebugConfig parameter \n",
- "\n",
- "job_name = sagemaker_simple_estimator.latest_training_job.name\n",
- "\n",
- "client = sagemaker_simple_estimator.sagemaker_session.sagemaker_client\n",
- "\n",
- "description = client.describe_training_job(TrainingJobName=job_name)\n",
- "\n",
- "# uncomment next line to see full details of training job \n",
- "# description\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "u'InProgress'"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "# The status of the training job can be seen below\n",
- "description['TrainingJobStatus']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Once your training job is started SageMaker will spin up a rule execution job to run the ExplodingTensor rule.\n",
- "\n",
- "### Tornasole specific parameters in the description\n",
- "\n",
- "DebugConfig parameter has details about Tornasole related configuration. The key parameters to look for below are\n",
- "\n",
- "*S3OutputPath* : This is the path where output tensors from tornasole is getting saved.\n",
- "\n",
- "*RuleConfig* : This parameter tells about the rule config parameter that was passed when creating the trainning job. In this you should be able to see details of the rule that ran for training.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "{u'DebugHookConfig': {u'DebugHookSpecificationList': [],\n",
- " u'LocalPath': u'/opt/ml/output/tensors',\n",
- " u'S3OutputPath': u's3://sagemaker-ca-central-1-072677473360/tensors-mxnet-tornasole-simple-demo-2019-08-30-04-24-08-626'},\n",
- " u'RuleConfig': {u'RuleSpecificationList': [{u'InstanceType': u'ml.c5.4xlarge',\n",
- " u'RuleEvaluatorImage': u'453379255795.dkr.ecr.ca-central-1.amazonaws.com/script-rule-executor:latest',\n",
- " u'RuleName': u'VanishingGradient',\n",
- " u'RuntimeConfigurations': {u'end-step': u'33'},\n",
- " u'VolumeSizeInGB': 10}]}}"
- ]
- },
- "execution_count": 8,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "description['DebugConfig']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "### Check the status of the Rule Execution Job\n",
- "\n",
- "To get the rule execution job that SageMaker started for you, run the command below and it shows you the `RuleName`, `RuleStatus`, `FailureReason` if any, and `RuleExecutionJobArn`. If the tensors meets a rule evaluation condition, the rule execution job throws a client error with `FailureReason: RuleEvaluationConditionMet`. These details are also available as part of the response description above under: description['RuleMonitoringStatuses']\n",
- "\n",
- "The logs of the training job are available in the `Cloudwatch Logstream` /aws/sagemaker/TrainingJobs with `RuleExecutionJobArn`.\n",
- "\n",
- "You will see that once the rule execution job starts, that it identifies the vanishing gradient situation in the training job, raises the `RuleEvaluationConditionMet` exception and ends the job.\n",
- "\n",
- "**Note: The next cell blocks till the rule execution job ends. Once it says RuleStatus is Started, and shows the RuleExecutionJobArn, you can look at the status of the rule being monitored. At that point, we can also look at the logs as shown in the next cell**"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Wait to get status for Rule Execution Jobs...\n",
- "=============================================\n",
- "RuleName: VanishingGradient\n",
- "RuleStatus: NotStarted\n",
- "=============================================\n",
- "Wait to get status for Rule Execution Jobs...\n",
- "=============================================\n",
- "RuleName: VanishingGradient\n",
- "RuleStatus: NotStarted\n",
- "=============================================\n"
- ]
- },
- {
- "ename": "KeyboardInterrupt",
- "evalue": "",
- "output_type": "error",
- "traceback": [
- "\u001b[0;31m\u001b[0m",
- "\u001b[0;31mKeyboardInterrupt\u001b[0mTraceback (most recent call last)",
- "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mstatuses\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msagemaker_simple_estimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdescribe_rule_execution_jobs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
- "\u001b[0;32m/home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages/sagemaker/estimator.pyc\u001b[0m in \u001b[0;36mdescribe_rule_execution_jobs\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 524\u001b[0m \u001b[0mruleMonitoringStatuses\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjob_details\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"RuleMonitoringStatuses\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 525\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mall_rules_completed\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 526\u001b[0;31m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m60\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 527\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 528\u001b[0m \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"RuleMonitoringStatuses not found in this training job\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
- ]
- }
- ],
- "source": [
- "statuses = sagemaker_simple_estimator.describe_rule_execution_jobs()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Check the logs of the Rule Execution Job\n",
- "\n",
- "If you want to access the logs of a particular rule job name, you can do the following.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "rule_job_name = statuses[0].get('RuleExecutionJobName')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we can attach to this job to see its logs"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from sagemaker.estimator import Estimator\n",
- "vanishing_gradient_tensor = Estimator.attach(rule_job_name)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Receive a CloudWatch Event for Rules\n",
- "When the status of training job or rule execution job change (i.e. starting, failed), TrainingJobStatus [CloudWatch events](https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html) are emitted. More details on this, see [below](#CloudWatch-Event-Integration-for-Rules). "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Making this a good run\n",
- "\n",
- "In above example, we saw how a VanishingGradient Rule was run which analyzed the tensors when training was running and produced an alert in form of cloudwatch event.\n",
- "\n",
- "You can create the estimator with following *entry_point_script* and *bad_hyperparameters*. Start a new training job. You will see that VanishingGradient rule is not fired in that case as no tensors demonstrate vanishing gradient issue.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "entry_point_script = '../scripts/mnist_gluon_basic_hook_demo.py'\n",
- "good_hyperparameters = {'random_seed' : True, 'num_steps': 6}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "simple_estimator = MXNet(role=sagemaker.get_execution_role(),\n",
- " base_job_name='mxnet-trsl-test-nb',\n",
- " train_instance_count=1,\n",
- " train_instance_type='ml.m4.xlarge',\n",
- " image_name=cpu_docker_image_name,\n",
- " entry_point=entry_point_script,\n",
- " hyperparameters=good_hyperparameters,\n",
- " framework_version='1.4.1',\n",
- " debug=True,\n",
- " py_version='py3',\n",
- " rules_specification=[\n",
- " {\n",
- " \"RuleName\": \"VanishingGradient\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"start-step\" : \"1\",\n",
- " \"end-step\": \"5\"\n",
- " }\n",
- " }\n",
- " ])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "simple_estimator.fit(wait=False)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "simple_estimator.describe_rule_execution_jobs()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Enabling Tornasole with SageMaker\n",
- "#### Storage\n",
- "The tensors saved by Tornasole are, by default, stored in the S3 output path of the training job, under the folder **`/tensors-`**. This is done to ensure that we don't end up accidentally overwriting the tensors from a training job with the others. Rules evaluation require separation of the tensors paths to be evaluated correctly.\n",
- "\n",
- "If you don't provide an S3 output path to the estimator, SageMaker creates one for you as: **`s3://sagemaker--/`**\n",
- "\n",
- "This path is used to create a Tornasole Trial taken by Rules (see below).\n",
- "\n",
- "#### New Parameters \n",
- "The new parameters in Sagemaker Estimator to look out for are\n",
- "\n",
- "- `debug` :(bool)\n",
- "This indicates that debugging should be enabled for the training job. \n",
- "Setting this as `True` would make Tornasole available for use with the job\n",
- "\n",
- "- `rules_specification`: (list[*dict*])\n",
- "You can specify any number of rules to monitor your SageMaker training job. This parameter takes a list of python dictionaries, one for each rule you want to enable. Each `dict` is of the following form:\n",
- "```\n",
- "{\n",
- " \"RuleName\": \n",
- " # The name of the class implementing the Tornasole Rule interface. (required)\n",
- "\n",
- " \"SourceS3Uri\": \n",
- " # S3 URI of the rule script containing the class in 'RuleName'. \n",
- " # This is not required if you want to use one of the\n",
- " # First Party rules provided to you by Amazon. \n",
- " # In such a case you can leave it empty or not pass it. If you want to run a custom rule \n",
- " # defined by you, you will need to define the custom rule class in a python \n",
- " # file and provide it to SageMaker as a S3 URI. \n",
- " # SageMaker will fetch this file and try to look for the rule class \n",
- " # identified by RuleName in this file.\n",
- " \n",
- " \"InstanceType\": \n",
- " # The ML instance type which should be used to run the rule evaluation job\n",
- " \n",
- " \"VolumeSizeInGB\": \n",
- " # The volume size to store the runtime artifacts from the rule evaluation \n",
- " \n",
- " \"RuntimeConfigurations\": {\n",
- " # Map defining the parameters required to instantiate the Rule class and\n",
- " # parameters regarding invokation of the rule (start-step and end-step)\n",
- " # This can be any parameter taken by the rule. \n",
- " # Every value here needs to be a string. \n",
- " # So when you write custom rules, ensure that you can parse each argument from a string.\n",
- " #\n",
- " # PARAMS CAN BE\n",
- " #\n",
- " # STANDARD PARAMS FOR RULE EXECUTION\n",
- " # \"start-step\": \n",
- " # \"end-step\": \n",
- " # \"other-trials-paths\": (';' separated list of s3 paths as a string)\n",
- " # \"logging-level\": (can be one of \"CRITICAL\", \"FATAL\", \"ERROR\", \n",
- " # \"WARNING\", \"WARN\", \"DEBUG\", \"NOTSET\")\n",
- " #\n",
- " # ANY OTHER PARAMETER TAKEN BY THE RULE\n",
- " # \"parameter\" : \n",
- " # : \n",
- " }\n",
- "}\n",
- "```\n",
- "\n",
- "### Inputs\n",
- "Just a quick reminder if you are not familiar with script mode in SageMaker. You can pass command line arguments taken by your training script with a hyperparameter dictionary which gets passed to the SageMaker Estimator class. You can see this in the examples below.\n",
- "\n",
- "\n",
- "### Rules\n",
- "Rules are the medium by which Tornasole executes a certain piece of code regularly on different steps of the job.\n",
- "They can be used to assert certain conditions during training, and raise Cloudwatch Events based on them that you can\n",
- "use to process in any way you like. \n",
- "\n",
- "Tornasole comes with a set of **First Party rules** (1P rules).\n",
- "You can also write your own rules looking at these 1P rules for inspiration. \n",
- "Refer [DeveloperGuide_Rules.md](../../../../rules/DeveloperGuide_Rules.md) for more on the APIs you can use to write your own rules as well as descriptions for the 1P rules that we provide. \n",
- " \n",
- "Here we will talk about how to use Sagemaker to evalute these rules on the training jobs.\n",
- "\n",
- "\n",
- "##### 1P Rule \n",
- "If you want to use a 1P rule. Specify the RuleName field with the 1P RuleName, and the rule will be automatically applied. You can pass any parameters accepted by the rule as part of the RuntimeConfigurations dictionary. Rules constructor take trial as parameter. \n",
- "A Trial in Tornasole's context refers to a training job. It is identified by the path where the saved tensors for the job are stored. \n",
- "A rule takes a `base_trial` which refers to the job whose run invokes the rule execution. \n",
- "\n",
- "**Note:** A rule can be written to compare & analyze tensors across training jobs. A rule which needs to compare tensors across trials can be run by passing the argument `other_trials`. The argument `base_trial` will automatically be set by SageMaker when executing the rule. The parameter `other_trials` (if taken by the rule) can be passed by passing `other-trials-paths` in the RuntimeConfigurations dictionary. The value for this argument should be `;` separated list of S3 output paths where the tensors for those trials are stored.\n",
- "\n",
- "Here's a example of a complex configuration for the SimilarAcrossRuns (which accepts one other trial and a regex pattern) where we ask for the rule to be invoked for the steps between 10 and 100.\n",
- "\n",
- "``` \n",
- "rules_specification = [ \n",
- " {\n",
- " \"RuleName\": \"SimilarAcrossRuns\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"other_trials\": \"s3://sagemaker--/past-job\",\n",
- " \"include_regex\": \".*\",\n",
- " \"start-step\": \"10\",\n",
- " \"end-step\": \"100\"\n",
- " }\n",
- " }\n",
- "]\n",
- "```\n",
- "List of 1P rules and details about the rules can be found in *First party rules* section in [DeveloperGuide_Rules.md](../../../../rules/DeveloperGuide_Rules.md) \n",
- "\n",
- "\n",
- "##### Custom rule\n",
- "In this case you need to define a custom rule class which inherits from `smdebug.rules.Rule` class.\n",
- "You need to provide Sagemaker the S3 location of the file which defines your custom rule classes as the value for the field `SourceS3Uri`. Again, you can pass any arguments taken by this rule through the RuntimeConfigurations dictionary. Note that the custom rules can only have arguments which expect a string as the value except the two arguments specifying trials to the Rule. Refer section *Writing a rule* in [DeveloperGuide_Rules.md](../../../../rules/DeveloperGuide_Rules.md) for more details.\n",
- "\n",
- "Here's an example:\n",
- "```\n",
- "rules_specification = [\n",
- " {\n",
- " \"RuleName\": \"CustomRule\",\n",
- " \"SourceS3Uri\": \"s3://weiyou-tornasole-test/rule-script/custom_rule.py\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"threshold\" : \"0.5\"\n",
- " }\n",
- " }\n",
- "]\n",
- "```\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### CloudWatch Event Integration for Rules\n",
- "When the status of training job or rule execution job change (i.e. starting, failed), TrainingJobStatus [CloudWatch events](https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html) are emitted.\n",
- "\n",
- "After GA, you can configure a CloudWatch event rule to receive and process these events by setting up a target (Lambda function, SNS) as follows:\n",
- "\n",
- "- The SageMaker TrainingJobStatus CW event (https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#sagemaker_event_types) will include rule job statuses associated with the training job\n",
- "- A CW event will be emitted when a RuleStatus changes\n",
- "- Customer can create a CloudWatch event rule that monitors the Training Job customer started\n",
- "- Customer can set a Target (Lambda funtion, SQS) for the CloudWatch event rule that processes the event, and triggers an alarm for the customer based on the RuleStatus. \n",
- "\n",
- "Refer [this page](https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html) for more details. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "conda_amazonei_mxnet_p27",
- "language": "python",
- "name": "conda_amazonei_mxnet_p27"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 2
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython2",
- "version": "2.7.15"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/examples/pytorch/README.md b/examples/pytorch/README.md
new file mode 100644
index 000000000..9818d371a
--- /dev/null
+++ b/examples/pytorch/README.md
@@ -0,0 +1,2 @@
+## Example Notebooks
+Please refer to the example notebooks in [Amazon SageMaker Examples repository](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger)
diff --git a/examples/pytorch/notebooks/PyTorch-SimpleInteractiveAnalysis.ipynb b/examples/pytorch/notebooks/PyTorch-SimpleInteractiveAnalysis.ipynb
deleted file mode 100644
index 520295125..000000000
--- a/examples/pytorch/notebooks/PyTorch-SimpleInteractiveAnalysis.ipynb
+++ /dev/null
@@ -1,1559 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Simple Interactive Analysis in Tornasole\n",
- "This notebook will demonstrate the simplest kind of interactive analysis that can be run in smdebug. It will focus on the [vanishing/exploding gradient](https://medium.com/learn-love-ai/the-curious-case-of-the-vanishing-exploding-gradient-bf58ec6822eb) problems on a simple MNIST digit recognition."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "First of all, we will import some basic libraries for deep learning and plotting."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "import numpy as np\n",
- "import torch\n",
- "import torch.utils.data\n",
- "from torch import nn\n",
- "import matplotlib.pyplot as plt\n",
- "import torch.nn.functional as F\n",
- "import torch.optim as optim\n",
- "from torchvision import datasets, transforms\n",
- "from torch.autograd import Variable"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's copy the Tornasole libraries to this instance, this step has to be executed only once. Please make sure that the AWS account you are using can access the tornasole-wheels-alpha bucket.\n",
- "\n",
- "To do so you'll need the appropriate AWS credentials. There are several ways of doing this:\n",
- "\n",
- "inject temporary credentials\n",
- "if running on EC2, use EC2 roles that can access all S3 buckets\n",
- "(preferred) run this notebook on a SageMaker notebook instance\n",
- "The code below downloads the necessary .whl files and installs them in the current environment. Only run the first time!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "#WARNING - uncomment this code only if you haven't done this before\n",
- "#!aws s3 sync s3://tornasole-external-preview-use1/sdk/ts-binaries/tornasole_pytorch/py3/latest/ tornasole_pytorch/\n",
- "#!pip install tornasole_pytorch/*\n",
- "\n",
- "# If you run into a version conflict with boto, run the following\n",
- "# !pip uninstall -y botocore boto3 aioboto3 aiobotocore && pip install botocore==1.12.91 boto3==1.9.91 aiobotocore==0.10.2 aioboto3==6.4.1"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's import the Tornasole libraries, all we need is a `SessionHook` to use as a callback, as well as some ancillary data structures."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "from smdebug.pytorch.hook import *\n",
- "from smdebug.core.save_config import SaveConfig\n",
- "\n",
- "import logging\n",
- "logging.getLogger(\"tornasole\").setLevel(logging.ERROR)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can define a simple network - it doesn't really matter what it is.\n",
- "Importantly - we **add the Tornasole Hook**. This hook will be run at every batch and will save selected tensors (in this case, all of them) to the desired directory (in this case, `'./ts_output/{run_id}'`.\n",
- "\n",
- "See the documentation for more details."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "class Net(nn.Module):\n",
- " def __init__(self):\n",
- " super(Net, self).__init__()\n",
- "\n",
- " # self.conv1 = nn.Conv2d(1, 20, 5, 1)\n",
- " self.add_module('conv1', nn.Conv2d(1, 20, 5, 1))\n",
- " self.add_module('conv2', nn.Conv2d(20, 50, 5, 1))\n",
- " self.add_module('fc1', nn.Linear(4*4*50, 500))\n",
- " self.add_module('fc2', nn.Linear(500, 10))\n",
- "\n",
- " def forward(self, x):\n",
- " x = F.relu(self.conv1(x))\n",
- " x = F.max_pool2d(x, 2, 2)\n",
- " x = F.relu(self.conv2(x))\n",
- " x = F.max_pool2d(x, 2, 2)\n",
- " x = x.view(-1, 4*4*50)\n",
- " x = F.relu(self.fc1(x))\n",
- " x = self.fc2(x)\n",
- " return F.log_softmax(x, dim=1)\n",
- "\n",
- "def create_net(tornasole_save_interval, base_loc, run_id):\n",
- " model = Net()\n",
- " # Create and add the hook. Arguments:\n",
- " # - save data in './{base_loc}/{run_id} - Note: s3 is also supported\n",
- " # - save every 100 batches\n",
- " # - save every tensor: inputs/outputs to each layer, as well as gradients\n",
- " hook = SessionHook(out_dir=base_loc + \"/\" + run_id, save_config=SaveConfig(save_interval=100), save_all=True)\n",
- " hook.register_hook(model)\n",
- " return model"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And we create a simple training script. No Tornasole-specific code here, this is a slightly modified version of the [digit recognition](https://github.com/pytorch/examples/blob/master/mnist/main.py) example on the PyTorch github."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [],
- "source": [
- "def transformer(data, label):\n",
- " data = data.reshape((-1,)).astype(np.float32)/255\n",
- " return data, label\n",
- "\n",
- "def test(model, device, test_loader):\n",
- " model.eval()\n",
- " test_loss = 0\n",
- " correct = 0\n",
- " with torch.no_grad():\n",
- " for data, target in test_loader:\n",
- " data, target = data.to(device), target.to(device)\n",
- " output = model(data)\n",
- " test_loss += F.nll_loss(output, target, reduction='sum').item() # sum up batch loss\n",
- " pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability\n",
- " correct += pred.eq(target.view_as(pred)).sum().item()\n",
- "\n",
- " test_loss /= len(test_loader.dataset)\n",
- "\n",
- " print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
- " test_loss, correct, len(test_loader.dataset),\n",
- " 100. * correct / len(test_loader.dataset)))\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "def train(model, epochs, learning_rate, momentum, batch_size, device):\n",
- " train_loader = torch.utils.data.DataLoader(\n",
- " datasets.MNIST('./data', train=True, download=True,\n",
- " transform=transforms.Compose([\n",
- " transforms.ToTensor(),\n",
- " transforms.Normalize((0.1307,), (0.3081,))\n",
- " ])),\n",
- " batch_size=batch_size, shuffle=True)\n",
- "\n",
- " val_data = torch.utils.data.DataLoader(\n",
- " datasets.MNIST('./data', train=False, download=True,\n",
- " transform=transforms.Compose([\n",
- " transforms.ToTensor(),\n",
- " transforms.Normalize((0.1307,), (0.3081,))\n",
- " ])),\n",
- " batch_size=batch_size, shuffle=False)\n",
- " \n",
- " # Collect all parameters from net and its children, then initialize them.\n",
- " optimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=momentum)\n",
- " model = model.to(device)\n",
- " total_step = len(train_loader)\n",
- " for epoch in range(epochs):\n",
- " model.train()\n",
- " count = 0\n",
- " for batch_idx, (data, target) in enumerate(train_loader):\n",
- " data, target = data.to(device), target.to(device)\n",
- " optimizer.zero_grad()\n",
- " output = model(Variable(data, requires_grad = True))\n",
- " loss = F.nll_loss(output, target)\n",
- " loss.backward()\n",
- " count += 1\n",
- "\n",
- " optimizer.step()\n",
- " if batch_idx % 10 == 0:\n",
- " print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n",
- " epoch, batch_idx * len(data), len(train_loader.dataset),\n",
- " 100. * batch_idx / len(train_loader), loss.item()))\n",
- " \n",
- "# torch.save(model.state_dict(),\"mnist_params.pt\")\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Clear up from previous runs, we remove old data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [],
- "source": [
- "!rm -rf ./ts_output/"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "At this point we are ready to train. For the purposes of this example, we will name this run as `'good'` because we know it will converge to a good solution. \n",
- "\n",
- "If you have a GPU on your machine, you can change the device line appropriately -- e.g for an NVIDIA GPU, it would be `device = torch.device(\"cuda\")`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Train Epoch: 0 [0/60000 (0%)]\tLoss: 2.295261\n",
- "Train Epoch: 0 [640/60000 (1%)]\tLoss: 1.748892\n",
- "Train Epoch: 0 [1280/60000 (2%)]\tLoss: 0.695785\n",
- "Train Epoch: 0 [1920/60000 (3%)]\tLoss: 0.629404\n",
- "Train Epoch: 0 [2560/60000 (4%)]\tLoss: 0.355879\n",
- "Train Epoch: 0 [3200/60000 (5%)]\tLoss: 0.461338\n",
- "Train Epoch: 0 [3840/60000 (6%)]\tLoss: 0.465871\n",
- "Train Epoch: 0 [4480/60000 (7%)]\tLoss: 0.286747\n",
- "Train Epoch: 0 [5120/60000 (9%)]\tLoss: 0.262511\n",
- "Train Epoch: 0 [5760/60000 (10%)]\tLoss: 0.276488\n",
- "Train Epoch: 0 [6400/60000 (11%)]\tLoss: 0.234418\n",
- "Train Epoch: 0 [7040/60000 (12%)]\tLoss: 0.523879\n",
- "Train Epoch: 0 [7680/60000 (13%)]\tLoss: 0.582521\n",
- "Train Epoch: 0 [8320/60000 (14%)]\tLoss: 0.257838\n",
- "Train Epoch: 0 [8960/60000 (15%)]\tLoss: 0.162327\n",
- "Train Epoch: 0 [9600/60000 (16%)]\tLoss: 0.202325\n",
- "Train Epoch: 0 [10240/60000 (17%)]\tLoss: 0.204970\n",
- "Train Epoch: 0 [10880/60000 (18%)]\tLoss: 0.517194\n",
- "Train Epoch: 0 [11520/60000 (19%)]\tLoss: 0.138784\n",
- "Train Epoch: 0 [12160/60000 (20%)]\tLoss: 0.174099\n",
- "Train Epoch: 0 [12800/60000 (21%)]\tLoss: 0.358449\n",
- "Train Epoch: 0 [13440/60000 (22%)]\tLoss: 0.234432\n",
- "Train Epoch: 0 [14080/60000 (23%)]\tLoss: 0.239579\n",
- "Train Epoch: 0 [14720/60000 (25%)]\tLoss: 0.143086\n",
- "Train Epoch: 0 [15360/60000 (26%)]\tLoss: 0.464434\n",
- "Train Epoch: 0 [16000/60000 (27%)]\tLoss: 0.213992\n",
- "Train Epoch: 0 [16640/60000 (28%)]\tLoss: 0.198865\n",
- "Train Epoch: 0 [17280/60000 (29%)]\tLoss: 0.426876\n",
- "Train Epoch: 0 [17920/60000 (30%)]\tLoss: 0.236948\n",
- "Train Epoch: 0 [18560/60000 (31%)]\tLoss: 0.019676\n",
- "Train Epoch: 0 [19200/60000 (32%)]\tLoss: 0.365207\n",
- "Train Epoch: 0 [19840/60000 (33%)]\tLoss: 0.177355\n",
- "Train Epoch: 0 [20480/60000 (34%)]\tLoss: 0.101089\n",
- "Train Epoch: 0 [21120/60000 (35%)]\tLoss: 0.217447\n",
- "Train Epoch: 0 [21760/60000 (36%)]\tLoss: 0.122373\n",
- "Train Epoch: 0 [22400/60000 (37%)]\tLoss: 0.083280\n",
- "Train Epoch: 0 [23040/60000 (38%)]\tLoss: 0.044131\n",
- "Train Epoch: 0 [23680/60000 (39%)]\tLoss: 0.164748\n",
- "Train Epoch: 0 [24320/60000 (41%)]\tLoss: 0.217344\n",
- "Train Epoch: 0 [24960/60000 (42%)]\tLoss: 0.168444\n",
- "Train Epoch: 0 [25600/60000 (43%)]\tLoss: 0.094563\n",
- "Train Epoch: 0 [26240/60000 (44%)]\tLoss: 0.160949\n",
- "Train Epoch: 0 [26880/60000 (45%)]\tLoss: 0.190556\n",
- "Train Epoch: 0 [27520/60000 (46%)]\tLoss: 0.049234\n",
- "Train Epoch: 0 [28160/60000 (47%)]\tLoss: 0.106162\n",
- "Train Epoch: 0 [28800/60000 (48%)]\tLoss: 0.104469\n",
- "Train Epoch: 0 [29440/60000 (49%)]\tLoss: 0.053504\n",
- "Train Epoch: 0 [30080/60000 (50%)]\tLoss: 0.110556\n",
- "Train Epoch: 0 [30720/60000 (51%)]\tLoss: 0.322309\n",
- "Train Epoch: 0 [31360/60000 (52%)]\tLoss: 0.086979\n",
- "Train Epoch: 0 [32000/60000 (53%)]\tLoss: 0.134221\n",
- "Train Epoch: 0 [32640/60000 (54%)]\tLoss: 0.201668\n",
- "Train Epoch: 0 [33280/60000 (55%)]\tLoss: 0.062239\n",
- "Train Epoch: 0 [33920/60000 (57%)]\tLoss: 0.107864\n",
- "Train Epoch: 0 [34560/60000 (58%)]\tLoss: 0.082699\n",
- "Train Epoch: 0 [35200/60000 (59%)]\tLoss: 0.187130\n",
- "Train Epoch: 0 [35840/60000 (60%)]\tLoss: 0.436204\n",
- "Train Epoch: 0 [36480/60000 (61%)]\tLoss: 0.216804\n",
- "Train Epoch: 0 [37120/60000 (62%)]\tLoss: 0.295533\n",
- "Train Epoch: 0 [37760/60000 (63%)]\tLoss: 0.151546\n",
- "Train Epoch: 0 [38400/60000 (64%)]\tLoss: 0.109190\n",
- "Train Epoch: 0 [39040/60000 (65%)]\tLoss: 0.089795\n",
- "Train Epoch: 0 [39680/60000 (66%)]\tLoss: 0.111167\n",
- "Train Epoch: 0 [40320/60000 (67%)]\tLoss: 0.148806\n",
- "Train Epoch: 0 [40960/60000 (68%)]\tLoss: 0.086549\n",
- "Train Epoch: 0 [41600/60000 (69%)]\tLoss: 0.166614\n",
- "Train Epoch: 0 [42240/60000 (70%)]\tLoss: 0.076532\n",
- "Train Epoch: 0 [42880/60000 (71%)]\tLoss: 0.207414\n",
- "Train Epoch: 0 [43520/60000 (72%)]\tLoss: 0.057692\n",
- "Train Epoch: 0 [44160/60000 (74%)]\tLoss: 0.135699\n",
- "Train Epoch: 0 [44800/60000 (75%)]\tLoss: 0.070328\n",
- "Train Epoch: 0 [45440/60000 (76%)]\tLoss: 0.287908\n",
- "Train Epoch: 0 [46080/60000 (77%)]\tLoss: 0.181923\n",
- "Train Epoch: 0 [46720/60000 (78%)]\tLoss: 0.109931\n",
- "Train Epoch: 0 [47360/60000 (79%)]\tLoss: 0.082871\n",
- "Train Epoch: 0 [48000/60000 (80%)]\tLoss: 0.336507\n",
- "Train Epoch: 0 [48640/60000 (81%)]\tLoss: 0.132857\n",
- "Train Epoch: 0 [49280/60000 (82%)]\tLoss: 0.124299\n",
- "Train Epoch: 0 [49920/60000 (83%)]\tLoss: 0.064722\n",
- "Train Epoch: 0 [50560/60000 (84%)]\tLoss: 0.102338\n",
- "Train Epoch: 0 [51200/60000 (85%)]\tLoss: 0.081316\n",
- "Train Epoch: 0 [51840/60000 (86%)]\tLoss: 0.023367\n",
- "Train Epoch: 0 [52480/60000 (87%)]\tLoss: 0.009987\n",
- "Train Epoch: 0 [53120/60000 (88%)]\tLoss: 0.021232\n",
- "Train Epoch: 0 [53760/60000 (90%)]\tLoss: 0.234437\n",
- "Train Epoch: 0 [54400/60000 (91%)]\tLoss: 0.076132\n",
- "Train Epoch: 0 [55040/60000 (92%)]\tLoss: 0.099620\n",
- "Train Epoch: 0 [55680/60000 (93%)]\tLoss: 0.153108\n",
- "Train Epoch: 0 [56320/60000 (94%)]\tLoss: 0.008535\n",
- "Train Epoch: 0 [56960/60000 (95%)]\tLoss: 0.069565\n",
- "Train Epoch: 0 [57600/60000 (96%)]\tLoss: 0.345571\n",
- "Train Epoch: 0 [58240/60000 (97%)]\tLoss: 0.502152\n",
- "Train Epoch: 0 [58880/60000 (98%)]\tLoss: 0.153598\n",
- "Train Epoch: 0 [59520/60000 (99%)]\tLoss: 0.009579\n",
- "Train Epoch: 1 [0/60000 (0%)]\tLoss: 0.290723\n",
- "Train Epoch: 1 [640/60000 (1%)]\tLoss: 0.145595\n",
- "Train Epoch: 1 [1280/60000 (2%)]\tLoss: 0.069823\n",
- "Train Epoch: 1 [1920/60000 (3%)]\tLoss: 0.064840\n",
- "Train Epoch: 1 [2560/60000 (4%)]\tLoss: 0.006950\n",
- "Train Epoch: 1 [3200/60000 (5%)]\tLoss: 0.000492\n",
- "Train Epoch: 1 [3840/60000 (6%)]\tLoss: 0.194620\n",
- "Train Epoch: 1 [4480/60000 (7%)]\tLoss: 0.240464\n",
- "Train Epoch: 1 [5120/60000 (9%)]\tLoss: 0.073014\n",
- "Train Epoch: 1 [5760/60000 (10%)]\tLoss: 0.177253\n",
- "Train Epoch: 1 [6400/60000 (11%)]\tLoss: 0.048263\n",
- "Train Epoch: 1 [7040/60000 (12%)]\tLoss: 0.036351\n",
- "Train Epoch: 1 [7680/60000 (13%)]\tLoss: 0.011458\n",
- "Train Epoch: 1 [8320/60000 (14%)]\tLoss: 0.341599\n",
- "Train Epoch: 1 [8960/60000 (15%)]\tLoss: 0.373527\n",
- "Train Epoch: 1 [9600/60000 (16%)]\tLoss: 0.047152\n",
- "Train Epoch: 1 [10240/60000 (17%)]\tLoss: 0.124424\n",
- "Train Epoch: 1 [10880/60000 (18%)]\tLoss: 0.138054\n",
- "Train Epoch: 1 [11520/60000 (19%)]\tLoss: 0.314980\n",
- "Train Epoch: 1 [12160/60000 (20%)]\tLoss: 0.244121\n",
- "Train Epoch: 1 [12800/60000 (21%)]\tLoss: 0.194258\n",
- "Train Epoch: 1 [13440/60000 (22%)]\tLoss: 0.092362\n",
- "Train Epoch: 1 [14080/60000 (23%)]\tLoss: 0.205115\n",
- "Train Epoch: 1 [14720/60000 (25%)]\tLoss: 0.202674\n",
- "Train Epoch: 1 [15360/60000 (26%)]\tLoss: 0.189018\n",
- "Train Epoch: 1 [16000/60000 (27%)]\tLoss: 0.168465\n",
- "Train Epoch: 1 [16640/60000 (28%)]\tLoss: 0.075228\n",
- "Train Epoch: 1 [17280/60000 (29%)]\tLoss: 0.024219\n",
- "Train Epoch: 1 [17920/60000 (30%)]\tLoss: 0.249284\n",
- "Train Epoch: 1 [18560/60000 (31%)]\tLoss: 0.055043\n",
- "Train Epoch: 1 [19200/60000 (32%)]\tLoss: 0.199740\n",
- "Train Epoch: 1 [19840/60000 (33%)]\tLoss: 0.264624\n",
- "Train Epoch: 1 [20480/60000 (34%)]\tLoss: 0.145213\n",
- "Train Epoch: 1 [21120/60000 (35%)]\tLoss: 0.182477\n",
- "Train Epoch: 1 [21760/60000 (36%)]\tLoss: 0.181954\n",
- "Train Epoch: 1 [22400/60000 (37%)]\tLoss: 0.041947\n",
- "Train Epoch: 1 [23040/60000 (38%)]\tLoss: 0.165648\n",
- "Train Epoch: 1 [23680/60000 (39%)]\tLoss: 0.075048\n",
- "Train Epoch: 1 [24320/60000 (41%)]\tLoss: 0.091085\n",
- "Train Epoch: 1 [24960/60000 (42%)]\tLoss: 0.267341\n",
- "Train Epoch: 1 [25600/60000 (43%)]\tLoss: 0.419169\n",
- "Train Epoch: 1 [26240/60000 (44%)]\tLoss: 0.397417\n",
- "Train Epoch: 1 [26880/60000 (45%)]\tLoss: 0.059258\n",
- "Train Epoch: 1 [27520/60000 (46%)]\tLoss: 0.678994\n",
- "Train Epoch: 1 [28160/60000 (47%)]\tLoss: 0.097712\n",
- "Train Epoch: 1 [28800/60000 (48%)]\tLoss: 0.078830\n",
- "Train Epoch: 1 [29440/60000 (49%)]\tLoss: 0.083803\n",
- "Train Epoch: 1 [30080/60000 (50%)]\tLoss: 0.373137\n",
- "Train Epoch: 1 [30720/60000 (51%)]\tLoss: 0.317618\n",
- "Train Epoch: 1 [31360/60000 (52%)]\tLoss: 0.076827\n",
- "Train Epoch: 1 [32000/60000 (53%)]\tLoss: 0.125064\n",
- "Train Epoch: 1 [32640/60000 (54%)]\tLoss: 0.057970\n",
- "Train Epoch: 1 [33280/60000 (55%)]\tLoss: 0.167010\n",
- "Train Epoch: 1 [33920/60000 (57%)]\tLoss: 0.026072\n",
- "Train Epoch: 1 [34560/60000 (58%)]\tLoss: 0.160082\n",
- "Train Epoch: 1 [35200/60000 (59%)]\tLoss: 0.046618\n",
- "Train Epoch: 1 [35840/60000 (60%)]\tLoss: 0.050997\n",
- "Train Epoch: 1 [36480/60000 (61%)]\tLoss: 0.370405\n",
- "Train Epoch: 1 [37120/60000 (62%)]\tLoss: 0.106518\n",
- "Train Epoch: 1 [37760/60000 (63%)]\tLoss: 0.101690\n",
- "Train Epoch: 1 [38400/60000 (64%)]\tLoss: 0.064859\n",
- "Train Epoch: 1 [39040/60000 (65%)]\tLoss: 0.079881\n",
- "Train Epoch: 1 [39680/60000 (66%)]\tLoss: 0.110059\n",
- "Train Epoch: 1 [40320/60000 (67%)]\tLoss: 0.067634\n",
- "Train Epoch: 1 [40960/60000 (68%)]\tLoss: 0.208821\n",
- "Train Epoch: 1 [41600/60000 (69%)]\tLoss: 0.088838\n",
- "Train Epoch: 1 [42240/60000 (70%)]\tLoss: 0.079848\n",
- "Train Epoch: 1 [42880/60000 (71%)]\tLoss: 0.193431\n",
- "Train Epoch: 1 [43520/60000 (72%)]\tLoss: 0.171546\n",
- "Train Epoch: 1 [44160/60000 (74%)]\tLoss: 0.178438\n",
- "Train Epoch: 1 [44800/60000 (75%)]\tLoss: 0.169155\n",
- "Train Epoch: 1 [45440/60000 (76%)]\tLoss: 0.055189\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Train Epoch: 1 [46080/60000 (77%)]\tLoss: 0.028144\n",
- "Train Epoch: 1 [46720/60000 (78%)]\tLoss: 0.021224\n",
- "Train Epoch: 1 [47360/60000 (79%)]\tLoss: 0.071242\n",
- "Train Epoch: 1 [48000/60000 (80%)]\tLoss: 0.010806\n",
- "Train Epoch: 1 [48640/60000 (81%)]\tLoss: 0.110852\n",
- "Train Epoch: 1 [49280/60000 (82%)]\tLoss: 0.010513\n",
- "Train Epoch: 1 [49920/60000 (83%)]\tLoss: 0.364887\n",
- "Train Epoch: 1 [50560/60000 (84%)]\tLoss: 0.013409\n",
- "Train Epoch: 1 [51200/60000 (85%)]\tLoss: 0.419877\n",
- "Train Epoch: 1 [51840/60000 (86%)]\tLoss: 0.067204\n",
- "Train Epoch: 1 [52480/60000 (87%)]\tLoss: 0.067301\n",
- "Train Epoch: 1 [53120/60000 (88%)]\tLoss: 0.179633\n",
- "Train Epoch: 1 [53760/60000 (90%)]\tLoss: 0.168005\n",
- "Train Epoch: 1 [54400/60000 (91%)]\tLoss: 0.053020\n",
- "Train Epoch: 1 [55040/60000 (92%)]\tLoss: 0.127693\n",
- "Train Epoch: 1 [55680/60000 (93%)]\tLoss: 0.000790\n",
- "Train Epoch: 1 [56320/60000 (94%)]\tLoss: 0.638172\n",
- "Train Epoch: 1 [56960/60000 (95%)]\tLoss: 0.190752\n",
- "Train Epoch: 1 [57600/60000 (96%)]\tLoss: 0.001666\n",
- "Train Epoch: 1 [58240/60000 (97%)]\tLoss: 0.040600\n",
- "Train Epoch: 1 [58880/60000 (98%)]\tLoss: 0.218874\n",
- "Train Epoch: 1 [59520/60000 (99%)]\tLoss: 0.056282\n",
- "Train Epoch: 2 [0/60000 (0%)]\tLoss: 0.076416\n",
- "Train Epoch: 2 [640/60000 (1%)]\tLoss: 0.112750\n",
- "Train Epoch: 2 [1280/60000 (2%)]\tLoss: 0.171557\n",
- "Train Epoch: 2 [1920/60000 (3%)]\tLoss: 0.014248\n",
- "Train Epoch: 2 [2560/60000 (4%)]\tLoss: 0.062175\n",
- "Train Epoch: 2 [3200/60000 (5%)]\tLoss: 0.182906\n",
- "Train Epoch: 2 [3840/60000 (6%)]\tLoss: 0.135498\n",
- "Train Epoch: 2 [4480/60000 (7%)]\tLoss: 0.123534\n",
- "Train Epoch: 2 [5120/60000 (9%)]\tLoss: 0.122674\n",
- "Train Epoch: 2 [5760/60000 (10%)]\tLoss: 0.111532\n",
- "Train Epoch: 2 [6400/60000 (11%)]\tLoss: 0.099646\n",
- "Train Epoch: 2 [7040/60000 (12%)]\tLoss: 0.051113\n",
- "Train Epoch: 2 [7680/60000 (13%)]\tLoss: 0.227051\n",
- "Train Epoch: 2 [8320/60000 (14%)]\tLoss: 0.138824\n",
- "Train Epoch: 2 [8960/60000 (15%)]\tLoss: 0.088158\n",
- "Train Epoch: 2 [9600/60000 (16%)]\tLoss: 0.103052\n",
- "Train Epoch: 2 [10240/60000 (17%)]\tLoss: 0.369061\n",
- "Train Epoch: 2 [10880/60000 (18%)]\tLoss: 0.165350\n",
- "Train Epoch: 2 [11520/60000 (19%)]\tLoss: 0.142054\n",
- "Train Epoch: 2 [12160/60000 (20%)]\tLoss: 0.034043\n",
- "Train Epoch: 2 [12800/60000 (21%)]\tLoss: 0.093324\n",
- "Train Epoch: 2 [13440/60000 (22%)]\tLoss: 0.129838\n",
- "Train Epoch: 2 [14080/60000 (23%)]\tLoss: 0.023088\n",
- "Train Epoch: 2 [14720/60000 (25%)]\tLoss: 0.110030\n",
- "Train Epoch: 2 [15360/60000 (26%)]\tLoss: 0.355520\n",
- "Train Epoch: 2 [16000/60000 (27%)]\tLoss: 0.072964\n",
- "Train Epoch: 2 [16640/60000 (28%)]\tLoss: 0.002617\n",
- "Train Epoch: 2 [17280/60000 (29%)]\tLoss: 0.415827\n",
- "Train Epoch: 2 [17920/60000 (30%)]\tLoss: 0.061368\n",
- "Train Epoch: 2 [18560/60000 (31%)]\tLoss: 0.059896\n",
- "Train Epoch: 2 [19200/60000 (32%)]\tLoss: 0.006614\n",
- "Train Epoch: 2 [19840/60000 (33%)]\tLoss: 0.072400\n",
- "Train Epoch: 2 [20480/60000 (34%)]\tLoss: 0.084101\n",
- "Train Epoch: 2 [21120/60000 (35%)]\tLoss: 0.060527\n",
- "Train Epoch: 2 [21760/60000 (36%)]\tLoss: 0.245168\n",
- "Train Epoch: 2 [22400/60000 (37%)]\tLoss: 0.104240\n",
- "Train Epoch: 2 [23040/60000 (38%)]\tLoss: 0.039879\n",
- "Train Epoch: 2 [23680/60000 (39%)]\tLoss: 0.127886\n",
- "Train Epoch: 2 [24320/60000 (41%)]\tLoss: 0.151734\n",
- "Train Epoch: 2 [24960/60000 (42%)]\tLoss: 0.156671\n",
- "Train Epoch: 2 [25600/60000 (43%)]\tLoss: 0.109427\n",
- "Train Epoch: 2 [26240/60000 (44%)]\tLoss: 0.080561\n",
- "Train Epoch: 2 [26880/60000 (45%)]\tLoss: 0.092277\n",
- "Train Epoch: 2 [27520/60000 (46%)]\tLoss: 0.210802\n",
- "Train Epoch: 2 [28160/60000 (47%)]\tLoss: 0.057986\n",
- "Train Epoch: 2 [28800/60000 (48%)]\tLoss: 0.124804\n",
- "Train Epoch: 2 [29440/60000 (49%)]\tLoss: 0.119243\n",
- "Train Epoch: 2 [30080/60000 (50%)]\tLoss: 0.279113\n",
- "Train Epoch: 2 [30720/60000 (51%)]\tLoss: 0.214091\n",
- "Train Epoch: 2 [31360/60000 (52%)]\tLoss: 0.106701\n",
- "Train Epoch: 2 [32000/60000 (53%)]\tLoss: 0.547162\n",
- "Train Epoch: 2 [32640/60000 (54%)]\tLoss: 0.377766\n",
- "Train Epoch: 2 [33280/60000 (55%)]\tLoss: 0.128663\n",
- "Train Epoch: 2 [33920/60000 (57%)]\tLoss: 0.078428\n",
- "Train Epoch: 2 [34560/60000 (58%)]\tLoss: 0.096952\n",
- "Train Epoch: 2 [35200/60000 (59%)]\tLoss: 0.047050\n",
- "Train Epoch: 2 [35840/60000 (60%)]\tLoss: 0.106311\n",
- "Train Epoch: 2 [36480/60000 (61%)]\tLoss: 0.092369\n",
- "Train Epoch: 2 [37120/60000 (62%)]\tLoss: 0.038745\n",
- "Train Epoch: 2 [37760/60000 (63%)]\tLoss: 0.474230\n",
- "Train Epoch: 2 [38400/60000 (64%)]\tLoss: 0.213040\n",
- "Train Epoch: 2 [39040/60000 (65%)]\tLoss: 0.665591\n",
- "Train Epoch: 2 [39680/60000 (66%)]\tLoss: 0.068594\n",
- "Train Epoch: 2 [40320/60000 (67%)]\tLoss: 0.036250\n",
- "Train Epoch: 2 [40960/60000 (68%)]\tLoss: 0.144957\n",
- "Train Epoch: 2 [41600/60000 (69%)]\tLoss: 0.355639\n",
- "Train Epoch: 2 [42240/60000 (70%)]\tLoss: 0.198450\n",
- "Train Epoch: 2 [42880/60000 (71%)]\tLoss: 0.221584\n",
- "Train Epoch: 2 [43520/60000 (72%)]\tLoss: 0.043087\n",
- "Train Epoch: 2 [44160/60000 (74%)]\tLoss: 0.053449\n",
- "Train Epoch: 2 [44800/60000 (75%)]\tLoss: 0.244004\n",
- "Train Epoch: 2 [45440/60000 (76%)]\tLoss: 0.051597\n",
- "Train Epoch: 2 [46080/60000 (77%)]\tLoss: 0.018794\n",
- "Train Epoch: 2 [46720/60000 (78%)]\tLoss: 0.047302\n",
- "Train Epoch: 2 [47360/60000 (79%)]\tLoss: 0.233751\n",
- "Train Epoch: 2 [48000/60000 (80%)]\tLoss: 0.523653\n",
- "Train Epoch: 2 [48640/60000 (81%)]\tLoss: 0.011048\n",
- "Train Epoch: 2 [49280/60000 (82%)]\tLoss: 0.185908\n",
- "Train Epoch: 2 [49920/60000 (83%)]\tLoss: 0.085652\n",
- "Train Epoch: 2 [50560/60000 (84%)]\tLoss: 0.065321\n",
- "Train Epoch: 2 [51200/60000 (85%)]\tLoss: 0.174393\n",
- "Train Epoch: 2 [51840/60000 (86%)]\tLoss: 0.031607\n",
- "Train Epoch: 2 [52480/60000 (87%)]\tLoss: 0.174475\n",
- "Train Epoch: 2 [53120/60000 (88%)]\tLoss: 0.217395\n",
- "Train Epoch: 2 [53760/60000 (90%)]\tLoss: 0.061645\n",
- "Train Epoch: 2 [54400/60000 (91%)]\tLoss: 0.141715\n",
- "Train Epoch: 2 [55040/60000 (92%)]\tLoss: 0.198288\n",
- "Train Epoch: 2 [55680/60000 (93%)]\tLoss: 0.254158\n",
- "Train Epoch: 2 [56320/60000 (94%)]\tLoss: 0.110041\n",
- "Train Epoch: 2 [56960/60000 (95%)]\tLoss: 0.270937\n",
- "Train Epoch: 2 [57600/60000 (96%)]\tLoss: 0.070328\n",
- "Train Epoch: 2 [58240/60000 (97%)]\tLoss: 0.024610\n",
- "Train Epoch: 2 [58880/60000 (98%)]\tLoss: 0.236358\n",
- "Train Epoch: 2 [59520/60000 (99%)]\tLoss: 0.117915\n",
- "Train Epoch: 3 [0/60000 (0%)]\tLoss: 0.146749\n",
- "Train Epoch: 3 [640/60000 (1%)]\tLoss: 0.039942\n",
- "Train Epoch: 3 [1280/60000 (2%)]\tLoss: 0.005945\n",
- "Train Epoch: 3 [1920/60000 (3%)]\tLoss: 0.118340\n",
- "Train Epoch: 3 [2560/60000 (4%)]\tLoss: 0.212263\n",
- "Train Epoch: 3 [3200/60000 (5%)]\tLoss: 0.108361\n",
- "Train Epoch: 3 [3840/60000 (6%)]\tLoss: 0.123859\n",
- "Train Epoch: 3 [4480/60000 (7%)]\tLoss: 0.151609\n",
- "Train Epoch: 3 [5120/60000 (9%)]\tLoss: 0.190431\n",
- "Train Epoch: 3 [5760/60000 (10%)]\tLoss: 0.044887\n",
- "Train Epoch: 3 [6400/60000 (11%)]\tLoss: 0.118531\n",
- "Train Epoch: 3 [7040/60000 (12%)]\tLoss: 0.175035\n",
- "Train Epoch: 3 [7680/60000 (13%)]\tLoss: 0.116727\n",
- "Train Epoch: 3 [8320/60000 (14%)]\tLoss: 0.233833\n",
- "Train Epoch: 3 [8960/60000 (15%)]\tLoss: 0.088643\n",
- "Train Epoch: 3 [9600/60000 (16%)]\tLoss: 0.384525\n",
- "Train Epoch: 3 [10240/60000 (17%)]\tLoss: 0.043991\n",
- "Train Epoch: 3 [10880/60000 (18%)]\tLoss: 0.851141\n",
- "Train Epoch: 3 [11520/60000 (19%)]\tLoss: 0.094810\n",
- "Train Epoch: 3 [12160/60000 (20%)]\tLoss: 0.083756\n",
- "Train Epoch: 3 [12800/60000 (21%)]\tLoss: 0.222417\n",
- "Train Epoch: 3 [13440/60000 (22%)]\tLoss: 0.448306\n",
- "Train Epoch: 3 [14080/60000 (23%)]\tLoss: 0.037722\n",
- "Train Epoch: 3 [14720/60000 (25%)]\tLoss: 0.318071\n",
- "Train Epoch: 3 [15360/60000 (26%)]\tLoss: 0.241556\n",
- "Train Epoch: 3 [16000/60000 (27%)]\tLoss: 0.034820\n",
- "Train Epoch: 3 [16640/60000 (28%)]\tLoss: 0.154392\n",
- "Train Epoch: 3 [17280/60000 (29%)]\tLoss: 0.103088\n",
- "Train Epoch: 3 [17920/60000 (30%)]\tLoss: 0.309260\n",
- "Train Epoch: 3 [18560/60000 (31%)]\tLoss: 0.129015\n",
- "Train Epoch: 3 [19200/60000 (32%)]\tLoss: 0.081739\n",
- "Train Epoch: 3 [19840/60000 (33%)]\tLoss: 0.194573\n",
- "Train Epoch: 3 [20480/60000 (34%)]\tLoss: 0.100797\n",
- "Train Epoch: 3 [21120/60000 (35%)]\tLoss: 0.130121\n",
- "Train Epoch: 3 [21760/60000 (36%)]\tLoss: 0.148123\n",
- "Train Epoch: 3 [22400/60000 (37%)]\tLoss: 0.107668\n",
- "Train Epoch: 3 [23040/60000 (38%)]\tLoss: 0.118747\n",
- "Train Epoch: 3 [23680/60000 (39%)]\tLoss: 0.145568\n",
- "Train Epoch: 3 [24320/60000 (41%)]\tLoss: 0.228613\n",
- "Train Epoch: 3 [24960/60000 (42%)]\tLoss: 0.125414\n",
- "Train Epoch: 3 [25600/60000 (43%)]\tLoss: 0.083142\n",
- "Train Epoch: 3 [26240/60000 (44%)]\tLoss: 0.394818\n",
- "Train Epoch: 3 [26880/60000 (45%)]\tLoss: 0.045244\n",
- "Train Epoch: 3 [27520/60000 (46%)]\tLoss: 0.005072\n",
- "Train Epoch: 3 [28160/60000 (47%)]\tLoss: 0.115797\n",
- "Train Epoch: 3 [28800/60000 (48%)]\tLoss: 0.095257\n",
- "Train Epoch: 3 [29440/60000 (49%)]\tLoss: 0.005111\n",
- "Train Epoch: 3 [30080/60000 (50%)]\tLoss: 0.110229\n",
- "Train Epoch: 3 [30720/60000 (51%)]\tLoss: 0.082010\n",
- "Train Epoch: 3 [31360/60000 (52%)]\tLoss: 0.055340\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Train Epoch: 3 [32000/60000 (53%)]\tLoss: 0.195543\n",
- "Train Epoch: 3 [32640/60000 (54%)]\tLoss: 0.150267\n",
- "Train Epoch: 3 [33280/60000 (55%)]\tLoss: 0.177324\n",
- "Train Epoch: 3 [33920/60000 (57%)]\tLoss: 0.038098\n",
- "Train Epoch: 3 [34560/60000 (58%)]\tLoss: 0.036462\n",
- "Train Epoch: 3 [35200/60000 (59%)]\tLoss: 0.076419\n",
- "Train Epoch: 3 [35840/60000 (60%)]\tLoss: 0.094155\n",
- "Train Epoch: 3 [36480/60000 (61%)]\tLoss: 0.276453\n",
- "Train Epoch: 3 [37120/60000 (62%)]\tLoss: 0.013989\n",
- "Train Epoch: 3 [37760/60000 (63%)]\tLoss: 0.033511\n",
- "Train Epoch: 3 [38400/60000 (64%)]\tLoss: 0.053062\n",
- "Train Epoch: 3 [39040/60000 (65%)]\tLoss: 0.002972\n",
- "Train Epoch: 3 [39680/60000 (66%)]\tLoss: 0.044364\n",
- "Train Epoch: 3 [40320/60000 (67%)]\tLoss: 0.201063\n",
- "Train Epoch: 3 [40960/60000 (68%)]\tLoss: 0.239112\n",
- "Train Epoch: 3 [41600/60000 (69%)]\tLoss: 0.301890\n",
- "Train Epoch: 3 [42240/60000 (70%)]\tLoss: 0.209023\n",
- "Train Epoch: 3 [42880/60000 (71%)]\tLoss: 0.340197\n",
- "Train Epoch: 3 [43520/60000 (72%)]\tLoss: 0.051514\n",
- "Train Epoch: 3 [44160/60000 (74%)]\tLoss: 0.163148\n",
- "Train Epoch: 3 [44800/60000 (75%)]\tLoss: 0.035544\n",
- "Train Epoch: 3 [45440/60000 (76%)]\tLoss: 0.105758\n",
- "Train Epoch: 3 [46080/60000 (77%)]\tLoss: 0.091835\n",
- "Train Epoch: 3 [46720/60000 (78%)]\tLoss: 0.218505\n",
- "Train Epoch: 3 [47360/60000 (79%)]\tLoss: 0.212545\n",
- "Train Epoch: 3 [48000/60000 (80%)]\tLoss: 0.001972\n",
- "Train Epoch: 3 [48640/60000 (81%)]\tLoss: 0.165325\n",
- "Train Epoch: 3 [49280/60000 (82%)]\tLoss: 0.099900\n",
- "Train Epoch: 3 [49920/60000 (83%)]\tLoss: 0.475469\n",
- "Train Epoch: 3 [50560/60000 (84%)]\tLoss: 0.102674\n",
- "Train Epoch: 3 [51200/60000 (85%)]\tLoss: 0.067554\n",
- "Train Epoch: 3 [51840/60000 (86%)]\tLoss: 0.376874\n",
- "Train Epoch: 3 [52480/60000 (87%)]\tLoss: 0.133132\n",
- "Train Epoch: 3 [53120/60000 (88%)]\tLoss: 0.042010\n",
- "Train Epoch: 3 [53760/60000 (90%)]\tLoss: 0.008966\n",
- "Train Epoch: 3 [54400/60000 (91%)]\tLoss: 0.073707\n",
- "Train Epoch: 3 [55040/60000 (92%)]\tLoss: 0.128305\n",
- "Train Epoch: 3 [55680/60000 (93%)]\tLoss: 0.039086\n",
- "Train Epoch: 3 [56320/60000 (94%)]\tLoss: 0.176628\n",
- "Train Epoch: 3 [56960/60000 (95%)]\tLoss: 0.025344\n",
- "Train Epoch: 3 [57600/60000 (96%)]\tLoss: 0.137705\n",
- "Train Epoch: 3 [58240/60000 (97%)]\tLoss: 0.035565\n",
- "Train Epoch: 3 [58880/60000 (98%)]\tLoss: 0.165229\n",
- "Train Epoch: 3 [59520/60000 (99%)]\tLoss: 0.070512\n"
- ]
- }
- ],
- "source": [
- "model = create_net(tornasole_save_interval=100, base_loc='./ts_output', run_id='good')\n",
- "train(model=model, epochs=4, learning_rate=0.1, momentum=0.9, batch_size=64, device = torch.device(\"cpu\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Data Analysis\n",
- "Now that we have trained the system we can analyze the data. Notice that this notebook focuses on after-the-fact analysis. Tornasole also provides a collection of tools to do automatic analysis as the training run is progressing, which will be covered in a different notebook."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We import a basic analysis library, which defines a concept of `Trial`. A `Trial` is a single training run, which is depositing values in a local directory (`LocalTrial`) or S3 (`S3Trial`). In this case we are using a `LocalTrial` - if you wish, you can change the output from `./ts_output` to `s3://mybucket/myprefix` and use `S3Trial` instead of `LocalTrial`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [],
- "source": [
- "from smdebug.trials import LocalTrial"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And we read the data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [],
- "source": [
- "good_trial = LocalTrial('myrun', './ts_output/good')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can list all the tensors we know something about. Each one of these names is the name of a tensor - the name is a combination of the layer name (which, in these cases, is auto-assigned by PyTorch) and whether it's an input/output/weight/bias/gradient."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "['Net_conv1.weight',\n",
- " 'Net_conv1.bias',\n",
- " 'Net_conv2.weight',\n",
- " 'Net_conv2.bias',\n",
- " 'Net_fc1.weight',\n",
- " 'Net_fc1.bias',\n",
- " 'Net_fc2.weight',\n",
- " 'Net_fc2.bias',\n",
- " 'conv1_input_0',\n",
- " 'conv1_output0',\n",
- " 'conv2_input_0',\n",
- " 'conv2_output0',\n",
- " 'fc1_input_0',\n",
- " 'fc1_output0',\n",
- " 'fc2_input_0',\n",
- " 'fc2_output0',\n",
- " 'Net_input_0',\n",
- " 'Net_output0',\n",
- " 'gradient/Net_fc2.bias',\n",
- " 'gradient/Net_fc2.weight',\n",
- " 'gradient/Net_fc1.bias',\n",
- " 'gradient/Net_fc1.weight',\n",
- " 'gradient/Net_conv2.weight',\n",
- " 'gradient/Net_conv2.bias',\n",
- " 'gradient/Net_conv1.weight',\n",
- " 'gradient/Net_conv1.bias']"
- ]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "good_trial.tensor_names()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For each tensor we can ask for which steps we have data - in this case, every 100 steps"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "[0,\n",
- " 100,\n",
- " 200,\n",
- " 300,\n",
- " 400,\n",
- " 500,\n",
- " 600,\n",
- " 700,\n",
- " 800,\n",
- " 900,\n",
- " 1000,\n",
- " 1100,\n",
- " 1200,\n",
- " 1300,\n",
- " 1400,\n",
- " 1500,\n",
- " 1600,\n",
- " 1700,\n",
- " 1800,\n",
- " 1900,\n",
- " 2000,\n",
- " 2100,\n",
- " 2200,\n",
- " 2300,\n",
- " 2400,\n",
- " 2500,\n",
- " 2600,\n",
- " 2700,\n",
- " 2800,\n",
- " 2900,\n",
- " 3000,\n",
- " 3100,\n",
- " 3200,\n",
- " 3300,\n",
- " 3400,\n",
- " 3500,\n",
- " 3600,\n",
- " 3700]"
- ]
- },
- "execution_count": 13,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "good_trial.tensor('gradient/Net_fc1.weight').steps()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can obtain each tensor at each step as a `numpy` array"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "numpy.ndarray"
- ]
- },
- "execution_count": 14,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "type(good_trial.tensor('gradient/Net_fc1.weight').value(300))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Gradient Analysis"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can also create a simple function that prints the `np.mean` of the `np.abs` of each gradient. We expect each gradient to get smaller over time, as the system converges to a good solution. Now, remember that this is an interactive analysis - we are showing these tensors to give an idea of the data. \n",
- "\n",
- "Later on in this notebook we will run an automated analysis."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Define a function that, for the given tensor name, walks through all \n",
- "# the batches for which we have data and computes mean(abs(tensor)).\n",
- "# Returns the set of steps and the values\n",
- "\n",
- "def get_data(trial, tname):\n",
- " tensor = trial.tensor(tname)\n",
- " steps = tensor.steps()\n",
- " vals = []\n",
- " for s in steps:\n",
- " val = tensor.value(s)\n",
- " val = np.mean(np.abs(val))\n",
- " vals.append(val)\n",
- " return steps, vals"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "metadata": {},
- "outputs": [],
- "source": [
- "def plot_gradients( lt ):\n",
- " for tname in lt.tensor_names():\n",
- " if not 'gradient' in tname: continue\n",
- " steps, data = get_data(lt, tname)\n",
- " plt.plot( steps, data, label=tname)\n",
- " plt.legend()\n",
- " plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can plot these gradiends. Notice how they are (mostly!) decreasing. We should investigate the spikes!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydeXhURdq37+rurAQIYRXCKgFZDAHCpiTCIJvgAgM6qOMGoiOKjjrzynyXAg6oI7wziMurqIgjjrLpiAuyKAjIEhKJCkEIhEASAoTs6aTTy6nvj04fujudpLOnk3NfFxfdZ61zTudXdZ566ldCSomGhoaGRvNF19gF0NDQ0NCoXzSh19DQ0GjmaEKvoaGh0czRhF5DQ0OjmaMJvYaGhkYzx9DYBXCnQ4cOslevXo1dDA0NDQ2fIiEh4YqUsqOndU1O6Hv16kV8fHxjF0NDQ0PDpxBCnKtonRa60dDQ0GjmaEKvoaGh0czRhF5DQ0OjmdPkYvSesFgspKenYzKZGrsoGhqVEhgYSHh4OH5+fo1dFA0NFZ8Q+vT0dFq3bk2vXr0QQjR2cTQ0PCKlJDs7m/T0dHr37t3YxdHQUPGJ0I3JZKJ9+/aayGs0aYQQtG/fXnvz1Ghy+ITQA5rIa/gE2u9UoyniM0LfEJjNZsxmc2MXQ0NDQ6NO0YTeifz8fAoKChq7GBoaGhp1iib0TiiKgqIoDXKuXr16ceXKFQBuuOGGGh9n3bp1XLhwwWXZp59+yvLly1m3bh06nY5ffvlFXTd48GBSU1MrPeaqVasoLi6udJtNmzYxYMAAxo8fX+E299xzD/3792fw4ME89NBDWCyWctvs2bOH6dOne9z/lltuIS8vr9JyaGhoVI0m9E5IKWsl9FartUb7HThwoMbn9CT027ZtY8qUKQCEh4ezfPnyah3TG6F///33effdd9m9e3eF29xzzz389ttv/Prrr5SUlPDee+9VqxzffPMNoaGh1dpHQ0OjPD6RXunM0i+Pk3ShbsMrA7u24YXpA1WRl1J67FT7+9//zvr16+nYsSPdu3dn+PDhfPXVV0RFRbF//37mzJlDv379WLZsGWazmfbt2/Pxxx/TuXNnsrOzmTNnDhkZGYwZMwbnKRxDQkIoKioCYMWKFWzcuJHS0lJmzJjB0qVLSU1NZerUqYwdO5YDBw7QrVs3vvjiC77++mvi4+O55557CAoK4uDBgwQGBpKYmMiwYcP49ddfmT59Onv37uXkyZP079/f5Xp27NjB4sWLKS0t5dprr+WDDz5g7dq1XLhwgfHjx9OhQwePQv7iiy+yf/9+5s6dy2233cYrr7zC//zP//Dtt9+i0+l4+OGHeeKJJ7jlllvUfUaOHEl6errH+19QUMC0adM4ffo048eP56233kKn06m+Rx06dOCOO+4gLS0Nk8nEk08+yfz587HZbMydO5f4+HiEEDz00EP8+c9/rv4PQEOjmaO16MtwFl5PrfojR46wZcsWfv75Z7Zt2+ZivGY2m4mPj+eZZ55h7NixHDp0iKNHj/KHP/yBV199FYClS5cyduxYjh8/zowZMzh//ny5c+zYsYPk5GTi4uJITEwkISGBvXv3ApCcnMyCBQs4fvw4oaGhbNmyhVmzZhEdHc3HH39MYmIiQUFBHD16lCFDhqgVlU6n469//SsvvfSSy7muXLnCsmXL2LVrFz/99BPR0dH885//ZOHChXTt2pXdu3dX2Fp/4YUX1POuWLGCNWvWkJqaSmJiIr/88gv33HOPy/YWi4WPPvpIfctwJy4ujtdff52kpCTOnDnDZ599Vm6btWvXkpCQQHx8PKtXryY7O5vExEQyMjI4duwYv/76Kw8++KDH42totHR8rkW/+NZB9XJc57CLoijo9XqX9T/++CO33347gYGBBAYGcuutt6rr7rrrLvVzeno6d911F5mZmZjNZnXgzN69e1UBmzZtGu3atStXhh07drBjxw6GDh0KQFFREcnJyfTo0YPevXsTFRUFwPDhwyuMs3/77bdMnTrVZdndd9/N8uXLOXv2rLrs0KFDJCUlceONNwL2ymrMmDGV36QK2LVrF48++igGg/3nFBYW5rL+scceIzY2lpiYGI/7jxw5kj59+gAwZ84c9u/fz6xZs1y2Wb16NZ9//jkAaWlpJCcn079/f1JSUnjiiSeYNm0akyZNqlH5NTSaOz4n9PVFVS36ymjVqpX6+YknnuDpp5/mtttuY8+ePSxZsqRaZVi0aBGPPPKIy/LU1FQCAgLU73q9npKSEo/H2LFjB1u2bHFZZjAYeOaZZ/jHP/7hcq6JEyfyySefeF2+mrB06VKysrJ45513KtzGPUzm/n3Pnj3s2rWLgwcPEhwczLhx4zCZTLRr146ff/6Z7du38/bbb7Nx40bWrl1bL9ehoeHLaKGbMpzF3Vn0Hdx44418+eWXmEwmioqK+OqrrzweJz8/n27dugHw4YcfqstjY2P5z3/+A9g7S3Nzc8vtO3nyZNauXavG6zMyMrh8+XKl5W7dujWFhYXqua1WK+3bty+33QMPPMCuXbvIysoCYPTo0fz444+cPn0aAKPRyKlTp8od0xsmTpzIO++8o74V5eTkAPDee++xfft2PvnkE3S6in9qcXFxnD17FkVR2LBhA2PHjnVZn5+fT7t27QgODua3337j0KFDgD38pCgKv//971m2bBk//fST12XW0GhJaEJfhrPQe2rRjxgxgttuu43IyEimTp3K9ddfT9u2bcttt2TJEmbPns3w4cPp0KGDunzx4sXs3buXQYMG8dlnn9GjR49y+06aNIm7776bMWPGcP311zNr1qwqBfeBBx7g0UcfJSoqiq1bt3LzzTd73M7f35+FCxeqFUfHjh1Zt24dc+bMITIykjFjxvDbb78BMH/+fKZMmVJp6qQz8+bNo0ePHkRGRjJkyBC1Qnv00Ue5dOkSY8aMISoqihdffBGA+Ph45s2bp+4/YsQIHn/8cQYMGEDv3r2ZMWOGy/GnTJmC1WplwIABPPfcc4wePRqwV4Tjxo0jKiqKe++9l5dfftmr8mpotDSEp9ZrYxIdHS3dZ5g6ceIEAwYMqNfzGo1G8vPzAWjTpg0hISHltikqKiIkJITi4mJiY2NZs2YNw4YNq9dyVYd58+Yxb948VQg1GoeG+L1qaLgjhEiQUkZ7WudVi14IMUUIcVIIcVoI8ZyH9QFCiA1l6w8LIXq5re8hhCgSQjxbkwtoCLyJ0c+fP5+oqCiGDRvG73//+yYl8mAPlWgir6Gh4U6VnbFCCD3wJjARSAeOCCG2SimTnDabC+RKKfsKIf4A/AO4y2n9P4FtdVfsusch7jqdrkKhd4QkWhKjRo2itLTUZdlHH33E9ddf30gl0tDQqC7eZN2MBE5LKVMAhBCfArcDzkJ/O7Ck7PNm4A0hhJBSSiHEHcBZwFhnpa4HFEVBp9Oh0+mw2WyNXZwmw+HDhxu7CBoaGrXEm9BNNyDN6Xt62TKP20gprUA+0F4IEQL8D7C0shMIIeYLIeKFEPGOrJCGRlEUhBDodDqPWTcaGhoavkp9Z90sAf4lpSyqbCMp5RopZbSUMrpjx471XKQKy6C26BvK2ExDQ0OjIfAmdJMBdHf6Hl62zNM26UIIA9AWyAZGAbOEEK8CoYAihDBJKd+odcnrGOfQjSb0GhoazQlvhP4IECGE6I1d0P8A3O22zVbgfuAgMAv4XtrjH+qYdyHEEqCoKYo8XLU9cAh9RcZmGhoaGr5GlaGbspj748B24ASwUUp5XAjxohDitrLN3scekz8NPA2US8Fs6ji36B3f6xPNj772eONXP27cONzHZQAkJibyzTff1Gl5NDSaKl7F6KWU30gp+0kpr5VSLi9b9oKUcmvZZ5OUcraUsq+UcqQjQ8ftGEuklCvrtvh1g5TSJUbvWFZdND96V2rrR18VtfGr14ReoyXhe6Zm256Di7/W7TG7DIYhT1bZotf86O3UpR/9ggULmDx5MrfddhszZsygXbt2rF27lrVr13LmzBmWL1/O+vXrWb16NWazmVGjRvHWW2+h1+td/Oo9PZtnn7WPz9u0aROPPfYYeXl5vP/++4waNYoXXniBkpIS9u/fz6JFi1wcSDU0mhua1w1XW++VCb3mR3+VuvSjj4mJYd++fYDduyYpyT48Y9++fcTGxnLixAk2bNjAjz/+SGJiInq9no8//tjrZwP2N624uDhWrVrF0qVL8ff358UXX+Suu+4iMTFRE3mNZo/vteinvlLnh7SazXDlippHD+WFXvOjr5ja+NHHxMSwatUqkpKSGDhwILm5uWRmZnLw4EFWr17Nhx9+SEJCAiNGjACgpKSETp06uRyjsmcDMHPmTKDy+6ah0ZzxPaGvB7xp0VeG5kdfMVX50Xfr1o28vDy+/fZbYmNjycnJYePGjYSEhNC6dWuklNx///21cqZ03Du9Xl/jfhQNDV9GC93g6nPjCHm4C73mR18xtfWjHz16NKtWrVJb/StXrlRb/xMmTGDz5s3qfcjJyeHcuXMu+3v7bJyp7jVqaPgymtBzVdSFEGr4xl3oNT/6iqmtH31MTAxWq5W+ffsybNgwcnJyVKEfOHAgy5YtY9KkSURGRjJx4kQyMzNdzu/ts3Fm/PjxJCUlERUVxYYNG7y6Tg0Nn8WRWthU/g0fPly6k5SUVG5ZXVJQUCAzMjKkzWaTUkp56dIlmZ2dXW67wsJCKaWURqNRDh8+XCYkJNRruarL3Llz5cGDBxu7GI1CU3o29f171dDwBBAvK9BVLUbP1Ri9c6aKpxj9/PnzSUpKwmQycf/99zdJP/qWSlN/NhoajYkm9FwdFess9J467TQ/ejtN0Y++JT4bDQ1v0YSeq0LvQDM2u4rmR6+h4ftonbFc9aJ34GxspqGhoeHraELPVS96B7Xxu9HQ0NBoamhCj+fQjWO5hoaGhq+jCT2a0GtoaDRvWrzQO/JM3WP0UL9C3xL86N944w369u2LEEK9Vnf27NnD9OnTPa7zxm9eQ0Ojalq80DvbHzioqdBrfvSu3HjjjezatYuePXtW6/wOauM3r6GhcRWfS6/8R9w/+C3ntzo7npSS7oHdWTRqkbqsIqFvLn70paWlSCnZuXNnvfrRO1w4q6KgoIBp06Zx+vRpxo8fz1tvvYVOp3Pxm7/jjjtIS0vDZDLx5JNPMn/+fGw2G3PnziU+Ph4hBA899BB//vOfq/cD0NBoAbT4Fr0D5xa9J2Oz5uJHL6UkLy+P9PT0BvOjr4q4uDhef/11kpKSOHPmjGrn7MzatWtJSEggPj6e1atXk52dTWJiIhkZGRw7doxff/2VBx98sFrn1dBoKfhci/5/Rv5PnR7PZDKRk5PjEqP3ZGzW3PzoDx8+3GB+9FUxcuRI+vTpA8CcOXPYv38/s2bNctlm9erVfP755wCkpaWRnJxM//79SUlJ4YknnmDatGlMmjSpRuXX0Gju+JzQ1zWOEIq7jW51Rsf6oh+9oigN4kfvDc6VrKfve/bsYdeuXRw8eJDg4GDGjRuHyWSiXbt2/Pzzz2zfvp23336bjRs3snbt2oYsuoaGT9DiQzeeOmMd352Fvrn50UdHR9e7H723xMXFcfbsWRRFYcOGDYwdO9ZlfX5+Pu3atSM4OJjffvuNQ4cOAfbpEBVF4fe//z3Lli3jp59+qtZ5NTRaCprQO3nRO+Mu9M3Fj94h9GFhYfXuR7969WrCw8NJT08nMjJS9aB396MfMWIEjz/+OAMGDKB3797MmDHD5fhTpkzBarUyYMAAnnvuOUaPHg3YK8Jx48YRFRXFvffeW6tZqDQ0mjOiqQ3zj46Olu6TO584cYIBAwbUy/ny8/MxGo107drVZXleXh4mk4kuXbqoy4qKiggJCaG4uJjY2FjWrFnTpOxw582bx7x581Qh9ERBQQFFRUUEBgZWO5au4R31+XvV0KgIIUSClDLa0zotRu/mc+PA2djM0dpv6p7n3vjRO95SmloFr6GhUX+0eKF3tz9w4Gxs5hD65uB57hB6bzuafcWPXkNDo2I0oa9C6Cta76s4WvLeCr3mR6+h4fs0HwWrIe5e9A6aq7GZFrrR0Gh5tHihryxGD81X6LWJVTQ0Wg4tXui9Cd00J5zFXRN6DY2WQYsWeodFcUsReimlS6hKE3oNjZZBixb6igZLOS+rL6FvDD96KSW/+93v1O0ruraG9KOvDd741Y8bNw73cRkAiYmJfPPNN3VeJg2Npogm9JS3PwDPxmZV0dT96B0teL1eD9RO6BvCj74qauNXrwm9RkvC59IrL770EqUn6saPXpESi8WCHDSI4BeeL7feXeh93Y/e4RDpEPqdO3eyfPnyRvWjX7BgAZMnT+a2225jxowZtGvXjrVr17J27VrOnDnD8uXLWb9+PatXr8ZsNjNq1Cjeeust9Hq9i1+9p2fz7LPPAva3j8cee4y8vDzef/99Ro0axQsvvEBJSQn79+9n0aJFLg6kGhrNjRbdoqdMbD2FbsBV6JuDH73jWvR6PTk5ObzyyiuN7kcfExPDvn37ALt3TVJSEgD79u0jNjaWEydOsGHDBn788UcSExPR6/V8/PHHLseo7NmA/U0rLi6OVatWsXTpUvz9/XnxxRe56667SExM1EReo9njcy36Ln/7W50dq7i4mLy8PDp27OhxvU6nU8MxzcmPXq/Xc+TIEU6cONHofvQxMTGsWrWKpKQkBg4cSG5uLpmZmRw8eJDVq1fz4YcfkpCQwIgRIwAoKSmhU6dOLseo7NkAzJw5E6j8vmloNGd8Tujrkoq86B14G6P3FT/6lStXAvbrklIyfvx4Nm/e7HX56oNu3bqRl5fHt99+S2xsLDk5OWzcuJGQkBBat26NlJL777+/Vs6Ujnun1+tr3I+ioeHLtOjQTWWdsY7ljoFFzcGPfvfu3WRnZ6PT6YiOjubw4cNNwo9+9OjRrFq1itjYWGJiYli5ciUxMTEATJgwgc2bN6v3IScnh3Pnzrns7+2zcaa616ih4ct4JfRCiClCiJNCiNNCiOc8rA8QQmwoW39YCNGrbPlIIURi2b+fhRAz3PdtTBw55ZXF6MHe0m4OfvSPPPIIV65cQafT0aFDB954440m4UcfExOD1Wqlb9++DBs2jJycHFXoBw4cyLJly5g0aRKRkZFMnDiRzMxMl/N7+2ycGT9+PElJSURFRbFhwwavrlNDw2dxDBqq6B+gB84AfQB/4GdgoNs2jwFvl33+A7Ch7HMwYCj7fA1w2fG9on/Dhw+X7iQlJZVbVhfk5OTIzMzMCtcbjUaZkZEhLRaLlFLKwsJCdfnw4cNlQkJCvZSrpsydO1cePHiwwvV5eXnywoULUkopL126JK9cudJQRat3mtKzqa/fq4ZGZQDxsgJd9SZGPxI4LaVMARBCfArcDiQ5bXM7sKTs82bgDSGEkFI6J2MHAk1qKGZVzpTuo2N93Y/e+XodcfrmQlN/NhoajYk3Qt8NSHP6ng6MqmgbKaVVCJEPtAeuCCFGAWuBnsAfpZTlesOEEPOB+YDHkEZ9ISuwP3DgLvS+7kfvbH8ghMBms1W5j6/40fv6s9HQqE/qPetGSnkYGCSEGAB8KITYJqU0uW2zBlgD9qkE67tMDhRFUVMCPdHc/G6cKzbn1NHK0PzoNTR8H286YzOA7k7fw8uWedxGCGEA2gLZzhtIKU8ARcDgmha2rqnIi95BcxN699BNc7kuDQ2NyvFG6I8AEUKI3kIIf+ydrVvdttkK3F/2eRbwvZRSlu1jABBC9ASuA1LrpOR1QFWhm/o2NmtoPMXom1OcXkNDwzNVhm7KYu6PA9uxZ+CslVIeF0K8iL2XdyvwPvCREOI0kIO9MgAYCzwnhLAACvCYlLLubQxrgEPkqhL65tTylU7z3zpXYg7vGw0NjeaJVzF6KeU3wDduy15w+mwCZnvY7yPgo1qWsV6oarCUg+Yi9O4Vm/MYAQ0NjeZNix0ZW5kXvTP1JfQN7UfvuIZRo0aRmppaaf+DNzbF9UliYiJjxoxh0KBBREZG1vmApvj4eBYuXFjpNqmpqQwe7Lk7ydM919Boyvic182+jae4klZU6+MoUsFqsWAwXKBTzzbE3NnP43beZqeA3SWxsiyeiqitH/3gwYPp2rWrumzbtm0sXLiQX3/9VfWjX79+vct+lfU/rFq1invvvZfg4OAal6s2BAcH8+9//5uIiAguXLjA8OHDmTx5co29592Jjo4mOjq6xvt7uucaGk2ZFtuidwzdqqJB79Ki//vf/07//v0ZO3Ysc+bMYeXKlYwbN46nnnqK6OhoXnvtNb788ktGjRrF0KFDufnmm7l06RIA2dnZTJo0iUGDBjFv3rxyfvQOVqxYwYgRI4iMjGTx4sWAvXU5YMAAHn74YQYNGsSkSZMoKSlh8+bNqh99VFQUJSUlSClVP3qA6dOnc/z4cdXewNm++IcffuCmm25i2LBhzJ49m6KiIlavXq360VdmgfDtt98ybNgwhgwZwoQJEwC7D80dd9xBZGQko0ePVme2WrJkCQ899BDjxo2jT58+rF69GoDnnnuON998Uz3mkiVLWLlyJf369SMiIgKArl270qlTJ7KyslzOb7PZ6N27N1JK8vLy0Ov1qqVzbGwsycnJGI1GHnroIUaOHMnQoUP54osvANizZw/Tp08HICsri4kTJ6rPpWfPnuqbls1m8+qea2g0eSoaMttY/xrKAsHd3qAi8vPzZUZGhjx8+LAcMmSILCkpkQUFBbJv375yxYoV8qabbpJ/+tOf1O1zcnKkoihSSinfffdd+fTTT0sppXziiSfk0qVLpZRSfvXVVxKQWVlZUkopW7VqJaWUcvv27fLhhx+WiqJIm80mp02bJn/44Qd59uxZqdfr5dGjR6WUUs6ePVt+9NFHUkopb7rpJnnkyBH1/AkJCfKPf/yjlFLKDz74QC5YsEB++OGH8p577pEZGRly4MCB8uzZszIzM1OOGjVKXrp0SUop5SuvvKKWr2fPnmrZPHH69GnZtWtXmZKSIqWUMjs7W0op5eOPPy6XLFkipZTyu+++k0OGDJFSSrl48WI5ZswYaTKZZFZWlgwLC5Nms1n+9NNPMjY2Vj3ugAED5Pnz513OdfjwYXnddddJm81WrhyTJ0+Wx44dk19++aWMjo6Wy5YtkyaTSfbq1UtKKeWiRYvU+5SbmysjIiJkUVGR3L17t5w2bZqUUsoFCxbIl156SUop5bZt29TnUp177o5mgaDRGFBLC4RmSXVi9OD7fvTLli1zmewkLi6OU6dOcfPNN6PT6arlR3/w4EFGjx6tXpvDf37//v2qRfLvfvc7srOzKSgoUK85ICCAgIAAOnXqxKVLlxg6dCiXL1/mwoULZGVl0a5dO7p3vzpkIzMzkz/+8Y98+OGHHjvNY2Ji2Lt3L2fPnmXRokW8++673HTTTap3/Y4dO9i6datqz2wymcpN+LJ//34+//xzAKZMmeLyXLy95xoaTZ0WK/SyCi96B94MmvIFP/onnniCN99806Vii42N5d///jdt2rTxuoyOcjr+VVVROnAvv6PfY/bs2WzevJmLFy+6VJgFBQVMmzaN5cuXM3r0aI/HjI2N5f/+7/+4cOECL774IitWrGDPnj2q86WUki1bttC/f3+X/RzhtOqWWQvTaPgqLTZGX5VFsQOH0I8ZM8an/ejvvvtu9u/fr8a6x4wZw5EjR6rtRy+lJCoqikOHDnHmzBngqv98TEyMOs3fnj176NChQ5WVyF133cWnn37K5s2bmT3bnqFrNpuZMWMG9913H7Nmzapw35EjR3LgwAF0Oh2BgYFERUXxzjvvEBsbC9jv5+uvv65W6kePHi13jBtvvJGNGzcC9krS03NxR/Oy1/A1WrTQV9Wah6tCP2zYMJ/2ozcYDDz00ENqxdGxY0dWr17N3Llzq+VHL6Wkffv2vPrqq8yaNYshQ4aoLfElS5aQkJBAZGQkzz33nEtFVxGDBg2isLCQbt26cc011wCwceNG9u7dy7p164iKiiIqKorExETAPl/t1q32gdkBAQF0795dbfHHxMRQWFioGq49//zzWCwWIiMjGTRoEM8/X34C+MWLF7Njxw4GDx7Mpk2b6NKlC61bt660zM73XGvla/gCwtHaaSpER0dL98mdT5w4wYABA+r0PNnZ2dhstnLzj7pjtVq5fPkyoaGhKIpCSEgIxcXFxMbGsmbNmiZlhztv3jzmzZvnMdSRl5eHyWSiS5cu6rKsrCx0Op3HN4CKcNwPgHbt2hEUFFT7gjcipaWl6PV6DAYDBw8e5E9/+pNaqdSU+vi9amhUhRAiQUrpMW+4xcboq9uiVxSlyXueV+ZH7+l6azIYzNna2Bub46bO+fPnufPOO1EUBX9/f959993GLpKGRp3TYoVeSumVx4vzwCJf9jz3JPRCiCqF3t2PXlEU/vWvfzFgwIBmIfQREREeY/caGs2JFiv03rbom4uxmSfzMm9mmXL3ozcajeTn5yOE8HrEsIaGRuOidcZ6QXMQeunBqdNxXdXpp3HcBz8/P5+/JxoaLYVmI/QWm0JWYSkWa9Xi4+1gKQfNQegrCt1A9RwsHWmpBoOhWYRuNDRaAs1G6K02SWZ+CUZz1eEEbwdLOfB1oa9ocFNNrIptNhs6nQ69Xo+iKD59XzQ0WgrNRugD/XTohKDYXHUr01svege+LvQVXW9Npkp0vBk44v1aq15Do+nTbIReCEGQn75aQu/ewrXmmLDmmMptX5NYdlU0pB/9zz//DNivY/DgwapnS0VWxZX50Ts6detT6L31o3/ggQfYvHlzueXe+M1raLQkfC7rZve6NVw+l+JxndmqYLFJEvz1UEn4XbEpWK1W/Pz8EDpBp559GP/AfJRSq2pf7IxziKOyuH5T9aN/+eWXee2117wO3VTmR98QLfra+tHX1m9e4yrp6els3LiRWbNmeRzNreEbNJsWPYBOCECieNvydtI9aVPAJkGR9s/Oxy0TRF/1o09KSuL06dPlQjffffcdt956K6NHj/bKj15KqQr9zp07mTx5MqNHj24UP3oHu3btIjo6mn79+qn+Q85+83FxcYwZM4ahQ4dyww03cPLkSZE2uGgAACAASURBVACOHz/OyJEjiYqKIjIykuTkZI/Hb+mcPHmSgoIC/vOf/1Tpu6TRhKnIv7ix/tXGj77UYpU/p+XKK4WmSrcrLCyUGRkZLh7nNpNFlqYVyNK0AmkrcfWoLykpkV9//bWMjIz0ST/6d999V86aNUuazWY5aNAgefbsWZmVlSXHjh0rk5OTZWFhoVd+9FarVWZkZMiUlBQZHh4uDx8+LHNzcxvNj/7++++XkydPljabTZ46dUp269ZNlpSUuPjN5+fnq3MO7Ny5U86cOVMt6/r166WUUpaWlsri4uJyx68pzcmP/oMPPpCvvfaaXLFihVy5cqXMzc1t7CJpVAAtxY/eT6/DoNNRUkWc3lOMXloU18+BV7fX6XQcOXKE6dOn+6Qf/ezZs3n55Zc5d+6cuuzQoUOcOHGC22+/XbUNrsqP3nHfEhISiI2NpXfv3litVtW8raH96AHuvPNOdDodERER9OnTRzVmc5Cfn8/9999PcnIyQggsFgtgd+9cvnw56enpzJw5U32D0LiKzWYjIyODYcOGMXToUD744APWr1/Pgw8+6GLNrdH0aVahGyEEQf56ii2VC70si7W7CL1Vsc8rqBcuog9VpyG6+9E//vjj/Prrr7zzzjuYTOU7dysr16JFi0hMTCQxMZHTp08zd+5coGI/d3d27NjBpEmTypX/kUceYcWKFS7nmjhxIrt27WLfvn0kJSXx/vvvV1o+9+wdvV7vdYy+Kj/6DRs2VNuPHsp3qLt/f/755xk/fjzHjh1TbabBbtu8detWgoKCuOWWW/j++++9uo6WxKVLl7BYLHTv3p0uXbowZ84ccnNz+c9//oPZbG7s4mlUg2Yl9ADB/npKLTZsSsVxek+Dh6RFQfjpEH56pFtFodPpGDFiBN98841P+tFLKbnzzjv57rvv1Fj36NGj+fHHH0lNTUVRFK/86B2iPnr0aPbu3UtaWho2m43s7Gyg4f3oATZt2oSiKJw5c4aUlJRyk4w4P49169apy1NSUujTpw8LFy7k9ttvV/sTNK6SlpYGoL5l9erVi1mzZnHhwgU2btyopdb6EM1O6IP89UigpJJWvUehtyoIgw7hp0NaXFMphRBERUUxdepUn/SjVxSFwMBAFi5c6OJHv27dOh577DHGjh3rlR+9o0XfuXNn1qxZw3333cfNN9/caH70AD169GDkyJFMnTqVt99+m8DAQJfj//Wvf2XRokUMHTrU5S1o48aNDB48mKioKI4dO8Z9991XZVlbGmlpabRp08bldz5gwACmT5/O6dOn+eKLL3x6fElLotn50VttCkmZBVzTNpCOrQM9bpOVlYUQQhViaVOwZBrRtw0AvcCWY8LQKRid/1UTsIsXL2K1WgkPD/c5P/qcnBwsFgudO3cut48jl9+5UqqIgoICioqKuOaaaxBCUFJSQm5uLh06dMDf379uLqIZ0Fz86P/1r3/RrVs37rzzznLr9u7dy/fff8+YMWOYPHlyI5ROw50W5Udv0OvwN+gqHTilKAp+fn7qd2m1V3bCoANDmf+LRQEnodfpdDz11FOcPn3a5/zoKzNwE0J4/QruOI4jDq6Njm2+FBQUkJ+fX2H/SExMDEVFRRw8eJCQkBBuvPHGBi6hRnVodkIPEFzFCFnp5uQorfZthZ8O9AKE5w7Zt99+26uWb1PD/Xqd0el0ldoNO/vR22w2pJR88sknXH/99ZrQN2Pc4/PuCCGYMmUKRqORnTt3cs0119CnT5+GLKJGNWiWQh/kbyCvxILFpuCnd4vFOw36UZdZrmbcCCHK4vTlO2R91X/dkxe9g6p8fJz96N1DXo57qAl942EymfD39/fat8lb0tLSMBgMLlNPuqPT6bjjjjs4fvw4586d04S+CdPsOmPBnnkDeMynd/RJuAu9MFwNSQhD+Q5ZXzY2qyp04xhU4c1xnCsMIUS1Uiw16habzcbq1atx79OqC9LS0ujatWuVlh5+fn60bduWnJycOi+DRt3RLIU+0E+PwLOTpcfBUlZ7aqUD4a8DxW6H4KA+jM0aAoeIOwu9YraVq/C8FXr3CkMT+sajpKSE4uLiOrcmsFgsZGZmVhi2cScsLEwT+iZOsxR6vU4Q4KfzmGJZTuAUCTbpKvRln53j9DXxbm8KuF+vYrFhvVyMNNlcl1fxtuKo5DShbzo4xltUlYJbXS5cuICiKJrQNyOapdCDPXxTbLaWE2b30Z0OMRcGJ6E3VCz0vha+KfcGU5ZhJK2uy70Reijvae+YgMTXKsDmgNFoBOpe6KvqiHUnLCyMkpISSkpK6rQcGnVHsxZ6myIxu00t6C58quA5t+j1OtDrkE6hn7oW+obyo/f39ycpKUktf+TwIaSmnVOv29Obiic/esd1u3fq1lfmzZQpUwgNDVVdKOsSb/zqU1NTGTx4sMd1nu55Y1CfQh8WFua1n01YWBiA1qpvwvhc1k3el2cwXzBWuZ0iJa3NNrL97EZn6nLFhmK1kutnzyCRVgVDWCBhf3AdOu8YIevAG6Fvin703bp1Y/Xq1cTExLjsJ21uIR2n6/LkR19Zix7sQl+Ta6+Iv/zlLxQXF/POO+/U2TEd1Nav3tM9bwwcQl9UVFStye4rQ0pJWlpatUzenIXeYTeh0bRoti16nT0dHsXN88bRcFX7YqVE6DyYY/npkFbFHsPHLnD/+te/iIyM9Ck/+qlTp3Lq1CnVx0adWMWqsGPHDmJiYpg8eTJ33313pX707kL/7bffMmzYMEaOHMmdd96JzWarMz96gAkTJtC6desKn6/NZqN3795IKcnLy0Ov17N3717A7iuUnJyM0WjkoYceYuTIkQwdOpQvvvgCcPWrz8rKYuLEiepz6dmzp/qmZbPZvLrnjYVD6KWU6ufakpOTQ3FxsddhG0B1YtVa9E2YivyLG+tfbfzo3Tl9qVAmXyp0WZafny8zMjJUz/jSC0XSkl3ei9xqNNu96UutUkopDx06JAcOHCizsrJ8yo9+/vz5ctWqVfLee++VUko5sP8AefLArzLjl7MyJiZGFhQUyIyMDLl06dJK/egd29lsNnn58mUZHh4uU1JSpM1mk8eOHZMFBQV17kfv7CvvicmTJ8tjx47JL7/8UkZHR8tly5ZJk8kke/XqJaWUctGiRep9ys3NlREREbKoqMjluAsWLJAvvfSSlFLKbdu2qc+lOvfcnYbyo//vf/8rFy9eLBcvXiwzMjLq5JhHjx6VixcvlhcvXqzWfitXrpSfffZZnZTBnaSkJPnPf/5TlpSU1Mvxmwu0FD96d4L99VwxmlGkLJt9ynUYvz3jRkEY/MrtK/zsIQmHFcKBAweYPHky/v7+tG7d2qf86GfMmMFbb73F2bNn1WWHE+JISkoiJiYGi8WCzWardBi7oigIIdDpdBw6dEj1owf7q7vNZqtzP/qqiImJYe/evZw9e5ZFixbx7rvvctNNNzFixAj1fm7dulV9SzCZTJw/f97lGPv37+fzzz8H7P0Czs/F23veWBiNRnUcRF3F6dPS0ggICKBjx47V2q8+M2+OHTtGfn4+Z86cYdCgQfVyjuaOV6EbIcQUIcRJIcRpIcRzHtYHCCE2lK0/LIToVbZ8ohAiQQjxa9n/v6vb4ldOkL8eKSWlTmmWDsECp4wbv/K3QRgcVghl9ghl/vWeYvRN2Y9eSonBYODpp5/mlVdeAVk2IEzCzRNuJjExkV27dnHo0KFK/egriwF7k2JZXT96b4iNjWXfvn3ExcVxyy23kJeXx549e9T+CCklW7ZsUe/n+fPnq2U25u09byyMRqM6SrkuhT48PLza8f76EnpFUUhJsc8RrU33WHOqfJpCCD3wJjAVGAjMEUIMdNtsLpArpewL/Av4R9nyK8CtUsrrgfuBj+qq4N7gGCHrPHBKOuWCqxk3Bg9Cr1oh2Le58cYb2blzJ8XFxT7lRy/LJll58MEH+e6777iScwURoGfUsBEcOHBAnUu2sLCwUj96Z6F3+NE73hAKCgqw2Wx15kfvLSNHjuTAgQPodDoCAwOJiorinXfeITY2FrDfz9dff13tDzl69Gi5Y9x4441s3LgRsFeSnp6LOxX59Tc0RqORzp07I4Sok/KYTCYuX75crbcqB2FhYRiNRtUXqa64ePEiJSUlBAQEcPr0aZ9Lb24qeFNtjwROSylTpJRm4FPgdrdtbgccqrYZmCCEEFLKo1JKRx7acSBICBFAA+GYWtBZ6J0Fy+5xg+pY6Y6zN/2IESOYMmUKMTExPuVH76jY/P39eeKxBVy+koXw09GxfQfe/793mTNnDuPHj2fKlCmV+tHbbDb1vrVvE8bbb/wfM2fOZMiQITz88MPYbLY686MHe1hm9uzZfPfdd4SHh7N9+3bA1Y8+ICCA7t27qw6LMTExFBYWcv311wP22aUsFguRkZEMGjSI559/vtz5Fy9ezI4dOxg8eDCbNm2iS5culXYCu9/zxu6Mbd26Na1ataoToc/IyAC8z593pr5SLB2teYdbpiO5QaOaVBS8d/wDZgHvOX3/I/CG2zbHgHCn72eADh6Os6uCc8wH4oH4Hj16lOtkqE3n1tmsInnyYoH6/eLFi+pk1uasYmm+WFThvtbCUlmaViAVi31i6nPnzslLly5Jo9Eohw8fLhMSEmpcrvpg7ty58uDBgy7Lrly5Ii9fviyldOpgNltlaUahtOTaO7eysrLUbSoiMzNTnRjafMkoSzMK1U5nT5Ot+womk0mdPPzAgQNqB3JtaIjO2NLSUrl48WK5b98++fbbb6sTndeG3bt3y8WLF9eo0/PChQty8eLF8tixY7UuhzMffvihfPPNN2VhYaFcvHix/OGHH+r0+E2J1NRUmZeXV+P9aezOWCHEIOzhnEme1ksp1wBrwD7xSF2eO8hfT0GBBZsi0etEuRa9LqDilxpnKwRh0PH000/z22+/YbVafcaP3rlPAkeoSq9D6IXqw1+VM6d0c/yUVgUUiSy1IQINLrn0de2iWN+cP3+eO++8E0VR8Pf35913323sInmFI52yVatWtG7dWu30rg1paWl07ty53Cxd3lAfLXqLxcK5c+cYMWIEISEhdO3aleTkZDU015yQUvLZZ5+pc/PWNd4IfQbg/C4XXrbM0zbpQggD0BbIBhBChAOfA/dJKc/UusTVxNnJslWAXg1lXM24qfgWXM28sUGQQY2fd+7cuULb36aG8yQr0iZBJxA6AXod2K7mxlcU+3T40VssFvR6PTqdjrUr3mbwgEEoJVZ0bkLvPKGLLxAREeExdt/UcRd6R9ilpiiKQnp6uhr2qi4BAQG0atWqToXeMSexw/44IiKCvXv3Ulxc7DKYryEoLS0lLy/P4yxtdcGlS5fIz8/npptuqpfjeyP0R4AIIURv7IL+B+But222Yu9sPYg9RPO9lFIKIUKBr4HnpJQ/1l2xvSeoTKyLLVaC/e2tTZ1O59H6wB2hE2C42iEbFBREUVERJSUlLoOcmjKOig3sQi/K/PmFQaCUXnW2rEjoDx8+jMViISsri9DQUAINAVgvF4NOoJRYkaFXj6+ZmzUc7kJvNBqx2Ww1boBkZWVRWlpao/i8g7rOvDlz5gw6nY6ePXsC0LdvX3744QfOnDlT4wqpJiiKwieffML58+d54oknPKZG15aTJ08CVGtEcnWo8j1bSmkFHge2AyeAjVLK40KIF4UQt5Vt9j7QXghxGngacKRgPg70BV4QQiSW/etU51dRCY6pBUvMNpfRnaq9gYeMG2eEk9D7+fnh5+eH0Wj0CRMv95ALNkXteBZ6nX2YsCKrNDZzuW9lFaQ+xE8N32gzTTU8jswsh9BD7VIsq2tk5om6FvqUlBS6d++uprl269aNoKCgBk+zjI+PJzU1FUVR2L9/f72c4+TJk3Tr1q3KRICa4lVAVUr5jZSyn5TyWinl8rJlL0gpt5Z9NkkpZ0sp+0opR0opU8qWL5NStpJSRjn9q1vzbC8I9jNQ7CT0Do8bhOfUSmeEv6sVQqtWrbDZbJjN5novd21xVEbq5CJWidA7JlcpG0tgc2rxV1B5ORuaqWZorfxA2Fv12gQkDY97ix5qL/StWrWqVWs1LCyMwsLCOvnbKC4uJjMz02XWKp1OR9++fRs0zTInJ4edO3dy7bXXEh0dzdGjR8nPz6/TcxQUFHDhwgX69+9f9cY1xLd6zmpIkL8ei03BYrvqwe4+q1RF6Ny86QMDAxFClHN3bIo4hFun09k9buTV0A36qx2rVRm2ufjcWBXQ6xB6HbpAvT18I6Um9A2M0WjE398fPz+/OhP67t27V/n3UBmODllvxiJUhWOMhvv0hBERERQXFzeIe6iiKHzxxRfodDpuu+02xo4dC8CPP9ZtFNrxhqIJfS1xdMiazOWFvipcOmTL9g0ODqakpKRWwtZQNsXh4eEcP35cbYkPGTWU1NTUq9duKx+6cbcptjlXkFapvg2IIINL+KYuhd4bm+IHHniAzZs3l1vujQ2xr2M0GtUR2bUV+qKiInJycmoVtoG6FfozZ84QEBBQziG0b9++QMOMko2Li+PcuXNMmTKFtm3bEhoaypAhQ0hISKjTAXMnT54kNDSUTp3qL6rdIoQ+qGxqQbPFnkIoEPaMm0o6YlX0ZVYITr72jh5/98EyNR0iX1ubYneh37ZtG1OmTEFRFK655hpeffVVsLnadgqdAJ1A2pRyoRt3oXexjXCadlEXaFDDNw6hr6u+i7/85S989FHNBlJHR0erDpnNFaPRqCYEBAcHq6Oba0J6ejpQu/g81G2KZUpKCr169SrXuRwcHEx4eHi9C312dja7du0iIiJC9TsCGDt2LIqi1Opv1hmz2UxKSgr9+vWr1dtUVficqdm2bdu4ePFitfcrsdjQIRFSwd/PD2kuEyydoEuXLuWMwByoVghmhb///e+sX7+ejh070qlTJyIjI9mzZw9RUVHs37+fOXPm0K9fP5YtW4bZbKZ9+/Z8/PHHdO7cmezsbObMmUNGRgZjxowpZ1Ps6FxbsWIFGzdupLS0lBkzZrB06VJSU1OZOnUqY8eO5cCBA3Tr1o0vvviCr7/+WrXMDQoK4uDBgwQGBqo2xT/99BM333wz8fHx/HbiBH079bKPBC5j177veXHlckqtZrp168batWtZs2aNalPcoUMHdu/efXVS8LJ5dIVex7fffsvf/vY3rGYr7UPD+Gr71+Tm5vKnP/2Js2fPEhwczJo1a4iMjGTJkiWcP3+elJQUzp8/z1NPPcXChQt57rnn6N69OwsWLADso4pDQkJ49tlnmTBhAnv27Knyue7atYtXXnmFgoIC/vnPfzJ9+nT27NnDypUr+eqrr4iLi+PJJ5/EZDIRFBTEBx98QP/+/Tl+/DgPPvggZrMZRVHYsmVLvWU81AdGo1EVVp1OR0hISI2FPi0tDZ1O5zIy2YFislJ0KJNWwzujb+1f6XGCgoIICgqqtdDn5OSQl5dX4ZtuREQEu3fvpqioqF6y3xRF4b///S8Gg4Fbb73VRYDbt2/P9ddfT3x8PGPHjvV6cpaKSElJwWq11mvYBlpIix5AV9YhCVz1ZPeyBhV+Oo4cOcKWLVv4+eef2bZtG7/88guKoqAoCmazmfj4eJ555hnGjh3LoUOHOHr0KH/4wx/srWlg6dKljB07luPHjzNjxoxyLopg91pJTk4mLi6OxMREEhISVI/15ORkFixYwPHjxwkNDWXLli3MmjWL6OhoPv74YxITEwkKCuLo0aMMGTJENWDT6XQ8++yzvLziHy4if+XKFV5e9SrfbviS+Ph4hgwZwurVq1m4cCFdu3Zl9+7d7N69G7hqG+F4q7mSm83DDz/Mli1bOBqXwCf/9yE6Bf73f/+XIUOG8Msvv/DSSy9x3333qef77bff2L59O3FxcSxduhSLxcJdd92l+swAbNy4sdrGZqmpqcTFxfH111/z6KOPljORu+6669i3bx9Hjx7lxRdf5G9/+xsAb7/9Nk8++SSJiYnEx8cTHh5erfM2Ns6hG6id/05aWhpdu3b1OAbCGHeRgm9TufjPBIzxF6t8Y6uLzBuH7YF7fN6Bo0I+c6Z+huUcOnSItLQ0pk6d6tGryeH4eujQoVqf69SpUwQEBKgppPWFz7XoK2p5V0WO0UxeXh5BOoWOwe1Qisz4dQ3x6nVJ+Ok4cOQgt916G4GBgQQGBqo2xYqiNFmbYscf5d13380ry1/mbMbVyuXQoUOcOPUbN916M8JPR3FxseoZ445jIJRjJO3hhDjVplgqkrCw9ihme0zz2WefBRrGphjgzjvvRKfTERERQZ8+fVS/Hgf5+fncf//9JCcnI4TAYrEAMGbMGJYvX056ejozZ870qda8oigUFxeXE/rs7OxqH8tqtZKRkcHIkSM9rjedzkPfLgB92wByNydTfPQy7WZEYOgQ5HH7sLAwNVWzpqSkpNCmTRsXcz5nunTpQqtWrUhOTmbIkCG1Opc7WVlZfP/99/Tv35/IyEiP23Ts2JFBgwZx+PBhbrjhBoKCPN+LqlAUhZMnT9K3b986nZ3NEy2mRR/sr0cgy6yHvcu4ceDokHWMJAV7SMfPzw9FUVyGjDclm2KH0Pv7+/PUowv53zf/BcClAhMWq42bx0/gyPYfOZrwEz/88AOvv/66x2O6t+jRX71vQifQBeoRpfZ1FXXI1odNMXiYGczt+/PPP8/48eM5duwYX375pfo87r77brZu3UpQUBC33HIL33//fbXP3Vg4ZhJr1aoVhT9mYMkqrnGL/uLFi9hsNo8VrLQolKbkEzSgPR3nRxI6oy/m9CIurvqJwh/S1OkonQkLC1NdVGuCw5a4T58+Ff59Oir206dP12kCgCNk4+fnx/Tp0yvVh5iYGMxmM4cPH67x+S5cuIDRaKz3sA20IKEPMOjQCXuYWVq9y7hxIPx03BA9mq++/hqTyaTaFPv722OWztasTcmm2DmP/o+z7+b7fbu5nJVFjtHMwCHDOXD4IKfPnkHaJCUlJWoHl/MxHaZIer0eyu7bmDFjXGyKc0sKEIrdLuHTTz8FGsamGGDTpk0oisKZM2dISUkp90fj/DzWrVunLneIycKFC7n99tvVaQ99AUcOfbBfIPlfpmA8fJHWrVtjMpnUNxZvcbS+PYWuSs/lg1UhoF87hE4QMuoaujw9nMB+7cjflsrlN45iTnetXMLCwtTpHWtCZmYmJpOpwrCNg759+2IymWpt/eDMgQMHyMjI4JZbbqly4FKXLl3o378/hw4dqlZjzpmTJ08ihFAzieqTFiP0Qgj8dGBRsAuWNxk3jn11ghHRI5g2+RYiIyNVm+KwsDCEEJhMJlVUm5JNsSzzopdS4q/3Y8EjC8gqqzhCQtuz9r33ue/xh4gaHsX06dPVYdjONsXlRsUadHTs2JE1a9aoNsV3P3AvCMFfnvkLR48ebTCbYoAePXowcuRIpk6dyttvv13OkOuvf/0rixYtYujQoS6tzI0bNzJ48GCioqI4duyYS39CU8ch9AGKPaZuzS6pcYrlxYsXCQkJ8Vghm5LzQC8I6H3VjlvfNoAO9w2k/b0DsBWZufxmInlfp6hve7XNvKkqPu/g2muvRQhRZ9k3ly9fZvfu3QwYMIDBgwd7tc9NN92EyWTiyJEjNTrnyZMn6dGjR4P49oimNpQ/OjpaxsfHuyw7ceJEtWYGqoiLFy9htgm6yCD0YYHog7034LJml1CQV0DYtZ0pLi4mNjaWNWvW0L9/f/Lz82nfvr1LeKIxmDdvHvPmzVNj7dnZ2dhsNjqGdcBy0Yi+XSCXLFauFJXSKsBAn7BgLJlGe/y11B5Ld66cwJ7+deXKFdq1a4c+x4aulQFDaHl3Q2t2CbmlheAnqj0NXXOjrn6vFXHs2DE2b97M3Mn3IL64iKFTMMbbQ/noo4948MEHq9Wx995772EwGHjggQfKrbv02k+IQAOdHvEcq1ZKrOR/cxbjkYuEzuxLyMhrMBqNrFixgilTplTY51MZH374IcXFxfzpT3+qctu1a9diNpt59NFHq30eZ2w2G++//z55eXk89thj1crkWb9+PRcuXOCpp55S3/C9ITc3l9dee41JkybVahyNM0KIBClltKd1LaZFDyClgp/D96UaoRuwh28ee+ZxoqKiGDZsGL///e8ZNmwYQUFBTWak7Hvvvefyx6U6ddoc9sSC0rKWl8liA13ZGAGbUuE0iWqLHmEfWVvBfRNBBvRSYGti0+01R9QWvcn+LKw5JYS0sotTdVv02dnZHjs9bUVmLJlGAvuFVrivLshA6My+iAA9lgtl4aTgYAICAmrUojebzZw/f77K1ryDiIgILl68WGuL5v3793PhwgWmTZtW7XTN2NhYiouLcW+cVoVjNreGiM+DD2bd1BRFsc8UFWTQIW2QZ7bS3t97pz/hp+ffb6zF0DEYXcDV/XQ6HUFBQRQXF9O2bdsm5ceuKAoGg+Fqp5leUGq1d17ZFIlVsY9ylVaJTu/Zkz4mJobi4mIMeoN9FK1B8NH69eXcA3WBBnToUNyN1HyY0tJS/P3963UgS01wTAruZwQzgFUSjP1tsjpCX1xcTElJiUehLz1tj7EHRlTufSOEwK9zMJZLxer3mqZYnj9/3sWW2IFUJEqRGX0b1zfmiIgIvvvuO06fPl3juSEyMzP54YcfGBAeQW9r9UemOjLiDhw4wIgRI7y26T558iQdOnSoMLOorvH9v0YvcfTOG9BjFXClyFytUZxXJyEp38vviLE1hVa9M6rglqVFohNYrAqt/O31u8liU33pK7Iq3r17Nzt37iThQFxZhs5RjxaxQifQ+9mP2xw8bywWC9nZ2WrruSlRVFREcHAwSv5V8zBDoX0S+OoIvSMd05PYmE7logs24Ne16hauoVMw1stX71NNhT4lJcXFltiB8VAmma8ewVbgapbWuXNnWrduXeM4vdVq5fPPPyc4OJjo9HByt5zCWMpYCgAAIABJREFUZqxeZzbYW/VFRUVez2tgMplITU2lX79+1T5XTfEZoa9tX4Lq12Kzi3ap1UahqRphBn2ZZYClvBg6zKWKi4ublH2x2hlrU0AnMCsSCbQJsrc6TBbF3qK3Xe20dS+/Kv5W7AOu9BW3bv0CyzoHTdX/Y2lqOBwYq5tR0RDP3zFYypprwtDRnsNtyzFVO8WyIqGXUmI6nUdA31C7VUYV+HVuhWK0Yiuy37OwsDDy8vKqXeE7bIndY93FP2eBVWI65Vp5CCGIiIggJSWlRo2L3bt3c/nyZSZdF0OAyT76vehA9c3SevXqRY8ePdi/f79XWU8O982GCtuAjwh9YGAg2dnZtfojcvwQ9DaBf4ABP72OK0Xez1ivWiF4EHqwt+qtVmu109vqC4do22P00iU+3ypAj0Gnw2Sx2d0sFYlOeLYqdkwPKK2KfQrCSsIYhrIKxNYMhN7xHM1ms9ciIqUkOzu7RlPxVQeH0NvySvHv2QYMAmt2zYReCFFu8J71cjFKgZnAvt5ZFvt1tr/ROsI3YWFhKIpSLTtfo9HIxYsXufbaa12W2wrMmM/bY/Cmk+VTkiMiIigtLa32IK3z589z4MABhg0bRuczfvh1aUXgwPYUHbiAUlq9fiYhBOPGjaOgoIBNmzZVOYbg1KlTBAUF1dpbqDr4RIw+PDyc9PR0srKyanyMkpISSktLyZaB6Fr5YVQU0kusFF0KwE/vXX2nFFtQzAr6K+XjtlJKCgoKyMzMbPBpzjyhKAoFBQUEBQVhKLW/jRQbIL/Eir4gkJwiM1lICgL8UIwWbEGCElMJubm5LvF1o9GIoihckQGgE+hzK45BSinJz8/HHwPBWSEulgu+RmFhoTpxS1ZWltcZFYGBgfVup2A0Ggnv2g2lyIKhXSCGsCA1xbI6PlDZ2dn2bCo34zBTsj0+HxBRcUesM4YyobdeLoZrQ11SLB2fq6IiW+KSpGyQ4N+jNaZTufbEAae/1969e6PT6UhOTqZXr15enctsNvPf//6Xtm3bMm7gDRQcOEHojL74dw3hclI2xkMXaX1T9Z5hnz59mDZtGl9//bU6JsTTbF82m41Tp07Rv3//Bu3H8gmh9/PzU60EasqmTZvISE1nVnY0nf88jKIQP8a88h23D+nGP2YN8uoY5owiLr9+lDYTe9JmQvk8+C+//JKff/6ZZ555psbDouuKrKwsNm3axMyZM2n3TRGthndmWWkRe0/lEPf/bmbpl8f5NC6No/Nv5MoniWT/LoDtB7bzyCOPuOSzr1mzhuCgYMad7EXIDd0IHVH5c1jxyqt0Kwpl5gOzvW4RNjXMZjMvv/wyMTExJCYmcs0119TLhM01xWg0EmSwd0zq2wVgaB+I9UoJrQdUL15dUcZNaXIuho5BGNp592aib+Nvz7xxatFD9XLpHbbE7sZqJcevYGgfSEhMODkfn8B8rpCAPlfz+gMDA+nZsyfJyclMnDjRq3Pt3LmTnJwcHnjgAayHriAC9ARHdUIXoCcgIpTCfemE3HDN1RHxXjJixAgURWHbtm1s3ryZWbNmlRP7tLQ0TCZTg4ZtwEdCN3VBXl4ebQytQCcwtA+iXSt/Zg4L5/PEDLK9DOH4dwshcEAYhfszUDzE94cNG4bVaiUpKamui19tHLHlAL2/3S8+NICUrCL6dLT7o/Tv3JoSi43LZQ5vfqX2n4K79bLRaCTIPxCsEkOHqv/wQ8NCMepNlPxypS4vp0HJzMxESkl4eDjXXXcdZ86caTIzilksFsxmM0FlWTaG0EAM7YOw5ZgICQnBbDa7jNSuCEVRPAq9tNptDwL6eteah/KZNyEhIfj5+Xkt9FJKUlJS6N27t4swKiVWSs/kEzioA4ERoaATmE6WP2bfvn25fPmyardcGWfOnOHIkSOMHj2a7h26UvzrFVoN76xm0rUe1x2lyIIx/pJXZXdn1KhRTJ48mRMnTvDZZ5+VC/udPHkSvV5fLkRV37QYoc/NzSVECcTQIVDNBX/oxl6YrQr/OVzeSbIi2kzogSyxUnSwfKdN165dad++Pb/++mudlbumOITe32K/Vn1oAClXjPTpaM+i6NfFPpLyZEExwl+HX7F02Q/sf4BGo5Eg7GELQ/uq31LatG1LcYCVkuNXXDz8fQmHYHTt2pXrrrsOq9Vab06J1cWRBRRYNipW3y4AQ4dApEUhxGB/Pt7E6QsLC13sMhyUnitAWpQq0yrdcc68qW6KZU5ODvn5+eXCNqbfckCRBA1ujy7QQECvNh6FfuDAgfj7+/Pee++xbt06kpKSPParmEwmvvjiCzp06MCECRMwHrkENkmr0VffIgL6tMW/ZxsKf0hXx59UlzFjxjBx4kSOHz/Of//7X5dstpMnT9KrV68GH1zZIoTebDZTXFxMiMkPv05X4+d9O7Xmpn4d+fehc5i9FCX/8NYEXhdG0b6Mcp02QggiIyNJTU2tsddHXeFomfuZ7Y/YGKgnr9hCnw72Fn1EJ7vgn7pchL5dIIay7DhnoTebzVitVgJtdlHx61i10Ldt25YiWzE2o8UeX/VBMjIyCA0NJSQkhJ49exIYGFjOFbOxUAdLmQ2gA32bALUCDlbsFbI3Qu/IuHEfCV2anAs6QcC1bT3tViGeMm+8FXqH7YF7K7fk2BV0rf3xD7c3SgL7h2G5WIw13/WNpV27djz55JNMmDCB3NxcNm7cyGuvvcbevXtV3yiwO7sWFhYyY8YMDHoDxsOZBFzb1kUThBC0Ht8dW14pxYn/n70zD4irPtf/58w+zMAAww4hkBAgbEnIvidmMWrilrpWrfXaXqtWbWu1vbfrbe9tba3WarXe1rbeVq0mcY9m3/eFEBKSQFYIhJ1hmX07vz8OMywzwAyBxKa/5x8NnHPmDOec97zf533e5x16TXD27NksWrSIY8eO+YN9c3Mzra2tV5y2gX+RQO8LujqrAkVC70LpQ3Myaep08GlZ6LKqqEXpeK1uzHvrAn7n05gfP378Ms748uEL2Mouaf/FLhXJ2K6MPlKjJDVaS0V9p1TQ6/D22g96ZI9OOYJKhmyQwRMgBXqXx407Rh707/PPgNraWr8RmlwuJzs7m8rKyi9Ef4D/mthlyKPUCHLBH+i1dumFHE6g75vR20+3oUqPRKYOr3wXTHljMplCGuJ96tQpDAZDr8Kt6PJgrzShzTf6JZ6aXGmVESyr1+l0zJ07lyeffJK7776buLg4tmzZwosvvsiaNWvYvXs3paWlzJ07l9TUVOynWvG0OdDPTAk4liYnBmWyjs5tFxG9Q1f6zZ07l4ULF1JWVsbHH3/sTxaupH7eh3+pQB/p1aKM7x3o542LIytBzxu7zg8o3zxe285X/3KA5S/vREjRocmJwbyjBq+j98MfGxtLWlraVXdD9Gf0FkAucMYiZUE+jh4gJymSyoZO5DFqhDYPgiD04uj92aNVCiahdIgaDFIm6MmPwHm+HVfDF6/haCCYzeZejpcgDS+x2WxBh8VcafiuicoiII/uKshGq0EuoLFKj3Oogb7nYHHosj24ZA6btoE+yhuk58Dj8QxqT9DU1MTZs2cpLi7udX/ZK9sQXV60+d0vIkVCBHKDGvup/mfSymQycnNzeeCBB3jssceYPHkyFRUVbNy4kaSkJObNmweAee8l5FEqNOMDi9G+rN7dZMNWfnm1pvnz5zN//nxKS0vZunUrSUlJREeHXv8YLvxLBHqfJbBe1PgbTHwQBIGvzs6g/FIHBy8E3kDnmsw8/nYJy1/exZ6zLRyv7WBbRRORXVm9ZV/gSqCoqIjGxsYhjTwcLtjtdmlYSIcbuUHN2RYLKrmMtJjuF11OUiRnm8wIBhXYPWjUmqAZvapT6HfQRF/4Ar0zTQlyAfO+f66s3md72zPQ+wZDfBHoG/816cCvihFkAopYDbI2NyqVKuRAbzQaewVXx9k2EHvLKjvsLl7Zcpo391xg3fE6SqpNXGqz4erDXw9VebN//37kcjlTpvT24rKVNyNoFb0UNoIgoMmNwXGmLaT6T3x8PDfeeCPf+c53uPXWW7n77rtRKBS4mm04Trehm56M0E8DoLYgDkWcls6tFy+7CW7BggXMnTsXj8dzVWgb+CeRV14u2traUMjkaFEFDVi3T0rj1+sr+POu80zLlG7QunYbL206zarDNagVMh5fmMVDczJZ+uJ2Vh26yJIHpqDOjqFzRy26mSnIevjm5Ofns27dOsrKykhKSrpi37Mn7HY7Go0GT5sDRbSac00WRhsjkPfodMxJjMTlEWmRC6gBjUodNNAr20ExMbxA3+mwkFQUj7WkEcOyzF7+QF9k1NbWIghCL5mfSqXyT69atmzZVfW+sVgsKJVKZJ0ef0YPUqE8nKaplpaWACmj/XQbgkbh58QBPjpSy/MbKgP2FwQw6lQkRGpIidbyHzfmou+hvOkZ6PszKbNarZSWllJUVNRrWpbo8WI72Yo2N7aXZh4knt6yvx7HhQ40ISqD1Gp1rwHfln11IBPQTe3/2RRkApEL0jCtPi1RSDmh9QMEPZYgcN111zFmzJirNrLyXyKjb2trI1KhQx6pRqYJfLdpVXLunZbOhhP1HL3Yxs8/PcH8X2/j/ZJa7p8xmu3fXcjT1+cQ2yXJ3HKqkWazQ+LqLS7pxukBnU5HVlYWx44dC4mjHAnYbDZ/oO8rrfQhO1F6oC908fdquSo4R+9VhKS4AalDWC6X097ejm5GMqLDg7V04OEpXyTU1taSkJCAUqagbe053G0S5ZWbm0t7e/tVXaVBV1esNgJESXHjg8Ko8TdNDRbo3W43JpOpFz8viiKO0yY0WYZetgcl1W3ER6o59IPFrH1iDn95cCq/uL2QJxeNY0leEskGDVsrGllTUtNLeRMZGYlCoRgwoy8pKcHtdgfYGTvOtyPa3L1oGx/UY6NBHlxmGQq8Tg+WQw1oC4zIowauOUVMTEBuUNO55fKzekEQyMzMDNn0bLjxLxHoTSYTkWgHVI3cP3M0MkHglt/v5s+7z3PzhBS2PD2fn9ycT3xk9wN1x+Q03F6RD4/Uoh4dJTVY7KjB6+zN1RcVFdHZ2UlVVdWIfa+BYLfb0Wq0eDocCFEqqlutfmmlD2PidchlAietUjBTCcpegd5sNqNWqlAgD0lDDxJHajAY6OjoQJUeiTJZh2Vf3RfKA6g/iKLoL8TaTrRg3lmLpUtGm5OTgyAIV52+MZvNRKik+7jnXABFnBbR6UWv0Q0a6E0mE6Io9gr07iYbnnYn6j78/OEqE8Xp0cTp1eSnGFiYm8A909J5anE2v7i9kDcenEpBShQHz5t6KW9kMhkxMTH9BnqPx8OBAwfIzMwkMTGx1+9s5S0IShnq7MBagUwtR51pCGqHEApsR5sQ7W70MwKLsH0hKGREzk/DWdWB8/zlWSFfbfxLBPq2tjb0zuC0jQ/JBi2PLczi5gkprH9qHs/fMaEXn+3DuMRIJo6K5r1D0ls+alG61GCxv3dWn52djUqlumpFWZvNhlqpAhHaVTJcHtEvrfRBo5STYYygrMUsPViiIqAYG6GUgkmoHD1AVFQU7e3tCIKAbkYyrjoLzurw55leabS2tmK320lNTfVL62zHm/3zWUeNGnXVA73FYkEr7+6K9cG34tLJNX77hv4QTHFjPy0Fzp6F2KZOB9WtViaPHrg4OzUjltKaNuhKBnrSN/0F+pMnT9LR0RGQzYteEVt5C+rsmF50aE9ocmJxN1pxt4ZvOGfeewlFYgSqzIFHXPqgm5qITK+kY+vVL8RfDq75QG+327Hb7ehd6oBCbF98a0k2v7tnEuMSB54XeceUNCobzJTVtKPOMKDOiqZze++sXqVSMX78eE6cOHFVjM7sdjtqQVom1nml8+qb0UOX8qbRjDxGjdItD6BuNIIaQSNHpgt9yWkwGPyGVhETExDU8gB664sIXyE2OS4Je0UrskgV7hY77q7AlZubS0NDQ9B5v1cKUgObryu2N3UDECGqcbvdAR3OPeEL9D3ljI7TbSiMGhSx3auEkmrpexanDxLoM2Nxur1UeqS+kp7Km9bW1qD05b59+4iNjWXcuHG9fu6s6cTb4QxK2/jgl1lWhkffOC924rpkQT8zOeQ6i6CUo5+TiuN0W8B83H8mXPOB3vdQRooaFPHDYza2YkIKaoWMVYclxzx/Vn+gN39bVFSEw+HwT5O5krDb7ahFqR5x3unT0OsCtstJjKKq1YpgUKNyCQGBXutVoogLTVrpQ0xMDB0dHbzxxhscOV6KfEI01rKmIXl9X0nU1tZKksN6ATwiMbdlgSBl9SAFeuCqZfVerxer1YrGq0SmV/byYpFHa0AmoHUNrqVvaWkhIiLCb74n2R60BdA2JdUmlHKBgtSBm6emZkgvjH2NHQHKG7fb3atpCSS/l5qaGqZPnx5g7GUvb5G+R27/xU9FnDQKdCCZZTBY9tZJvjaTwhswop+RjKBR0L7uAq56y2Vp668WrvlA79fQi1qUYdAPAyFKo+SGgiQ+Kr2E3eVBnWlAPcZA5/aLvQaTZGZmotfrr7glgtfrxeFwoPRIgf6UxU6sTkV0RGDxKSdJjyhCp1qO0iHrZbVssVhQO+UhF2J9mDFjBosXL8Zms/HJJ5/wpxPvs012nJMbSq5acToU1NbWkpycjP1YCwqjBs34WFTpUdjKuzPghISEqxbo7XY7Xq8XjVPeS3ED0phIRawGrV0K/oMF+p60jbO6A9EZaHtwpKqN/BQDmkHMvWJ1KrIS9BysMvXyvOlPYrl///4AJQxI1IqtvAX1WAOyAeY5C4KAJicGx9m2fm3D+8JjdmItayKiOCHsZjCZRkHUdaNwnGmj4bclXPrZPpr/Wk7n9ouSZcQwWX1YjzbhvGQefMMh4F8m0OtlWuQhuvGFgjumjKLT7mZ9uZTFRy1Ox9vpwtwjq5fJZBQWFlJZWXlFp0/5fW7ccmQRCk61WAP4eR98ypsGweu3S/AFFKvVisYhD4ufB8lRcM6cOTz22GM8/PDDTJw4kWpFC6uPfs5LL73Eli1bhjSBaCThdrupq6sjJT4Zx9k2tBPiEQQBbYERV50Fd4tEheTm5lJdXX1VJk91z4qVB3WWVBg1aDqklVc4gd5+ug1k9LI9cLq9HK1pG5S28WFqRiyHL5iQ91DeBAv07e3tlJeXU1xcHOD34m604m62DUjb+KDJiUV0eXGcD83z3nJI8rXRz0gefOMgiJyXRtIzU4m5I5uIgjjcLTbaP79A02tHqf3JXhpfL6Njc3XIL56+cNZ00rqqgo6NIyPe+JcI9EpBgS42st/miKFg5hgjqdFaVh+WDLDUY6KlrH5bb66+qKgIr9d7RR0t/YHeISA3qDnXHCit9GG0UYdKIeO8y+Wneux2u//FpBFVQ14JCYJAWloay5cv5/EVD7HQWUBsRDQ7d+7klVdeCXtYxEiisbERj8dDnFMHolRbANDmS14wvqw+NzcXURSvCh3nb5ayCgEZPUgFWVWXxVJ/gd7hcGA2mwMKsapRUb2kxyfrOnC4vRSPDk2rPi0zhk6Hm1atzK+88c1Q7hnoDx48KG0/bVrAMWzHpb+xNm/wQK8eYwBFaDJLr9WFZW8d6jEGlInBn4PPjtWxtmzgOpIiVoNuciIxK8eR9J0pJP9gOsb7xqOfnoTo9NCxsYrWNZVhK8y8Vhctfz+JXKci5ksjY49wzQd6k8lEpKBF2U+gGypkMoEvTU5j15lmatukbC9q6Wi8Ziem1d0XOykpifj4+CuqvvEV4hQ2EKNUNJudQQuxAHKZwLgEPScsDlR0jRi02/1BRSsOrFYKFZFFiYyLSOMm9TSeeuoptFot27Ztu+zjDhd8hVhDrRxlis5vdKWI1aBM0fl5+uTkZKKioq4KfeO/Jm5lr0KsD3KjBrlDWlH1F+j7Km48FheuWrNkA9wDvkLsYIobH3w8/cmuepCrwRogsXQ6nRw6dIjc3NyAqVYgDRlRpUcGDAEPBplKjnpM9KAyS9Hlpfn/TuAxO4laOjroNm6Plx9+eJwff1yONwz+Xa5XoS2II3rFWBK/OYmo6zOwlTbRuTl0hY7oFWl9twJPpxPjfeORhyF6CAfXfKBva2tD7x5ccTMUfGlyGqIIa3xZfYYBw7IMbGXNdGySLrYgCBQWFlJdXX3F1Bq+jF5hgU6VtIrpj7oBqUP2sMnSK6P3N0uh8is6QoXXK9LZZ5ygoJChmyqpWXQeNTNnzuTs2bMheYgHg8fjGVZtfm1tLRHaCDSXPP5s3gdtfhzO6k6pJ0EQrppHvf+aiMqgNKSvljKQlr5voO+2PQjUzycbNCQbQntu0mIiSDFo2NMunWNf5Q1AWVkZdrs9QFIJ4G6146o1+1dQoUCbE4O72Ya7ObjCSPSKtL5XgfNCB7F35qDOCF5U3n++lRaLk2azgyMXh+46G7kgjYjiBDo2VWM9GlqTYOfWi9grTESvGINq1MBqv8vBNR3oRVGkzdQmmZkNUyG2J0bFRjBrrJHVh2v8mYB+XhoRkxPp3Nx9sX2OlleqKOvL6FUOGU2CdF79ZfQgedNXmO2o5Er//r6gotNoByyMBcNr288y65dbaOzorXPWTZdazi0H6pk6dSoajYadO3eGdWyQgvwfX3mdV196xZ+JXy5qa2tJ1BoRENAWxff6nbZACoo96Zur4VHf8+UblLrpusd1Cu2ggd7HnztOtyGo5b1sDwCOVIfOz/swNTOWzTVtAcobn8Ry3759JCcnk54eOJ3NZ2kdCj/vg6bLliAYfSOKIu2fnsN2rBnDTWOImBAfsI0Pn5bVEaGSo5QLbDgx9M5nQRCIuX0cqowoWldV4qgauMnKXmmiY1MVEZMS0E0fWu0gVFzTgd5qteJ0ObuklSMz2u+OKWlUt1rZf1662QRBIOa2rO6LXd1BTEwM6enplJWVXZEOUf90KVHBRbcbuUwgPbZ/aWlO1xASuU7r398XVPSxoTWW+OD1iry9v5pOu5vfbek91k4RrUGTG4vlUD0quZIZM2ZQUVERtq3Awf0HqDc10mFq509/+hObNm0adCDzQLDb7TQ1NWE0a1FlRgXQIoqECBTxWn+gv1oe9RaLBa1SgwwheDE2Wg0y0AnqAQO9wWCQDO9EEftpE+qx0b3qV/XtdmrbbBSHSNv4MDUjliazA69R0yvQO51OysrKaG5uZubMmQFS3Tf3XKBsywWEeG1YNKEiTtreXhm4UjbvrMW85xL6OalEzk0NsrcEt8fLuuN1LB6fyIwxRjaWD22ylA+CQobx/jzkBjUt/3ei36Yut8lO6z9OoUyMIPq2rBH3T7qmA31PaeVwaej7Yll+MpFqhV9TDz0udlTXxW5zUFRURHNz8xXxSvEHepSctjtJj41Apej/Uud0KW9sXQOwfYFeAPTx4Q2g2He+hdo2G5lxOv5x4CIXmnurU/QzU/CaXdiONzN9+nTUajU7duwI+fhms5mtW7eS6onlTucscrWj2bVrF6+//vqQaaC6OqkIZzRHBNA2QJf6Jg7HuTY8Fpffo76ioqKXR70oirS0tFBaWsonn3zCq6++yqpVq4bt5W6xWIiQqxHUcmTaQImgoJAhj9ag9agwm81Bpaw9FTeeVjueNke//Hxxenh2uj5DwEaVEKC82bRpE3q9nry8vF77uD1e/m/TadKsHt5rN/s/O1RocmKwn23vJYCwljbS/tl5tEVxGG4ceMbx3nMtmKwubipKZmleIueaLZxpvDyJo1ynJO7BfESPSPOb5QFjR0W3l5a3TiJ6RGLvy+u3A3g48S8R6KPUuhErcmhVcpZPSOHzY/W9eGm5TkncV/IQXV5a3ixnfFYuMpnsihRlbTYbMkGGHBnHOuwD8vMAyQYNkWoFzTIBOTIp0HeaJcVNmC/INYdr0asVvPnVaSjlMp7fUNHr9+qsaORGDeZ9dWi1WqZNm8aJEydoagptms/mzZtxuVzMVowncUUus1vHsrL4BhwOB2+88QYbN24MuxPZR//EE4W2IDhHrM03ghfsJ6WVW25uLna7nYMHD7Jr1y7eeecdfv3rX/Pyyy/z4Ycfcvz4cWQyGeXl5cOm0LFYLFLNJKb/YqXCqEHrUPjlsT3hexH5Ar39jPR89J0PW1JlQqWQkZ8S3ks+K15PdISSCrfLr7zxBXqz2cy0adNQKHq/oHaebma8VUSOQIlG5K7X9/K3fVUhvxw1ObHQNedW+k4mWldVoh5jIPbOnF4GbcGwtqwOnUrO/Ox4FudJnjuXQ9/4oIyPwHjfeNxNNlrePoXo6f4+bZ+ew1VjJvaO7BGhlIMhpEAvCMIyQRAqBEE4IwjC94L8Xi0Iwrtdv98vCEJG18+NgiBsFQTBLAjCK8N76oPDF+ijjUO3GA0Fd0xJw+byBMizlIk6jPfm4qq3YPu4mnHjxl0RR0u73Y5GoUKQyyg1WfqVVvogCALZSZGcd7pRi0psFitmUyeaMBU3Foebz4/XcVNhMunGCP5tTiafltVxrKZb6yzIBPTTk3Fe6MBVb2HGjBkolcqQuPqamhqOHDlCgZhOcmEGupnJqLNjiD3o5mt3PcTEiRPZvXt32Nl9bU0tUUQQnZ3Yb0KgTNUjj1b7B1H4POrXrVvHpk2baG5uJjs7mxUrVvDoo4/y7LPPct/c24nRGdi0adOwTKeyWCxoPUqpC7YfKIxaNJbgA0gsFgsOh6O7EHumDblBHXCNS6pNFKUaBlwFBoNMJjBldCx7O6QXjKvBisFgQBAEFAoFkydPDtjngyO1LJOpkMWoefnJ2czJiuOHHx7n6VVl2F2D/83UmQYEpQx7RSvOS2Za/nYSRZwW4/15/tnQ/cHl8bKuvJ4leYlolHKSDVompBnYcJn0jQ+arGhibs3CUWmi7VOpnmMpacCyrw79vLR+k4qRwKBXUhAEOfB74AYgD7hHEIS8Ppv9G2ASRTELeBF4ruvnduCHwNPDdsZhwGQMXOoYAAAgAElEQVQyoUaJLiE8njlcTBoVzdh4HasOBwYXTU4shuVjsJ9oIcuThNls5siRIyPK1dtsNlSCElGvxOHxDliI9SE7MZJyix2VqMDWacXcaUYrKsPqil13vB6r08PKyZLn9tfnjyEmQslz63pz2RGTE0EuYDlQj06nY8qUKRw7dmzAJiqv18tnn32GThPBREcG2qI4aQj1ynEgl2H76AI3r7iZ++67D6fTyRtvvMHWrVtDOu+a6ovEeyKJmNh/wU4QBLT5RuynTXgd0oCP++67j3vuuYfvfve7fPOb3+TWW29l8uTJJCQk4Gmy0fZOJcVt6TQ1NVFaWhrSuQwEs9mM2qnoZWbWF4o4LRGO4DYIPefEil4R+5k21FnRvYePuD0cr+0Im5/3YVpmDPu6Ar270YpCoSAtLY0pU6b08pwHMDvcHC9vYKJXjn5KEtE6FW98ZSpPLR7H+0dquP3VPVS3DNxoKChlqMdGYy9vofkv5cg0cuIeKghKbfXF7jPNtFld3FTU7WS5JC+R0ottNHSEZ5jWH3TTktDPTcWyt462T87S9sEZVJkGDNdnDMvxQ0Uor+xpwBlRFM+JougE/gHc0mebW4A3u/5/NbBIEARBFEWLKIq7kAL+FYep1YTeO3weN/1BEATunDKKw1WmoPyeflYKuulJxB8XSDTE88knn/D222+PWHeoz+fGrpW4v8GoG4CcRD1nnS7UKLCZrVisln4HtfSHNSU1pMdGMDVDChJRGiWPLcxi15lmdp3uHskm1ynRFsZhKWnA6/Qwa9YsZDIZu3bt6vfYpaWlXLp0iVmGQjR6LepMiW6QG9TE3DIWZ3UnnTtqyMrK4tFHH6WwsJDt27dz9OjRAc+5o6ODTquZBCE66Fi5ntAWxIFb9Gu3MzIyyMnJCQhgottL6zsVyDQKxurSSJLHsHXrVhwOR7DDhgS3243D4UDjVvSyJ+4LhVFDhCi9CPoL9EajEdclM6LNHcDPH6/twOnxhq248WFqRixNiHiUMn9B9qGHHuL6668P2PbzY3Vc71YgCpJLJEirgqcWZ/Pnr0ylxmRlxSu72FoxsFRRkxODp8OJ6PIS91ABCsPgOnyQaJtItYJ52d2Z9dJ8SRm26eTwZPUAhhsy0YyPxbz7EoJGgfHe3GFt3gwFoQT6VKBnC2NN18+CbiOKohtoB0LWSQmC8HVBEA4JgnAoVK42FLS1mIgUNQP60A8XbitORS4T/J2yPdFhd1NTHI8tUc8NjQXMmjibqqoqXn31VbZt2zbs7pZ2ux2VR4Gp62YKKaNPiqQOLypRIXXGOm1oldqQJ0PVttnYe66F24tTe2WI980YTWq0lufWnerVjKKfloRo92A71kxkZCTFxcWUlpb66baesNlsbNq0ibTUNEZfikRbGNfrQdFOjEdbGEfHxiqcl8xoNBpuueUW0tPTWbt2rT/ABT3vi9L1Ss0YNeh3VY2OQqZX+pun+kP7esn8KuZL2cTcPo6p1rGYzWb27t074H4Dwd8shWrgjN6oRYtUVA8W6OVyOQaDoZufH9s70B/xFWJD7Ijti4JUA1qlnGZNd6AXBCGoquSTIzWsEFRo84wBTVILcxP49JtzSYnW8tBfD/Krdac422QOuhLWFsShHhdN3Ffy+u187Qun28v68nqW5CeiVnRf93EJejKMEcNG34BEV8benYtuZjJxD+Qhjxx44MlI4AtRjBVF8X9FUZwiiuKU+Pj+l89hHpP2zvYuxc3IB/qESA0Lc+JZfbiG59dX8M13jnDLK7uY+F8bmPDTDax4bTd3NjTQKELyYS2Pfv0RcnJy2LZtG6+99hqnT58e/ENChM1mQ+WSUYeHSI2COP3gN1ZOYiQmRFQo6bRZcHnd6HShr4Q+KKlBFGFlce9RaRqlnG8tyeZYbTufHe+uYagyDSjitX4f/9mzZwOwe/fugGNv27YNq9XKonEzwSUS0UfnLggC0bdmIdMqML1Xiej2IpfLWblyJXK5nFWrVvUrv6w6fg5BFEifMi7o73t9jkxAm2fEfsrUr6eJ/YwJ885adDOS0ebGos2NJWNCFhneBHbv2h3SmL9g6NWp3ENaaXa4e71AFbEa5IIMrTKwO7alpYXY2FhkMhmOM20ok3QBQedwlYm0GC0JkUPzhVLKZUxKj+a0x+1X3gRDXbsN1dkOokSpZhMM6cYI3v/GLG6blMqr286y6DfbmfGLzXzr3VLeO3iRi63Si0QeqSL+3wpRZ4ZePN59ppkOu5vlRb0/WxAEluYnsedsc0DT3+VAppYTc0vWiDZFDfj5IWxTC4zq8e+0rp8F3UYQBAVgAPpPo64AzGYzbq9H0tCH6b44VHx5+miazQ5e236W0osmIjVKbixM5vs35PKH+4p594m5VEwxEuXyUvtRDXfccQf3338/giDw1ltv8e677/p93C8HdpsNlajggtPNmHh9SBpdo16NUa8GmRKrS2q40keFdlOKosiaklqmZcYyKohe/7ZJqeQkRvL8+gr/UGlBENBNS8JZ3Ymr3kJ0dDQTJ06kpKSkV4BqaGjgwIEDTJkyhchqAXmUCtXowJqLXKck5vZxuOot/q5kg8HALbfcQn19PRs3bgx67rVVFzEKkUTmhZZgaAviEJ0e7GcCZYBeqwvTe5Uo4rW9ZH3Ry8cwXZ6Nx+0esu1Dd1dsd7OU1elmznNb+MOO7sYtQSFDblCjkwUP9EajEdHlwXGhPUBtI4oiJdWmkG0P+sPUjFiOWO1+5U0wfFR6iVtQIRpUAefRE1qVnBfunMjWpxfwP7cVMjUjlh2VTTyzpoy5v9rKnOe28Mzqo3xy9BKeMOwLPi2rI1KjYE5W4HVfkpeIyyOyrWL42IWrjVAC/UFgnCAImYIgqIC7gY/7bPMx8JWu//8SsEW8yrPj/NJKfdSg1ffhwsLcBA7+52JO/WwZO5+5jr8/PJ3/ua2Qf58/lmUFyeSlRHHnreNZqxUxnG6n41gTY8eO5Rvf+AbXXXcdp0+f5pVXXhmUVx4IXq8Xu8OBGiUnLXbGhsDP+5CTpMchdC9jI0NsliqpbuN8s4UvFQcffCyXCXz3+hwutFh592A3CxhR3F2UBZgzZw5er5c9e/YAUuD5/PPP0Wg0LJg1D3tFq0Tb9COZ0+YZpa7kLvtYkGSQ06dPZ//+/QENTh6Hm3pLC8mxiSHfI+oxBgSN3G/A5YMoipg+OIPH4iL27txe2mi5XsXom4vIdadSUlISspS0J/wZvVyNTC8VW/eda6HN6uLt/dW9s/o4LRFi76Ypr9dLa2srRqMRx4UOcIuo+/DztW02GjocQ+bnfZiWGcs5JMWMj77pi/0Ha5mEAsPMlEElkACZcTrunZ7OK/cWc+gHi9nwrXn8ZEUe+SlRrC9v4JvvHOGlzaGtih1uDxtO1HN9flJQZVFxegxGnYqNJ4aPvrnaGPTu7uLcHwfWAyeB90RRLBcE4b8EQbi5a7M3AKMgCGeAbwN+CaYgCBeAF4AHBUGoCaLYGRH4An3PKTpXAvGRapTy/v+saoWcvDtyqcRD06pKPJ1OFAoF8+bN47HHHiMlJYWPPvpoyM0/TqcTURRRiQpOWR2DSit7IjsxknZ397lHxoW2FF5TUoNGKeOGwqR+t1k0PoEpo2N4afNprE6JRulblI2NjaWwsJBDhw5hsVgoLy/nwoULXHfddQjnbeAR0Q7Qyg4QvWIMcoMa06pKfxPNkiVLSEpK4qOPPuq1Yrp08BwuwU16bkZI3xOkjFk73ojtRAu/21DJ7jMSX28tacR2rJmoJaNRpQbWRLQT45mZMQmFV86Gz9aH/Hk++DuVDZH+Fdr2royzxmTj4IXuwr7CqCHCqegV6Nvb2/F4PBiNRomflwsB3i8l1dIzc7mBflJ6NDVd1hs+z5ueOHGpg8JmJ14BdFMSA34/GARBIDsxkgdnZ/L6/VMo+eESbpuUyu+3nukl5e0POyub6bS7uakoOGUklwksHp/I1lONOIfJa/5qI6Q0RhTFz0RRzBZFcawoiv/d9bMfiaL4cdf/20VRvEMUxSxRFKeJoniux74ZoijGiqKoF0UxTRTFK+LXa2qVltYxiVc20IeCBXmJbB4TgeD0UPfuKX+BKSYmhrvuuovIyEhWrVo1JA/7nl2xDYQmrfQhJzGSNrH7lohKHvyBt7s8fHr0Esvyk4jUDDws4ns35NLU6eAvuy/4f96zKAswd+5cXC4XO3bsYMOGDSQlJTF58mRsZU3Io9WDcpwyjYKYO7JxN9swvVuB9VgzYquTL92+Eo/Hw5o1a/ya9qpSKQNML8oa9Hv2hDbfiGhzs23LOX6zoQJ3i422j86iyowicl7wVY0gCCSvzGeimMHp82c4f/58WJ9psViQI0PTY47x9sompmfGolPJeb+km01VGLVo3VJ3rO+7NjdLf1+j0YjjTBuq9MiA4nNJlQmtUk5u8uXxyBEqBQmpkdiE4Bn9x4cucgMqlONjkYdQPxoMcpnAT1bkE6dX8e33SgfV3689VodBq2T22P517EvyEul0uNl3bvgY6FD6AkYKX4hi7EigtakFjahEmzSyGvqh4pEvFfBHmRPOtGM92L1EjIiI4M4778RsNvPBBx+E3VzltyiWK7FBeBl9Up9Anzr4S3LzyUY67G6/dn4gTMmIZfH4RP6w7Swmi8Td9i3KxsfHk5eXx/79++no6OCGG25AtHmwn+4eBjIYNGOjibxuFLbyFlrfOknDC4ex/7aCObI8qqur2fDnj7EcauBSQx1KmYL4hPAEADXRSuyILJKpOFrdRv3bJ0EGsXcN3ImpiNEwa/FcdKKa9R99Hta1lWbFqlDGSPWmC80WLrRYuaEgiRsKk1l7rA5b1wpGYdQSIar8+0G3tDJaG4XrkhlNVuBLvKTaRFGaYcAVaaiYlhnLOdGDs753QdbjFWktaSASgZjZ/XvQhAtDhJLnVhZxutHMixv770S2uzxsPNHA9fmJAzaEzRkXh1YpH1KXrN3l4XhtO6sP1/Dfa09w/xv7mfbfm8j94Toe+utB6tr7n+c7UrhmA31bc6ukuIkbWQ39UDEqNoLEhaM4hJuWj8/4JxgBpKamsmzZMk6fPj2gtjwYfBm9R6FCECDDGB5104yU5SmQo9YNXsReU1JDUpSGWQNkRz3xzLIcLE43r247AwQWZQHmzZsHSK6fo0ePluaIegPVNgPBsDSDlJ/OIuHxicTclUPkvDTyUrLJUaSxv+Yo5e/vo0noIDkxOWBu6UCwuzx8c/VRSuRelmu03I8Kai3E3Jo1oL7dh+g56UyPyqe+rZFjJaHXYiydZrTe7kLsjtMSbbMgJ4GVxWmYHW5/UFLEBWrpW1paUKvVKOpcXbbEvfl5u8vDiUtDb5Tqi6kZEk9v7xPo95xtZqFdwB6llIaHDCMW5CRwz7R0/nfnOQ5XBe9R2VHZhNnhZnmPJqlg0CglW4RNJxpD8qi/1GbjqX8cYdFvtpH/4/Usf3kXT686ypt7q2i1OJkzLo6H52Sy52wzS1/YwTsHqq+IwaEP126gb2+/Yhr6oeKRBVn8JVrE5vHS8m5FLz+MKVOmUFBQwNatWzl37twAR+kNX0ZvkSlIjdYOOu+zJ/RqBZ6u4K6VD9500tTpYHtlk7+HIBRkJ0Zye3Eab+6t8svjfEVZc1dWn5SUxCOPPMItt0h9edayJhRGaQBIOJB12e/qJiVgWJZB3AN5rHzmAYxxRrZHV9KiMDNqTKBl7kD45eenOFXfyZhZacitbh5Gw6EIIagZWjAIMoHp9y4kVtSzeX3orpvmTnOXD710XbZXNDHaGEFGnI7pmbGkRmtZ00XfKGK16AgM9EajEefZdsmWOLU3PVNW047bKzL5Mvl5H6ZmxHIeL3K7p5fyZufuagpRYJydGtLqLFz8503jSY3W8p33jvprQT2x9lgdMRFKZo4dvM1naX4i9R12jtUOzPu321w8+JcDbDjRwJh4PY8uGMsr905i07fnceKn17P2ibm8cOdEfrA8j/VPzaMg1cD33z/Gl/+0f9DO3+HCNRnovV4vHXYzkTIdsqgr35wQKjRKOY/fnM+vRRuu6k46d/RwwBQEVqxYgdFoZM2aNXR0DOxtDVKQLy8vB6DJIw+Ln/chpssuIkI9+Avyo9JaPF4xQDs/GL69JBuFTODHH5cjiqK/KGs90ugvoCYlJaFQKPCYndIM16LQaJvBoFKpuOOOO7A7pLm4qamh0wdbTjXw1z0X+OrsDIqvywC5gFUj4wfW9rCW4+rkSBYWzKLDZWHPp9tC2sdisUga+mgNDreHPWdbmJ8trXBkMoHbi1PZdbqJhg47glKGXi8F8r6B3n6mLcCWGCT9PEiF1OFAjE6F0yA9ez6e3up0E13ZjlsAw9T+C/eXA71awfN3TOBCi5XnPu+tsrK7PGw60cCygqSQ6KnrchOQywb2qHe6vXzj74c532zhTw9M4Y8PTOE7S3NYXpRCVkIkij6fM9qo4+2vSWq8spp2rv/tDt7YdT4saehQcE0G+s7OTryiF4M+asR9ni8Xi8cn4M6JYZvMTcfGapy13RYKarWaO++8E6fTyerVq/s1xhJFkbKyMl555RVOnDhBoTudKqcsJOuDvshMlnh5feLgD/zqwzVMGBVNVkJ4L5SUaC3fXpLNllONfH5ceoj6FmV9sB1rlma4DqK2CQdJSUnceOONqNVqRo0aNfgOQGOHnadXlTE+OYpnl+Ui00qt7Mq7sjEDnx0Lj8stuG0GoxTx7C7dj9M2sDWCKIpY7VZp4EiMmkMXTNhcHn+gB6lXwSvCh0ekrD7SaEBAehZcLhft7e3ERBjwtNrRBNGtl1SbyIzTSb0Uw4T4LpsKH0+/qbSORV4FzixD2MNswsGMMUa+OjuDN/dW+VVRANsqmrA4PdxUODBt40N0hIppGbH9yixFUeR7a8rYc7aF51YWMSsrNPpSEATunZ7Ohm/NY8aYWH726Qnu+MMezjQOrZkuFFyTgd43si8mZniyk5GEIAj8eEU+L2KnUw6t755C7FGdT0hIYMWKFVRXV7Nly5aA/ZuamnjzzTd5//33MRgMfHXlfUx3j6PG42bsEObk5qTEIorAIBl9+aV2TtV38qXioRXUHpyVQUFqFD/5uJwOuyugKOuDtawJRYIWReLw1lomT57Ms88+S2Tk4AoTr1fkO6skKuB3d0/002Ha/Dgyx8eTmxTJZ8cGHizdF4JCxsw5s3Hg4tSuga2rHQ4HHq9Xsj+IUrO9sgmVXMaMMd30w5h4PcXp0awpqZHktfERaJG09D5PpSi7FMT78vOiKHKk2jRs2bwP+dlGzIg0XZCojwu7a9AhkH5d8Nmtw4lnrs9lTJyOZ1aX+TtcPy27RKxOxYwxoSvxluYnUtlg5nxzYJfvCxsref9ILU8vzeb2MFe1ICU8f35wKi/eNYFzzRZufGkXr28fmall12Sgb2uRbuyYhNDHkl1NZMbpuHt+Jj90mXE32mj+20lcDd03VlFREVOmTGH37t3+ph+n08nmzZt57bXXqK+vZ/ny5Tz88MPEKyWOtQFxSNRNTlIUVd4YxMiBOec1h2tRyWWsmBBadtQXCrmMX9xWRLPZwa/XVQQtyno6HDgvdBAxTLRNX4RahH1j13l2nm7mh8vzGJcY+GK4qTCZw1WmsNUU2bPyUaHgRFn5gNuZzdIqL0IbgSAX2FbRyNTMGHTq3g6NKyenUdlgpvxSh6Sl96roaO/wK250rTLkhkCjuupWK81m52Xr5/ti6hgj5/FgvdRJY4ed8Q0O2nRy1Bkjr4TTquQ8f+cE6tpt/PzTk9icHjafbGRZQVIAnTIQlnR51G/sQ9/840A1L285w91TR/HYwvDkuT0hCAK3TUpj47fmszgvAatzZCSY12Sgb6mTlvuxqcO33B9pPLYwi0sGJW9HijirOmj4bQmt71X4R5EtW7aMlJQUPvjgAw4fPsyrr77Kzp07KSws5PHHH2fS+AmYt9ZgWl2JV4BavGFJK30Ym6Bjp2ccb5zw8NNPyvn8WB3N5t7Ugsvj5aPSWhaNTyA6Yug1kMI0A1+ZlcHf91dxuMoUUJS1lknXcbAmqZHEsZp2frX+FMvyk7h3WvDC7Y1djTefh0nfKFRKxsSM4nxHLW5r/4PG/c1Sej2X2mxUNph70TY+LC9MQaWQsfpwTZfEUk1nW3eg19Z4UWfFBLw0fVOdLtf6oC9So7U0qWSo25xs23GB8cjRTU+6YnRqcXoMj8wfy7uHLvLjj49jc3lYXhjebNa0mAjykqN6mZxtq2jkPz88zvzseH52a8GwfJ/4SDWvfnkyTy4a3HNpKLgmA72pqZUI1GiSro6B0FAQoVLwg+V5vNrZyZb5CejnpmIta6b+N4cwfXgGwerhjjvuQBAEPvnkE5RKJQ8++CDLZy3FtaGOul8coGNjFcpkPZ/lReJQyUiKCpT7iV4vJ3ZsweUI7hytVsj52S0FpBi0vHOgmm+8VcKUn2/iuue38ezqMlYfrmHVoRpaLM6wi7DB8J2lOSRFafjPD47h1ch7FWVtZU0ok3VhT7kaLlgcbp74xxGMOjW/XFnY7wM9Nl4/JPoGIG9SAQ7BReWO/gfH+wN9dCQ7KiVZ5fzswBWXIULJkvGJfHz0EmKMWgr05k5aWlrQR+hQ2AjOz1e1oVcryA6yWrlcCHEaIjygPdSEQ4DR88JTOV0unlw8jtykSN47VEOcXuUfdxgOluYncrjaRLPZwfHadh57q4ScxEh+/+XiYek56AlZiOq1sI87Ike9ymhra5PMzK7QmK7hwg0FSSzIiednm09zviCG5GemoJuahOVAPXW/OoSwt417br+Lm266iQcX3YVui5mGF0uwlDSiK04g8VvFxD9UwB6XkzHxuqCB6dyRg3z++xc4vPajfs/j3unpvPP1GZT9+HrWfGMW37shl8w4HZ8fr+PpVUf5jw+OYdSpmJ9z+Zm2Xq3gv24p4FR9J3/aed5flDXvrMVZ3Yk2DO38cEIURX7ycTkXWiy8eNfEQVcuNxYmc6jKRH17eKMXcmcUokDGyWP90zcWc/eg9u2VTSQbNGQnBqflbi9OpdXiZE+zGZ2oxua009jYSLRSCuLBDMQOV5mYMMoQskQ2HBi7bBYm2aEpLQKZZvCBIMMJtULO83dMQCETuKkwOSzaxoeleUmIIvzf3ioe+utBDFolf/nqVPTqK/tdLgfXZKBvt3YSpdCF7KX+RYEgCPz2rokkGtR84+8ltAgiMbdmkfSdyUQUxmHeWYPirTpG7RAw/fUkrjozUUtGk/y9qcTcPs7vxX2uycyYuOCB4OTObQCUbVqHd5DxdiqFjMmjpeXvGw9OpfRHS1n31Fx+dmsBL98zadiymSV5iVyfn8hLmytpiFKiiNfSsakKgIiiKzduzYfDVa2sfG0Pqw7X8NiCrJA01zd2UQKfHw8vq1epVGTEpnHWXIu7Pbj6prNVKmZqjVHsOt3M/Oz+axbzsuOJ06tYc+wSOo20EqqrqyPKrUGZFBFgS2xxuDlV3zFs+vm+GNfDFTRzccaIfMZgKEg1sO6puXx3We6Q9h+fHElqtJbfbT6NzeXhrw9NIzHIavmLjGsu0Hs8HswuCwbdPw9t0xPRESpev28KbTYnj71VgtPtRWHUEntXDolPFaPJjkEWoSTmjmySn51G1KL0Xn4hdpeH2jZbUH7eabNy9vABYpJT6Wxp4lzJwbDOTSYTyE2K4v4Zo0OWkoWKn95cgEIm4wcflxMxNQlEUKbpg1pMN3ba+c2GCtaFGVQHw7kmM4/87TArX9tLjcnGL24v5NtLskPaNyvhcuibQqyCg3M7g9tAmU2dqEUFFz1eOh3uoPy8D0q5jFsmprL5ZCMRXVp6URTRdyhRB7E9OFrThleEScPMz/swdkwMFkRq1QIJ2VfPdyorIXLIGbggCCyfkIxSLvD6fZNHhOIaaVxzgb69vR0RiDaMzI17JZCXEsVzK4s4eMHEz9d2P/zKRB3GL48n4RsT0E0Obq17ocWCKAafKnX6wF7cTgdLv/5N9MY4SjesHdHvEQ6SDBqeXprNjsomdqi9CFoFuim9m2outdn48UfHmfPcVl7ecoanV5UFFIqHgmazgx99dJylL+5g5+kmvr0km23fXcA909LD4kx99E2480bHTylAhsDJ48HpG0unGY2oYl9zJ3KZMOhL9vbiVJweL2ZZdwJg8GgDaBuPV/Tr7otHjczzIpPJUKzIZPSXx3/he1oGwtNLc9j5zHXDnuD0ROOFcziGYGQYCq65QN9aLzVI/LNIK/vDLRNT+drcTP5vbxXvHbo4+A5d8M1mDdYsdXLXNqLiE0kdn8+ERcuoKjuCqa7vDJmrh/tnZjBhVDQ/2ViB7lvF6KZLgb66xcr33y9j/q+38tb+am6dmMJfHpyKzeXht5v6N7AaDDanh1e2nGbBr7fx1v5q7p42im3fXcgTi8YRoQo/+7uxMBlRlGahhgOtVsuo2BTOWS/hDDKVyWyWDM3WX2ylOD0ag3bgZqP8FAO5SZGctHR3WxoEXa8JTLVtNu753328d6iGL09PxzCCDUxjZ48iKfuf+3lUymUkGUaOrnFYrXzwq/9i7UvPjcjxr71AXyOpEmJTrzy3O9x4dlkus7OM/ODD4xy9GDhLtSdazA4ef7uEn689SVGaIWB5aWkzUX3sKOPnzEcQBAoXXY9MLufoxs9G8iuEBblM4H9uK8BkdfHcpgrONln49nulLPzNNtYcruXuqels++4CfvWlCSzMTeDL09N558DFIXUUHq9tZ+Hz23h+QyUzxxpZ/9Q8fn5rIfGRQ+8MzUrQk5MYGXaXLEjqm3aZlZo9gcMzrA4rWrmakksdA9I2PbGyOI3DHU5kooAgChhHJfhrVp8dq+OG3+6g/FI7L9w5gZ/fWtDvcUx1tex8502ctivjyfKvil3/eBNzawszv3TviBz/mt4fADwAACAASURBVAv0psYW6cZOD81k6osMhVzGy/cUE69X88jfDwelKURR5KPSWha/sJ315fV8e0k2qx+ZFWDBWrFnB6LoZfycBQDoomMYN20Wx7dt6ldqeTWQn2Lg3+Zk8s6Biyx5cTufHavjwVkZ7Hx2IT+7tYC0Hn7sTy4aR4RSzi8+OzXAEQNhc3p44p0jALz37zP54wNTwrZx6A83FiZzsKo1fPpmQj4AJ8tPBLgaWl12FF0mc8FklcFwy6QU6gSIQE2kqEE3zojV6eZ7a8p49K0SMuP1fPbkXG4vTuuXUmk4f5Z3fvQMBz5cxfrXXrqibov/SrhUeZLSDZ8xadlyksfljMhnXHuB3tSGDjVK4xfTnjhcxOpUvH7/ZFotUnHWN3MVoL7dzsNvHuLJf5SSbtSx9om5PLFoXFCf7ZO7txOfMQZjWreOeeLSm3BYLJzas+OKfJdQ8dTicSwen8Aj88ey69nr+OHyvKAqB6NezaMLs9h8qpE9Z5qDHCk4fvn5Sc41W/jNnROGpKseCDcVJQ2JvomKiiI5OpFzjjqcF7tXKB6PB4fXiUNUEqdXkZ8SWldpQqSGjKwYYr16EkQDtQYFy1/exbuHLvKNBWNZ/chMRg9gYV1z4jjv/fT7KFQqim+4mcr9uyn5rH9J7v/H0OBxu9jw+stExsYx5677R+xzrrlA32ZuJ0qhC2kO5T8LClIN/HJlIfvPt/I/n51EFEXeOVDNkhe2s/tsMz+4aTzvf2NWv2oAU10t9Wcq/dm8D6nj84kbNZrS9Wu/UNlahErBn74ylWeX5RI3iMnWV2dnkBqt5edrT4bkALi9sok391bx0OxMZo9AYS0rIZLsRP2Q6Jv8SQW0yDqp399tS+0fHOIUmDcuPqzi8PIpo5jkyme6Zzy3vl+KxeHmrX+bzrPLcgeUxp49fIA1//Mj9DGx3P3TX7HgK18ja+pMtv/9z9ScPB729/r/6B8HPlxNS001ix9+FJV25JLTay7QdzjNREX888mfBsNtk9J4aHYmf9l9gRte2sn33z9GfmoU656cx8NzxwzY7HJy13YQBHJnz+v1c0EQmLD0JhrPn6X+7NCLmlcTGqWcZ5blcKKugw+ODFxYNlmcfHfVUcYl6Hlm2cgskaGbvmkMl74pkMYpnzpxCrFr5dbZJGnomz3ygAY1j9vNR8//N2cP7w96vCV5iZyQwRavm3m5Cax7ct6gqpETO7fy0fM/xzhqNHf99Dmi4iTN/rJHnyI6MYlPf/scZlPwoR7/H+GhpeYi+z94l5xZ8xhTPHVEP+uaCvQuhxOr6CDG8MV3rRwK/uNGqThbY7LxP7cV8vbDM8gYxIpYFEVO7d7GqLxCImMDH/K8uQtQarSUrv/iSC3DxYqiFCakGXh+fYV/nF5fiKLIDz48jsnq5MW7JoY1kCVc3ORT3xwPL6s3Go3ERcVy3lMvDfAGOhul/7aICub0CdKndm/nzMG9bP7zH3A7A71yNEo5xntzib8zh/+9fzIxuoG7e0s+/5jPX/kNo/IKuPNH/01EVLdKRx2h4+Zv/wcOm5VPf/scnhAHpgwH3E7nNfdyEb1eNv7xZZRqDQu/8rUR/7xrKtC3VDcCEBP3xRsIPhxQyGX89avT2Pcfi7h3emga74azpzHVXQqgbXxQaSPIm3cdFXt3Yu0YeJLOFxUymcAPludR32HnjzuDT+P6qPQSa4/V8dTibApSh3eEXV+MS4xkXIKetUNpnppQQIOsjZaD1QB0tnR1xcZG9fKKF71eDny4ighDNJ3NTZSu/zTo8ZYVJA9YcAXpJbhn1Vts/ev/kjV1Jrc9+5OgNEJcegZLv/Y4tafK2fnOm2F/t3AgiiL1ZyrZ9KdX+cMj9/P6Iw+w462/4HG7RvRzg8Fm7uTY1g3sWfU2DefODAvNWbZ5PbWnTjD//n9DFz3yPT//PGYNg0EUaTm4GYDY5KFxr067jZaaauxmM3aLGUfXf+3mTunfFjO66Bjm3fcQKs3V8dFRymVhWQ+c3LUNuULBuOmz+t1m4tIbObphLeXbNjH15pXDcZpXHFMzYlmWn8Qftp/l7qmjSOhRvK1ts/HDj4777RxCgdvppHLfLkYXTRrSg3hjYTK/23Kaxg57r3MZDOPzxrNj5w4qKitIcRbS1hXos7J7N4+dObiP1ks13PTEdynfvpn9H7xHwXVL0ejCUw+JXi9b/vq/lK7/lPwFi1n69W8ik/e/2hk/dyGXTp/i8KcfkDIuh+wZc8L6vMFgaTNxYudWyrdtoqWmGoVSRda0mciVSg5+vIaLJ45x0xPPEJ04MhOqfLCZOzlzcC+V+3ZTfazUbxeyd/XbxCSnkDNzLjmz5hE3Knxv/c7WZna89RfSC4rIX7B4uE89KK6dQH9uK6aTB0EeR1xS+Pa8DefO8OGvf4a5tSXgdwq1Go1Oj0an5+zhA9SdqeT27/3kiryJLwdej4dTe3YwpnjagAEgbtRo0vIKOLrxM6Ysvw0hjGHZ4cLjduOy20Eg7KA0GJ69IZdNJxt4cVMlv7i9CJCGhjz93lG8XpEX75wYknFXw7kzfP77F2ipqUYbGcXihx8NO6DdVJTMS5tPs668ngdmZoS8X1JSEgZ9FBfaG7GfaKGxpR2ZKDB7QrdTqCiK7P/wPaKTksmeOYfY1FH87dknOPjRaube+2BY57n973+mdP2nTF5+G/Pveyik7tUFDzxMw7kzrHvtJYyjRmNMDW1KVzB43G4cVgs1J49Tvm0T50sPI3q9JGfnsuRrj5Mzay7qCOl5HlM8lQ2v/46/PfsES772GLmz5w/5c4MhWHA3JCQyeflt5MyYQ1R8AqcP7KVizw72f7CKfe+/izEtndxZ88iZNZeY5NCG8Gz58+t43W4Wf+3xK9YtfO0E+rHXYYqqRGZuIOrTu+HL70J0aJaoFXt3se7VF9FGRbHiW99DF2OUArtej1qnR6Hs7ho8V3KQT377S9754dPc/v3/IjZlaBOWrgSqjx/F2t5G7pzBH4iJS2/i098+x/mjhxkzKbTCkMftxtJmwtzagrm1GXNrC52tLZhbW7B2tOOy2XDabbgcdpx2Oy6btZvbFQRmrrybmSvvGbYXS2acjvtnjubNPRd4cFYmOUmR/GXPBfaea+GXtxeSPojk1uN2se/999j/wbvoDNEs/fcnOLrxMz558Zfkzp7PdQ89glYfWqE/u4u+eXFjJYerTBSnx1CcHkNucuSAKzJBEBhfkMeBfQdoO1yL2WxBiYqJPSwKqsqO0HDuDEu+/k1kMjkJGWMYP2cBJZ99zMRly4PWYoLh3JGDHF77IROvvynkIA8gVyhZ8a3v87fvPcknL/yCe//7NwErXNHrpa2xnqaq8zRVncfc2orDErhCdtq6h7XoY2KZuuJ28hcsJjYl0AI7e/psksaMY+3vfs3a3/2aqmOlXPfgv6PU9L9iclitVJWVcL60BGtHG16PB6/bjdfjweNx43V78Hqkf5vqaruD+023kjNzLgmZY3v9XYoWXU/RouuxtJk4vX8Pp/bsYPeqt9j93t9JyBhL1tQZjJk8jYSMMUH/nqf37+HMwb3MvfdBYpKGNrRnKBC+SLI6gClTpoiHDh0a0r5/++UfaXa28i3l70GhhnvegbQp/W4viiJ7V7/D3tVvk5I9npu/8x8hZel1Zyr44Ln/QhRFbnvmR6RkD80Vb6Tx+e9f4Oyh/Tzy+t9QqAYuxHncLv742EMkjsnitmd/3O92zRer2LvmH9SePI6lvQ363D9yhQJ9rJEIQzQqbQQqjRalRoNKq0Wp0aJSa1BqtNSfreTU7u1kTZ3BDY99+7KlZV6vB3tnJ/UNTTzyx+3kRsv50rQMvrG1g5k5KfzxgSkDBrKm6gus+/2LNF44S97chSx88P+1d95xelT1/n+fmXn6s71nk93NpkF6I6QZ6U0hogiIBv3RVEDxKlfhelXkol5AxQZyRVREkRLARLxUKRdCeu/Jbur28mx72jxTzu+PebLZTQ/skmWZ9+t19pw9M8/MZ74z850z55w558v4w2Es02TFomdY9uyTBDKzuOCmr51wD4k1+9r43Vu7WLOvjaYu52M3v0dh4tBsppRlM7Ush7Elmfg8Ch5FwaMpaIqgvraGx/70R842x7FFNBBTU9x+17e6t/vUD++gvaGe63/1++5CSEdTI3/8ty8zdt45XPDlrx9XW6y9jcf+/VbC2Tlc86OfH/f6OBL7Nq1n4T3fY/TMOUz7xKdo3rubpr27ad6zi+Z9ezCSjhMXQiGYnd1dePKHM/CHnEKUP+y8KecMGUrZ+IkoyvEbyW3L4t1nnmD5358mp6SUT972bQorKruXdzQ1UL16BdWrV1CzZRO2ZeIPhckoKERVVRRVQ9GcWFVVFE1DUVTnDWnmXIoqR55USbsr0sKOpUvYsewd6nZuAynJyCugctoMRkybwbBxE9E8HpKxKH/61s0EM7P4/I8fQNX6tpwthFgtpTyiwxtUjv7Bux7AHwpw/ZfOhr9+FqKNcPnDMO7yw9Y19CQv/faX7Fj6NmPnncP5N32tV8n9eLQ11PHcj39AtC3CJ277NiOnn/meNPcXhp7ktzctYMysuVz4ldtO6DdLnv4Ly557iht+9QhZhb3rQCN1Nbz7zBNsX/o2Xr+fUTNmk5FfSEZuHuEeIZBxYhOySylZ++Ji3nz8UXKHDOVT//49sotPbPafWHsbq154nsZdVcQ72ol3dpDo6jzsoQNgCZVh4yZy+pmzGDFtBhl5vUu7tmWxcvGzvPvME/jDYc678RZGnTHrsO007q7mpQd/Tsv+vYw/+wLOuvYGfMETezhJKanrSLJmbxtr9rWxZl87W+o6MKwj33sCyed86xhj5xIVSQj4ueXOmwGo3b6VJ7//75x17Y1M+8T8Xr9747FHWPviP/jiTx8kb+jRq1OkbfPsT35A7dbNfOG/f9HrI7qTZfnfn+GdHg2z3kCQgvLh3aGwopK8YWV4vH036fgB9m1az//+5mckuzqZdcU13aOzttY4Ddm5Q4Z2O9sho08/ZttDXxHvaGfX2lVUr1rOng1rMHUdjz9AxcQpWKbB7rWrueZHP6N4RN/PJPWRcPR20uS+n9zLqCGVfObLn4NYCzz5edi/DM75HnzsW5B2QF2RFhbdfw+Nu6uZd82XmH7pp99TXVm8o53n7/0hjbuqOff6rzLp/ItPehv9xbZ3/49//vI+Pvu9H1E2ftIJ/aartYVHbr2O6Zd+mnnput72hnqWPvs3tr79JprXy5SLL2X6Jy8nkNE3837u3bCOF37x3wB88ht3UD5x8lHXTUS7WLX4Wda89A8sw6Bk5BhC2TkEs7IIZGYTzMwkmJWNFsrg689X0R6JcFNFAnP3JtobnR4wRZUjGTHtTEZMPxPV4+Glhx6goWoHo8+cw7k33NyrS+GhmIbB0mf+ysrFzxHOy+Oir37jhG172LHoBivXb2N/awKZU4Rh2ZiWxLBtDFMS3bkM2bgHj1QpLy7j6q9+AYDn7/0hdTu3c9Nv/nBYlUW8s4NHv34DZeMnMf/2/zzqvlf/8++8+effc94NNzPp/Evek/4DSNtmy9tv4A0EKKyoJLOg6AMdpTLe2cFLDz3A7rWrUFSV0tPGMWLaDCqnzfhAq0aOhJlKsX/zBqpXL6d61XKibRGmffJyzlpwfb/s7yPh6Nu3N/CLvz3M7DHTOP+qTzj1vkYSFt8KG5+BSdfApb+gYc9e/v7Te0glElzytdvfd0ncSCZ54Zf3smvNSs68/CrmXPWFATEc6/P33U3TripufOiPJ/Q6fIDFP/sxNVs3cfXd97Fy8XNsfus1VFVj0oWfYMZlnyHYD98otDfU8/f7/4tIbQ0fX3AdUy+Z38uGejzOmv9dxKoXnieVTHDa7HnM/uw1x2z82lzXwbb6Lj4zbShSSiK1Nd033IHXawB/OINzr/sKY2bPO+HzVrdjGy899ABt9bWUjBpDbukwcocM7Y6zCot6vZZLKelqbaGhegcN1TtpqNpB466q7oHCLvzKbYw/+/xe+9i5cyd//etfAZgxagqXfH4+TXt28fh3vs6cK7/AzM9cfURty557iiVPPc7Vd99P6ZjTD1veuLuaJ777LYZPmc782787IK7V94uUksZdVWQXl/R5A39fIaWkrb6W7OKSk7ofT4aPhKPfvGQ5z7z6Ihl1NSjRFkI5OYRzcp2Q2EuoYQnklLNst5dgdi6f+vb3KSirOPEdmCmoXw9GHCo+Bj0aEG3L4rVHH2Ljv17mtDkfZ9zHzyWnZAgZ+QX9dlKPRaKrk4e/vIApF192eOmhfR+8+n2YeQsMO7yuee/GdSy8xykNqprGxPMvZsb8zxLO6d9vE1KJOC8++ABVK5c6VWk33oqUNute+V9WLFpIsquTkWfMYvaVnz+583YE4h3t7FqzkraGOqZcdOl7OjZDT7Ji0UJqtm6ira6WWHtb9zJFVckuKiG3dCi2bdNQtYN4R3t6mUZB+XCKR4yieORoti15i30b1/PJb3y7V88e0zS5/7770VM65338HOaePY8Xfnkfu9eu5Mbf/BF/+MgOzUgmefS2G8kuLuGqu+7t5ciNZJLH7/wGRiLOgvt+fcy3F5cPH8dy9IOm1032sGKGFhQycsJYVD1OtC1CtC1Ca20N+9o60OPDoRmGBFqZX7SB4NtRKDsThs2E0qngOaRfvB6FmhWwdynsWwo1q8BM9xAoHAvzboexnwJFRVFVzr/xVjJy83l34RNsW/IW4DjK7OIh5JQMScel5JaUUlg5ol/74e9Y9g62ZR3+kVRnHTx2KbTtgR2vwOeegMre65SNn8SY2fPwh8KcefmVh9Vp9xfeQJDLvnknS599kqULn6B5727inR3E2iJUTJrKnKsW9Fm9ZjAr+7AS9Mni8fmZc+UXuv9PxqK01dUSqatxQq0TA1RMmtrt2AvKK3u1BY2ZOZeFP/oe//zVT/H4AwyfPA0ATdMYNXoUmzZtIpyTSVt9LTuWvsP0yz59VCcP4PH7mXXFNbz2+wfZtWYFI6YdfGN948+P0FZfyxXf/S/XyX/EGDQl+sbdnSy8dxXlE/KYfnEFxZW9L2QjmSReu42M9o0otStg33Jo2e4sVDxQMgnKZoK0HcdevwGkBUKB4glQNhvKZ4GRgLd/7vw2f7RT9z/+ClCdZ2a8o925yetraW+oo62+lrb6Otob6rq7FiqqStGIUZSNm8iwsRMZMuY0PL6+m9TgyR98h0RXJ1/62UMHS3TRZvjTJY6z//Tv4PV7oLUarnwMxgyctgWAnSve5aWHHqCgvJK5Vy9g6OlHHy99MJCMRXn67v+gra6Wz/zHD7uPd8uWLTz99NMsWLCA6tf+yda33+SG3zx63J5hlmny2O23oKgq197/axRFZcfyJfzj5z/hjPlXdLe/uAwuPhJVN3rcYOObtaz/136SMYOhp+Uw/eIKhozOPno9ZDwC+1c4Dbb7lkHtGqfBtnS649TLZsHQM8B/SMOjbcPWRfB/P4XGTZAzHD72TZh4NWhH7qZm2xZdLc201u6ndutm9m/ZSEP1TqRto6gaJaNGM2zsBIaNm0h2cQma14fm8aB5fUftLSClxEgm0BNxUvEEqUScaKSVxT//MXOuWsDMT1918Dgfuwxaq9g57SmWLfEwYkIm0yK342teCZf/D0y44qRt3p9Yptnn3c8GMvGOdp686w5ibRGu/P6PKaociW3b7Nixg5K8XP5w201MPO9Czr3uqye0vQOO/cKv3EbZhMk8/u2vkV1cwtV33/+RsutHiY+Eo9/ZtpNvvvlNwiKT8v2TKKkaj6b7MQo6MKc04KswCXlDTMyfyNSiqSjiCB+tmOmBoY7irA/DtmHHi/DWfVC/DrKGwZzb4PRLIeP4n2inEnFqt29l/+YN7N+ykcbqKqS0D1tPKIrj+L1eNI+3+7d6In7ELoWKqnLdL/7H6SKZ7IQ/z8ds2M47BY+zeb0gM99PZ0sSf0hjRsFLjEs8jHLZAzDtiyd02JZpY5k2Xr/rMPqSzpZmnvzBtzF1navuure7i+Qbjz3Cupdf4PpfPkJmwYlNPCKl5In//BbRtghZBUU07a5mwX2/OuU9UU4aKZ3CWPNWOP0yCA7Ocaz6go+Eo9/buZdfr/01cSNO3IyTTOrk7R1Bxa5pBPUsmkP7WF36Kp3+FkpFOdMzZjLGN45gKot4R4pYu06sXcc0bFRNoGoKiqagqgLVo6CoCqomCGZ4KRufR8WEfIKZ6QeClFD1muPwa1Y4eQWnw4izofJsKJ8NvuP3BtDjceq2byHaFsFM6ZiGgZnSsdKxmTK6Ryn0BgP4giG8gSC+QBBvIIA3GMQXCBHOzSOrsAhSMfjLFbTtruFl+QCtLQpTLijjzPmVtNZEWbKwirqd7eQEIsz2P0j5pVcgZt9yRG2WabN/a4Tq1U3sWt9MKmHhC6iE8wJk5Pq7QzjXR0aen8y8AIEMz8G3qVQMOuuhq86JU10w9nIIfbjnEu1r2uprefIH30FRFK6++z48/gCP3HodY2bO5aKb/+2w9dsb4wQyvfgChz9092/ZyNM/vBOAi27+N8Z9/NyTFySl80bozwS1/+aVPYy2vbDhKYy1C9nfkEW7NYSywBbyps9GzLoZCvpumOlU0mTnykbMlE3ZuFyyi4Ifyt5IHwlHfzQs02b78gZWv7SXzubEYct1TxxvpqAgP4eC/Bw8XhXLkliGRUyP05WIEtcTxFNJdF3HEw3hTThjbxRUhBkxuZCKifnkloQQAA0bofp12PWmU9dvJp02gGEzHKc/4mwonnjibw3vFSMJf7uKbZsFb8W+hub38vFrR7ExsJRX977KhIIJXFp5KalqH+8+u5OO5iTDvOuYc45G3vyvgRBYlk3ttjaqVjWwa20jehK8apJK3zKylf1E7Xy67CKidiFdZj4pu3c7g19LkOtrIFfsIlepIlfbT662j4CSnkHJlwlzvg4zbwbvyY9PNFhp3reHp++6A184TPn4yWx4/WW+9NOHen0E1dmSYMmTm9i1qQufTzLxvOFMOncYvkMm+X7t9w+iah7O+uKNJ+y8pGUS37acyJqlRKr2Eu0ShNVWskJxMrMlmbk+tKx8yCiBcBFkDoHSaSf0FntM9C7Ysoj4ykXsqbLZrZ/B/tRULHnwIZat1THCt4SRoyzyzr0SMfLc7u9jTpaO5gQb36ph65J6UomDwy5nFgSoGJ9H+YQ8SkfloHqOP0SHoVskYwbhbN8pm/ToI+3oD2BbNns3tWKZklC2Dzugs6zjHV7a/yIrGlZgS5tROaMYGh7Kvs597O/aT8o+OMa3X/VTllmGikpzbRdlkbFUtE2gMOp8VejNhhGTixg5oZhAphevX8WjGHiaV6PtexOx+02neyaA6nV67pRMcsKQyVA4Djx91CBrpkj97f/x9urhbEueQ8GIEC2zNvBU7V+IJCMUBYtojDcCMKN4BpcNn8+Q3aez/h/VpEyV08obEXmV7NocJalreESC4b7ljPIvYViZhTpyLmSXQ7IDku2QaIdkO3o0QVenIBrT6EjmEJEjiBilROK5pMyDDigQUsgrVCmxl1Pa9RzFOe2oZ30Lpl77wZQakx1Qtw4i1aD6wBvCUkMkjQBxw0cy5See1NB1lUBeFlkFAbIKAoc50f6kbsc2Ft7znxh6klFnzuayb/4H4DiU1X/fwLq3WhDSYHJwMa1mGbv1mXi9konnVTDp3DL8oRPTmkqYNO/rIlLTTmT7Tlr3txHpCKDbB99AFcXGtns6O0lYayNLqSNTrSdLbSBLa3DsNHos3jHzoGLO8R/eRgJadiKbttG2bjm7N3eyOzGVRmMUoJCRrVExpZjhk/LJKQqxZ2ML1Strqa3qQkpBllrHyNxtjJh7GvlnfxrhPf6XylJKara3seH1GvZsbEERMGJIIxPlHwladez1Xcre5FRqWnKxLIHmUxl2Wg7l4/MoKMsg2qbTFUk6oTVJV3OUrtYE6dEe8PksiktVSkZmUzJ2GAWVuXi8H0wX64+Eo7eiMdoXPkNwyhT8p5+OOImxO1oSLbyy5xVe3vMyHXoHZZlllGeWO3FGOeWZ5RQGC7tLRNFUlLVNa1nduJr1ezeTqNIoi4yltGM0mjzCDSYkilegecGrpgioMYKyhVCqhqBsJqB0EFCj+HOzCZaU4sktQQllooRyUELZKKFc1IxclFAewtejW6aUYBlOt08j2R23vPAwr6yeSptVSnTiXp4J/xZdJvlY6cdYMHYBM0tmUh+rZ3H1YhZVLaImWkNQC3Jh8cVMWlFA095RKMJguG8lIwt2UTa+EG3kXPSyM+nSPERTUeJmnISZIGkmSZiJXiFpJrGkRcgTIsObQUgL40uGEG0+zFYVvRk661O01sRAgioMij1bGZpVS+m8uRSe9UnUHhODJGMGHU0J2pvitDfF6WhK0NEUR4+byLQdpAQkSOcPUjoFPSFAIYViJVGsBIoZRVhxFCwMITCtDJJ2Fro8ftWaT0uSFU6SmWmTleclszCI8ARIpRR0XaCnFFI66EnQkzZ60hHhD3vxBz34wh78QQ/+kAdfSMMf8hDK8pE3NIR2hIlQ9m1az5t//j0X3/ot8oeVs+NfG1j6j/3E9CCjA28za2YX4bNvgP3LaH7lSVY3zKVan43HK5l4TgWTzhtGINz7Poh3pqivaqduZzv12xtpqU8hpXNde0WMXG8dufmQW1lK7oTJ5FUUEsjwkIwadDQnukNns3MOOppjJKK9J3sJKO1kqY1kZZlkleSQWTkCoSgkGhuIt7aR6IiTiJokdC8JO5O4nY0pnUJOQbHC8OllDJ9cQF5pGCEEUkpM28STLgQkulLsWt1A1TtbqK1Rkahkas1kZ+kEczMJFhUTLBlCMMtHKMtLMNOHL6hRvbaZjW/WEKmL4fdbjMtZxnjzD4Q9nTD6IqeNrWYl1K/HCeVadgAAD45JREFUsAS1+gT2yLPYm5xCVO/90NIUgwy1mQzRQFhtJlNtwqvEaTZG0JA6jTbLGZRNwSI/1EhJXhdFpQI1mImhZJESGRiESNl+UqaGoVsYSYui4VlMueC9DUnxvh29EOIi4JeACvxeSvnfhyz3AX8GpgGtwFVSyj3pZXcC1wMW8HUp5cvH2td7dfTxlSvZu+BaR4/Xi3/8eAKTJxOYPInA5Ml4Ck+sEeu9EDfibGjZwKr9q9ld1UA8pqMnDFK6iTBUPJYPr+XHY/nwWD78ZpiAEcZvhvEbIZSTmP9FYKEICwUnCCyEsBHYCGEBFnE7j5QnxYsj/0Qkdz+XjbiMz4/9PJVZlYdtT0rJmqY1LKpaxMt7XiZuxhkuhlDoD9DpFURtnWgqStSIYth9O+nDUE8Z41JnUNo5ikBdBla742w1JUXRUC+WCNDerJOM92yglmSEDLLDMfweHaTt2EDaIC2EtAAbIS2sVIKoHiMmVGLCQ1z1kVR86CikbOdhkPRESXnj4EshfCk8fgOv3yToMwhrBhkJL8HOAJ6oHyUawoxnoKfyiFsFyEM+Q/GKGF4Rw6fE8Ik4XsWZ71WXmSRlJkmZgW4Gsent1BUhySuQFJUFKBxRQNGYIeQUH5z3uGn1Wt5+ejsNHYXke6opmFrFngnFbOzax9bIVvID+UzMG8/EWAcVK1ezp34u1ck5aB6YeHY5uUNC1FW1U7etmfYWp4pCFSmKPdsp9mwlkN1MalQJkYpKGkM5NCSbaYg10BhrpCHWQESPEFADBD1Bwp4wIU+oOx30BAmTSVaygFAiB19XANGYwGoxSHZ6SBi9uzkrmPi1OAG/STCsEsgKEMjLwT8kB88Ikxa1ntquWupiddR21VIbq6W2q5a4GSfLl0VhsJCiYFF3KBBFeDal0LclMLtAT4VI2tnYR/lEKD+jnYmepxjl+RdqQQVdE6+iffR5RBSBLW3C3jCZwku4bR/B+g0otauQ+1YSiSi0W0MIqy0EMqCrMJ/m7BIaw7k0+YI0qYJ2aZEpNPJsyIra+BsC2E1B4q25tEWLsOThhU+BhVck8KgpPKrJ8OEpZn3j2hO+h3pt6/04eiGECuwAzgdqgJXA56SUW3qsczMwUUr5FSHE1cDlUsqrhBBjgb8BM4AhwGvAaCnlked74/1V3RiNTSTWrSOxdi2JdetIbt6MNBzn5CktJTB5MlpBAcKjITwe0JxYaB4n9niQlolMJLDjCexEAjse6/W/tEzUcBglnIESDqNmhFFCYZSMsJMfCoGqIRSBVBRStkGXHaPLiNFpdNFlxUjYOgmZIm4nSdgp4ikbPQkpXcHQBXE9RSwVx7AsBCpCqghUFKnix4tPatgIbFsgbQFSQZEKAgXFVjCVBJ35y/lE4XTm+Mbia49hNjc7oakZs70NNRRGzc9Hy81Fzc9Dy83Dyg6z3tzD69HVJGydMH4yhJ8M6SOEl6D0ErQ9BKSGT3jwKh68mg+v4sWjefGq3u60QEG3dXQrRdLWSVhJktaBOEnUilNnRdhnNFJt1NGu6tgiSI4+kvLOUZR2VmCrSeK+JqL+Jjr9zbT7m2jzR0iqFikhsJAIQLVBSBBSoEpQ0sEADlzdUkDIE2JoeCilmUMZFh5Kvj+feLKLWKKDaKKDWKKDWLKLeLKTeLKLZDJGghSmCqaCE6eDJRSyZCGaomEocUwlgcAGGxTpvFUIKdFQCCgKQUUlKCAoJWFLI5TyENK9BBIhRKIcmajE0CuRtlP9oCoJMsKNaJpOS2QUltbF9uH/YmXG2/gMi4AO5WohY/xltJmdbE3uplM1SHghHAwxqSODypqzsbrOBBQUJY7q304itJOW7P3sz++gXrNpTcXAtPAb4DXAb0CG5aVEzaFQySJfZJIp/Ji2SVKmSNqp9PWrd6fjVpJO4qQ0gaFBKh0MDWzFS64sQVUVkp44uqpjY2LbFtK2sW0L27bQLIlfh2AKArok2/QxRMmhWGaQb4cI2hpdaoqISNBKjGY6aSFK0itIeiDpAUsFSwFN8ZAlQ2TrQbIMH2Ejg6CZScK3m3bfXhJaBrqpYsYTeHSTQAoCzuCiJL2Q8ELCJ9C9AhEKooUy0IJ+pGLTHuskHmvHlwKfSTqWhE2NLBEiaieIixSGBqYqMA5cL4pKLsMRQsHQkuhKgpSWwBApLMXGQmIpkrMyh3PPlYvek/97v45+FnCXlPLC9P93Akgpf9JjnZfT6ywVQmhAA1AA3NFz3Z7rHW1/fVlHb6dSJDdvJrFuvfMA2LABu6MDaZrOA+B4x+73owQCKMEgSjCACAQRioIdi2FFo9jpcLztDAREMIhWkI9WUICWk4sdi2K2tGJGIliRiNNVdIBgeVR0j0QKUGzHaQspUKRE2I4DFbZEDHyzHxMbsNMPEFsBQxHEg4V0hSuIh8uJhypIefPJa3mX8n0vE04kUE7wmG3hOK2OUB6mx0coWo9mSzQbNFugWqDaH3IDDkK65k1ixu+efE+/fb9DIJQC+3v8XwMcOhJY9zpSSlMI0QHkpfOXHfLbw0aiEkLcBNwEUFb23odMPRTF6yU4ZQrBKVMOWyalBMvqdvrSNJEpA6EIlGAQEQic0IQY0rad0n60y3H8sRjSsiFdYiEdpC2dPMty/jct53/TQlqmoyWtB1s6v5MHfnd4GqEcrIQWzoQVTu8DgfBoaPlpp54OSujoDWPSsrA6OjBbWrAiEczWVrAlwutF+LwoXm867XNijxehqQcfcFIenEez23ekK8qldOwgD8mzbGQijhWLIeO9YzsWw4473wgIRQVVdc6FqjrVGYoKquIsU4SzrGdaKAhVOdgbo8eD+NCCjdA8CE1DeDRn+4f8j20jU4ZzjRwhAI4moYCi9EqjiHQ7io20rfQ5tsEyneM3TQwjiWmkMI0klpEiy9ApNAwsowbL3AWWTXhUPplnXIESDnW/NSqhdBwMIi2z22Z2PN6dTnS0IiJ12JZJwH86wUAmAV8off607uMUXi8iEEDxB1CCAZRAAHEg7fcjfL6D59G2nWv4wP1z4FpPpbD1FFJPInUdO6kjU3p3uuc1220jIZxzJgR4PKgZGQeP68CxpoPwepHJ5MFjjMexY+k4EUfG4+l72XTuJ9NM31sW0jTANBE+f9p26e2HQs7203kI4djuKEEahmOXQAAl4O9tI38A4fM6+zUMZCqFNAzsVKo7LVNGbx9gWT2uDSceWnl49WpfMCC+eJFS/g74HTgl+g9in0IIp+pG0+AYM9QcdzuKghoOoYY/vN0Dhaqi5eai5bofoww2Tn5G04GLCDhOljz324uT5URaAWuBnrMYDE3nHXGddNVNFk6j7In81sXFxcWlHzkRR78SGCWEGC6E8AJXA4sPWWcxcOD7+SuA16XzjrwYuFoI4RNCDAdGASv6RrqLi4uLy4lw3KqbdJ37rcDLON0r/yCl3CyEuBtYJaVcDDwKPC6EqAIiOA8D0us9DWwBTOCWY/W4cXFxcXHpewbNB1MuLi4uH2WO1evmxL/UcXFxcXH5UOI6ehcXF5dBjuvoXVxcXAY5rqN3cXFxGeQMuMZYIUQzsPd9bCIfaOkjOf2Fq7FvcDX2Da7GvuFUayyXUhYcacGAc/TvFyHEqqO1PA8UXI19g6uxb3A19g0DWaNbdePi4uIyyHEdvYuLi8sgZzA6+t+dagEngKuxb3A19g2uxr5hwGocdHX0Li4uLi69GYwlehcXFxeXHriO3sXFxWWQM2gcvRDiIiHEdiFElRDijlOsZY8QYqMQYp0QYlU6L1cI8aoQYmc6zknnCyHEr9K6NwghpvaTpj8IIZqEEJt65J20JiHEF9Pr7xRCfPFI++pjjXcJIWrTtlwnhLikx7I70xq3CyEu7JHfb9eCEGKYEOINIcQWIcRmIcRt6fwBY8tjaBxotvQLIVYIIdandf4wnT9cCLE8vc+n0sOjkx7u/Kl0/nIhRMXx9Pejxj8JIXb3sOXkdP4puXeOi0xPA/dhDjjDJ1cDlYAXWA+MPYV69gD5h+TdB9yRTt8B3JtOXwK8CAhgJrC8nzTNA6YCm96rJiAX2JWOc9LpnH7WeBdw+xHWHZs+zz5gePr8q/19LQAlwNR0OgPYkdYyYGx5DI0DzZYCCKfTHmB52kZPA1en8x8GvppO3ww8nE5fDTx1LP39rPFPwBVHWP+U3DvHC4OlRD8DqJJS7pJSpoAngfmnWNOhzAceS6cfAz7VI//P0mEZkC2EKOnrnUsp/w9nroD3o+lC4FUpZURK2Qa8ClzUzxqPxnzgSSmlLqXcDVThXAf9ei1IKeullGvS6S5gK848yAPGlsfQeDROlS2llDKa/teTDhI4B1iYzj/UlgdsvBA4VwghjqG/PzUejVNy7xyPweLojzSB+bEu7P5GAq8IIVYLZ+JzgCIpZX063QAUpdOnUvvJajpVWm9Nvwb/4UCVyEDQmK46mIJTyhuQtjxEIwwwWwohVCHEOqAJx/lVA+1SSvMI++zWk17eAeT1t85DNUopD9jyR2lbPiCE8B2q8RAtp9RHDRZHP9CYK6WcClwM3CKEmNdzoXTe5QZUv9aBqCnNb4ERwGSgHvjZqZXjIIQIA88C35BSdvZcNlBseQSNA86WUkpLSjkZZz7pGcBpp1jSYRyqUQgxHrgTR+sZONUx3zmFEo/LYHH0A2oScillbTpuAp7HuYAbD1TJpOOm9OqnUvvJavrAtUopG9M3mg08wsFX8lOmUQjhwXGgf5VSPpfOHlC2PJLGgWjLA0gp24E3gFk41R0Hpjntuc9uPenlWUDrB6Wzh8aL0tVjUkqpA39kANnySAwWR38iE5h/IAghQkKIjANp4AJgE70nUP8isCidXgxcm26tnwl09KgC6G9OVtPLwAVCiJz0a/8F6bx+45D2istxbHlA45Emnu/XayFdJ/wosFVK+fMeiwaMLY+mcQDaskAIkZ1OB4DzcdoT3gCuSK92qC0P2PgK4PX029PR9PeXxm09HuoCpw2hpy0HxL3Tiw+q1be/A05r9w6cOr7vnkIdlTg9ANYDmw9owalL/BewE3gNyJUHW/UfTOveCEzvJ11/w3ldN3DqB69/L5qA63Aau6qA//cBaHw8rWEDzk1U0mP976Y1bgcu/iCuBWAuTrXMBmBdOlwykGx5DI0DzZYTgbVpPZuA7/e4h1ak7fIM4Evn+9P/V6WXVx5Pfz9qfD1ty03AXzjYM+eU3DvHC+4QCC4uLi6DnMFSdePi4uLichRcR+/i4uIyyHEdvYuLi8sgx3X0Li4uLoMc19G7uLi4DHJcR+/i4uIyyHEdvYuLi8sg5/8DQVJW+80Ub8cAAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "plot_gradients(good_trial)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can also print inputs and outputs from the model. For instance, let's print the 42nd sample of the 2700th batch, as seen by the network. \n",
- "\n",
- "Notice that we have "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAALkklEQVR4nO3dX4hc9RnG8eeJ1RsjmlS6xBir1dxItbEsoVCpFlHSgEYRxCAlpdL1QkEhFw32wkApSKnWXgkrBpNilYCKQaWaBmnam5JVkpg/1aQaMcsmqQQ1ubK6by/mpKxx58xmzjlzJn2/H1hm5vfOzHk5+uT8m5mfI0IA/v/Na7sBAINB2IEkCDuQBGEHkiDsQBLfGOTCbHPqH2hYRHi28UpbdtsrbL9r+6DtdVXeC0Cz3O91dtvnSHpP0s2SDkvaIWl1ROwreQ1bdqBhTWzZl0s6GBHvR8Tnkp6XtKrC+wFoUJWwL5b00YzHh4uxr7A9ZnvC9kSFZQGoqPETdBExLmlcYjceaFOVLfukpCUzHl9ajAEYQlXCvkPSUttX2D5P0t2SttTTFoC69b0bHxFf2H5A0uuSzpG0ISL21tYZgFr1femtr4VxzA40rpEP1QA4exB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQRN9TNuPscMkll5TWX3311dL6tddeW1p/4oknSutr164trWNwKoXd9iFJJyR9KemLiBitoykA9atjy/7jiPi4hvcB0CCO2YEkqoY9JL1h+y3bY7M9wfaY7QnbExWXBaCCqrvx10fEpO1vSdpq+58RsX3mEyJiXNK4JNmOissD0KdKW/aImCxuj0l6SdLyOpoCUL++w277fNsXnLov6RZJe+pqDEC9quzGj0h6yfap9/lTRPy5lq5Qm8suu6y0fs0115TWI8qPvHrVMTz6DntEvC/pezX2AqBBXHoDkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAmmbEYlN9xwQ2n9wgsv7Fr79NNP624HJdiyA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASXGdPrphyu2+jo6Ol9fnz53etcZ19sHpu2W1vsH3M9p4ZYwttb7V9oLhd0GybAKqay278M5JWnDa2TtK2iFgqaVvxGMAQ6xn2iNgu6fhpw6skbSzub5R0e819AahZv8fsIxExVdw/Immk2xNtj0ka63M5AGpS+QRdRITtKKmPSxqXpLLnAWhWv5fejtpeJEnF7bH6WgLQhH7DvkXSmuL+Gkkv19MOgKb03I23/ZykGyVdbPuwpEckPSpps+17JX0o6a4mm0RzIqodWU1PT9fUCZrWM+wRsbpL6aaaewHQID4uCyRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBI9Z3HF2e3IkSOl9YMHD5bWly5dWlqfN4/txdmi538p2xtsH7O9Z8bYetuTtncWfyubbRNAVXP5Z/kZSStmGf99RCwr/l6rty0AdesZ9ojYLun4AHoB0KAqB1wP2N5d7OYv6PYk22O2J2xPVFgWgIr6DfuTkq6UtEzSlKTHuj0xIsYjYjQiRvtcFoAa9BX2iDgaEV9GxLSkpyQtr7ctAHXrK+y2F814eIekPd2eC2A49LzObvs5STdKutj2YUmPSLrR9jJJIemQpPsa7BEVHDp0qLS+a9eu0vpVV11VWp+enj7TltCSnmGPiNWzDD/dQC8AGsTHn4AkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKfkkajLrrooq61ycnJAXYCtuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kIQjYnALswe3MMzJPffcU1rfuHFjad12aX3z5s1da6tXz/bDxagqImb9j8KWHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeS4PvsyfW6Tt6rPm9e+fai1+sxOD237LaX2H7T9j7be20/WIwvtL3V9oHidkHz7QLo11x247+QtDYirpb0A0n3275a0jpJ2yJiqaRtxWMAQ6pn2CNiKiLeLu6fkLRf0mJJqySd+izlRkm3N9UkgOrO6Jjd9uWSrpP0D0kjETFVlI5IGunymjFJY/23CKAOcz4bb3u+pBckPRQRn82sRefbNLN+ySUixiNiNCJGK3UKoJI5hd32ueoE/dmIeLEYPmp7UVFfJOlYMy0CqEPP3Xh3rp08LWl/RDw+o7RF0hpJjxa3LzfSIRp16623ltZ7fQV6enq60usxOHM5Zv+hpJ9Kesf2zmLsYXVCvtn2vZI+lHRXMy0CqEPPsEfE3yV1+2TETfW2A6ApfFwWSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEk+Cnp5DZt2lRav/POOyu9/yeffFLp9agPW3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSMKD/F1v2/yI+JBZsmRJaf2DDz4orW/fvr20ftttt3WtnTx5svS16E9EzPpr0GzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJntfZbS+RtEnSiKSQNB4Rf7C9XtIvJP27eOrDEfFaj/fiOjvQsG7X2ecS9kWSFkXE27YvkPSWpNvVmY/9ZET8bq5NEHaged3CPpf52ackTRX3T9jeL2lxve0BaNoZHbPbvlzSdZL+UQw9YHu37Q22F3R5zZjtCdsTlToFUMmcPxtve76kv0r6TUS8aHtE0sfqHMf/Wp1d/Z/3eA9244GG9X3MLkm2z5X0iqTXI+LxWeqXS3olIr7b430IO9Cwvr8IY9uSnpa0f2bQixN3p9whaU/VJgE0Zy5n46+X9DdJ70iaLoYflrRa0jJ1duMPSbqvOJlX9l5s2YGGVdqNrwthB5rH99mB5Ag7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJ9PzByZp9LOnDGY8vLsaG0bD2Nqx9SfTWrzp7+3a3wkC/z/61hdsTETHaWgMlhrW3Ye1Lord+Dao3duOBJAg7kETbYR9vefllhrW3Ye1Lord+DaS3Vo/ZAQxO21t2AANC2IEkWgm77RW237V90Pa6NnroxvYh2+/Y3tn2/HTFHHrHbO+ZMbbQ9lbbB4rbWefYa6m39bYni3W30/bKlnpbYvtN2/ts77X9YDHe6ror6Wsg623gx+y2z5H0nqSbJR2WtEPS6ojYN9BGurB9SNJoRLT+AQzbP5J0UtKmU1Nr2f6tpOMR8WjxD+WCiPjlkPS2Xmc4jXdDvXWbZvxnanHd1Tn9eT/a2LIvl3QwIt6PiM8lPS9pVQt9DL2I2C7p+GnDqyRtLO5vVOd/loHr0ttQiIipiHi7uH9C0qlpxltddyV9DUQbYV8s6aMZjw9ruOZ7D0lv2H7L9ljbzcxiZMY0W0ckjbTZzCx6TuM9SKdNMz40666f6c+r4gTd110fEd+X9BNJ9xe7q0MpOsdgw3Tt9ElJV6ozB+CUpMfabKaYZvwFSQ9FxGcza22uu1n6Gsh6ayPsk5KWzHh8aTE2FCJisrg9JukldQ47hsnRUzPoFrfHWu7nfyLiaER8GRHTkp5Si+uumGb8BUnPRsSLxXDr6262vga13toI+w5JS21fYfs8SXdL2tJCH19j+/zixIlsny/pFg3fVNRbJK0p7q+R9HKLvXzFsEzj3W2acbW87lqf/jwiBv4naaU6Z+T/JelXbfTQpa/vSNpV/O1tuzdJz6mzW/cfdc5t3Cvpm5K2STog6S+SFg5Rb39UZ2rv3eoEa1FLvV2vzi76bkk7i7+Vba+7kr4Gst74uCyQBCfogCQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJ/wIrTKNeLgYTHwAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "# The raw tensor\n",
- "raw_t = good_trial.tensor('Net_input_0').value(2700)[42]\n",
- "# We have to undo the transformations in 'transformer' above. First of all, multiply by 255\n",
- "raw_t = raw_t * 255\n",
- "# Then reshape from a 784-long vector to a 28x28 square.\n",
- "input_image = raw_t.reshape(28,28)\n",
- "plt.imshow(input_image, cmap=plt.get_cmap('gray'))\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can also plot the relative values emitted by the network. Notice that the last layer is of type `nn.Linear(500, 10)`: it will emit 10 separate confidences, one for each 0-9 digit. The one with the highest output is the predicted value.\n",
- "\n",
- "We can capture and plot the network output for the same sample."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAP8UlEQVR4nO3df4xlZX3H8feHpWKXxmLCJgjLMGu6tAFiUScoNZq20EqpcaPWhGaqsf4xpQW1PxIj3aQ2mk1MNW1tsNXxR5Om0xKDUoiiwKaN6T+os7Kl/DQLsrCUpqOm2HYNuvLtH/euzC4zuzt77p1z95n3K7mZe55z5zzfPWE/PHvOc5+TqkKS1KbT+i5AkjQ+hrwkNcyQl6SGGfKS1DBDXpIadnrfBSx39tln1/T0dN9lSNIpZc+ePd+uqi0r7ZuokJ+enmZxcbHvMiTplJJk/2r7vFwjSQ0z5CWpYYa8JDXMkJekhhnyktSwsYd8kquSPJxkX5L3jbu/viwswPQ0nHba4OfCQt8VSdKYp1Am2QR8DPgV4ADw9SS3VdUD4+x3vS0swNwcHDw42N6/f7ANMDvbX12SNO6R/GXAvqp6tKp+ANwE7Bhzn+tu587nAv6wgwcH7ZLUp3GH/HnAE8u2DwzbfizJXJLFJItLS0tjLmc8Hn98be2StF56v/FaVfNVNVNVM1u2rPit3Ik3NbW2dklaL+MO+SeB85dtbx22NWXXLti8+ci2zZsH7ZLUp3GH/NeB7Um2JXkBcA1w25j7XHezszA/DxdcAMng5/y8N10l9W+ss2uq6lCS64E7gE3AZ6rq/nH22ZfZWUNd0uQZ+yqUVXU7cPu4+5EkPV/vN14lSeNjyEtSwwx5SWqYIS9JDTPkJalhhrwkNcyQl6SGGfKS1DBDXpIaZshLUsMMeUlqmCEvSQ0z5CWpYYa8JDXMkJekhhnyktQwQ16SGmbIS1LDDHlJapghL0kNG1vIJ/nTJE8m2Tt8XT2uviRJKzt9zMf/i6r6yJj7kCStwss1ktSwcYf89UnuTfKZJC9e6QNJ5pIsJllcWloaczmStLGkqk7+l5PdwDkr7NoJ3A18Gyjgg8BLquqdxzrezMxMLS4unnQ9krQRJdlTVTMr7et0Tb6qrjzBAj4JfKFLX5KktRvn7JqXLNt8E3DfuPqSJK1snLNr/izJpQwu1zwG/M4Y+5IkrWBsIV9VbxvXsSVJJ8YplJLUMENekhpmyEtSwwx5SWqYIS9JDTPkJalhhrwkNcyQl6SGGfKS1DBDXpIaZshLUsMMeUlqmCEvSQ0z5CWpYYa8JDXMkJekhhnyktQwQ16SGmbIS1LDDHlJalinkE/y1iT3J3k2ycxR+25Isi/Jw0le361MSdLJOL3j798HvBn4xPLGJBcB1wAXA+cCu5NcWFU/6tifJGkNOo3kq+rBqnp4hV07gJuq6pmq+hawD7isS1+SpLUb1zX584Anlm0fGLY9T5K5JItJFpeWlsZUjiRtTMe9XJNkN3DOCrt2VtWtXQuoqnlgHmBmZqa6Hk+S9JzjhnxVXXkSx30SOH/Z9tZhmyRpHY3rcs1twDVJzkiyDdgOfG1MfUmSVtF1CuWbkhwALge+mOQOgKq6H/gs8ADwZeA6Z9ZI0vrrNIWyqm4Bblll3y5gV5fjS5K68RuvktQwQ16SGmbIS1LDDHlJapghL0kNM+QlqWGGvCQ1zJCXpIYZ8pLUMENekhpmyEtSwwx5SWqYIS9JDTPkJalhhrwkNcyQl6SGGfKS1DBDXpIaZshLUsO6Psj7rUnuT/Jskpll7dNJvp9k7/D18e6lSpLWqtODvIH7gDcDn1hh3yNVdWnH40uSOugU8lX1IECS0VQjSRqpcV6T35bkniRfSfLaMfYjSVrFcUfySXYD56ywa2dV3brKrz0FTFXVd5K8EvinJBdX1fdWOP4cMAcwNTV14pVLko7ruCFfVVeu9aBV9QzwzPD9niSPABcCiyt8dh6YB5iZmam19iVJWt1YLtck2ZJk0/D9S4HtwKPj6EuStLquUyjflOQAcDnwxSR3DHe9Drg3yV7gZuDaqvput1IlSWvVdXbNLcAtK7R/Dvhcl2NLkrrzG6+S1DBDXpIaZshLUsMMeUlqmCEvSQ0z5CWpYYa8JDXMkJekhhnyktQwQ16SGmbIS1LDDHlJapghL0kNM+QlqWGGvCQ1zJCXpIYZ8pLUMENekhpmyEtSwwx5SWpYp5BP8uEkDyW5N8ktSc5atu+GJPuSPJzk9d1LlSStVdeR/F3AJVX1MuCbwA0ASS4CrgEuBq4C/jrJpo59SZLWqFPIV9WdVXVouHk3sHX4fgdwU1U9U1XfAvYBl3XpS5K0dqO8Jv9O4EvD9+cBTyzbd2DY9jxJ5pIsJllcWloaYTmSpNOP94Eku4FzVti1s6puHX5mJ3AIWFhrAVU1D8wDzMzM1Fp/X5K0uuOGfFVdeaz9Sd4BvAG4oqoOh/STwPnLPrZ12CZJWkddZ9dcBbwXeGNVHVy26zbgmiRnJNkGbAe+1qUvSdLaHXckfxw3AmcAdyUBuLuqrq2q+5N8FniAwWWc66rqRx37kiStUaeQr6qfOca+XcCuLseXJHXjN14lqWGGvCQ1zJCXpIYZ8pLUMENekhpmyEtSwwx5SWqYIS9JPVpYgOlpOO20wc+FNa8Admxdv/EqSTpJCwswNwcHh4vC7N8/2AaYnR1NH47kJaknO3c+F/CHHTw4aB8VQ16SevL442trPxmGvCT1ZGpqbe0nw5CXpJ7s2gWbNx/ZtnnzoH1UDHlJ6snsLMzPwwUXQDL4OT8/upuu4OwaSerV7OxoQ/1ojuQlqWGGvCQ1zJCXpIYZ8pLUMENekhrWKeSTfDjJQ0nuTXJLkrOG7dNJvp9k7/D18dGUK0lai64j+buAS6rqZcA3gRuW7Xukqi4dvq7t2I8k6SR0CvmqurOqDg037wa2di9JkjQqo7wm/07gS8u2tyW5J8lXkrx2tV9KMpdkMcni0tLSCMuRJB035JPsTnLfCq8dyz6zEzgEHF7u/ilgqqpeDvwh8A9JXrTS8atqvqpmqmpmy5Yt3f9EknQCxv2wjklx3GUNqurKY+1P8g7gDcAVVVXD33kGeGb4fk+SR4ALgcWuBUtSV+vxsI5J0XV2zVXAe4E3VtXBZe1bkmwavn8psB14tEtfktowCSPo9XhYx6ToukDZjcAZwF1JAO4ezqR5HfCBJD8EngWurarvduxL0iluUkbQ6/GwjkmR4RWWiTAzM1OLi17RkVo1PT0I9qNdcAE89tjGq2NUkuypqpmV9vmNV0nrZlJG0OvxsI5JYchLWjfr8bi7E7EeD+uYFIa8tEFMwg3PSRpBz84OLs08++zgZ4sBD4a8tCEcvuG5fz9UPXfDc72DfiONoCeFN16lDaC1G406kjdepQ1uUm54av0Z8tIGMCk3PLX+DHlpA5ikG55aX4a8tA76ntniDc+Nq+uyBpKOY1K+yj87a6hvRI7kpTHbSIthafIY8tKYObNFfTLkpTFzZov6ZMhLY+bMFvXJkFfT+p7VAs5sUb+cXaNmTcqslsP9GerqgyN5NctZLZIhr4Y5q0Uy5NUwZ7VIhrwa5qwWaQQhn+SDSe5NsjfJnUnOHbYnyV8l2Tfc/4ru5Uonzlkt0ggeGpLkRVX1veH7dwMXVdW1Sa4G3gVcDbwK+GhVvepYx/KhIZK0dmN9aMjhgB86Ezj8f40dwN/VwN3AWUle0rU/HdskzAuXNDlGMk8+yS7g7cDTwC8Nm88Dnlj2sQPDtqeO+t05YA5gyjtinUzSvHBJk+GERvJJdie5b4XXDoCq2llV5wMLwPVrKaCq5qtqpqpmtmzZsvY/gX7MeeGSjnZCI/mquvIEj7cA3A68H3gSOH/Zvq3DNo2J88IlHW0Us2u2L9vcATw0fH8b8PbhLJtXA09X1VPPO4BGxnnhko42innyHxpeurkX+FXgPcP224FHgX3AJ4HfG0FfOgbnhUs6Wucbr1X1llXaC7iu6/F14g7fXN25c3CJZmpqEPDedJU2LlehbIyrHUpazmUNJKlhhrwkNcyQl6SGGfIaC5dXkCaDN141ci6vIE0OR/IaOZdXkCaHIa+Rc3kFaXIY8ho5l1eQJochr5FzeQVpchjyGjkfuydNDmfXaCxcXkGaDI7kJalhhrwkNcyQl6SGGfKS1LAmQt51UiRpZaf87BrXSZGk1Z3yI3nXSZGk1XUK+SQfTHJvkr1J7kxy7rD9F5M8PWzfm+RPRlPu87lOiiStrutI/sNV9bKquhT4ArA8zP+1qi4dvj7QsZ9VuU6KJK2uU8hX1feWbZ4JVLdy1s51UiRpdZ2vySfZleQJYJYjR/KXJ/m3JF9KcnHXflbjOimStLpUHXvwnWQ3cM4Ku3ZW1a3LPncD8MKqen+SFwHPVtX/Jrka+GhVbV/l+HPAHMDU1NQr9+/ff5J/FEnamJLsqaqZFfcdL+TX0MkUcHtVXbLCvseAmar69rGOMTMzU4uLiyOpR5I2imOFfNfZNctH5zuAh4bt5yTJ8P1lw36+06UvSdLadf0y1IeS/CzwLLAfuHbY/hvA7yY5BHwfuKZG9U8GSdIJ6xTyVfWWVdpvBG7scmxJUnen/DdeJUmrG9mN11FIssTgss/JOhs45s3dDcRzcSTPx3M8F0dq4XxcUFVbVtoxUSHfVZLF1e4wbzSeiyN5Pp7juThS6+fDyzWS1DBDXpIa1lrIz/ddwATxXBzJ8/Ecz8WRmj4fTV2TlyQdqbWRvCRpGUNekhrWRMgnuSrJw0n2JXlf3/X0Kcn5Sf4lyQNJ7k/ynr5r6luSTUnuSfKFvmvpW5Kzktyc5KEkDya5vO+a+pTkD4Z/T+5L8o9JXth3TaN2yod8kk3Ax4BfAy4CfjPJRf1W1atDwB9V1UXAq4HrNvj5AHgP8GDfRUyIjwJfrqqfA36eDXxekpwHvJvBCrmXAJuAa/qtavRO+ZAHLgP2VdWjVfUD4CYGK2JuSFX1VFV9Y/j+fxj8JT6v36r6k2Qr8OvAp/qupW9Jfhp4HfBpgKr6QVX9d79V9e504CeTnA5sBv6j53pGroWQPw94Ytn2ATZwqC2XZBp4OfDVfivp1V8C72WwUupGtw1YAv52ePnqU0nO7LuovlTVk8BHgMeBp4Cnq+rOfqsavRZCXitI8lPA54DfP+pZvBtGkjcA/1VVe/quZUKcDrwC+Juqejnwf8CGvYeV5MUM/tW/DTgXODPJb/Vb1ei1EPJPAucv2946bNuwkvwEg4BfqKrP911Pj14DvHH4ZLKbgF9O8vf9ltSrA8CBqjr8L7ubGYT+RnUl8K2qWqqqHwKfB36h55pGroWQ/zqwPcm2JC9gcOPktp5r6s3wiVyfBh6sqj/vu54+VdUNVbW1qqYZ/Hfxz1XV3EjtRFXVfwJPDB/0A3AF8ECPJfXtceDVSTYP/95cQYM3ors+Gap3VXUoyfXAHQzujn+mqu7vuaw+vQZ4G/DvSfYO2/64qm7vsSZNjncBC8MB0aPAb/dcT2+q6qtJbga+wWBW2j00uMSByxpIUsNauFwjSVqFIS9JDTPkJalhhrwkNcyQl6SGGfKS1DBDXpIa9v83PK0Ut0h3xAAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The network predicted the value: 1\n"
- ]
- }
- ],
- "source": [
- "plt.plot(good_trial.tensor('Net_output0').value(2700)[42], 'bo')\n",
- "plt.show()\n",
- "print('The network predicted the value: {}'.format(np.argmax(good_trial.tensor('Net_output0').value(2700)[42])))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Vanishing Gradient"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We have now worked through some of the basics. Let's pretend we are debugging a real problem: the Vanishing Gradient. When training a network, if the `learning_rate` is too high we will end up with a Vanishing Gradient. Let's set `learning_rate=1`.\n",
- "\n",
- "Notice how the accuracy remains at around ~10% - no better than random."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Train Epoch: 0 [0/60000 (0%)]\tLoss: 2.330259\n",
- "Train Epoch: 0 [640/60000 (1%)]\tLoss: 2.459480\n",
- "Train Epoch: 0 [1280/60000 (2%)]\tLoss: 2.342018\n",
- "Train Epoch: 0 [1920/60000 (3%)]\tLoss: 2.370183\n",
- "Train Epoch: 0 [2560/60000 (4%)]\tLoss: 2.306495\n",
- "Train Epoch: 0 [3200/60000 (5%)]\tLoss: 2.287591\n",
- "Train Epoch: 0 [3840/60000 (6%)]\tLoss: 2.284006\n",
- "Train Epoch: 0 [4480/60000 (7%)]\tLoss: 2.360036\n",
- "Train Epoch: 0 [5120/60000 (9%)]\tLoss: 2.298006\n",
- "Train Epoch: 0 [5760/60000 (10%)]\tLoss: 2.341758\n",
- "Train Epoch: 0 [6400/60000 (11%)]\tLoss: 2.304522\n",
- "Train Epoch: 0 [7040/60000 (12%)]\tLoss: 2.286000\n",
- "Train Epoch: 0 [7680/60000 (13%)]\tLoss: 2.293531\n",
- "Train Epoch: 0 [8320/60000 (14%)]\tLoss: 2.358820\n",
- "Train Epoch: 0 [8960/60000 (15%)]\tLoss: 2.307475\n",
- "Train Epoch: 0 [9600/60000 (16%)]\tLoss: 2.310409\n",
- "Train Epoch: 0 [10240/60000 (17%)]\tLoss: 2.324923\n",
- "Train Epoch: 0 [10880/60000 (18%)]\tLoss: 2.341324\n",
- "Train Epoch: 0 [11520/60000 (19%)]\tLoss: 2.346567\n",
- "Train Epoch: 0 [12160/60000 (20%)]\tLoss: 2.398468\n",
- "Train Epoch: 0 [12800/60000 (21%)]\tLoss: 2.327744\n",
- "Train Epoch: 0 [13440/60000 (22%)]\tLoss: 2.355408\n",
- "Train Epoch: 0 [14080/60000 (23%)]\tLoss: 2.342527\n",
- "Train Epoch: 0 [14720/60000 (25%)]\tLoss: 2.381174\n",
- "Train Epoch: 0 [15360/60000 (26%)]\tLoss: 2.327620\n",
- "Train Epoch: 0 [16000/60000 (27%)]\tLoss: 2.312094\n",
- "Train Epoch: 0 [16640/60000 (28%)]\tLoss: 2.349320\n",
- "Train Epoch: 0 [17280/60000 (29%)]\tLoss: 2.325104\n",
- "Train Epoch: 0 [17920/60000 (30%)]\tLoss: 2.410883\n",
- "Train Epoch: 0 [18560/60000 (31%)]\tLoss: 2.312054\n",
- "Train Epoch: 0 [19200/60000 (32%)]\tLoss: 2.328432\n",
- "Train Epoch: 0 [19840/60000 (33%)]\tLoss: 2.326895\n",
- "Train Epoch: 0 [20480/60000 (34%)]\tLoss: 2.294943\n",
- "Train Epoch: 0 [21120/60000 (35%)]\tLoss: 2.310558\n",
- "Train Epoch: 0 [21760/60000 (36%)]\tLoss: 2.353846\n",
- "Train Epoch: 0 [22400/60000 (37%)]\tLoss: 2.319908\n",
- "Train Epoch: 0 [23040/60000 (38%)]\tLoss: 2.286135\n",
- "Train Epoch: 0 [23680/60000 (39%)]\tLoss: 2.337246\n",
- "Train Epoch: 0 [24320/60000 (41%)]\tLoss: 2.360455\n",
- "Train Epoch: 0 [24960/60000 (42%)]\tLoss: 2.375340\n",
- "Train Epoch: 0 [25600/60000 (43%)]\tLoss: 2.329563\n",
- "Train Epoch: 0 [26240/60000 (44%)]\tLoss: 2.395788\n",
- "Train Epoch: 0 [26880/60000 (45%)]\tLoss: 2.309679\n",
- "Train Epoch: 0 [27520/60000 (46%)]\tLoss: 2.316720\n",
- "Train Epoch: 0 [28160/60000 (47%)]\tLoss: 2.346542\n",
- "Train Epoch: 0 [28800/60000 (48%)]\tLoss: 2.307389\n",
- "Train Epoch: 0 [29440/60000 (49%)]\tLoss: 2.317679\n",
- "Train Epoch: 0 [30080/60000 (50%)]\tLoss: 2.395548\n",
- "Train Epoch: 0 [30720/60000 (51%)]\tLoss: 2.491510\n",
- "Train Epoch: 0 [31360/60000 (52%)]\tLoss: 2.282305\n",
- "Train Epoch: 0 [32000/60000 (53%)]\tLoss: 2.254564\n",
- "Train Epoch: 0 [32640/60000 (54%)]\tLoss: 2.333660\n",
- "Train Epoch: 0 [33280/60000 (55%)]\tLoss: 2.362551\n",
- "Train Epoch: 0 [33920/60000 (57%)]\tLoss: 2.384102\n",
- "Train Epoch: 0 [34560/60000 (58%)]\tLoss: 2.326107\n",
- "Train Epoch: 0 [35200/60000 (59%)]\tLoss: 2.320798\n",
- "Train Epoch: 0 [35840/60000 (60%)]\tLoss: 2.464493\n",
- "Train Epoch: 0 [36480/60000 (61%)]\tLoss: 2.331449\n",
- "Train Epoch: 0 [37120/60000 (62%)]\tLoss: 2.415666\n",
- "Train Epoch: 0 [37760/60000 (63%)]\tLoss: 2.473129\n",
- "Train Epoch: 0 [38400/60000 (64%)]\tLoss: 2.397358\n",
- "Train Epoch: 0 [39040/60000 (65%)]\tLoss: 2.407380\n",
- "Train Epoch: 0 [39680/60000 (66%)]\tLoss: 2.462022\n",
- "Train Epoch: 0 [40320/60000 (67%)]\tLoss: 2.399495\n",
- "Train Epoch: 0 [40960/60000 (68%)]\tLoss: 2.306419\n",
- "Train Epoch: 0 [41600/60000 (69%)]\tLoss: 2.341526\n",
- "Train Epoch: 0 [42240/60000 (70%)]\tLoss: 2.294206\n",
- "Train Epoch: 0 [42880/60000 (71%)]\tLoss: 2.290251\n",
- "Train Epoch: 0 [43520/60000 (72%)]\tLoss: 2.317634\n",
- "Train Epoch: 0 [44160/60000 (74%)]\tLoss: 2.358360\n",
- "Train Epoch: 0 [44800/60000 (75%)]\tLoss: 2.332942\n",
- "Train Epoch: 0 [45440/60000 (76%)]\tLoss: 2.312563\n",
- "Train Epoch: 0 [46080/60000 (77%)]\tLoss: 2.328177\n",
- "Train Epoch: 0 [46720/60000 (78%)]\tLoss: 2.319674\n",
- "Train Epoch: 0 [47360/60000 (79%)]\tLoss: 2.349114\n",
- "Train Epoch: 0 [48000/60000 (80%)]\tLoss: 2.294027\n",
- "Train Epoch: 0 [48640/60000 (81%)]\tLoss: 2.344978\n",
- "Train Epoch: 0 [49280/60000 (82%)]\tLoss: 2.322978\n",
- "Train Epoch: 0 [49920/60000 (83%)]\tLoss: 2.308064\n",
- "Train Epoch: 0 [50560/60000 (84%)]\tLoss: 2.344061\n",
- "Train Epoch: 0 [51200/60000 (85%)]\tLoss: 2.403563\n",
- "Train Epoch: 0 [51840/60000 (86%)]\tLoss: 2.312183\n",
- "Train Epoch: 0 [52480/60000 (87%)]\tLoss: 2.287836\n",
- "Train Epoch: 0 [53120/60000 (88%)]\tLoss: 2.333920\n",
- "Train Epoch: 0 [53760/60000 (90%)]\tLoss: 2.312499\n",
- "Train Epoch: 0 [54400/60000 (91%)]\tLoss: 2.359949\n",
- "Train Epoch: 0 [55040/60000 (92%)]\tLoss: 2.363973\n",
- "Train Epoch: 0 [55680/60000 (93%)]\tLoss: 2.304453\n",
- "Train Epoch: 0 [56320/60000 (94%)]\tLoss: 2.454728\n",
- "Train Epoch: 0 [56960/60000 (95%)]\tLoss: 2.338699\n",
- "Train Epoch: 0 [57600/60000 (96%)]\tLoss: 2.289698\n",
- "Train Epoch: 0 [58240/60000 (97%)]\tLoss: 2.325969\n",
- "Train Epoch: 0 [58880/60000 (98%)]\tLoss: 2.334784\n",
- "Train Epoch: 0 [59520/60000 (99%)]\tLoss: 2.363783\n",
- "Train Epoch: 1 [0/60000 (0%)]\tLoss: 2.304468\n",
- "Train Epoch: 1 [640/60000 (1%)]\tLoss: 2.237942\n",
- "Train Epoch: 1 [1280/60000 (2%)]\tLoss: 2.379854\n",
- "Train Epoch: 1 [1920/60000 (3%)]\tLoss: 2.321807\n",
- "Train Epoch: 1 [2560/60000 (4%)]\tLoss: 2.326054\n",
- "Train Epoch: 1 [3200/60000 (5%)]\tLoss: 2.366278\n",
- "Train Epoch: 1 [3840/60000 (6%)]\tLoss: 2.262960\n",
- "Train Epoch: 1 [4480/60000 (7%)]\tLoss: 2.338786\n",
- "Train Epoch: 1 [5120/60000 (9%)]\tLoss: 2.378443\n",
- "Train Epoch: 1 [5760/60000 (10%)]\tLoss: 2.333674\n",
- "Train Epoch: 1 [6400/60000 (11%)]\tLoss: 2.328306\n",
- "Train Epoch: 1 [7040/60000 (12%)]\tLoss: 2.338466\n",
- "Train Epoch: 1 [7680/60000 (13%)]\tLoss: 2.356887\n",
- "Train Epoch: 1 [8320/60000 (14%)]\tLoss: 2.377309\n",
- "Train Epoch: 1 [8960/60000 (15%)]\tLoss: 2.312181\n",
- "Train Epoch: 1 [9600/60000 (16%)]\tLoss: 2.397353\n",
- "Train Epoch: 1 [10240/60000 (17%)]\tLoss: 2.364430\n",
- "Train Epoch: 1 [10880/60000 (18%)]\tLoss: 2.379686\n",
- "Train Epoch: 1 [11520/60000 (19%)]\tLoss: 2.351562\n",
- "Train Epoch: 1 [12160/60000 (20%)]\tLoss: 2.350115\n",
- "Train Epoch: 1 [12800/60000 (21%)]\tLoss: 2.244029\n",
- "Train Epoch: 1 [13440/60000 (22%)]\tLoss: 2.360412\n",
- "Train Epoch: 1 [14080/60000 (23%)]\tLoss: 2.315639\n",
- "Train Epoch: 1 [14720/60000 (25%)]\tLoss: 2.389025\n",
- "Train Epoch: 1 [15360/60000 (26%)]\tLoss: 2.397625\n",
- "Train Epoch: 1 [16000/60000 (27%)]\tLoss: 2.324974\n",
- "Train Epoch: 1 [16640/60000 (28%)]\tLoss: 2.326982\n",
- "Train Epoch: 1 [17280/60000 (29%)]\tLoss: 2.397022\n",
- "Train Epoch: 1 [17920/60000 (30%)]\tLoss: 2.341864\n",
- "Train Epoch: 1 [18560/60000 (31%)]\tLoss: 2.316780\n",
- "Train Epoch: 1 [19200/60000 (32%)]\tLoss: 2.290725\n",
- "Train Epoch: 1 [19840/60000 (33%)]\tLoss: 2.302054\n",
- "Train Epoch: 1 [20480/60000 (34%)]\tLoss: 2.341123\n",
- "Train Epoch: 1 [21120/60000 (35%)]\tLoss: 2.367768\n",
- "Train Epoch: 1 [21760/60000 (36%)]\tLoss: 2.341992\n",
- "Train Epoch: 1 [22400/60000 (37%)]\tLoss: 2.338322\n",
- "Train Epoch: 1 [23040/60000 (38%)]\tLoss: 2.355606\n",
- "Train Epoch: 1 [23680/60000 (39%)]\tLoss: 2.284112\n",
- "Train Epoch: 1 [24320/60000 (41%)]\tLoss: 2.374856\n",
- "Train Epoch: 1 [24960/60000 (42%)]\tLoss: 2.331543\n",
- "Train Epoch: 1 [25600/60000 (43%)]\tLoss: 2.321192\n",
- "Train Epoch: 1 [26240/60000 (44%)]\tLoss: 2.265647\n",
- "Train Epoch: 1 [26880/60000 (45%)]\tLoss: 2.298278\n",
- "Train Epoch: 1 [27520/60000 (46%)]\tLoss: 2.317490\n",
- "Train Epoch: 1 [28160/60000 (47%)]\tLoss: 2.272723\n",
- "Train Epoch: 1 [28800/60000 (48%)]\tLoss: 2.400963\n",
- "Train Epoch: 1 [29440/60000 (49%)]\tLoss: 2.507440\n",
- "Train Epoch: 1 [30080/60000 (50%)]\tLoss: 2.393094\n",
- "Train Epoch: 1 [30720/60000 (51%)]\tLoss: 2.419714\n",
- "Train Epoch: 1 [31360/60000 (52%)]\tLoss: 2.351777\n",
- "Train Epoch: 1 [32000/60000 (53%)]\tLoss: 2.419491\n",
- "Train Epoch: 1 [32640/60000 (54%)]\tLoss: 2.429312\n",
- "Train Epoch: 1 [33280/60000 (55%)]\tLoss: 2.297789\n",
- "Train Epoch: 1 [33920/60000 (57%)]\tLoss: 2.351755\n",
- "Train Epoch: 1 [34560/60000 (58%)]\tLoss: 2.342575\n",
- "Train Epoch: 1 [35200/60000 (59%)]\tLoss: 2.359362\n",
- "Train Epoch: 1 [35840/60000 (60%)]\tLoss: 2.323574\n",
- "Train Epoch: 1 [36480/60000 (61%)]\tLoss: 2.405147\n",
- "Train Epoch: 1 [37120/60000 (62%)]\tLoss: 2.372452\n",
- "Train Epoch: 1 [37760/60000 (63%)]\tLoss: 2.360568\n",
- "Train Epoch: 1 [38400/60000 (64%)]\tLoss: 2.419126\n",
- "Train Epoch: 1 [39040/60000 (65%)]\tLoss: 2.283723\n",
- "Train Epoch: 1 [39680/60000 (66%)]\tLoss: 2.336538\n",
- "Train Epoch: 1 [40320/60000 (67%)]\tLoss: 2.346513\n",
- "Train Epoch: 1 [40960/60000 (68%)]\tLoss: 2.304324\n",
- "Train Epoch: 1 [41600/60000 (69%)]\tLoss: 2.341439\n",
- "Train Epoch: 1 [42240/60000 (70%)]\tLoss: 2.361294\n",
- "Train Epoch: 1 [42880/60000 (71%)]\tLoss: 2.406241\n",
- "Train Epoch: 1 [43520/60000 (72%)]\tLoss: 2.300334\n",
- "Train Epoch: 1 [44160/60000 (74%)]\tLoss: 2.309165\n",
- "Train Epoch: 1 [44800/60000 (75%)]\tLoss: 2.361495\n",
- "Train Epoch: 1 [45440/60000 (76%)]\tLoss: 2.443631\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Train Epoch: 1 [46080/60000 (77%)]\tLoss: 2.314088\n",
- "Train Epoch: 1 [46720/60000 (78%)]\tLoss: 2.392107\n",
- "Train Epoch: 1 [47360/60000 (79%)]\tLoss: 2.323653\n",
- "Train Epoch: 1 [48000/60000 (80%)]\tLoss: 2.294007\n",
- "Train Epoch: 1 [48640/60000 (81%)]\tLoss: 2.314728\n",
- "Train Epoch: 1 [49280/60000 (82%)]\tLoss: 2.355071\n",
- "Train Epoch: 1 [49920/60000 (83%)]\tLoss: 2.349204\n",
- "Train Epoch: 1 [50560/60000 (84%)]\tLoss: 2.307742\n",
- "Train Epoch: 1 [51200/60000 (85%)]\tLoss: 2.347462\n",
- "Train Epoch: 1 [51840/60000 (86%)]\tLoss: 2.405621\n",
- "Train Epoch: 1 [52480/60000 (87%)]\tLoss: 2.423558\n",
- "Train Epoch: 1 [53120/60000 (88%)]\tLoss: 2.382326\n",
- "Train Epoch: 1 [53760/60000 (90%)]\tLoss: 2.320254\n",
- "Train Epoch: 1 [54400/60000 (91%)]\tLoss: 2.369054\n",
- "Train Epoch: 1 [55040/60000 (92%)]\tLoss: 2.321925\n",
- "Train Epoch: 1 [55680/60000 (93%)]\tLoss: 2.321786\n",
- "Train Epoch: 1 [56320/60000 (94%)]\tLoss: 2.339195\n",
- "Train Epoch: 1 [56960/60000 (95%)]\tLoss: 2.323470\n",
- "Train Epoch: 1 [57600/60000 (96%)]\tLoss: 2.421056\n",
- "Train Epoch: 1 [58240/60000 (97%)]\tLoss: 2.329478\n",
- "Train Epoch: 1 [58880/60000 (98%)]\tLoss: 2.315077\n",
- "Train Epoch: 1 [59520/60000 (99%)]\tLoss: 2.340677\n",
- "Train Epoch: 2 [0/60000 (0%)]\tLoss: 2.308716\n",
- "Train Epoch: 2 [640/60000 (1%)]\tLoss: 2.318739\n",
- "Train Epoch: 2 [1280/60000 (2%)]\tLoss: 2.294240\n",
- "Train Epoch: 2 [1920/60000 (3%)]\tLoss: 2.417459\n",
- "Train Epoch: 2 [2560/60000 (4%)]\tLoss: 2.342914\n",
- "Train Epoch: 2 [3200/60000 (5%)]\tLoss: 2.373566\n",
- "Train Epoch: 2 [3840/60000 (6%)]\tLoss: 2.307191\n",
- "Train Epoch: 2 [4480/60000 (7%)]\tLoss: 2.340860\n",
- "Train Epoch: 2 [5120/60000 (9%)]\tLoss: 2.305294\n",
- "Train Epoch: 2 [5760/60000 (10%)]\tLoss: 2.383138\n",
- "Train Epoch: 2 [6400/60000 (11%)]\tLoss: 2.337879\n",
- "Train Epoch: 2 [7040/60000 (12%)]\tLoss: 2.336192\n",
- "Train Epoch: 2 [7680/60000 (13%)]\tLoss: 2.339699\n",
- "Train Epoch: 2 [8320/60000 (14%)]\tLoss: 2.323756\n",
- "Train Epoch: 2 [8960/60000 (15%)]\tLoss: 2.305490\n",
- "Train Epoch: 2 [9600/60000 (16%)]\tLoss: 2.325570\n",
- "Train Epoch: 2 [10240/60000 (17%)]\tLoss: 2.288280\n",
- "Train Epoch: 2 [10880/60000 (18%)]\tLoss: 2.306230\n",
- "Train Epoch: 2 [11520/60000 (19%)]\tLoss: 2.342124\n",
- "Train Epoch: 2 [12160/60000 (20%)]\tLoss: 2.346761\n",
- "Train Epoch: 2 [12800/60000 (21%)]\tLoss: 2.428949\n",
- "Train Epoch: 2 [13440/60000 (22%)]\tLoss: 2.404235\n",
- "Train Epoch: 2 [14080/60000 (23%)]\tLoss: 2.278017\n",
- "Train Epoch: 2 [14720/60000 (25%)]\tLoss: 2.326802\n",
- "Train Epoch: 2 [15360/60000 (26%)]\tLoss: 2.358422\n",
- "Train Epoch: 2 [16000/60000 (27%)]\tLoss: 2.343786\n",
- "Train Epoch: 2 [16640/60000 (28%)]\tLoss: 2.293986\n",
- "Train Epoch: 2 [17280/60000 (29%)]\tLoss: 2.336337\n",
- "Train Epoch: 2 [17920/60000 (30%)]\tLoss: 2.321667\n",
- "Train Epoch: 2 [18560/60000 (31%)]\tLoss: 2.371925\n",
- "Train Epoch: 2 [19200/60000 (32%)]\tLoss: 2.374017\n",
- "Train Epoch: 2 [19840/60000 (33%)]\tLoss: 2.320786\n",
- "Train Epoch: 2 [20480/60000 (34%)]\tLoss: 2.342391\n",
- "Train Epoch: 2 [21120/60000 (35%)]\tLoss: 2.308513\n",
- "Train Epoch: 2 [21760/60000 (36%)]\tLoss: 2.271694\n",
- "Train Epoch: 2 [22400/60000 (37%)]\tLoss: 2.332782\n",
- "Train Epoch: 2 [23040/60000 (38%)]\tLoss: 2.360431\n",
- "Train Epoch: 2 [23680/60000 (39%)]\tLoss: 2.289818\n",
- "Train Epoch: 2 [24320/60000 (41%)]\tLoss: 2.305624\n",
- "Train Epoch: 2 [24960/60000 (42%)]\tLoss: 2.311587\n",
- "Train Epoch: 2 [25600/60000 (43%)]\tLoss: 2.331149\n",
- "Train Epoch: 2 [26240/60000 (44%)]\tLoss: 2.313762\n",
- "Train Epoch: 2 [26880/60000 (45%)]\tLoss: 2.349113\n",
- "Train Epoch: 2 [27520/60000 (46%)]\tLoss: 2.355408\n",
- "Train Epoch: 2 [28160/60000 (47%)]\tLoss: 2.304258\n",
- "Train Epoch: 2 [28800/60000 (48%)]\tLoss: 2.377938\n",
- "Train Epoch: 2 [29440/60000 (49%)]\tLoss: 2.321165\n",
- "Train Epoch: 2 [30080/60000 (50%)]\tLoss: 2.364525\n",
- "Train Epoch: 2 [30720/60000 (51%)]\tLoss: 2.406883\n",
- "Train Epoch: 2 [31360/60000 (52%)]\tLoss: 2.400862\n",
- "Train Epoch: 2 [32000/60000 (53%)]\tLoss: 2.334538\n",
- "Train Epoch: 2 [32640/60000 (54%)]\tLoss: 2.282245\n",
- "Train Epoch: 2 [33280/60000 (55%)]\tLoss: 2.300971\n",
- "Train Epoch: 2 [33920/60000 (57%)]\tLoss: 2.308848\n",
- "Train Epoch: 2 [34560/60000 (58%)]\tLoss: 2.333123\n",
- "Train Epoch: 2 [35200/60000 (59%)]\tLoss: 2.333816\n",
- "Train Epoch: 2 [35840/60000 (60%)]\tLoss: 2.313128\n",
- "Train Epoch: 2 [36480/60000 (61%)]\tLoss: 2.320728\n",
- "Train Epoch: 2 [37120/60000 (62%)]\tLoss: 2.311455\n",
- "Train Epoch: 2 [37760/60000 (63%)]\tLoss: 2.312425\n",
- "Train Epoch: 2 [38400/60000 (64%)]\tLoss: 2.301049\n",
- "Train Epoch: 2 [39040/60000 (65%)]\tLoss: 2.287769\n",
- "Train Epoch: 2 [39680/60000 (66%)]\tLoss: 2.368213\n",
- "Train Epoch: 2 [40320/60000 (67%)]\tLoss: 2.329561\n",
- "Train Epoch: 2 [40960/60000 (68%)]\tLoss: 2.296645\n",
- "Train Epoch: 2 [41600/60000 (69%)]\tLoss: 2.339840\n",
- "Train Epoch: 2 [42240/60000 (70%)]\tLoss: 2.400887\n",
- "Train Epoch: 2 [42880/60000 (71%)]\tLoss: 2.366787\n",
- "Train Epoch: 2 [43520/60000 (72%)]\tLoss: 2.371027\n",
- "Train Epoch: 2 [44160/60000 (74%)]\tLoss: 2.338437\n",
- "Train Epoch: 2 [44800/60000 (75%)]\tLoss: 2.389745\n",
- "Train Epoch: 2 [45440/60000 (76%)]\tLoss: 2.362866\n",
- "Train Epoch: 2 [46080/60000 (77%)]\tLoss: 2.440138\n",
- "Train Epoch: 2 [46720/60000 (78%)]\tLoss: 2.340149\n",
- "Train Epoch: 2 [47360/60000 (79%)]\tLoss: 2.426742\n",
- "Train Epoch: 2 [48000/60000 (80%)]\tLoss: 2.357159\n",
- "Train Epoch: 2 [48640/60000 (81%)]\tLoss: 2.400013\n",
- "Train Epoch: 2 [49280/60000 (82%)]\tLoss: 2.337224\n",
- "Train Epoch: 2 [49920/60000 (83%)]\tLoss: 2.369920\n",
- "Train Epoch: 2 [50560/60000 (84%)]\tLoss: 2.327389\n",
- "Train Epoch: 2 [51200/60000 (85%)]\tLoss: 2.318965\n",
- "Train Epoch: 2 [51840/60000 (86%)]\tLoss: 2.357245\n",
- "Train Epoch: 2 [52480/60000 (87%)]\tLoss: 2.421128\n",
- "Train Epoch: 2 [53120/60000 (88%)]\tLoss: 2.365572\n",
- "Train Epoch: 2 [53760/60000 (90%)]\tLoss: 2.359708\n",
- "Train Epoch: 2 [54400/60000 (91%)]\tLoss: 2.317222\n",
- "Train Epoch: 2 [55040/60000 (92%)]\tLoss: 2.371051\n",
- "Train Epoch: 2 [55680/60000 (93%)]\tLoss: 2.360173\n",
- "Train Epoch: 2 [56320/60000 (94%)]\tLoss: 2.345640\n",
- "Train Epoch: 2 [56960/60000 (95%)]\tLoss: 2.355781\n",
- "Train Epoch: 2 [57600/60000 (96%)]\tLoss: 2.335961\n",
- "Train Epoch: 2 [58240/60000 (97%)]\tLoss: 2.336265\n",
- "Train Epoch: 2 [58880/60000 (98%)]\tLoss: 2.383019\n",
- "Train Epoch: 2 [59520/60000 (99%)]\tLoss: 2.294914\n",
- "Train Epoch: 3 [0/60000 (0%)]\tLoss: 2.302218\n",
- "Train Epoch: 3 [640/60000 (1%)]\tLoss: 2.321162\n",
- "Train Epoch: 3 [1280/60000 (2%)]\tLoss: 2.301874\n",
- "Train Epoch: 3 [1920/60000 (3%)]\tLoss: 2.406926\n",
- "Train Epoch: 3 [2560/60000 (4%)]\tLoss: 2.365343\n",
- "Train Epoch: 3 [3200/60000 (5%)]\tLoss: 2.323746\n",
- "Train Epoch: 3 [3840/60000 (6%)]\tLoss: 2.344622\n",
- "Train Epoch: 3 [4480/60000 (7%)]\tLoss: 2.351114\n",
- "Train Epoch: 3 [5120/60000 (9%)]\tLoss: 2.407657\n",
- "Train Epoch: 3 [5760/60000 (10%)]\tLoss: 2.418502\n",
- "Train Epoch: 3 [6400/60000 (11%)]\tLoss: 2.337087\n",
- "Train Epoch: 3 [7040/60000 (12%)]\tLoss: 2.303796\n",
- "Train Epoch: 3 [7680/60000 (13%)]\tLoss: 2.401513\n",
- "Train Epoch: 3 [8320/60000 (14%)]\tLoss: 2.337463\n",
- "Train Epoch: 3 [8960/60000 (15%)]\tLoss: 2.324577\n",
- "Train Epoch: 3 [9600/60000 (16%)]\tLoss: 2.335718\n",
- "Train Epoch: 3 [10240/60000 (17%)]\tLoss: 2.384667\n",
- "Train Epoch: 3 [10880/60000 (18%)]\tLoss: 2.267396\n",
- "Train Epoch: 3 [11520/60000 (19%)]\tLoss: 2.306527\n",
- "Train Epoch: 3 [12160/60000 (20%)]\tLoss: 2.367751\n",
- "Train Epoch: 3 [12800/60000 (21%)]\tLoss: 2.309073\n",
- "Train Epoch: 3 [13440/60000 (22%)]\tLoss: 2.315047\n",
- "Train Epoch: 3 [14080/60000 (23%)]\tLoss: 2.347873\n",
- "Train Epoch: 3 [14720/60000 (25%)]\tLoss: 2.268999\n",
- "Train Epoch: 3 [15360/60000 (26%)]\tLoss: 2.333838\n",
- "Train Epoch: 3 [16000/60000 (27%)]\tLoss: 2.349008\n",
- "Train Epoch: 3 [16640/60000 (28%)]\tLoss: 2.375700\n",
- "Train Epoch: 3 [17280/60000 (29%)]\tLoss: 2.331388\n",
- "Train Epoch: 3 [17920/60000 (30%)]\tLoss: 2.335067\n",
- "Train Epoch: 3 [18560/60000 (31%)]\tLoss: 2.332542\n",
- "Train Epoch: 3 [19200/60000 (32%)]\tLoss: 2.345043\n",
- "Train Epoch: 3 [19840/60000 (33%)]\tLoss: 2.300745\n",
- "Train Epoch: 3 [20480/60000 (34%)]\tLoss: 2.416367\n",
- "Train Epoch: 3 [21120/60000 (35%)]\tLoss: 2.282617\n",
- "Train Epoch: 3 [21760/60000 (36%)]\tLoss: 2.317955\n",
- "Train Epoch: 3 [22400/60000 (37%)]\tLoss: 2.329546\n",
- "Train Epoch: 3 [23040/60000 (38%)]\tLoss: 2.333439\n",
- "Train Epoch: 3 [23680/60000 (39%)]\tLoss: 2.432110\n",
- "Train Epoch: 3 [24320/60000 (41%)]\tLoss: 2.389215\n",
- "Train Epoch: 3 [24960/60000 (42%)]\tLoss: 2.317299\n",
- "Train Epoch: 3 [25600/60000 (43%)]\tLoss: 2.398170\n",
- "Train Epoch: 3 [26240/60000 (44%)]\tLoss: 2.354642\n",
- "Train Epoch: 3 [26880/60000 (45%)]\tLoss: 2.310941\n",
- "Train Epoch: 3 [27520/60000 (46%)]\tLoss: 2.352980\n",
- "Train Epoch: 3 [28160/60000 (47%)]\tLoss: 2.370045\n",
- "Train Epoch: 3 [28800/60000 (48%)]\tLoss: 2.332853\n",
- "Train Epoch: 3 [29440/60000 (49%)]\tLoss: 2.328536\n",
- "Train Epoch: 3 [30080/60000 (50%)]\tLoss: 2.410731\n",
- "Train Epoch: 3 [30720/60000 (51%)]\tLoss: 2.315743\n",
- "Train Epoch: 3 [31360/60000 (52%)]\tLoss: 2.362804\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Train Epoch: 3 [32000/60000 (53%)]\tLoss: 2.268909\n",
- "Train Epoch: 3 [32640/60000 (54%)]\tLoss: 2.324456\n",
- "Train Epoch: 3 [33280/60000 (55%)]\tLoss: 2.313516\n",
- "Train Epoch: 3 [33920/60000 (57%)]\tLoss: 2.345426\n",
- "Train Epoch: 3 [34560/60000 (58%)]\tLoss: 2.328141\n",
- "Train Epoch: 3 [35200/60000 (59%)]\tLoss: 2.392569\n",
- "Train Epoch: 3 [35840/60000 (60%)]\tLoss: 2.333704\n",
- "Train Epoch: 3 [36480/60000 (61%)]\tLoss: 2.352234\n",
- "Train Epoch: 3 [37120/60000 (62%)]\tLoss: 2.323742\n",
- "Train Epoch: 3 [37760/60000 (63%)]\tLoss: 2.318627\n",
- "Train Epoch: 3 [38400/60000 (64%)]\tLoss: 2.340932\n",
- "Train Epoch: 3 [39040/60000 (65%)]\tLoss: 2.401247\n",
- "Train Epoch: 3 [39680/60000 (66%)]\tLoss: 2.390721\n",
- "Train Epoch: 3 [40320/60000 (67%)]\tLoss: 2.372447\n",
- "Train Epoch: 3 [40960/60000 (68%)]\tLoss: 2.381556\n",
- "Train Epoch: 3 [41600/60000 (69%)]\tLoss: 2.370942\n",
- "Train Epoch: 3 [42240/60000 (70%)]\tLoss: 2.382010\n",
- "Train Epoch: 3 [42880/60000 (71%)]\tLoss: 2.337266\n",
- "Train Epoch: 3 [43520/60000 (72%)]\tLoss: 2.327033\n",
- "Train Epoch: 3 [44160/60000 (74%)]\tLoss: 2.379825\n",
- "Train Epoch: 3 [44800/60000 (75%)]\tLoss: 2.323223\n",
- "Train Epoch: 3 [45440/60000 (76%)]\tLoss: 2.283228\n",
- "Train Epoch: 3 [46080/60000 (77%)]\tLoss: 2.334064\n",
- "Train Epoch: 3 [46720/60000 (78%)]\tLoss: 2.381998\n",
- "Train Epoch: 3 [47360/60000 (79%)]\tLoss: 2.324826\n",
- "Train Epoch: 3 [48000/60000 (80%)]\tLoss: 2.344363\n",
- "Train Epoch: 3 [48640/60000 (81%)]\tLoss: 2.407687\n",
- "Train Epoch: 3 [49280/60000 (82%)]\tLoss: 2.405679\n",
- "Train Epoch: 3 [49920/60000 (83%)]\tLoss: 2.347231\n",
- "Train Epoch: 3 [50560/60000 (84%)]\tLoss: 2.381284\n",
- "Train Epoch: 3 [51200/60000 (85%)]\tLoss: 2.320855\n",
- "Train Epoch: 3 [51840/60000 (86%)]\tLoss: 2.332896\n",
- "Train Epoch: 3 [52480/60000 (87%)]\tLoss: 2.331153\n",
- "Train Epoch: 3 [53120/60000 (88%)]\tLoss: 2.318925\n",
- "Train Epoch: 3 [53760/60000 (90%)]\tLoss: 2.324926\n",
- "Train Epoch: 3 [54400/60000 (91%)]\tLoss: 2.312320\n",
- "Train Epoch: 3 [55040/60000 (92%)]\tLoss: 2.316199\n",
- "Train Epoch: 3 [55680/60000 (93%)]\tLoss: 2.314889\n",
- "Train Epoch: 3 [56320/60000 (94%)]\tLoss: 2.341814\n",
- "Train Epoch: 3 [56960/60000 (95%)]\tLoss: 2.343977\n",
- "Train Epoch: 3 [57600/60000 (96%)]\tLoss: 2.324932\n",
- "Train Epoch: 3 [58240/60000 (97%)]\tLoss: 2.346128\n",
- "Train Epoch: 3 [58880/60000 (98%)]\tLoss: 2.328274\n",
- "Train Epoch: 3 [59520/60000 (99%)]\tLoss: 2.302693\n"
- ]
- }
- ],
- "source": [
- "model = create_net(tornasole_save_interval=100, base_loc='./ts_output', run_id='bad')\n",
- "train(model=model, epochs=4, learning_rate=1.0, momentum=0.9, batch_size=64, device=torch.device(\"cpu\"))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "metadata": {},
- "outputs": [],
- "source": [
- "bad_trial = LocalTrial( 'myrun', './ts_output/bad/')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can plot the gradients - notice how every single one of them (apart from one) goes to zero and stays there!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 22,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOy9eXgUZbr+f7/Va5LORna2hLATCGEHIQgjoLjAIKLiMvpTZBy343HmHHDm/BBwF0cZxhk33Fcc0RlERYYRBBGBgAlISAgkgSRkT6fTe3dVvd8/qqvTnfSeTneI9bkuLpKuqu63uytPPfUs90MopZCQkJCQ6L8w0V6AhISEhETvIhl6CQkJiX6OZOglJCQk+jmSoZeQkJDo50iGXkJCQqKfI4/2ArqSmppKc3Jyor0MCQkJiUuKY8eOtVBK0zxt63OGPicnB0VFRdFehoSEhMQlBSHkvLdtUuhGQkJCop8jGXoJCQmJfo5k6CUkJCT6OZKhl5CQkOjnSIZeQkJCop8jGXoJCQmJfo5k6CUkJCT6OZKhl5CQ6FW+r2jB2SZDtJfxi0Yy9BISEr3KI58UY/OeM9Fexi8aydBLSITInW8dwTs/VEd7GX0ajqdoMVhRozVHeym/aCRDL9EnKalph8nGRnsZXuF4igMVLdhX3hTtpfRptCYbeArUaU3RXsovGsnQS/Q5Oix2LH/5B7x7yKt0R9RpM9rA8RTnmo3RXkqfpsVgdfxvg9nGRXk1v1wkQy/R57jQagLLU5xp0Ed7KV5p0lsAADVaEyx2yYB5o0Vvc/5cK3n1UUMy9BJ9jlpHPLeype96y016wVOlFKhu7bvrjDaiRw90fq8SkUcy9BJ9DtHzq2w2gFIa5dV4prmj04Cda5IMvTfcDb3k0UcLydBL9DlEz6/DwkJrskd5NZ4RQzeAcEGS8EyzwQqljIFSzkgefRTpc4NHJCRq2jo9v8pmAwbEDYjiajzTpLciQS1HvFqBc5Kh90qL3oZUjRJqhUwy9FFE8ugl+hy1WjNGZ8QD6Ltx+ma9FekJauSmxUmVNz5oMViRGq/CoOQYKXQTRSRDL9GnoJSiRmvCzNwBUMgIqvqooW/SW5GmUWF4mqZP5xKiTYvBipQ4JQYnx0pNU1FEMvQSfQqtyQ6TjUN2ShyGDojts/HvJr0F6QkqDE+Lg9HGodElOSvRSYvBilSNCoOTY9BmtMFo7btNcL4ob9Dj7YNV0V5GyEiGXqJPIcbnByfHIDdN0yc9ekopmjqsSI8XPHoAUpzeAzxP0WqwITVeMPQAUNd+aXr17/1YjfVflEJn7pvFAf6QDL1En0JM2A0ZEIvc1DhUt5rA8X0rLNJhYWFleaTHqzE8XTL03uiw2MHy1OHRxwK4dEssq1uEdVc09t0mPl9Ihl6iG/vKm6L2Bym+7qDkGAxLjYON5XGxj3mBzY7SyvQEFdLjVdCo5DgXggwvpbRfx/bFGvpUjRJDBgge/aVaeSM2xZX14W5tX0iGXsINO8dj9bvH8Pd956Ly+jVaExJjFEhQKzAsNQ5A36u8aXLE49PiVSCEIDctLqQ1/rO4DjOe+g9sLB/uJfYJmh3yB2kaFdI0KqjkjFvp7KWCleWczka5ZOgl+gM1bSbYOD5qgyJqtWan95friH9X9bGwiCh/kB6vBgAMT9OE5NF/W9aMJr3VrXu0P+H06B0XRKHEMjoe/b+K63A+RKmKmjYzxOhhuRS6kegPVDpqwiujVBte02bC4CQhnpuqUSJeJe97Hr1L6AYAhqfF4aLOEnRFSUlNOwD0f0OvET6nwcmxUTH0FjuHh7cVY+uB0KpmxAvEmMx4lDfoL8lwm2To+xnNeiu2HqgM+WQUk4otBit0EZYfoJS6efSEEAxLi+tzlTdNHVaoFQziVUJjuVh5E8w624w2XHCEMVoNNj97X5q0GKyQMQRJMQoAQiVVNHI/1a1GUBp6wry6VVjzlXmZ0Jntl2QprWTo+xk7Si7iiS9PoyLE0IurJ3+uJbIhk2aDFVaWd1ZoAEBualzU7i680aS3Ij1eDUIIgM4QUzCGpKS23flzv/Xo9TakxCnBMMLnNCQ5FlqTHYYI19JXOy7AIRv6FiPi1XLMzE0BcGmGbyRD389o7BDCChWNIRr6FgPS44Vb7VDizj2hs7QyxvnYsFQNLurMfUrzvVlvRZrjMwKA7JRYMARBSSGIYRsAaDX2X49eDNsA6Kylj3D4Rgz9NXZYobcEf5da3WpETkocxmQKshzlDR1hXV8kCMjQE0KuIoSUE0LOEkLWetiuIoRsc2w/TAjJcTyeQwgxE0KKHf9eCe/yJbriNPRNoXkdlc1GzB2VBoWMRDw23tks1enRD0uL63Oa7016i/NiCABqhQxDBsQG59HXtGNkugZqBYMWfT/16B06NyKioY905U21y3kcyt3h+VYTclLjkBynRHq86pIssfRr6AkhMgB/A7AYwDgAKwkh47rsdjcALaV0BIAXATzrsu0cpbTA8e/eMK1bwgudhj54b7zdZEOr0YZRGRrkpMRFzaMXDQIghG4AoKoPhW+E0I3K7bFgKm8opSip1aFgSBJSNap+7NHbkBqndP4eraapqhYjUjXCOiqDDEfaWB61WhNyUoS1j3YkZC81AvHopwM4SymtpJTaAHwMYGmXfZYCeMfx86cAriBiAFMioog13qF08Imhh+FpGsFwRbissVZrQkqcErHKTvXsvlZLb7Fz0FtYpCeo3R4f7kga8wF08dZqzWgz2jBxSBJSNKp+GaOnlKK5i0cvyBVHXpe+qsWEuSPTIGNI0ENiarUm8BTIThHOw9EZ8ahoMvS5bm1/BGLoBwGocfm91vGYx30opSwAHYAUx7ZhhJCfCCHfEUIKPb0AIWQ1IaSIEFLU3Nwc1BuQcEf06KtajLBzwTXiiAJiuWka5KbF4XyrKejn6Am1WrObNw8AcSo5MhJUfSYh69os5UpumgZWlg9Iy0VMxBYMSUJqnLJfVt3orSxsLO/0pAGhiirSJZZ6ix0tBitGZsQjO8jwGiCEbQBgWGqnR29j+T4VSgyE3k7G1gMYSimdBOARAB8SQhK67kQpfY1SOpVSOjUtLa2Xl9R/MVhZGG0cxmTGw87RoBtEzjUboZARDEmOwfA0DVieRjSeWqs1Y/CA2G6P56ZqUBXhCiBvOGvoPYRugMAqO0pq2qGUMxidGY8UjRKtxv7n0Yt5B9dkLOAosWyP3DklatQMS41Fbgh3qWLJrOjRj8kUzNelFr4JxNDXARji8vtgx2Me9yGEyAEkAmillFoppa0AQCk9BuAcgFE9XXRfpdVgxWP/+jlqFSINOsEIzRmRCiD4ypvKZgOyU+IglzEuYl2R8Vx4nqLOg0cPoE/V0nftihUZniYYgkA+r5IaHcYPTIBCxggxeoMtoJDPpUSL4y7Fo6GPoEcvxuSHpWowPD0O1S3BieSdbzVCo5IjxZFrGJmhASH909AfBTCSEDKMEKIEcDOAHV322QHgDsfPNwD4llJKCSFpjmQuCCG5AEYCqAzP0vseX52sxzuHzqPYpXQukjQ5wjazR6SCkOATspUtRmfyM9dpuCLjSTfprbBxPIYke/Lo46A12aHtA0lL8TMWu2JFBsQpkRSr8Kufz3I8TtbpkD84CQCQolGB5Sk6Qij768u0Grx59LFoN9lDKnMMBdGjz06JxfA0DWwcH1QyuLrVhJzUWGfPhFohQ05KXP8z9I6Y+wMAvgFwGsAnlNJThJCNhJAljt3eAJBCCDkLIUQjlmDOBXCCEFIMIUl7L6W0Ldxvoq9QUqsD0OlZR5pGR1ghOyUWg5NjcCaIhCzL8TjfanQ2/ySoFUiLV0Ws8qZG26lD3xXxotMXErJNeivkDMGAWKXb44QQ5KbG+b0wVjQZYLZzKBgiGHoxht3Sz+L0nTo37p+T+P1GyquvajFgUFIM1AqZy11X4Od0davRGbYRGZ0Rf8k1TQUUo6eUfkUpHUUpHU4pfdLx2DpK6Q7HzxZK6QpK6QhK6XRKaaXj8e2U0jxHaeVkSukXvfdWos8JR5KtoSNKht6RKExPUGNUenxQwmQ1WjPsHHUaVUAIR0TKuIpe1hAPMfphqcFLDPQWTXqhCUjs9nRFqFTyvUaxUWqiw9CnxAkeb2s/q7xpNthACLpdEDtLLCNk6B0eOSDkegAEXHlj53jUas3O0kqR0ZnxqG41wmzrO018/pA6Y8OEwco6QyVR8+g7LNCo5NCo5BiRoUFlsxFsgFUzYshBTCqKP59tisw81Jo24Q9/UFJ3j35wcgzkDOkTYwW7dsW6Mjxdg2a91WcYpqS2HQlqudN4iB5vf/ToB8QqIZe5m5ghTo++9xOylFJUNRucJbrJcUqkxCkD9ujrtGZwPEVOV48+Mx6UImoKr6EgGfow8XOdDqI9rNdFR4q1qcPqjB2PTI+HjeNxPsCqmUpnDb2rR6+BzmxHWwRi47VaE9LjVVArZN22KWQMhg6I7TMefdeKGxHxIumrFLS4RoeJQ5KcMV+nR9/PKm9a9NZu8XlAyGXEKGQR8ei1Jjs6LKyboQ6mP6TKUbWWk9rd0ANA2SUkhSAZ+jAhhm3GD0pAQ5TU7Ro6LMh0NPKMyhCMTqCVN+eaDY6EYuetdm4QlSQ9pabNc8WN61r6gqFvdgwF94Tz8/Li6ZltHM406p3xeQBIjlWAkP7p0XeNzwNiLX1MRMp2xfPFLRyZHrhI3nlnaaV76CYnJQ4qOXNJJWQlQx8mSmp0GJwcg3FZCWiIkkff2GFBRkLnMAwg8A7Zymajmzfv+hyRCJnUtpvcNG66Miw18M7T3oLleLQabUjrUlopMnRALOQM8eoxnrqoA8dTTBzcaejlMgbJscp+F6NvMdg8evRA5EosRUPv6tHnpmrQarQFVMFV3WpCnFKGtC7vQ8YQjMzQXFIJWbn/XSQCoaS2HROHJCEzQY1mvRUsx3eLT/YmlFK30E2cSo5BSTEBl1hWthhwxZgMt8cGJcVAJWd6vcSS5XhcbLdgyUTvHv2wVKHz9KLO7POC0BPsdjtqa2thsXjOsXA8xWvXZSEp1orTp0973Gfr0izIGYvH7byFxetLspBJW3H6dGfx2QuLUiFnOK/PeSnyWGEiYpUyj+9pdb4KJpu819/vEGLH1iVZsDRfwOkWIVQ2NYnD60uycKGyAg1y33+fV2TZMf+aDJSVlXXb9uisBFhZPirfmVqtxuDBg6FQKAI+RjL0YaDVYEWt1ozfzMqGRqUATwVt9axE74Yr3LSb7LBxPDJcvM1RGZqADL3OZEeLweZ2iwsADEMc3YS9GzJp6LCA46lPAy6urarF2GuGvra2FvHx8cjJyYEnqSaTjQXfJDSVJcZ4/iOLaTHCxvIY5YjjunKh1YQEG4sxWe7N4apmAyiFs0ntUofjKewXdchMVHdrLAOE8Fe9zoKRAxMgZ3rPGTrfakSSncPozM7P28pyKG/QY3ByLAbEdQ8tuVLeoIdawXQrrwRc3kNWQsQdutbWVtTW1mLYsGEBHyeFbsLACUf9fP7gJGQlCid2fYQrb8Qa+gwXsa2RGfE41+xfgEkcMJKb1t3Q5Kb5rw3vKU4del+GXhQ368WLjsViQUpKikcjDwAsJ3yOCpl3vT6VgoGV4z1WKpnsLGKU3ZPNcoaA7UedsSwvVHp5M+JKh2G0s737nm0sD6Xc/fNWyhgQQmBlfZdGUkphY3movHj9YtGAJcKD3QkhSElJ8XrX6Q3J0IeBktp2MASYMCjRaWgjXWIpvl6GS6JwZLoGNpZ3jqzzhqeKG5HhaRrUtJn8/mH0hE4deu93QGnxKsQpZb2ekPUluioKvPnyQlVymdNIuMJyPGwsj1hPhl7GOI1jf0C8IMo99BoAgMJhPG29KJhHKYXVg6EmhEAlZ2C1+35tG8eDgna7UIg4DX0U5E5CEQaWDH0YOFGrw4h0DeJUcqdHH2lDL6oqdvXoAfjtkK1sNkDOEI/NSsPT4sDTThW/3qBWawYhwEAPNfQihAhhpGh2x4pet9yXR+8wLNYuht7kMAgxyu7RUjlDwPG03+jd+PucRI++68Uw3GvgKfXokavkTLfvpyvi2pRePHo5QyBjSJ+afOYLydD3EEopSmrandolSbEKqORMxLtjGz1osIxwxHz9NXacazZgaEosFB5ijU5Vxl5sDqnRmpCZoPb6RyUyLDUuqk1Tdo6HnGHA+PCoOg29uwEQuyhjPPQJyBwGsb+Eb1g/dz4yhoAhpFclsEWP3dM5pZLLYGN58D4aAcULgbfQDSEEaoUMFj93Bn0FydD3kLp2M1odQyQA4QTITFRHJUafHKuAyuVWUyNW3vj16I1uHbGuRELcrFZr9hmfFxmWGoe69ujNj2U56tObB4QwjJzpHhow2TioFTLIPIQzRIMY7vBNTk4OWlpaAACXXXZZyM/z9ttv4+LFi26Pffzxx3jyySfx9ttvg2EYnDhxwrlt9vTJqKu54PWzIoTgwzdfhq7D9zn1j3/8A2PHjsX8+fO97nPrrbdi9OjRGD9+PO666y7Y7UJXspUTzhGVnMG+fftw7bXXOo9RKRhQCOG1q6++Gu3t3UUIbSwPhhCv4SdACN9Y7VxEOsd7ilR100PEROzEwYnOxzIT1BGvpW/ssLqFbURGpGtwxkfTFMdTnG814Vdj0z1uj1XKMTBR3auVN7VtJszMTfG7X65jfuyFNhNGZXSvagknG744hdKL7p2PZjsHAnjs3nVFvBC57mdyzAnYtGJit/1FYxKIR8+yLOTy4P9sf/jhh6CPEXn77bcxfvx4DBw40PnY119/jYceeggnT57E4MGD8eSTT2Lbtm3CRgrIGPi883n39ZexZPnNALzPn3jjjTfw+uuvY86cOV73ufXWW/H+++8DAG655RZs3boVv/vd72BjeRBCPN6luobXvvrqK4/PKyRyGZ/xcLWcAUcp7Fz3pG9fQ/Loe0hJTTuUMsY5kAAAshLVEQ/dNHVYuo23A4SErK/Km1qtCTaOx/BU76V9w9M1vRYysbE8GjosHgeOdEUUpYrWtClKA0uEEULcwgKUUlDq/W5AfJzlKB5//HGMHj0ac+bMwcqVK/H8889j3rx5ePjhhzF16lT85S9/wRdffIEZM2Zg0qRJWLBgARobGwEAra2tWLRoEfLy8rBq1So3T1Oj6fx+N23ahGnTpiE/Px+PPfYYAKC6uhpjx47FPffcg7y8PCxatAhmsxmffvopioqKcOutt6KgoABmsxmUUhQXF2Py5MkAgGuvvRanTp1CeXm58H5BIXPcpezevRuzZs3C5MmTsWLFChgMBmzZsgVNDfX4zfJrvHrrGzduxPfff4+7774b//M//wOO4/CHP/wB48ePR35+Pv76178CAK6++moQQkAIwfTp01FbWwtACN2oXAx1R0cHrrnmGowePRr//eD94HkeVjvndtfz61//GlOmTEFeXh7efesNqOQMOI7DnXfeifHjx2PChAl48cUXnWvsTMj2/fCN5NH3kJLadowdmOAWC8xIVKNRZwXPU48qh71BQ4fFo5c7KiMeVseAY0/1wOec4wO7bxMZnqbBp8dqQSkNKePvi3qdGTz1XXEjIqoQBjvgORQeuy7P7XdKKX6+2IFUjdJvf0Sz3op6nRnjHDXW7SYbLrSZMNJLnbwYujl69Ai2b9+OkpIS2O12TJ48GVOmTAEA2Gw2FBUVAQC0Wi1+/PFHEEKwdetWPPfcc/jzn/+MDRs2YM6cOVi3bh2+/PJLvPHGG91ea/fu3aioqMCRI0dAKcWSJUuwf/9+DB06FBUVFfjoo4/w+uuv48Ybb8T27dtx22234aWXXsLzzz+PqVOnAgCOHz+OiRMnOs8FhmHwv//7v3jqqafwzjvvgFLhLqWlpQVPPPEE9uzZg7i4ODz77LN44YUXsG7dOjz/5z/j9W07MHu851rwdevW4dtvv3W+7ssvv4zq6moUFxdDLpejrc1d7dxut+O9997DX/7yFwDoVnFz5MgRlJaWIjs7G1dddRX2fbMTy5ff4PYcb775JgYMGACTyYT8yVOx/IbrUVzdiLq6Ovz8888A4BbmUSuE57ewHBIQePNSNJAMfQ/geIqf6zpw/WT3EbpZCWrYOB5tJu9t4OFeR7PeisxED6Ebh+bNmUaDR0Nf6TIQ3Bu5aXEwWFk06T2Hh3qCWEMfiKGPd2jkV0XBo+d4wStXBNDgo1J0hgbkMgZmGyeU9XkJ+TBECHMcPnQIS5cuhVqthlqtxnXXXefc56abbnL+XFtbi5tuugn19fWw2WzOxpn9+/fjs88+AwBcc801SE5O7vZau3fvxu7duzFp0iQAgMFgQEVFBYYOHYphw4ahoKAAADBlyhRUV1d7XO+uXbuwePFit8duueUWPPnkk6iqqgIgJFx//PFHlJaWYvbs2QCEi9WsWbMcRwgXCTvLQ670/5nu2bMH9957rzNsNWDAALft9913H+bOnYvCwkKhvJXjkRDTad6mT5+O3NxcAMDKlStx4PBhXLv0erfn2LJlCz7//HNQCjRerEVtdSWmF0xAZWUlHnzwQVxzzTVYtGiRc38Zw0ApYy4Jj14K3fSAymYDDFbWWXEjkunw+CJVYtlqsIKn8Bi6EStvKpo8J2TPNRuRHKtAso8uwd6svBFr6ANJxgJC41Q0xM3sAZRWinStvDHZOMQoZF5j1sSR9ON8JPXi4jov0g8++CAeeOABnDx5Eq+++mpQzTOUUjz66KMoLi5GcXExzp49i7vvvltYt6rTKZHJZGBZ1uNz7N69283gAYBcLsfvf/97PPvss6AQFEcppVi4cKHztUpLS513GeJHYeN6nsjcsGEDmpub8cILLwAQqqNol9LKrneicoZxq4zat28f9uzZg0OHDuHgkSKMycsHb7cjOTkZJSUlmDdvHl555RWsWrXK7XlUCtklUWIpGfoeUOIhEQvA6VlHytCLA0cyPMjnJqgVyEpUe1WxrGw2eOyIdcVp6HvBwNZqzZAxxNl/4I/cCA5DcUUsGfSU3OtKZ/elYHDMds5jo5QrchmDSVNn4IsvvoDFYoHBYMDOnTs97qvT6TBokHAX+c477zgfnzt3Lj788EMAQrJUq9V2O/bKK6/Em2++CYNBOB/q6urQ1NTkc23x8fHQ6/XO12ZZFikp3ZPnd955J/bs2YO21hYwhGDmzJk4ePAgzp49CwAwGo04c+aM8zmNBkPAtfQLFy7Eq6++6rz4iKGbrVu34ptvvsFHH30EhnHvYXBNkB45cgRVVVXgeR7btm3DZXNmu+WtdDodkpOTERsbi1OnTuPET0VQyITwE8/zWL58OZ544gkcP37cbV1qhVCT76tUsy8gGfoecKK2HRqVvJuhdMogRCghK9bQewurjEjX+PToc1O9x+eF5xW6UnvDo6/VmpCVqA5YL2RYahzajDa0myIr62v30+3pimv3pcUuGAG/hp4hyJs4GUuWLEF+fj4WL16MCRMmIDExsdu+69evx4oVKzBlyhSkpqY6H3/sscewf/9+5OXl4bPPPsPQoUO7Hbto0SLccsstmDVrFiZMmIAbbrjBacS9ceedd+Lee+9FQUEBduzYgQULFnjcT6lU4r77H0RbSzPkMiAtLQ1vv/02Vq5cifz8fMyaNcspELb6nntw/+03YOnVizw+V1dWrVqFoUOHIj8/HxMnTnRe0O699140NjZi1qxZKCgowMaNG2FleZwq+QkP3fdb5/HTpk3DAw88gLFjx2LYsGG4ftkyAHDOkLjqqqvAsizGjh2L9ev+hPxJUyFjCOrq6jBv3jwUFBTgtttuw9NPP+22LrXCcyd0n0OsCOgr/6ZMmUIvFZb89QC96dUfuj3OcjzNffRLumlXWUTW8f6P1TR7zU5a3272uH3DjlN09P99RTmOd3tcZ7bR7DU76cv7zvp9jWu3HKC3bf0xLOt15fq/H/T4GXrj36caaPaanfT4+bawr6W0tNTrtsYOMy2p0VK2y2fojeoWAy2r76CtBgstqdFSi431uX9Nq5GWXtRRvV5PKaXUaDTSKVOm0GPHjgX+BiLA3XffTQ8dOuR1u9FqpyU1Wqoz2fw+V3lDB61qNoRzeZRSSuu0Jnqytp3yvPfvymrnaEmNlrboLd22VTUbaHlDR0CvZbKytKRGS7VGa8jrDQVP5yqAIurFrkoefYjYWB6n6/Vu2uIiMoYgPV4VsaapRp0FDOkcNN2VURkaWOx8Nw1wMRHrz6MHHPNjeyEJWqs1BRyfB4Bhab0vbuYJlqOQEeKx4ckTYvel0cpBxhC/Xb8yGQHLUaxevRoFBQWYPHkyli9f7ixh7Cts3boVM2fO9Lrdn86NK0oZ0yt6N2LFja8KMYVM6M71JIXgSSPHGyoFA4K+L4UgVd2ESFlDB2wc7+yI7UpmohoNHZFpmmrsEMa2eQt/jMzoTMgOdZmWI4Zi/MXoASFO/8/iizDbOI8KjKFgsXNo7LAGJTs8JDkWMoZEPCFrD3K+gNh92WG2I1Yl91uWKmeE/d997/2Iyt6GG6dyZQBJa6WcgdHKYsaMGbBa3QevvPfee5gwYUJIa7CynEepCVec4bUuhp56qNjxBeN4nr5eeSMZ+hApqRHqafMHd4+hAkKcPlKjxhr1Fp9ljyPSRXEzA64Y2zlcpLJFEDPrOirNE6JWemWLAXkDPb/nYLnY7pAnHhC4br9SHp35sYHIH7gieoQcpX6NDtApfczyFH28ydInnR69/4uVQiZ0lh784VDYLm48pbCzPJJifGvNA8Jdl8nmXlkkVuz4uwNzRa1gnKJ1fZVL13WIMiW1OqTEKTHIi+JiRoKgd0MjkI0X5A+81+snxiiQkaDqlpCtbDZi6ADPYmZd6Y35sTXOGvrgBokMS+19jfyusDwPRRDNb66aQ/4SsQCcIaFLXdiM5YUQVyCNgqIxDae4mY3lQeFdjMwVlUIIHbmqhopJVVVQd29CmM7f3IdoIhn6EDnhGB3o7ZY8K1ENk42D3uq5FjmceJM/cGVkenw3FcvKZqPPjlhXclLiQEh4a+lrtf516D0xLDUO1a2RnR9r52hQXqeM6dRZCSTUJXrAXC8qOkYCYYRmYBdEpWO/cMbp/ckLu+JJUpvbiHIAACAASURBVNpTaaY/RCkEax/26iVDHwIGK4uKJoPXsA3Q2TTV2MsJWRsrDKzO8DKwWmRkhgYVjQanceR4iqpWY0DxeUA4mYckx4bVk65pM0MhI0F32+amxcFi5yOmJ8Q5tM2DCd0AgiFRypiA7pjk/USq2M7TgMcDKpy69OF7z/7khV0R77pcG6dE1UpfU8S64iqF0FeRDH0I/FynA6XwWHEjkpkQ2kjBP+8ux2fHawPev8kxQjAz0bfUwsj0eJjtHOoccfE6rRk2lvc4Vcob4a68qdWaMDApJuBKFpFhERgr6IqzWSrI+aZZSTEeh7l4QqxSsV/ihp4LIpchYwhkhITVo7eyQpVTIHdf3jx6f6qVXVHKhBkFfTkhG9CZSwi5ihBSTgg5SwhZ62G7ihCyzbH9MCEkp8v2oYQQAyHkD+FZdnQ5Ues7EQsgpElTPE+x9UAVPjh8IeBjxK5Yv6GbDPchJL7mxHpjeJoGlS2GsIVMagLUoe+KqGJZFQFxMyA4+QNXYhQyxKkCq3cQZBCYsIZuoqFHz/I8rpwzzatOjsjmzZthNpuhkDOwe2k2CkWP3mi2uuVHfMEwBEqZ++wAG8s7J2AB8KpX78r8+fNRcaqkW4llcXGxVxnkSOP3LCSEyAD8DcBCALUAjhJCdlBKS112uxuAllI6ghByM4BnAdzksv0FAF+Hb9nRpaRGh8HJMUjxIVgmTnoKJrxQozXBbOdwur4jYOXLJrEr1l/oJl0UN9Nj/pj0ztLKAGroRXLThHr8izpz0AlUT9RpTVjgUgUUKBkJKsQqZb0rhfD1WqDhJABAzfPItTvmvfZEvTNzArD4Ga+bAxkS3pf16D/6+GNh/QF8Rps3b8Ztt90GpSzGq0cfih79h+++hVWr7w34vakUMmfohlIKK8cj3qW0MlBD7UncrLi4GEVFRbj66qsDXk9vEYhHPx3AWUppJaXUBuBjAEu77LMUgCi68SmAK4jj3ocQ8msAVQBOhWfJ0aektt1n2AYQ4n8pccqgQjdljnJMk43DeT8DvUU65Q98h26SYpVIi1ehwmHgK1uMSIpVYIAPMbOuDA9j5Y3JxqLFYAs4tOEKIQQj0zX4uU7X43UEgmh7w6zQ3A25jGDzpmeC0qOfffl8nK0WQn3R1qMvPS3IG7h+TN706C9evIj58+fjlmVXC5UyXarTQtGjnzZtGi7W1XVLxN5///3YsWMHAGDZsmW46667AAiyxC8+vQFWlsd7772H6dNnYMWiOVj73w+Cc0yocr0r8jQrQOSbnf/EjVfPx8hRo3DgwAHYbDasW7cO27ZtQ0FBQedQligRiGswCECNy++1AGZ424dSyhJCdABSCCEWAGsg3A14DdsQQlYDWA3Aoz5HX6LVYEWt1ozbZ2b73TczMbhJU2dc6u5LL3Y4Y9G+aNRboZARJMf6N9gj0zXOsYKVzQbkpsYFFYt01tI3G3D5KO+TgQKhLgh5Yk/MHpGKV/dXQme2IzGmF7TAXTzvVp0ZLQYbxg9M6FVrf6r4OHbt/FfAevR2jsdTL/4Nzz73HF7/+5ao69E/8/RT+N+n/+r8iHzp0b/wwgvYu3cvqCoe9TozON49th+KHv27772Ph//viW6J2MLCQhw4cABLlixBXV0d6uvrAQAHDhzA1UuX4+yZMny8bRt2f7sPNTob/vr4WnzwwQf4zW9+43yOo0ePep0VAACgHD7c+R+UHfkOGzZswJ49e7Bx40YUFRXhpZdeCuV0CCu9nYxdD+BFSqnPYCql9DVK6VRK6dS0tJ4ZkN7GOTrQS0esK8KkKavf/UTKGvXITFBDzhCU1gfmrTZ2WJAerw4ozDMqIx4VTQZQSh2llYHH5wEgJU6JxBhFWCpvakOsoReZNzodHE9x8GxLj9fiD5ajUDAk7ENXunK86DDmLVoMtVqN+Ph4n3r0V155JSYVTMTbr/wVZaeFKOr+/ftx2223AQhMj37y5MkoKytDRUUFAPRYj/7w4cOovXAeok/vqkdfUFCAd955B+fPn3c7TvS+/SVk9+zZg9/+9rc+9ehnXTYbk2dc5tXQl5aWYty4ccjIyEB9fT0OHTqEOXNm4/DB73D82HHMnT0LN15ZiO/27UVlZaXbcxw8eNA5K6DrdwMAN96wHEo5g5zRE/zmJ6JBIB59HYAhLr8PdjzmaZ9aQogcQCKAVgie/w2EkOcAJAHgCSEWSmn0L3EhUlLbDkKA8YP8d4dmJKhx7Hx3qVhvnGnQY/ygRNRqTTjVZV6pNxo7LH7DNiIj0jUw2TicaTSgSW8NuIZehBCC3LQ4nGvqeeimRivq0Ifm0U8emoR4tRz7yptw9YSsHq/HF8HKH4QKQwQ1RZ7Sbtr1XfXoH3nkEcyafyV27Po3XnnhGWdlkD+oQ4/+t7/9rdvj1dXV3fTozWbPd6O7d+/G9u3b3R6Ty+W478GH8dbLm52hG+rQo//oo4+8rkespbezPBB4FNENUY/+lbc/RJPB1q0GftCgQWhvb8euXbswd+5ctLW14ZNPPoFGo0FqciIoBW685VasXbfReecW7EVdrVYjOVaJxibqVcc/mgRy9h4FMJIQMowQogRwM4AdXfbZAeAOx883APjWIahWSCnNoZTmANgM4KlL2cgDgkc/Ik0DTQDVFFmJamhN9oAEj6wsh8oWI0ZnajBuYEK3wdTe8DYU3BNiQvabUw0AfE+V8sbwNE3YPHqlnAl5ApdcxqBwZCq+O9Pc693HLE8DEunqKTMvuwzf7dkFg9EckB69leWw4x+CETXZuKjr0a+45TYcPvAdWlqahffjR49er9dDEaBHH4gevZ0XzgtP5bozZ87E5s2bnVOonn/+eRQWFkLOEFxWeDl2/PNz1F1sgFLGQKvVdrvzmD17tt9ZAWL4VBwg4/q5RRu/hp5SygJ4AMA3AE4D+IRSeooQspEQssSx2xsQYvJnATwCoFsJZn+AUoqSmvaAwjaAS9NUAJU355qM4HiK0ZkJGJeVgCa9Fc16/2EfwaMPzNCLM2V3/Swa+uA8euEYDZr0Vugt9qCPdaWmzYTByTE9mqk7b1Q6GjusOF3fu39Mdo4PqoEmVKZPm455CxdjyqSCgPTor5o3G2kOPXqTjYu6Hj0jU+DWu37rvHD41KNfvRpXXXUVFl5xBWQM8TtpKhA9+ivnzsRrm58DABQVFblNgyosLATLshgxYgQmT56MtrY2FBYWghCCcePy8N9r/3/cvmIpll4xCwsXLnTG8UWmTZvmd1aAUs4gTikDzwu2Yv78+SgtLe0Tydio6893/deX9ehr2ow0e81O+u4PVQHtf+BMM81es5MeOtfid9/Pj9fS7DU7aVl9Bz14Vjjuu/Imn8cYrXaavWYn/dveioDWQymlUx7fTbPX7KS5j35JrXYu4ONEvvm5nmav2UmLL2iDPtaVa7ccoLe/cbhHz9GgMwf9/n3hSeOb43laUqOlDTrPWv/hxGCx00NlNVRntvnVo+d5nv5c105r2oy0vKGDnmvS9/r6KPWtR3++1UhPX9QF/ZxnwqRLf6pOR2tajUEfd6HVSE9d1NGTte20Tmvyul8gswLajFZaUqOleos96HUEg6RH34uIidiuM2K9EcxIwbIGPRQygmGpccjLEjyF0nrf4Zsm5wjBwCUERjqULIcOiA1KoU9ETOD2NHxTozWFHJ8XyUhQY2xWAvaVN/foeXzROUKw9z16uYxg45qHMXv6VL969CxPwfEUKrkMsUoZzHYuIgJ6vvTo2RBzGUo5A0sP18/xPFieh1IR/OurFAxYTpgE5ks6IZBZAQlqBWSEQGuM7AQ0f0gyxUFw7LwWShmDMVnxAe3vNPQBhG7ONOqRm6qBUs5AKWcwKCnGb0JWfN7MAOetAkKH7KHK1qAapVzJTomFnCE9MvR6ix3tJntYmq7mjU7Da/sr0WGxI0Ed/jJLexCyuz1FzjB45qWtyEpUI83PxVts21crhJh0m9EGK8s7BbaiActTt67SQFl+1XwYTGYoXOLrwerRd2rcBP/+XY/x5fyI4SJfyBiCxBgF2s12DORp0PIevYVk6INgb3kTZuQOCPhk0qjkiFfJA/Loyxv0mJLdWQ4nJGR9l1gG2izlipiQDbbiRkQhYzA0JbZHlTdiaWUwOvTemDcqDS/vO4eDFS1Y3AvVN2KnaiQ8eoYIgyxYP/FqoFMpUSVnINp2k42LrqHnKGKVwX9OR48cRnmjHjJCMCJdE1IZqy0IMbOuuB4TyvFdSY5Tos1kQ4fZjuQgGhJ7Eyl0EyDVLUZUNhtxxZj0oI7LTFSj3k/TlN5iR127GaMzO+8UxmUloLLF2G0wgitNAercuDLSkZANtobelUlDkrHvTBNqAuze7crnPwnVuWMyA7sz8sXk7GTEq+S9Fr4RQzeRKK8U9G78yyAAggcrqCwyUMkZyAiB2ce50ttQSsHxfEh3PoQIozfNdg6GEGW9nfLCIYaOCIQ+iUCURv0Rq5RBJWfQFuEB9r6QDH2AfFsmVBL8akxw2iyZATRNnXF0q47OcDH0AxNAKXxOqWrssCBGIUN8gMJZADAlOxn/vWAUrh4fuvf7+0WjICMEf/rnz0HHVX+u02HrgUqsnD7UOfmqJyhkDOb0YpmlnaMgCGwGajiQywIz9BY755yLSghBjFIGky16MrksT0ERvPCbSFKsEgoZg6YAKs08YXWIkYVSxcUQYaavUhacaqU3CBE61Y1WFrY+Il0sGfoA+basCSPSNW4zVwMhKwAZhPIGId7d1aMH4DNO36gXJksFc3IqZAz+a8FIJMaGHs8emBSD/7lyNPafacaOkov+D3DAcjzWfnYCKRoV1i4eE/Lrd2X+6HQ0dFicWkHhhOV4yJjwGIBAkDNMQM1PVpaHSuE+xcpi5yM6jMUV8eIU6gWRIQRpGhWMVhbGELx6G8uFVFwgMiBOGZTukz+SHDX1WlPPypDDhWToA8BgZXG4qhW/CjJsAwi69M16q88/3vKGDsQpZW5jCQcnxyBBLfdZedMYwGSp3uL2WTmYOCQJG78oRXuAt6hvHazGz3UdWH9dXlj1aS4fLchm9Eb4xs7TiMTnRQIJ3XA8hZ3joXYxbDFKOSgozFGachSOENeAOCXkTPBePaVUuPD1YNhuWrwKafGhNe95QilnoFHJoTXZIlIN5Q/J0AfA9xXNsHM0NEOfGAOeAs0G7ydvWYMeozLj3W47CSF+O2QbOyzOASeRRsYQPL1sAtrNdjz11Wm/+9e0mfDCv89gwdh0XD0hM6xr6Syz9N3hGQqhlgyGiswRuvFlHERZXZVLKaE4l9Y1fBNJPfqSEkGPXs4QjB8/PiA9epPJPcfDMASpGiX0FjvMNjYgPfqXXnoJI0eOxPhBSdDr2jzus2/fPlx77bUetwWiNx8qA+KUsLE8jNboh2+kqpsA+LasCfFquVtVTKCIA0jqdRZkJXavMqGU4kyjHlfmdTd+47IS8eGR8+A8lGlRSoPSuekNxg1MwD2FuXjlu3NYNmkwZg3v3hYPCGv9v3/+DIYAG5eO75UwyLzRaXh9fyX0Fjviw1Bm+eyRZ1HWVgaTTZhYFI5qjDEDxmDN9DU+95EzjCOx6XlSE8uyzkEZrh6sQibEmIXkffdzorf16J9/7hls/MvrAYduRD362Fj3UGiKRolmgxVNemtAevSzZ8/GtLkLcP01i0KqOOrNwSDOmnqTDRp1dE2t5NH7gecpvi1rxuWj0kLKyIvyBN5KLJv1VmhNdrf4vMi4gQmw2HlUeRiw0WFhYbHzQc9bDTf/dcVIDBkQgz99ftKrps+Okov47kwz/nDlaAxM6nlJpSfmjUoD2wtqlpRSRLISevOmp7Hk8mmYO7fQhx79Dtx63QLMmj4VCxYsQGNjIwDAYmjHLddfFxU9+rLTpaiuPNvNIfGnR9/VW5cxDFLiVHj26ScD0qMfMnIcYgZkgiHEp/5UR0cHrrnmGowePRr33nsveF64WLre9fz617/GlClTkJeXh9deew0AwHEc7rzzTowfPx4TJkzAiy++GPB3yTAEibEK6Mx2cNEeEemtZTZa//qaBEJJjZZmr9lJtx+rCen4NoOVZq/ZSd84UOlx+/4zTTR7zU56sKK527ZTdTqavWYn/edPtd22nWnooNlrdtJ/FdeFtK5w8l258B7+/E1Zt21tBiudvHE3XfLS95Tl+F5bg43l6Ph1u+iaT0tCfo6ubeU2lqMlNVrarLf0dHkBceTIETohP58eqain9c1tdMSIEXTTpk308ssvp7/73e+c+xVX1DilBl5//XX6yCOPUEopXfXb++jvfv8otbEc3blzJwVAm5uF8youLo5SSuk333xD77nnHsrzPOU4jl5zzTX0u+++o1VVVVQmk9GffvqJUkrpihUr6HvvvUcppfTyyy+nR48edb7+sWPH6O23304ppfStt96i999/P33hb6/RpStWUkopzcvLo1VVVbS5uZkWFhZSg0GQN3jmmWfohg0bKKWUZmdnO9fWFTvL0ZO17XTm7ELn6/7973+ny5cvp3a7IC3Q2tpKdSYbPVGjpVXNBp/Pt3fvXqpSqei5c+coy7J0wYIF9B//+Ee3dbS2tlJKKTWZTDQvL4+2tLTQoqIiumDBAudzabXBSX8YLHZaUqOlrQarz/04jqcNOrPf/UR+0RIIvmrOQ+U/p5tAiKB/HgpJsQqo5IzX7lixfNKTRz8iXQOljPGYkG10yh9EL3QjMndUGpZNGoSXvzvnHGwi8tRXp6Ez2/HM9RN6tUtQLLPcVx6+MstIyh8Agub5tdctgUqtRkxsnFc9+vM1tbjnlmWYMGECNm3ahFOnhOFthw99j2uX3QizjYu4Hv2119+IE8ePoqqqyvlYIHr0npDLGAyIU4LleNgd+YiuevRqTQIutJmgVsoCmlI2ffp05ObmQiaTYeXKlfj++++77bNlyxZMnDgRM2fORE1NDSoqKpCbm4vKyko8+OCD2LVrFxISEvy+litCTb0MWh8FC3qLHRVNejR2WHrFhgH9KHRztsmAmU/9By99WxGQLHCg7C1vwqQhSSGXXhFCHE1T3g19qkblcf6sUs5gZIbGY0K2MQT5g97k/64ZiziVHI9+dtJZ4vfD2Rb841gtVs/Nxdis4P5AQmHe6DQ0dFhQ3hieMstIyh+IiDr0XStvRD16nlI8/sc/YNXqe3Hy5Em8+uqrsFgszmMJiE9jQR169MXFxSguLsbZs2dx9913A0A3PXpvuuq7d+/GokWL3B8kDFbd9xCeffZZt9dauHCh87VKS0s9Tr3yRJrj78FTeaKN5VDdaoKcIchJiQvIgeiaF+r6+759+7Bnzx4cOnQIJSUlmDRpEiwWC5KTk1FSUoJ58+bhlVdecVPEDAShpl4Bo5V1JtFd38f5VqMjNCvoXIVDFsQT/cbQxyhluGx4Kp7ffQYLXvgOX5+s77Fn19RhwYlaHa4IYYC1K5kJajR6M/SNeozO9N6lOi5LqLzp+l7EO4T0IATNepMUjQp/unosis5r8dHRC7DYOfzx85PITonFQ1eMjMgaLh8l3HXtLQtPmSXLR9ajnz17Nr76ciesFgt0HXqPmuc2lode34GhQ4RZQO+8845z29y5c7H7i09hsnER16NneYqbbrkde/bsQXNz4Hr03lDIGchlDDosLOwc79Sjt9hsqG41ob2tDTmpcQHnzY4cOYKqqirwPI9t27Z1S/DqdDokJycjNjYWZWVl+PHHHwEI4xB5nsfy5cvxxBNP4Pjx4wG9nitda+p5StHUYcGZRgP0FhaZCWqMzNCEpYjAG/3G0A9KisErt0/Bh6tmIE4px+8+OI5bXj+M034UIH0h1mXPDzFsI5KVqEZ9R/emKY4XKm5GZ3j3dscNTECr0dZNm76pw4IEtRwxyuhpm3TlhimDMSs3Bc98VYb1O06hutWEp5ZNiJj+SmaiGmMy48NWZhlpj17UPF+xaA5WLl/qUfPcynL43X+vxV2334IpU6Yg1aFHDwCPPfYYjh/+AYvmTMP27dsjpkdPKRV0bmJUeOihh4LSo/dVOqmQCSO3WvRWrFq1CkOGDMGECfm4bv4sHNr9L6gVMmzZsgWDBw9GbW0t8vPznR53Vz36adOm4YEHHsDYsWMxbNgwLFu2zO21rrrqKrAsi7Fjx2Lt2rVOhc66ujrMmzcPBQUFuO222/D000/7/Kw8IdbUtxttQpim0YCGDgvi1XKMyohHeoK620SxsOMteB+tf+FIxtpZjr57qJpO3PANHbZ2J/3T5ycCTnK4svrdo3TmU3soz/csifjUV6V05B+/6vY8Vc0Gmr1mJ/34yHmvx/54roVmr9lJvz3d6Pb4b98togv+vK9H6+oNKpsNdOSfvqLZa3bS339SHPHXf/qr03T4o1/SDrMt6GO7Jrhq20z057r2cC0tIPR6PS2v76CnLzR51Dxv1JlpSY3Wa2K71SDooZttbK+sz5Mevd2RtG7qCH/S+nyrkZ6sbad2lqMXWo20pEZL20L4W442WodOfUmNlpbVd4R0frryi07GishlDG6fmY19f5iH38zKwUdHajBv0168dbAK9gBna1pZDt9XtGD+mPQe131nJahh43i0ddGoLnMmYr179GMHCtu6JmQb9YFPlookw1Lj8MfFYzAqQ4M/XT024q8/b3RgZZaU0m4x066wIYp09YTVq1fj1wtm45pfzfaoeW5heTc53654apwKJ5706HtT4TM9XgWeUlS2GKE12ZCRoO4zipDBkKBWIEGtiEiYxhP9umEqKVaJ9UvycMuMoXh8Zyk2fFGK0/UdeO6GiX6PPVLVBqONC1qt0hPiSMF6ncUt6SqKmYnSwZ5IUCswdEBst4RsU4cVM3JDkxrube6cPQx3XJYTMX0YV6a4qFle5UW47ec6HZ7++jR+ONeKBWMzcE9hLqbldK9QsXORlT8ABM3zC61GmO28x0osq0PMzBsquXARMNnYsGq3+ILleqZzM2PGDFit7qFJUY9erZAhQa1Ah8WO5Fgl0vtAlVkoMAxBTogzIMJBvzb0IqMy4vHuXdPx7K5yvPLdOVwxNsNjJ6or35Y1QSVncNnwVJ/7BYLrpKnxgzpjruUNegwdEIs4P+qT47IS3Dx6nqdo0kdP/iAQomHkAaHMcvaIzjJL13XUtZvx52/K8XlxHZJiFLh52lDs+rke/y5tRP7gRDxWmAie0s7KF45HbBDKoOFCLmPAehD2og5NF18GnBCCGIUM5ggqWYpJ61ClIg4fPuxz+8AkNWJNMqTGByfgJ9FJvwzdeIIQgkcWjsL4QQl49LOTPgdvU0rxn9NNuGx4SliSnVleJk0JFTf+pXrHDUxAVYvRqdXdZrLBztE+GbrpC3Qts+yw2PHsrjLMf34fdp6sx2/nDse+/5mPp6+fgB/WXoEnl42HwcKizWhHeYMezXoLWJ6PuKCZiIwh4HgKvkullT2AcXcAEKuUw2LnItaN2VPlSn8o5bLIJCz7Mb8YQw8I2e8XbyyA0cpi7fYTXssvzzUbcaHNhF/1sKxSJFWjgowhbjIIVpZDVYvRTYPeG6JkcZnDqw9lstQvCVHNck9pI94+WIV5m/bh5X3ncM2ELHz7+8uxdvEYp3pmjFKGW2dkY88jlyNVo4RSzqBeZ0FZvR6U0ojH6IFOg9l10pRzXJ6fKqZYpQwUiJiSJcvxICB9ZmyeRHd+EaEbV0ZmxGPt4jHY8EUpPj5ag5XTu5eg7XUOGel5fB4QPLT0eJVb09S5JiM4ngbk0ecN6kzITs0ZENJkqV8SWYkxGJMZj+d3CzXblw1PwR+vHusWNusKwxCoFTIMT9PAbGPRbLBBZ7YjJgqj+cQQCMfzcPXFLPbAxuWJCVmzjfWp/xIuWI5CJiNSWKUP84vy6EXumJWD2SNS8PjOUlR7EAz7T1kjxmTGu+nD95TMRLXTEweA8kbBOw/E0GcmqJEcq3AmZDs9esnQe+O2mdmYNDQJb905DR+smuHTyHclRinH0AGxmDAoMSqqg06Pnu/q0QtKmv5CJHKZMGB+Ut7oiMgUD03V4GzZKee2UGWKuxKoTPGIESNACHG+13ASiIzxvHnzUFRU1O3x4uLiXlXHDIZfpKFnGILnV0yEnCF45JNit6EgOrMdRdVazA+TNy+S1WV2bFmDHgqZ0PbsD6c2vTN0I3j0aR5kEyQEbpuZjc/vmx2W8thII8oT27uGbuw85KABvZ9YhRyukclQZYoppXjrrbdRV1fn9vjXX3+Nq666CgCQOXAQXv3L80E9byCGXpQp3rt3r9d9Zs+ejT179iA7Ozuo1w+Ur776CklJSSEd25cM/S8udCOSlRiDx389Hv/1cTFe+e4cHviV0KZ/oKIZLE/DUlbpSkaCGt+5TEA606DH8DRNwC3c47IS8M6h87BzPBr1Fmc8WaJ3aHjqKVhPl4X1OVVjxyDzj3/0uc/jjz+O999/H7GJyRg2dCgumzkdO3fuREFBAfbs/Q7LbrgR0wvG44knnoDNZkNKSgo++OADZGRkoLW1FStXrkRdXR0mTZ0OSnln34hGo3HKHmzatAmffPIJrFYrli1bhg0bNqC6uhqLFy/GnDlz8MMPPyAjayBeffdj7PzySxwtKsKNK29BbEwM9u0/iJTEOKdM8cmTJ3H5gitx/PAPKC8vx+jRo93ez+7du/HYY4/BarVi+PDheOutt/Dmm286ZYpTU1M9GvKNGzc6ZYqXLFmCZ555BmvWrMGuXbvAMAzuuecePPjgg5g0aZLfz/3+++/HlVdeiSVLlmDZsmVITk7Gm2++iTfffBPnzp3Dk08+iffffx9btmyBzWbDjBkz8Pe//x0ymQw5OTkoKipCamqq87tJS0vDkCFDMGXKFPzhD38AINx93HfffWhvb8cbb7yBGTNmYN26dTCbzfj+++/x6KOPugnTRZqALAUh5CpCSDkh5CwhZK2H7SpCyDbH9sOE3HTAzQAAIABJREFUkBzH49MJIcWOfyWEkGVdj40mSwsG4bqJA7F5TwVO1uoACGWVSbEKTBoa/JARX2QlqmG0cdBbBL2L8obAKm5Exg1MgI3lUdlsRKPO0mc0biTCx9GjR7F9+3YUFxfjlfc+RfFPnboqFqsVH365Fw89/AjmzJmDH3/8ET/99BNuvvlmPPfccwCADRs2YM6cOTh16hSWLVuG+rrabmWWu3fvRkVFBY4cOYLi4mIcO3YM+/fvh53jUVFRgWW33IVtu3+AIkaD7Z9ux/XXL8ekyZOx+ZU38fGuA6g3ctj+7wMYMSYPOrPdoevO4P7/egRPPfWU22u1tLTgiSeewJ49e3D8+HFMnToVL7zwAh566CEMHDgQe/fu9eqtr1u3DlOnTsUHH3yATZs24bXXXkN1dTWKi4tx4sQJ3HrrrQF/roWFhThw4AAAQdKgtLQUAHDgwAHMnTsXp0+fxrZt23Dw4EEUFxdDJpPhgw8+8PjdlJSU4Ouvv+4WqmFZFkeOHMHmzZuxYcMGKJVKbNy4ETfddBOKi4ujauSBADx6QogMwN8ALARQC+AoIWQHpbTUZbe7AWgppSMIITcDeBbATQB+BjCVUsoSQrIAlBBCvqCU9o4WZwg8vjQPR6va8PC2n7DjgTnYV96MeaPSwl5BIDZNNegsoAAu6iwYFUDFjUjeQCHGXFqvc3TFSmGb3sSf590bHDx4EEuXLkVMTAwSExJwxZWdUsC/vn4FAGF84PnqWtx0002or6+HzWbDsGHDAAD79+/HZ599BgBYtuQ6JCQmwWR3/1NzlSkGgA69HgePnUSBIgmDhmQjd0we4mMUmD1jGuy6RmSnCMJhg5JiMC4rAQYriw8O7MPMyxfgQpsJF9stoKBYcdPN+NuLm7zKFAOAzWbDrFmzQvps9uzZg3vvvdcpUzxgwICAjy0sLMTmzZtRWlqKcePGQavVor6+HocOHcKWLVvwzjvv4NixY5g2bRoAwGw2Iz3d/Y5e/G7UajXUarWbhDQAXH/99QB8yztHk0BCN9MBnKWUVgIAIeRjAEsBuBr6pQDWO37+FMBLhBBCKXUNwqkBRH9KbheSYpXYtCIft79xBHe/cxRtRlvY4/MAnM1N9ToLdGbBqx8ThEefmxoHpZxB6cUONHZYMX5g4MlFiUsPuYzANRcrVwnnj1rO4MEHH8QjjzyCJUuWYN++fVi/fn234xmGgBB08+ipQ6b4jrtWoa7dDKOVhVohg67pIjSxMRidGQ9CCGLVSmeoR0TGECTGKHD4+7349NNPEROvgUYth0LGIClOjd///vceZYo/+uij8H0wITBo0CC0t7dj165dmDt3Ltra2vDJJ59Ao9EgPj4elFLccccdIQmWiYgSz77knaNJIKGbQQBqXH6vdTzmcR+Ht64DkAIAhJAZhJBTAE4CuNeTN08IWU0IKSKEFIkSp5GkcGQa7rwsBz9WtkHGEFw+Ki3sr+HaNCU28gQTupHLGIzJjMeJWh1aDFaptLIfMnv2bHzxxRewWCywmkz4dvfXzm12jgdDCBQyBjqdDoMGCX+CXWWKP/zwQwBCslTX3g6LnXfrF1mwcBFefX0rSiobYLFzYExtSIQJafEqEOK5o9mTTHFqairiVHIkxSqRGKNAjFKOO++8M2wyxV0RZYpFI9rW5nkQuDdmzpyJzZs3Y+7cuSgsLMTzzz+PwsJCAMAVV1yBTz/91Km62dbW1m1Aiut3YzAYPEpIdyXY99ib9Ho2j1J6mFKaB2AagEcJId0sFKX0NUrpVErp1LS08BvZQFi7eAxGpmtw2fAUp350OEl3hFoadBaUN+ihUcmDLt8cl5WAY+e1oBR9Wv5AIjREmeL8/Hz8fyuvx8gx45wyxTZO6IglhGD9+vVYsWKFR5ni/fv3Iy8vD5999hkGDxkKnlJYHI1WOrMNORNnYeF1y3HHskW46co5uOeOW7t57l3xJ1MsolQqwypT7MqqVaswdOhQ5OfnY+LEic4LWqAyxYWFhWBZFiNGjMDkyZPR1tbmNPTjxo3DE088gUWLFiE/Px8LFy5EfX291+9m8eLFHiWkuzJ//nyUlpaioKAA27ZtC+h99hreZC3FfwBmAfjG5fdHATzaZZ9vAMxy/CwH0AKAeHiubyHE7PvkzFiDxU4NFnuvPf/kjbvp2u0n6IpXfqDL/vZ90Me/80MVzV6zk2av2Un3lDb0wgp/2XiSfo00er2eUkrp2YstdFx+AS0qKqKUUnr6oo6ebzUG9VwWG0tLarS0vt1EK5sNtKRGS8sbOnp0jnuSKf6lIH43RqPRo4R0JAlWpjiQGP1RACMJIcMA1AG4GcAtXfbZAeAOAIcA3ADgW0opdRxTQ4VkbDaAMQCqQ7wm9Tr+xMV6SmaiGg06M8406rF4vG9RNU/kDeyUM5aapfonq1evRmlpKYwmM66+/iZMnDQJHE9h43gkB1lOq3QoWTbprWAIQVZiDFI1yh71FWzdujXkYy91xO/GYrHgjjvu6CYh3Zfxa9kcRvoBCF67DMCblNJThJCNEK4gOwC8AeA9QshZAG0QLgYAMAfAWkKIHQAP4D5Kafjb1y4RshLVKK7Rod1kD0jjpiujMxNAhKE7zlCQRP9CDEm0m2y40GYCy1HwVAi9qIM09IQQpGpUsLE8MhPUUPTRvgtfMsV9CfG7uRQJyIWllH4F4Ksuj61z+dkCYIWH494D8F4P19hvyEhQo8UgxC9HBZGIFdGo5MhJicOFNhNS4iRD358Ry3tFbx7wL2bmiUvhzs+fTLFEz/nFdsZGA7HyBgDG+Jgq5YsJgxLB8rykFNjPEVUzWY6H1c6BgEid0BIhIxn6CCI2TaXFq0Ke/vN/146FzjFNXqL/IurdsLwwbEQpZyQ9domQkQx9BBFLIkOJz4ukx6sl+YNfAK4KlhY771eaWELCF9LZE0HEkYLBNEpJ/DIhhEDOMLBzPGwsD5VC+lOVCB3p7IkgQwbEoGBIEq4YG36JBYn+h5whMFk5UFCo5cEnYnNyciKiR88wDE6cOOHcFi49+t6kuLgYs2bNQl5eHvLz88Pe0FRUVISHHnrI5z7V1dUYP368x22ePvOeIBn6CKKSy/DP+2eHZeC4RP9HJiOwsIJWjejRh6qjEqoePeDZ6Ljq0Q8ePBhPPvlkUM8ZbUMfGxuLd999F6dOncKuXbvw8MMP+x0wEgxTp07Fli1bQj4+3IZeitFLSHjgwCdn0FLjWxogWFKHaFB44yif+7hqnqekZ2HEuHzs/883uGz6FBw8eBArV67EqFGj/OrRz5o1y03jJlg9+kGDBuFf//oXvvzySxQVFeHWW29FTEwMDh06BLVa7aZHf+2112L//v1h1aMHgF27duGPf/wjOI5Damoq/vOf/6CtrQ133XUXKisrERsbi9deew35+flYv349Lly4gMrKSly4cAEPP/wwHnroIaxduxZDhgzB/fffDwBYv349NBqNU0ceAAYOHIj09HQ0Nze7DRnhOA4jRoxAZWUldDodUv5fe3cfHVV9JnD8+yRAklUEilrUhAqSg0qEQBHJsRmPccubaBTDSzlb0UXQXVH22MISu1GgTddW1+5Za3lbFOvuUV6ynqagEpSkkqKG0ERIQEqIKYTwJmJ4kZCB/PaPuRkmk0lmQmYyl+vzOWdO7tx7594nv5l55je/e+e5fftSWFiIy+XC5XKxcuVKrr/+ep566ikqKipwu90sXLiQzMxMioqKeOmll1i/fj3Hjh1j+vTp1NXVkZaWxqZNm9i+fbt3H7NmzQra5gkJnbvanfbolbIJ/5rnOz4rAzzj9W63m9LSUn7yk5+EXI9+//79rfbRVj16gL179/Lkk09SWVlJ7969ycvLIysry1sXvry8nISEBMrKyhg2bJj3F7YxMTHMnz8/rPXojx07xqxZs7ztsXbtWsBTz2f48OHs2LGDX/7ylzz88MPex3z++eds3LiRkpISFi1ahNvtZurUqaxZs8a7zpo1a1rVhi8pKaGxsZGbbrqpxfzY2FgGDx7Mrl27KC4uZsSIEWzZsoVz585x4MABkpOTyc3NJSMjg5KSEgoLC5k3bx5nzrS8POmiRYvIyMigsrKSrKysFs9LqG3eWdqjVyqAYD3vSPCveT52/AQAYoQWyam2Nng9+nvvvZc+fVpfPMe/Hv3p06fZu3cv/fv3Z8CAAaSmpgLt11V///33GT9+fIt506dPJzc3N2z16D/55BNcLpf3f2uuP19cXExeXh4AGRkZHD9+nJMnT3r/57i4OOLi4rj22ms5cuQIw4cP5+jRo9TV1XHs2DH69OlDUlKSdz+HDh3ixz/+MW+88QYxMa37venp6Xz00Ud88cUXZGdns2LFCu666y5v7fqCggLy8/N56SXPpRQbGhpafcAWFxfzzjvvADBu3LgWz0uobd5ZmuiVsinB6jGLcMUVF68tHEo9+rYYqx79448/3mJ+TU2Nt6Y6eHqzZ8+e9X844Eluzcm2Wbdu3aJej94//ubjGZMnT2bdunUcPny4xQfmyZMnuffee8nNzWX06NEBt+lyuViyZAl1dXUsXryYF198kaKiIm/lS2MMeXl5rYasjhw5ckkxt9XmnaVDN0rZhH/N800bPVVH/H8nFWo9+hMnTrTax9ixY3nttde84/UHDx70lhVuS6B69H379m21Xjjr0Y8ePdrbk4aL9efT09O9l/krKiri6quv5qqr2v+V+dSpU3n77bdZt24dkyd7KrU0Njby4IMP8vDDD5OVldXmY0eNGsXWrVuJiYkhPj6e1NRUli1bhsvlAjzt+corr3iPh5SVlbXaxp133ukdPiooKAj4vPgLdy17TfRK2USrmucpt9Hzql6tyl2EWo++f//+rfYxZswYpk+fTlpaGrfddhtZWVlBE0o06tFfc801LF++nEmTJjFs2DBvT3zhwoVs376doUOHsmDBghYfdG0ZMmQIp06d4oYbbuC6664DPGP1H330EatWrSI1NZXU1FTKy8sBz/Vq8/PzAU+POykpydvjT09P59SpU96Cazk5ObjdboYOHcqQIUPIyclptf/nn3+egoICUlJSWLt2Lf369aNnz/Z/S+Pb5mHp5bdVvzhat2jWo1ffbnaqR2+HmueBfJvr0V+qhoYG43Z7rgGwdetWM2zYsE5vMxL16JVSXcTuNc+/zfXoL9X+/fuZMmUKTU1N9OjRgxUrVnR5DJrolbKRy7nm+aW6XOrRX6rk5OSAY/ddSRO9UiqqtB595OnBWKWUcjhN9Eop5XCa6JVSyuE00SullMNpolfKobQefdtCrUf/yCOPsG7dulbzQ6k3byea6JW6jGg9+vDobD36ztab72p6eqVSARSuWs7Rv1WHdZvXfm8gdz8yu911fOvRJyUl8f3vf5/169eTmppKcXGx1qPvwnr0zT744ANeeOEFTp48ycsvv8zEiRNb1JsvKSlh7ty5NDQ0kJCQwOuvv87gwYOprKzk0UcfpbGxkaamJvLy8khOTg7txRJm2qNXyib869GXlpZ6lzU2Nmo9erq2Hn2zmpoaSkpK2LBhA0888QQNDQ0tlt98881s2bKFsrIyFi9ezLPPPgvA0qVLmTt3LuXl5ZSWlpKYmBhw+11Be/RKBRCs5x0J/vXo77vvPu8yrUcfnXr0AFOmTCEmJobk5GQGDhzoLczWrL6+nhkzZrB3717vRWIA0tLSyM3Npba2lkmTJkWtNw/ao1fqsuBfj37OnDns3LmTZcuWtephtsdY9ejLy8spLy+nqqqKmTNnAm3Xc/dXUFDAmDFjWsxrrx5987527drFypUrQ461o4LVo1+9enWH69ED3m8ubd3Pycnh7rvvpqKiwltmGjwffvn5+SQkJDBhwgQ2b97c6f/xUoWU6EVknIjsEZEqEVkQYHmciKy2ln8qIjda838oIttFZKf1NyO84SvlHP716NevXx9wPa1H33X16AHWrl1LU1MT+/bto7q6utVxCN/nY9WqVd751dXVDBw4kKeffprMzMwWZyZ1taCJXkRigVeB8cCtwI9E5Fa/1WYCJ4wxg4DfAM0f618C9xljbgNmAG+GK3ClnKZVPfrbbqNXr16t1tN69F1Xjx6gf//+jBo1ivHjx7N06VLi4+NbbH/+/PlkZ2czfPjwFt+C1qxZQ0pKCqmpqVRUVLQ4ntDl2qpf3HwD0oCNPvezgWy/dTYCadZ0NzwJXvzWEeArIK69/Wk9ehUtWo8+OK1Hbw+RqEd/A3DA534tcEdb6xhjzotIPdDXSvjNHgL+Yow55/dYRGQ2MBsI2AtR6ttC69GrSOiSs25EZAie4ZwxgZYbY5YDywFGjhxpAq2j1LeB1qP3cFI9ejsIJdEfBJJ87ida8wKtUysi3YBewHEAEUkE3gEeNsbs63TESilH0Xr0kRfKWTfbgGQRGSAiPYBpQL7fOvl4DrYCZAGbjTFGRHoDG4AFxpg/hytopZRSoQua6I0x54E5eA647gbWGGMqRWSxiNxvrbYS6CsiVcAzQPMpmHOAQcBzIlJu3a4N+3+hlFKqTSGN0Rtj3gXe9Zv3nM90AzA5wON+AfyikzEqpZTqBP1lrFJKOZwmeqUcSuvRt2/cuHH07t2biRMnhn3bodSrr6mpISUlJeCyQG3eGZrolbqMaD368Jk3bx5vvhmZH+t3tl59uBO9Vq9UKoCv/7iPxrozYd1mj+uvoPd9gUvhNtN69BdFuh79PffcQ1FRUZvPxYULFxg0aBDV1dXU19fTt29fCgsLcblcuFwuVq5cyfXXX89TTz1FRUUFbrebhQsXkpmZ2aJe/bFjx5g+fTp1dXWkpaWxadMmtm/f7t3HrFmzgrZ5QkJC0NdXe7RHr5RNaD36i7qyHn1bYmNjGTx4MLt27aK4uJgRI0awZcsWzp07x4EDB0hOTiY3N5eMjAxKSkooLCxk3rx5nDnTsoOwaNEiMjIyqKysJCsrq8XzEmqbd5b26JUKIFjPOxK0Hv1FXVWPPpj09HRvFc3s7GxWrFjBXXfdxe233w542jM/P5+XXnoJgIaGhlYfsMXFxbzzzjuA57iA7/MSapt3liZ6pS4D/vXon3nmGe6//36KiopYuHBhyNsxVj36xx9/vMX8mpqaVvXcz549G3AbBQUF3mTbrL169G+99VbI8XVGsHr0hw8fDrk338zlcrFkyRLq6upYvHgxL774IkVFRaSnpwOe/zEvL6/VkNWRI0cuKea22ryzdOhGKZvQevQXRboefahGjRrF1q1biYmJIT4+ntTUVJYtW4bL5QI87fnKK694j4eUlZW12sadd97pHT4qKCgI+Lz4C9Y+HaWJXimb0Hr0F0W6Hj14PjQmT57Mhx9+SGJiIhs3bgRa1qOPi4sjKSnJewWq9PR0Tp065S24lpOTg9vtZujQoQwZMoScnJxW+3/++ecpKCggJSWFtWvX0q9fP3r27NluzL5tHpZeflv1i6N103r0Klq0Hn1wWo++4xoaGozb7TbGGLN161YzbNiwTm8zEvXolVJdROvRO8/+/fuZMmUKTU1N9OjRgxUrVnR5DJrolbIRrUfv4aR69MnJyQHH7ruSJnqlVFRpPfrI04OxSinlcJrolVLK4TTRK6WUw2miV0oph9NEr5RDaT369oVSj/6RRx5h3bp1reaHUm/eTjTRK3UZ0Xr04dOZevSdrTff1fT0SqUCeO+99zh8+HBYt9mvX79WVR/9aT36i6Jdj77ZBx98wAsvvMDJkyd5+eWXmThxYot68yUlJcydO5eGhgYSEhJ4/fXXGTx4MJWVlTz66KM0NjbS1NREXl4eycnJQfcXCdqjV8omtB79RXaoR9+spqaGkpISNmzYwBNPPEFDQ0OL5TfffDNbtmyhrKyMxYsX8+yzzwKwdOlS5s6dS3l5OaWlpSQmJnZov+GkPXqlAgjW844ErUd/kV3q0QNMmTKFmJgYkpOTGThwoLcwW7P6+npmzJjB3r17ERHcbjcAaWlp5ObmUltby6RJk6LWmwft0St1WfCvRz9nzhx27tzJsmXLWvUw22OsevTl5eWUl5dTVVXFzJkzgbbrufsrKChgzJgxLea1V4++eV+7du1i5cqVIcfaUcHq0a9evbrDvXnA+82lrfs5OTncfffdVFRUeMtMg+fDLz8/n4SEBCZMmMDmzZs7vO9w0USvlE1oPfqL7FKPHmDt2rU0NTWxb98+qqurWx2H8H0+Vq1a5Z1fXV3NwIEDefrpp8nMzGxxZlJX00SvlE1oPfqL7FKPHqB///6MGjWK8ePHs3TpUuLj41tsf/78+WRnZzN8+PAW34LWrFlDSkoKqampVFRUtDie0NXE98i8HYwcOdL4HoRSqqvs3r2bW265JaoxnD59miuvvJJvvvkGl8vF8uXLbVWq+LHHHuOxxx7zXohDRUeg16qIbDfGjAy0fkg9ehEZJyJ7RKRKRBYEWB4nIqut5Z+KyI3W/L4iUigip0Xktx3+b5T6lpk9ezapqamMGDGChx56yFZJHjz16DXJX36CnnUjIrHAq8APgVpgm4jkG2N2+aw2EzhhjBkkItOAXwFTgQYgB0ixbkqpdmg9eg8n1aO3g1BOrxwFVBljqgFE5G0gE/BN9JnAQmt6HfBbERFjzBmgWEQGhS9kpSLHGNPqrAoVWVqPvmMuZbg9lKGbG4ADPvdrrXkB1zHGnAfqgdaH5dsgIrNFpFRESpuP2CvV1eLj4zl+/PglvZGU6grGGI4fP97qgHAwtvjBlDFmObAcPAdjoxyO+pZKTEyktrYW7WwoO4uPj+/wr2xDSfQHAd+fkiVa8wKtUysi3YBewPEORaJUlHXv3t37S0ylnCSUoZttQLKIDBCRHsA0IN9vnXxghjWdBWw2+v1XKaVsIWiP3hhzXkTmABuBWOA1Y0yliCwGSo0x+cBK4E0RqQK+wvNhAICI1ABXAT1E5AFgjN8ZO0oppSIopDF6Y8y7wLt+857zmW4AAv622BhzYyfiU0op1UlaAkEppRxOE71SSjmcJnqllHI4TfRKKeVwmuiVUsrhNNErpZTDaaJXSimH00SvlFIOp4leKaUcThO9Uko5nCZ6pZRyOE30SinlcJrolVLK4ZyT6N1n4eNXPX+VUkp5OSfR15XBxmfh499GOxKllLIVxyR6kzia+j7zMR+9AqcORzscpZSyDcck+rrinZw65OLg6SzY/PNoh6OUUrbhmER/zahk9rv30mTu55ttn8ChHdEOSSmlbMExif7LE1/z52uPcdAcpe7cPMz7/wZ6fXKllHJOou/Vqxfd4+IoumIPF2L7cmD392DPu8EfqJRSDueYRN9Qb7jyy1u4EBvDu91LcUsmZ/J/B+cbox2aUkpFlWMSfVxCd7pJAj1PDKY+9hxbuu+m9ug0TMl/Rzs0pZSKKsck+t3ndrCk/wJiuidw5ZkB1HQ7TlUc1OTvgW++inZ4SikVNY5J9KnXpPLA0Pv4/aCfc0ESiGu4mm3d9lErP+DUH/4z2uEppVTUOCbRd4/tTvYd2fzi7xexYcgSGi/0pLs7nj/12M2uvyTRdHRPtENUSqmocEyiN8Zw4fRpxt44lt8/8DqVo9+nwd0HmqA44St2L3k72iEqpVRUhJToRWSciOwRkSoRWRBgeZyIrLaWfyoiN/osy7bm7xGRseELvaXGqir+esdoaqb9iL9b9QeWXf8k3cfW0dDYizPSwJbzV/HVn/4Yqd0rpZRtBU30IhILvAqMB24FfiQit/qtNhM4YYwZBPwG+JX12FuBacAQYBzwO2t7YRdzxRX0nfUYxjRxfNlyDj86iyk/zee+/X+iZ30Ch2Pr+ePGKprOuyOxe6WUsi0xQX49KiJpwEJjzFjrfjaAMebffdbZaK3zsYh0Aw4D1wALfNf1Xa+t/Y0cOdKUlpZ26p+6cOoU32zbxpmtH3Pmk485W/03/jzuHzh01Vl6XohDkE5tXymlIuGqczE89uv5l/RYEdlujBkZaFm3EB5/A3DA534tcEdb6xhjzotIPdDXmv+J32NvCBDgbGA2QP/+/UMIqX2xPXvSMyODnhkZALiPHqVXUSEF22po7OaYwxJKKYfp1nQhMtuNyFY7yBizHFgOnh59uLff/dprSZoylZlTwr1lpZSyv1C6tweBJJ/7ida8gOtYQze9gOMhPlYppVQEhZLotwHJIjJARHrgObia77dOPjDDms4CNhvP4H8+MM06K2cAkAyUhCd0pZRSoQg6dGONuc8BNgKxwGvGmEoRWQyUGmPygZXAmyJSBXyF58MAa701wC7gPPCkMSYyg1BKKaUCCnrWTVcLx1k3Sin1bdPeWTd6CopSSjmcJnqllHI4TfRKKeVwmuiVUsrhbHcwVkSOAX/rxCauBr4MUziRojGGh8YYHhpjeEQ7xu8ZY64JtMB2ib6zRKS0rSPPdqExhofGGB4aY3jYOUYdulFKKYfTRK+UUg7nxES/PNoBhEBjDA+NMTw0xvCwbYyOG6NXSinVkhN79EoppXxooldKKYdzTKIPdgHzLo6lRkR2iki5iJRa874jIptEZK/1t481X0Tkv6y4d4jIiAjF9JqIHBWRCp95HY5JRGZY6+8VkRmB9hXmGBeKyEGrLctFZILPsoAXno/ka0FEkkSkUER2iUiliMy15tumLduJ0W5tGS8iJSLymRXnImv+ABH51Nrnaqs8Ola589XW/E9F5MZg8UcwxlUi8oVPW6Za86Py3gnKGHPZ3/CUT94HDAR6AJ8Bt0Yxnhrgar95vwYWWNMLgF9Z0xOA9wABRgOfRigmFzACqLjUmIDvANXW3z7WdJ8Ix7gQ+GmAdW+1nuc4YID1/MdG+rUAXAeMsKZ7An+1YrFNW7YTo93aUoArrenuwKdWG60BplnzlwL/ZE3/M7DUmp4GrG4v/gjHuArICrB+VN47wW5O6dGPAqqMMdXGmEbgbSAzyjH5ywTesKbfAB7wmf8MgVzbAAADGklEQVR74/EJ0FtErgv3zo0xH+G5VkBnYhoLbDLGfGWMOQFsAsZFOMa2ZAJvG2POGWO+AKrwvA4i+lowxhwyxvzFmj4F7MZzHWTbtGU7MbYlWm1pjDGnrbvdrZsBMoB11nz/tmxu43XAPSIi7cQfyRjbEpX3TjBOSfSBLmDe3gs70gxQICLbxXPhc4DvGmMOWdOHge9a09GMvaMxRSvWOdbX4Neah0TsEKM1dDAcTy/Plm3pFyPYrC1FJFZEyoGjeJLfPuBrY8z5APv0xmMtrwf6RjpO/xiNMc1tmWu15W9EJM4/Rr9YopqjnJLo7eYHxpgRwHjgSRFx+S40nu9ytjqv1Y4xWZYANwGpwCHgP6IbjoeIXAnkAf9ijDnpu8wubRkgRtu1pTHmgjEmFc/1pEcBN0c5pFb8YxSRFCAbT6y34xmO+dcohhiUUxK9rS5Cbow5aP09CryD5wV8pHlIxvp71Fo9mrF3NKYuj9UYc8R6ozUBK7j4lTxqMYpIdzwJ9H+NMf9nzbZVWwaK0Y5t2cwY8zVQCKThGe5ovsyp7z698VjLewHHuypOnxjHWcNjxhhzDngdG7VlIE5J9KFcwLxLiMgVItKzeRoYA1TQ8gLqM4A/WNP5wMPW0frRQL3PEECkdTSmjcAYEeljfe0fY82LGL/jFQ/iacvmGANdeD6irwVrTHglsNsY87LPItu0ZVsx2rAtrxGR3tZ0AvBDPMcTCoEsazX/tmxu4yxgs/Xtqa34IxXj5z4f6oLnGIJvW9rivdNCVx31jfQNz9Huv+IZ4/tZFOMYiOcMgM+AyuZY8IwlfgjsBT4AvmMuHtV/1Yp7JzAyQnG9hefruhvP+ODMS4kJ+Ec8B7uqgEe7IMY3rRh24HkTXeez/s+sGPcA47vitQD8AM+wzA6g3LpNsFNbthOj3dpyKFBmxVMBPOfzHiqx2mUtEGfNj7fuV1nLBwaLP4IxbrbasgL4Hy6emROV906wm5ZAUEoph3PK0I1SSqk2aKJXSimH00SvlFIOp4leKaUcThO9Uko5nCZ6pZRyOE30SinlcP8PkXs+3nm1ky0AAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "plot_gradients(bad_trial)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The `VanishingGradient` rule provided by Tornasole alerts for this automatically."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAANEUlEQVR4nO3db6hc9Z3H8c8nmoKaPEg2bIhJ3LRFhLqwVoIsblwjtSXqgxjE0oBLdK97KzTQgA9WXLHiUpDFdtn4IHCLIWnMWgpajLXS2hA27gNLrpqNUTfRjdHmEhNjxBpQoua7D+akXPXOmZtzzswZ832/4DIz5ztnzpeTfO75d8/8HBECcPab0XYDAAaDsANJEHYgCcIOJEHYgSTOHeTCbHPqH+iziPBU02tt2W2vsL3P9uu276rzWQD6y1Wvs9s+R9J+Sd+WdEjSLkmrI+KVknnYsgN91o8t+xWSXo+IAxFxUtIvJK2s8XkA+qhO2BdK+uOk14eKaZ9he9T2uO3xGssCUFPfT9BFxJikMYndeKBNdbbsE5IWT3q9qJgGYAjVCfsuSRfb/qrtr0j6nqRtzbQFoGmVd+Mj4hPbayX9VtI5kjZGxMuNdQagUZUvvVVaGMfsQN/15Y9qAHx5EHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5BE5SGbMTgzZpT/Tr788su71m655Zam2zkjZct/8cUXS+ddtWpVaf3EiROVesqqVthtH5T0gaRPJX0SEUubaApA85rYsl8TEcca+BwAfcQxO5BE3bCHpN/Zft726FRvsD1qe9z2eM1lAaih7m78soiYsP2Xkp6x/b8RsXPyGyJiTNKYJNmOmssDUFGtLXtETBSPRyX9StIVTTQFoHmVw277AtuzTz+X9B1Je5tqDECzHFFtz9r219TZmkudw4H/jIgf95iH3fgKzjvvvNL6/v37u9YuvPDCWsu2XVqv+v9Hkj7++OPS+nXXXVda37FjR+Vln80iYsp/tMrH7BFxQNLfVO4IwEBx6Q1IgrADSRB2IAnCDiRB2IEkKl96q7QwLr31xcKFC7vW5syZ09dlr1+/vrR+9dVXd631unR27bXXVuopu26X3tiyA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASfJX0WWBiYqJSbTpuv/320vqyZctK68ePH+9au//++yv1hGrYsgNJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEtzPnlyvr2t+6qmnSuv79u0rrd9www1dawcOHCidF9VwPzuQHGEHkiDsQBKEHUiCsANJEHYgCcIOJMF19rPcokWLSuu9rqO/++67pfWbbrqptP7ee++V1tG8ytfZbW+0fdT23knT5tp+xvZrxWN/RyIAUNt0duM3SVrxuWl3SdoeERdL2l68BjDEeoY9InZK+vx3C62UtLl4vlnSjQ33BaBhVb+Dbn5EHC6evy1pfrc32h6VNFpxOQAaUvsLJyMiyk68RcSYpDGJE3RAm6peejtie4EkFY9Hm2sJQD9UDfs2SWuK52skPdFMOwD6peduvO1HJS2XNM/2IUk/kvSApF/aHpH0pqTv9rPJ7OwpL5v+2UUXXdS19uSTT5bOe+mll5bWr7rqqtI619G/PHqGPSJWdyl9q+FeAPQRfy4LJEHYgSQIO5AEYQeSIOxAEtzi+iUwMjJSWh8bG+vbsjdu3Fhr/q1bt3at7dy5s3TeU6dO1Vp2VnyVNJAcYQeSIOxAEoQdSIKwA0kQdiAJwg4kwXX2IXDNNdeU1p9++unS+syZM5ts5zNmzCjfHtS5Fn7rrbeW1rds2VL5szPjOjuQHGEHkiDsQBKEHUiCsANJEHYgCcIOJFF7RBjU9/7775fWd+3aVVq/5JJLutbK7idvwuLFi0vrq1at6lrrNZw0msWWHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeS4H521FI2XLQkPffcc11re/bsKZ13xYoVlXrKrvL97LY32j5qe++kaffZnrC9u/i5vslmATRvOrvxmyRN9Sv23yPisuLnN822BaBpPcMeETslHR9ALwD6qM4JurW29xS7+XO6vcn2qO1x2+M1lgWgpqph3yDp65Iuk3RY0k+6vTEixiJiaUQsrbgsAA2oFPaIOBIRn0bEKUk/k3RFs20BaFqlsNteMOnlKkl7u70XwHDoeT+77UclLZc0z/YhST+StNz2ZZJC0kFJ3+9jjxhic+Z0PV0jSTr//PO71t56662m20GJnmGPiNVTTH64D70A6CP+XBZIgrADSRB2IAnCDiRB2IEk+Cpp1DJv3rzS+uzZs7vW3njjjabbQQm27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBNfZUWru3Lml9fXr11f+7A8//LDyvDhzbNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAmGbEapK6+8srT+7LPPltbfeeedrrUlS5aUzvvRRx+V1jG1ykM2Azg7EHYgCcIOJEHYgSQIO5AEYQeSIOxAEtzPjlL33HNPrfkffPDBrjWuow9Wzy277cW2d9h+xfbLtn9YTJ9r+xnbrxWP5QN1A2jVdHbjP5F0Z0R8Q9LfSvqB7W9IukvS9oi4WNL24jWAIdUz7BFxOCJeKJ5/IOlVSQslrZS0uXjbZkk39qtJAPWd0TG77SWSvinpD5LmR8ThovS2pPld5hmVNFq9RQBNmPbZeNuzJD0maV1E/GlyLTp300x5k0tEjEXE0ohYWqtTALVMK+y2Z6oT9K0R8Xgx+YjtBUV9gaSj/WkRQBN67sbbtqSHJb0aET+dVNomaY2kB4rHJ/rSYQLnnlv+zzBjRvnv5JMnT1b+7DvuuKO0vmLFitJ6r2GXt2zZUlrH4EznmP3vJP2DpJds7y6m3a1OyH9pe0TSm5K+258WATShZ9gj4r8lTXkzvKRvNdsOgH7hz2WBJAg7kARhB5Ig7EAShB1Igltch8DNN99cWl++fHlp/aGHHupaW7duXem8t912W2n92LFjpfW1a9eW1o8cOVJax+CwZQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJBiyeQisXr26tP7II48MqJMv6nUdfcOGDQPqBNPFkM1AcoQdSIKwA0kQdiAJwg4kQdiBJAg7kATX2YfArFmzSuv33ntvaf3OO++svOxNmzaV1kdGRip/NtrBdXYgOcIOJEHYgSQIO5AEYQeSIOxAEoQdSKLndXbbiyX9XNJ8SSFpLCL+w/Z9kv5J0jvFW++OiN/0+CyuswN91u06+3TCvkDSgoh4wfZsSc9LulGd8dhPRMSD022CsAP91y3s0xmf/bCkw8XzD2y/Kmlhs+0B6LczOma3vUTSNyX9oZi01vYe2xttz+kyz6jtcdvjtToFUMu0/zbe9ixJ/yXpxxHxuO35ko6pcxz/r+rs6v9jj89gNx7os8rH7JJke6akX0v6bUT8dIr6Ekm/joi/7vE5hB3os8o3wti2pIclvTo56MWJu9NWSdpbt0kA/TOds/HLJD0r6SVJp4rJd0taLekydXbjD0r6fnEyr+yz2LIDfVZrN74phB3oP+5nB5Ij7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJNHzCycbdkzSm5NezyumDaNh7W1Y+5Loraome/urboWB3s/+hYXb4xGxtLUGSgxrb8Pal0RvVQ2qN3bjgSQIO5BE22Efa3n5ZYa1t2HtS6K3qgbSW6vH7AAGp+0tO4ABIexAEq2E3fYK2/tsv277rjZ66Mb2Qdsv2d7d9vh0xRh6R23vnTRtru1nbL9WPE45xl5Lvd1ne6JYd7ttX99Sb4tt77D9iu2Xbf+wmN7quivpayDrbeDH7LbPkbRf0rclHZK0S9LqiHhloI10YfugpKUR0fofYNj+e0knJP389NBatv9N0vGIeKD4RTknIv55SHq7T2c4jHefeus2zPitanHdNTn8eRVtbNmvkPR6RByIiJOSfiFpZQt9DL2I2Cnp+Ocmr5S0uXi+WZ3/LAPXpbehEBGHI+KF4vkHkk4PM97quivpayDaCPtCSX+c9PqQhmu895D0O9vP2x5tu5kpzJ80zNbbkua32cwUeg7jPUifG2Z8aNZdleHP6+IE3Rcti4jLJV0n6QfF7upQis4x2DBdO90g6evqjAF4WNJP2mymGGb8MUnrIuJPk2ttrrsp+hrIemsj7BOSFk96vaiYNhQiYqJ4PCrpV+ocdgyTI6dH0C0ej7bcz59FxJGI+DQiTkn6mVpcd8Uw449J2hoRjxeTW193U/U1qPXWRth3SbrY9ldtf0XS9yRta6GPL7B9QXHiRLYvkPQdDd9Q1NskrSmer5H0RIu9fMawDOPdbZhxtbzuWh/+PCIG/iPpenXOyP+fpH9po4cufX1N0v8UPy+33ZukR9XZrftYnXMbI5L+QtJ2Sa9J+r2kuUPU2xZ1hvbeo06wFrTU2zJ1dtH3SNpd/Fzf9ror6Wsg640/lwWS4AQdkARhB5Ig7EAShB1IgrADSRB2IAnCDiTx/1ppMKNb4Fc+AAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "input_image = (bad_trial.tensor('Net_input_0').value(2700)[42]*255).reshape(28,28)\n",
- "plt.imshow(input_image, cmap=plt.get_cmap('gray'))\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAPCElEQVR4nO3dTYilV53H8e+v0zpaMWMLaRiSTncFIkpo4kQKiQYcsLOIbUjDrCKlELOoGfAlI0IwUwtXPZuIKEQMRdSNhS4yCSPS0e6gDswiwcoLMUnHENTuvInlQkesRabxP4t7Q6rbe6u7+rlVT+We7weKW8+5p+7510PXL0/OPfc8qSokSdNvV98FSJK2h4EvSY0w8CWpEQa+JDXCwJekRuzuu4CNXH755TU7O9t3GZL0lvH444//oar2jnpuRwf+7OwsKysrfZchSW8ZSU6Ne84pHUlqhIEvSY0w8CWpEQa+JDXCwJekRnQK/CT3JHk+ydNJHkqyZ0y/m5P8KsmLSb7cZcy3iuVlmJ2FXbsGj8vLfVckqXVdr/BPAAer6jrgBeDuczskuQT4JvBx4Frgk0mu7Tjujra8DAsLcOoUVA0eFxYMfUn96hT4VXW8qs4MDx8F9o3o9iHgxar6dVW9DvwAONJl3J1ucRHW1s5uW1sbtEtSXyY5h38H8PCI9iuBl9YdvzxsGynJQpKVJCurq6sTLG/7nD69uXZJ2g7nDfwkjyR5ZsTXkXV9FoEzQOdJi6paqqq5qprbu3fkp4N3vP37N9cuSdvhvFsrVNVNGz2f5HbgFuBQjb591ivAVeuO9w3bptbRo4M5+/XTOjMzg3ZJ6kvXVTo3A3cBt1bV2phuvwDem+TqJG8HbgN+2GXcnW5+HpaW4MABSAaPS0uDdknqS9fN0+4F/g44kQTg0ar61yRXAPdX1eGqOpPkc8BPgEuA71TVsx3H3fHm5w14STtLp8CvqmvGtL8KHF53fAw41mUsSVI3ftJWkhph4EtSIwx8SWqEgS9JjTDwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL6lXy8swOwu7dg0el5f7rmh6dbqJuSR1sbwMCwuwtjY4PnVqcAwwP99fXdPKK3xJvVlcfDPs37C2NmjX5Bn4knpz+vTm2tWNgS+pN/v3b65d3Rj4knpz9CjMzJzdNjMzaNfkGfiSejM/D0tLcOAAJIPHpSXfsN0qrtKR1Kv5eQN+u3iFL0mNMPAlqREGviQ1wsCXpEYY+JLUCANfUvNa2cDNZZmSmtbSBm5e4UtqWksbuHUK/CT3JHk+ydNJHkqyZ0Sfq5L8LMlzSZ5NcmeXMSVpklrawK3rFf4J4GBVXQe8ANw9os8Z4EtVdS1wA/DZJNd2HFeSJqKlDdw6BX5VHa+qM8PDR4F9I/q8VlVPDL//M3ASuLLLuJI0KS1t4DbJOfw7gIc36pBkFrgeeGyDPgtJVpKsrK6uTrA8ta6VlRjanJY2cEtVbdwheQT4hxFPLVbVfw37LAJzwD/XmBdM8i7gv4GjVfXghRQ3NzdXKysrF9JV2tC5KzFgcBU3rX/YaleSx6tqbuRz5wv8C3jx24F/AQ5V1dqYPm8DfgT8pKq+dqGvbeBrUmZnB8vtznXgAPz2t9tdjbR1Ngr8Tuvwk9wM3AX80wZhH+DbwMnNhL00SS2txJDG6TqHfy9wGXAiyVNJ7gNIckWSY8M+NwKfBj427PNUksMdx5U2paWVGNI4na7wq+qaMe2vAoeH3/8PkC7jSF0dPTp6Dn8aV2JI4/hJWzWhpZUY0jjupaNmeCs9tc4rfElqhIEvSY0w8CWpEQa+JDXCwJekRhj42nJuWibtDC7L1JZq6fZx0k7nFb62VEu3j5N2OgNfW8pNy6Sdw8DXlnLTMmnnMPC1pVq6fZzU1VYvcDDwtaXctEy6MG8scDh1CqreXOAwydDvfMerreQdryS1YlJ3Zdvojlde4UvSDrAdCxwMfEnaAbZjgYOBL0k7wHYscDDwJWkH2I4FDm6tIEk7xFbflc0rfElqhIEvSY0w8CWpEQb+FHMfeknr+abtlHIfeknn8gp/SrkPvaRzGfhTyn3oJZ3LwJ9S7kMv6VwG/pRyH3pJ5zLwp5T70Es6l6t0pthWf0xb0luLV/iS1AgDX5IaYeBLUiMMfElqRKfAT3JPkueTPJ3koSR7Nuh7SZInk/yoy5iSpIvT9Qr/BHCwqq4DXgDu3qDvncDJjuNJki5Sp8CvquNVdWZ4+Ciwb1S/JPuATwD3dxlPknTxJjmHfwfw8Jjnvg7cBfz1fC+SZCHJSpKV1dXVCZYnSW07b+AneSTJMyO+jqzrswicAf5mx/UktwC/r6rHL6Sgqlqqqrmqmtu7d+8mfhVJ0kbO+0nbqrppo+eT3A7cAhyqqhrR5Ubg1iSHgXcAf5/ke1X1qYuoV5J0kbqu0rmZwVTNrVW1NqpPVd1dVfuqaha4DfipYS9J26/rHP69wGXAiSRPJbkPIMkVSY51rk6SNDGdNk+rqmvGtL8KHB7R/nPg513GlCRdHD9pK0mNMPAlqREGviQ1wsCXpEYY+JLUCANf2mbLyzA7C7t2DR6X/+bz6dLW8J620jZaXoaFBVgbfkzx1KnBMXj/YW09r/ClbbS4+GbYv2FtbdAubTUDX9pGp09vrl2aJANf2kb792+uXZokA1/aRkePwszM2W0zM4N2aatNXeC7AkI72fw8LC3BgQOQDB6XlnzDVttjqlbpuAJCbwXz8/57VD+m6grfFRCSNN5UBb4rICRpvKkKfFdASNJ4UxX4roCQpPGmKvBdASFJ403VKh1wBYQkjTNVV/iSpPEMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY+JLUCANfkhrRKfCT3JPk+SRPJ3koyZ4x/fYkeWDY92SSD3cZV5K0eV2v8E8AB6vqOuAF4O4x/b4B/Liq3g98ADjZcVxJ0iZ1CvyqOl5VZ4aHjwL7zu2T5N3AR4FvD3/m9ar6Y5dxJUmbN8k5/DuAh0e0Xw2sAt9N8mSS+5NcOsFxJUkX4LyBn+SRJM+M+Dqyrs8icAZYHvESu4EPAt+qquuBvwBf3mC8hSQrSVZWV1c3/QtJkkbbfb4OVXXTRs8nuR24BThUVTWiy8vAy1X12PD4ATYI/KpaApYA5ubmRr2eJOkidF2lczNwF3BrVa2N6lNVvwNeSvK+YdMh4Lku40qSNq/rHP69wGXAiSRPJbkPIMkVSY6t6/d5YDnJ08A/Av/RcVxJ0iadd0pnI1V1zZj2V4HD646fAua6jCVJ6sZP2kpSIwx8SWqEgS9JjTDwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY+JLUCANfkhph4EtSIwx8SWqEgS9JjTDwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY3oFPhJ7knyfJKnkzyUZM+Yfl9M8mySZ5J8P8k7uowrSdq8rlf4J4CDVXUd8AJw97kdklwJfAGYq6qDwCXAbR3HlSRtUqfAr6rjVXVmePgosG9M193AO5PsBmaAV7uMK0navEnO4d8BPHxuY1W9AnwVOA28Bvypqo5PcFxJ0gU4b+AneWQ4937u15F1fRaBM8DyiJ9/D3AEuBq4Arg0yac2GG8hyUqSldXV1Yv5nSRJI+w+X4eqummj55PcDtwCHKqqGtHlJuA3VbU67P8g8BHge2PGWwKWAObm5ka9niTpInRdpXMzcBdwa1Wtjel2GrghyUySAIeAk13GlSRtXtc5/HuBy4ATSZ5Kch9AkiuSHAOoqseAB4AngF8Ox1zqOK4kaZPOO6Wzkaq6Zkz7q8DhdcdfAb7SZSxJUjd+0laSGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY+JLUCANfkhph4EtSIwx8SWqEgS9JjTDwJakRBr7UqOVlmJ2FXbsGj8t/c786TZtO2yNLemtaXoaFBVgb3rbo1KnBMcD8fH91aWt5hS81aHHxzbB/w9raoF3Ty8CXGnT69ObaNR0MfKlB+/dvrl3TwcCXGnT0KMzMnN02MzNo1/Qy8KUGzc/D0hIcOADJ4HFpyTdsp52rdKRGzc8b8K3xCl+SGmHgS1IjDHxJaoSBL0mNMPAlqRGpqr5rGCvJKnDqIn/8cuAPEyznrcxzcTbPx9k8H2+ahnNxoKr2jnpiRwd+F0lWqmqu7zp2As/F2TwfZ/N8vGnaz4VTOpLUCANfkhoxzYG/1HcBO4jn4myej7N5Pt401ediaufwJUlnm+YrfEnSOga+JDVi6gI/yc1JfpXkxSRf7ruePiW5KsnPkjyX5Nkkd/ZdU9+SXJLkySQ/6ruWviXZk+SBJM8nOZnkw33X1KckXxz+nTyT5PtJ3tF3TZM2VYGf5BLgm8DHgWuBTya5tt+qenUG+FJVXQvcAHy28fMBcCdwsu8idohvAD+uqvcDH6Dh85LkSuALwFxVHQQuAW7rt6rJm6rABz4EvFhVv66q14EfAEd6rqk3VfVaVT0x/P7PDP6gr+y3qv4k2Qd8Ari/71r6luTdwEeBbwNU1etV9cd+q+rdbuCdSXYDM8CrPdczcdMW+FcCL607fpmGA269JLPA9cBj/VbSq68DdwF/7buQHeBqYBX47nCK6/4kl/ZdVF+q6hXgq8Bp4DXgT1V1vN+qJm/aAl8jJHkX8J/Av1XV//ZdTx+S3AL8vqoe77uWHWI38EHgW1V1PfAXoNn3vJK8h8FswNXAFcClST7Vb1WTN22B/wpw1brjfcO2ZiV5G4OwX66qB/uup0c3Arcm+S2Dqb6PJflevyX16mXg5ap64//4HmDwH4BW3QT8pqpWq+r/gAeBj/Rc08RNW+D/AnhvkquTvJ3Bmy4/7Lmm3iQJgznak1X1tb7r6VNV3V1V+6pqlsG/i59W1dRdwV2oqvod8FKS9w2bDgHP9VhS304DNySZGf7dHGIK38SeqpuYV9WZJJ8DfsLgXfbvVNWzPZfVpxuBTwO/TPLUsO3fq+pYjzVp5/g8sDy8OPo18Jme6+lNVT2W5AHgCQar255kCrdZcGsFSWrEtE3pSJLGMPAlqREGviQ1wsCXpEYY+JLUCANfkhph4EtSI/4fCqIpvyP0+eIAAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The network predicted the value: 1\n"
- ]
- }
- ],
- "source": [
- "plt.plot(bad_trial.tensor('Net_output0').value(2700)[42], 'bo')\n",
- "plt.show()\n",
- "print('The network predicted the value: {}'.format(np.argmax(bad_trial.tensor('Net_output0').value(2700)[42])))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This concludes this notebook. For more information see the APIs at\n",
- "- https://github.com/awslabs/tornasole_core"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.7.3"
- },
- "pycharm": {
- "stem_cell": {
- "cell_type": "raw",
- "source": [],
- "metadata": {
- "collapsed": false
- }
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/examples/pytorch/sagemaker-notebooks/pytorch.ipynb b/examples/pytorch/sagemaker-notebooks/pytorch.ipynb
deleted file mode 100644
index 4a40d7a9a..000000000
--- a/examples/pytorch/sagemaker-notebooks/pytorch.ipynb
+++ /dev/null
@@ -1,449 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Debugging SageMaker Training Jobs with Tornasole"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Overview"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Tornasole is a new capability of Amazon SageMaker that allows debugging machine learning training. \n",
- "It lets you go beyond just looking at scalars like losses and accuracies during training and gives \n",
- "you full visibility into all tensors 'flowing through the graph' during training.\n",
- "\n",
- "Using Tornasole is a two step process: Saving tensors and Analysis. Let's look at each one of them closely.\n",
- "\n",
- "### Saving tensors\n",
- "\n",
- "Tensors define the state of the training job at any particular instant in its lifecycle. Tornasole exposes a library which allows you to capture these tensors and save them for analysis. Tornasole is highly customizable to save the tesnsors you want at different frequencies. Refer [DeveloperGuide_PyTorch](../../DeveloperGuide_PyTorch.md) for details on how to save the tensors you want to save.\n",
- "\n",
- "### Analysis\n",
- "\n",
- "Analyses of the tensors emitted is captured by the Tornasole concept called ***Rules***. On a very broad level, \n",
- "A Rule is a python code used to detect certain conditions during training. Some of the conditions that a data scientist training a deep learning model may care about are monitoring for gradients getting too large or too small, detecting overfitting, and so on.\n",
- "Tornasole will come pre-packaged with certain rules. Users can write their own rules using the Tornasole APIs.\n",
- "You can also analyze raw tensor data outside of the Rules construct in say, a Sagemaker notebook, using Tornasole's full set of APIs. \n",
- "Please refer [DeveloperGuide_Rules](../../../rules/DeveloperGuide_Rules.md) for more details about analysis.\n",
- "\n",
- "This example guides you through installation of the required components for emitting tensors in a \n",
- "SageMaker training job and applying a rule over the tensors to monitor the live status of the job. \n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup\n",
- "\n",
- "As a first step, we'll do the installation of required tools which will allow emission of tensors (saving tensors) and application of rules to analyze them"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "!aws s3 sync s3://tornasole-external-preview-use1/ ~/tornasole-preview\n",
- "!pip -q install ~/tornasole-preview/sdk/sagemaker-tornasole-latest.tar.gz\n",
- "!aws configure add-model --service-model file://`echo ~/tornasole-preview/sdk/sagemaker-smdebug.json` --service-name sagemaker"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now that we've completed the setup, we're ready to spin off a training job with debugging enabled. \n",
- "\n",
- "## Enable Tornasole in the training script\n",
- "\n",
- "### Import the hook package\n",
- "Import the SessionHook class along with other helper classes in your training script as shown below\n",
- "\n",
- "```\n",
- "from smdebug.pytorch import SaveConfig, SessionHook\n",
- "```\n",
- "\n",
- "### Instantiate and initialize hook\n",
- "\n",
- "```\n",
- " # Create SaveConfig that instructs engine to log graph tensors every 10 steps.\n",
- " save_config = SaveConfig(save_interval=10)\n",
- " \n",
- " # Create a hook that logs tensors of weights, biases and gradients while training the model.\n",
- " \n",
- " hook = SessionHook(save_config=save_config)\n",
- "```\n",
- "\n",
- "For additional details on SessionHook, SaveConfig and Collection please refer to the [API documentation](api.md)\n",
- "\n",
- "### Register Tornasole hook to the model before starting of the training.\n",
- "\n",
- "\n",
- "After creating or loading your desired model, you can register the hook with the model as shown below.\n",
- "\n",
- "```\n",
- "net = create_model()\n",
- "# Apply hook to the model\n",
- "# and enable mode in which engine will log graph tensors\n",
- "hook.register_hook(net)\n",
- "```\n",
- "\n",
- "#### Set the mode\n",
- "Tornasole has the concept of modes (TRAIN, EVAL, PREDICT) to separate out different modes of the jobs. Set the mode you are running in your job. Every time the mode changes in your job, please set the current mode. This helps you group steps by mode, for easier analysis. Setting the mode is optional but recommended. If you do not specify this, Tornasole saves all steps under a GLOBAL mode.\n",
- "\n",
- "\n",
- "```\n",
- "hook.set_mode(smd.modes.TRAIN)\n",
- "```\n",
- "\n",
- "Refer [DeveloperGuide_PyTorch.md](../../DeveloperGuide_TF.md) for more details on the APIs Tornasole provides to help you save the tensors in different forms at the frequency you want.\n",
- "\n",
- "#### Note\n",
- "Tornasole currently only works for single process training. We will support distributed training very soon. \n",
- "\n",
- "## Start Sagemaker training with Tornasole enabled\n",
- "\n",
- "We'll be training a simple Pytorch model using the script [simple.py](../scripts/simple.py).\n",
- "This will be done using SageMaker Pytorch 1.13.1 Container in Script Mode.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import sagemaker\n",
- "from sagemaker.pytorch import PyTorch"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "### Inputs\n",
- "Configuring the inputs for the training job. The command line arguments taken by the script\n",
- "can be passed using the hyperparameters dictionary below.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "entry_point_script = '../scripts/simple.py'\n",
- "docker_image_name= '072677473360.dkr.ecr.us-west-2.amazonaws.com/tornasole-preprod-pytorch-1.1.0-cpu:latest'\n",
- "hyperparameters = {'epochs': 2, 'lr' : 0.01, 'momentum' : 0.9, 'save-frequency' : 3, 'steps' : 10, 'hook-type' : 'saveall', 'random-seed' : True }\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "#### Storage\n",
- "The tensors saved by Tornasole are, by default, stored in the S3 output path of the training job, \n",
- "under the folder **`/tensors-`**. This is done to ensure that we don't end up accidentally \n",
- "overwriting the tensors from a training job with the others. Rules evaluation require separation of \n",
- "the tensors paths to be evaluated correctly.\n",
- "\n",
- "If you don't provide an S3 output path to the estimator, SageMaker creates one for you as:\n",
- "**`s3://sagemaker--/`**\n",
- "\n",
- "\n",
- "This path is used to create a Tornasole Trial taken by Rules (see below). \n",
- "\n",
- "#### New Parameters\n",
- "The new parameters in Sagemaker Estimator to look out for are\n",
- "\n",
- "##### `debug` (bool)\n",
- "This indicates that debugging should be enabled for the training job. \n",
- "Setting this as `True` would make Tornasole available for use with the job\n",
- "\n",
- "##### `rules_specification` (list[*dict*])\n",
- "This is a list of python dictionaries, where each `dict` is of the following form:\n",
- "```\n",
- "{\n",
- " \"RuleName\": # The name of the class implementing the Tornasole Rule interface. (required)\n",
- " \"SourceS3Uri\": # S3 URI of the rule script containing the class in 'RuleName'. \n",
- " If left empty, it would look for the class in one of the First Party rules already provided to you by Amazon. \n",
- " If not, SageMaker will try to look for the rule class in the script\n",
- " \"InstanceType\": # The ml instance type in which the rule evaluation should run\n",
- " \"VolumeSizeInGB\": # The volume size to store the runtime artifacts from the rule evaluation\n",
- " \"RuntimeConfigurations\": {\n",
- " # Map defining the parameters required to instantiate the Rule class and\n",
- " # parameters regarding invokation of the rule (start-step and end-step)\n",
- " # This can be any parameter taken by the rule\n",
- " : \n",
- " }\n",
- "}\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Rules\n",
- "Rules are the medium by which Tornasole executes a certain piece of code regularly on different steps of the job.\n",
- "They can be used to assert certain conditions during training, and raise Cloudwatch Events based on them that you can\n",
- "use to process in any way you like. \n",
- "\n",
- "A Trial in Tornasole's context\n",
- "refers to a training job. It is identified by the path where the saved tensors for the job are stored. \n",
- "A rule takes a `base_trial` which refers to the job whose run invokes the rule execution.\n",
- "A rule can optionally look at other jobs as well, passed using the ar `other_trials`. \n",
- "\n",
- "Tornasole comes with a set of first party rules (1P rules).\n",
- "You can also write your own rules looking at these 1P rules for inspiration. \n",
- "Refer [DeveloperGuide_Rules.md](../../../rules/DeveloperGuide_Rules.md) for more.\n",
- " \n",
- "Here we will talk about how to use Sagemaker to evalute these rules on the training jobs.\n",
- "##### 1P Rule \n",
- "If you want to use a 1P rule. Specify the RuleName field with the 1P RuleName, \n",
- "and the rule will be automatically applied. You can pass any parameters accepted by the \n",
- "rule as part of the RuntimeConfigurations dictionary. The arguments `base_trial` (and `other_trials` if \n",
- "taken by the rule) can be passed as the S3 path where the tensors for \n",
- "the trial are stored in the RuntimeConfigurations dictionary above.\n",
- "\n",
- "Here's a example of a complex configuration for the SimilarAcrossRuns (which accepts another trial and a regex pattern) \n",
- "where we ask for the rule to be invoked for the steps between 10 and 100.\n",
- "\n",
- "``` \n",
- "rules_specification = [\n",
- " {\n",
- " \"RuleName\": \"SimilarAcrossRuns\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"other_trials\": \"s3://sagemaker--/past-job\",\n",
- " \"include_regex\": \".*\",\n",
- " \"start-step\": \"10\",\n",
- " \"end-step\": \"100\"\n",
- " }\n",
- " }\n",
- "]\n",
- "```\n",
- "\n",
- "##### Custom rule\n",
- "In this case you need to define a custom rule class which inherits from `smdebug.rules.Rule` class.\n",
- "You need to provide Sagemaker the S3 location of the file which defines your custom rule classes as the value for the field `SourceS3Uri`.\n",
- "Again, you can pass any arguments taken by this rule through the RuntimeConfigurations dictionary. \n",
- "Note that the custom rules can only have arguments which expect a string as the value except the two arguments \n",
- "specifying trials to the Rule. Refer [DeveloperGuide_Rules.md](../../../rules/DeveloperGuide_Rules.md) for more.\n",
- "\n",
- "Here's an example:\n",
- "```\n",
- "rules_specification = [\n",
- " {\n",
- " \"RuleName\": \"CustomRule\",\n",
- " \"SourceS3Uri\": \"s3://weiyou-tornasole-test/rule-script/custom_rule.py\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"threshold\" : \"0.5\"\n",
- " }\n",
- " }\n",
- "]\n",
- "```\n",
- "\n",
- "### Estimator\n",
- "Now we'll call the Sagemaker Pytorch Estimator to kick off a training job along with a rule to monitor the job.\n",
- "\n",
- "For the purposes of this demonstration let us use the simple.py script with the above hyperparameters dictionary.\n",
- "These good hyperparameters do not produce vanishing gradients, so you will see that the rule doesn't get fired.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Training Example Without Vanishing Gradients "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [
- {
- "ename": "NameError",
- "evalue": "name 'sagemaker' is not defined",
- "output_type": "error",
- "traceback": [
- "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
- "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0msagemaker_execution_role\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msagemaker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_execution_role\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0;31m#sagemaker_execution_role = 'AmazonSageMaker-ExecutionRole-20190614T145575'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m estimator = PyTorch(role=sagemaker_execution_role,\n\u001b[1;32m 4\u001b[0m \u001b[0mbase_job_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'pytorch-good-example'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mtrain_instance_count\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;31mNameError\u001b[0m: name 'sagemaker' is not defined"
- ]
- }
- ],
- "source": [
- "sagemaker_execution_role = sagemaker.get_execution_role()\n",
- "#sagemaker_execution_role = 'AmazonSageMaker-ExecutionRole-20190614T145575'\n",
- "estimator = PyTorch(role=sagemaker_execution_role,\n",
- " base_job_name='pytorch-good-example',\n",
- " train_instance_count=1,\n",
- " train_instance_type='ml.m4.xlarge',\n",
- " image_name=docker_image_name,\n",
- " entry_point=entry_point_script,\n",
- " framework_version='1.1.0',\n",
- " hyperparameters=hyperparameters,\n",
- " py_version='py3',\n",
- " debug=True,\n",
- " rules_specification=[\n",
- " {\n",
- " \"RuleName\": \"VanishingGradient\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"start-step\": \"1\",\n",
- " \"end-step\": \"50\"\n",
- " }\n",
- " }\n",
- " ])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator.fit()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Result\n",
- "\n",
- "As a result of the above command, SageMaker will spin off 2 training jobs for you - the first one being the job which produces the tensors to be analyzed and the second one, which evaluates or analyzes the rule you asked it to in `rules_specification`\n",
- "\n",
- "### Check the status of the Rule Execution Job\n",
- "To get the rule execution job that SageMaker started for you, run the command below and it shows you the `RuleName`, `RuleStatus`, `FailureReason` if any, and `RuleExecutionJobArn`. If the tensors meets a rule evaluation condition, the rule execution job throws a client error with `FailureReason: RuleEvaluationConditionMet`. You can check the Cloudwatch Logstream `/aws/sagemaker/TrainingJobs` with `RuleExecutionJobArn`"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator.describe_rule_execution_jobs()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Receive CloudWatch Event For your Jobs\n",
- "When the status of training job or rule execution job change (i.e. starting, failed), TrainingJobStatus CloudWatch events are emitted : https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html. You can configure a CW event rule to receive and process these events by setting up a target (Lambda function, SNS). \n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Training Example With Vanishing Gradients \n",
- "\n",
- "Now let us change the hyperparameters dictionary to the below bad set of hyperparameters, which produce vanishing gradients \n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "entry_point_script = '../scripts/simple.py'\n",
- "docker_image_name= '072677473360.dkr.ecr.us-west-2.amazonaws.com/tornasole-preprod-pytorch-1.1.0-cpu:latest'\n",
- "bad_hyperparameters = {'epochs': 2, 'lr' : 1.0, 'momentum' : 0.9, 'save-frequency' : 3, 'steps' : 10, 'hook-type' : 'saveall', 'random-seed' : True }\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "sagemaker_execution_role = sagemaker.get_execution_role()\n",
- "#sagemaker_execution_role = 'AmazonSageMaker-ExecutionRole-20190614T145575'\n",
- "estimator = PyTorch(role=sagemaker_execution_role,\n",
- " base_job_name='pytorch-bad-example',\n",
- " train_instance_count=1,\n",
- " train_instance_type='ml.m4.xlarge',\n",
- " image_name=docker_image_name,\n",
- " entry_point=entry_point_script,\n",
- " framework_version='1.1.0',\n",
- " hyperparameters=bad_hyperparameters,\n",
- " py_version='py3',\n",
- " debug=True,\n",
- " rules_specification=[\n",
- " {\n",
- " \"RuleName\": \"VanishingGradient\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"start-step\": \"1\",\n",
- " \"end-step\": \"10\"\n",
- " }\n",
- " }\n",
- " ])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator.fit()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator.describe_rule_execution_jobs()"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.7.2"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/examples/rules/sagemaker-notebooks/BringYourOwnRule.ipynb b/examples/rules/sagemaker-notebooks/BringYourOwnRule.ipynb
deleted file mode 100644
index e84a64901..000000000
--- a/examples/rules/sagemaker-notebooks/BringYourOwnRule.ipynb
+++ /dev/null
@@ -1,630 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Debugging SageMaker Training Jobs with Tornasole \n",
- "## Writing Custom Rules"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Overview\n",
- "Tornasole is a new capability of Amazon SageMaker that allows debugging machine learning training. \n",
- "It lets you go beyond just looking at scalars like losses and accuracies during training and gives \n",
- "you full visibility into all tensors 'flowing through the graph' during training. Tornasole helps you to monitor your training in near real time using rules and would provide you alerts, once it has detected inconsistency in training flow. \n",
- "\n",
- "Using Tornasole is a two step process: Saving tensors and Analysis. Let's look at each one of them closely.\n",
- "\n",
- "### Saving tensors\n",
- "Tensors define the state of the training job at any particular instant in its lifecycle. Tornasole exposes a library which allows you to capture these tensors and save them for analysis. Tornasole is highly customizable to save the tensors you want at different frequencies. Refer [DeveloperGuide_TensorFlow](../../DeveloperGuide_TF.md) for details on how to save the tensors you want to save.\n",
- "\n",
- "### Analysis\n",
- "\n",
- "Analysis of the tensors emitted is captured by the Tornasole concept called ***Rules***. On a very broad level, \n",
- "a rule is a python code used to detect certain conditions during training. Some of the conditions that a data scientist training a deep learning model may care about are monitoring for gradients getting too large or too small, detecting overfitting, and so on.\n",
- "Tornasole will come pre-packaged with certain rules. Users can write their own rules using the Tornasole APIs.\n",
- "You can also analyze raw tensor data outside of the Rules construct in say, a Sagemaker notebook, using Tornasole's full set of APIs. \n",
- "Please refer [Developer Guide for Rules](../../../../rules/DeveloperGuide_Rules.md) for more details about analysis.\n",
- "\n",
- "This example guides you through installation of the required components for emitting tensors in a \n",
- "SageMaker training job and applying a rule over the tensors to monitor the live status of the job. \n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup\n",
- "\n",
- "As a first step, we'll do the installation of required tools which will allow emission of tensors (saving tensors) and application of rules to analyze them. This is only for the purposes of this private beta. Once we do this, we will be ready to use smdebug."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "scrolled": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Installing requirements...\n",
- "\u001b[33mYou are using pip version 10.0.1, however version 19.2.3 is available.\n",
- "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
- "Installation completed!\n"
- ]
- }
- ],
- "source": [
- "! aws s3 sync s3://tornasole-external-preview-use1/sdk/ ~/SageMaker/tornasole-preview-sdk/\n",
- "! chmod +x ~/SageMaker/tornasole-preview-sdk/installer.sh && ~/SageMaker/tornasole-preview-sdk/installer.sh"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "## Using custom Tornasole rules with SageMaker \n",
- "\n",
- "This notebook assumes that you have gone through at least one notebook demonstrating training models\n",
- "in SageMaker with Tornasole with your framework of choice. That notebook would demonstrate the \n",
- "changes you need to make in your training script to enable Tornasole, starting a training job \n",
- "along with a rule execution job, and looking at the status of these jobs.\n",
- "\n",
- "In this notebook we will focus on how to write a custom Tornasole rule, and how to \n",
- "execute this custom rule in SageMaker. To make this notebook runnable, we are picking a TensorFlow script as the training job.\n",
- "Whatever framework or script you use, rule behavior would be similar. \n",
- "\n",
- "### Start training with a custom rule\n",
- "\n",
- "#### Configuring the inputs for the training job\n",
- "Set the docker image to the SageMaker TensorFlow container that we have built with Tornasole pre-installed, for the region you are in. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {
- "pycharm": {
- "name": "#%%\n"
- }
- },
- "outputs": [],
- "source": [
- "import sagemaker\n",
- "import boto3\n",
- "from sagemaker.tensorflow import TensorFlow\n",
- "\n",
- "REGION = boto3.Session().region_name\n",
- "TAG='latest'\n",
- "\n",
- "docker_image_name= '072677473360.dkr.ecr.{}.amazonaws.com/tornasole-preprod-tf-1.13.1-cpu:{}'.format(REGION, TAG)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "Let us now set `entry_point_script` to the simple TensorFlow training script that has SessionHook integrated.\n",
- "The 'hyperparameters' below are the parameters that will be passed to the training script as command line arguments in SageMaker's script mode."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "pycharm": {
- "name": "#%%\n"
- }
- },
- "outputs": [],
- "source": [
- "entry_point_script = '../../frameworks/tensorflow/examples/scripts/simple.py'\n",
- "hyperparameters = { 'steps': 1000000, 'save_frequency': 50 }"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "#### Configuring custom rule\n",
- "We have written an example custom rule `CustomGradientRule`, available [here](../scripts/my_custom_rule.py). We need to upload this to a bucket in the same region where we want to run the job. We have chosen a default bucket below. Please change it to the bucket you want. We will now create this bucket if it does not exist, and upload this file. \n",
- "We will then specify this path when starting the job as `SourceS3Uri`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {},
- "outputs": [],
- "source": [
- "ACCOUNT_ID = boto3.client('sts').get_caller_identity().get('Account')\n",
- "BUCKET = f'tornasole-resources-{ACCOUNT_ID}-{REGION}'\n",
- "\n",
- "CUSTOM_RULE_PATH = '../scripts/my_custom_rule.py'\n",
- "\n",
- "PREFIX = os.path.join('rules', os.path.basename(CUSTOM_RULE_PATH))\n",
- "\n",
- "import os\n",
- "s3 = boto3.resource('s3')\n",
- "bucket = s3.Bucket(BUCKET)\n",
- "if not bucket.creation_date:\n",
- " s3.create_bucket(Bucket=BUCKET, CreateBucketConfiguration={'LocationConstraint': REGION})\n",
- "s3.Object(BUCKET, PREFIX).put(Body=open(CUSTOM_RULE_PATH, 'rb'))\n",
- "SOURCE_S3_URI = f's3://{BUCKET}/{PREFIX}'"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Keep in mind that for SageMaker to be able to evaluate your rule, the rule class **will need** to have a signature conforming to the spec defined by smdebug. \n",
- "\n",
- "This custom rule that we have written takes the arguments `self`, `base_trial` and `threshold`. \n",
- "In order to initialize a custom rule class, you'll need to pass down values for everything except `self` and `base_trial`. \n",
- "This is done through putting the parameters and their values as a string-to-string map in `RuntimeConfigurations` in the `rules_specification` parameter to the SageMaker Estimator.\n",
- "\n",
- "After we run this example, in this notebook we will look at these concepts in more detail."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {
- "pycharm": {
- "name": "#%%\n"
- }
- },
- "outputs": [],
- "source": [
- "estimator = TensorFlow(role=sagemaker.get_execution_role(),\n",
- " base_job_name='tensorflow-custom-rule-tornasole',\n",
- " train_instance_count=1,\n",
- " train_instance_type='ml.m4.xlarge',\n",
- " image_name=docker_image_name,\n",
- " entry_point=entry_point_script,\n",
- " hyperparameters=hyperparameters,\n",
- " framework_version='1.13.1',\n",
- " debug=True,\n",
- " py_version='py3',\n",
- " rules_specification=[\n",
- " {\n",
- " \"RuleName\": \"CustomGradientRule\",\n",
- " \"SourceS3Uri\": SOURCE_S3_URI,\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"RuntimeConfigurations\": {\n",
- " \"threshold\" : \"0.5\"\n",
- " }\n",
- " }\n",
- " ])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "To kick off the job, we call the `fit()` method on the SageMaker TensorFlow estimator"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {
- "pycharm": {
- "name": "#%%\n"
- }
- },
- "outputs": [],
- "source": [
- "# setting wait as True will cause the logs to be streamed in the notebook directly,\n",
- "# in order to proceed to further cells you'll need to stop cell execution. So, \n",
- "# we set wait to False for demonstration purposes.\n",
- "estimator.fit(wait=False)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Result\n",
- "As a result of the above command, SageMaker will spin off two jobs for you - the first one being the training job which produces the tensors to be analyzed and the second one, which evaluates or analyzes the rule you asked it to in `rules_specification`.\n",
- "#### Check the status of the Rule Execution Job\n",
- "To get the rule execution job that SageMaker started for you, run the command below and it shows you the `RuleName`, `RuleStatus`, `FailureReason` if rule job started, the `RuleJobName` and `RuleExecutionJobArn`. \n",
- "If the tensors meets a rule evaluation condition, the rule execution job throws a client error with `FailureReason: RuleEvaluationConditionMet`. \n",
- "You can check the Cloudwatch Logstream `/aws/sagemaker/TrainingJobs` with `RuleExecutionJobArn`.\n",
- "\n",
- "Depending on how your tensors are emitted and how your custom rule reacts to the script, your rule evaluation job will either fail or succeed. \n",
- "You can get the rule evaluation statuses of the jobs through the following mechanism. This function will continue to poll till the rule execution jobs end. To proceed with the notebook, please stop the cell after RuleStatus changes to InProgress. At this point, you should see RuleExecutionJobName. This will be needed to execute the next cell of code where we attach to the rule execution job to see its logs."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "metadata": {
- "pycharm": {
- "name": "#%%\n"
- }
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Wait to get status for Rule Execution Jobs...\n",
- "=============================================\n",
- "RuleName: CustomGradientRule\n",
- "RuleStatus: RuleExecutionError\n",
- "FailureReason: ClientError: RuleEvaluationConditionMet: Evaluation of the rule CustomGradientRule at step 0 resulted in the condition being met\n",
- "Traceback (most recent call last):\n",
- " File \"train.py\", line 214, in execute\n",
- " exec(_SYMBOLIC_INVOKE_RULE.format(self.start_step, self.end_step), globals(), exec_local)\n",
- " File \"\", line 2, in \n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 84, in invoke_rule\n",
- " raise e\n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 79, in invoke_rule\n",
- " rule_obj.invoke(step)\n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule.py\", line 56, in invoke\n",
- " raise RuleEvaluationConditionMet(self.rule_name, step)\n",
- "smdebug.exceptions.RuleEvaluationConditionMet: Evaluation of the rule CustomGradientRule at step 0 resulted in the condition being met\n",
- "\n",
- "\n",
- "RuleExecutionJobName: CustomGradientRule-75136a67d770aaea32adbb833fb2ee4f\n",
- "RuleExecutionJobArn: arn:aws:sagemaker:us-west-2:072677473360:training-job/customgradientrule-75136a67d770aaea32adbb833fb2ee4f\n",
- "=============================================\n"
- ]
- }
- ],
- "source": [
- "rule_description = estimator.describe_rule_execution_jobs()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "#### Check the logs of the Rule Execution Job\n",
- "If you want to access the logs of a particular rule job name, you can do the following. First, you need to get the rule job name (`RuleExecutionJobArn` field from the training job description). Note that this is only available after the rule job reaches Started stage. Hence the next cell waits till the job name is available"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we can attach to this job to see its logs"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
- "metadata": {
- "pycharm": {
- "name": "#%%\n"
- }
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "2019-08-29 22:40:05 Starting - Preparing the instances for training\n",
- "2019-08-29 22:40:05 Downloading - Downloading input data\n",
- "2019-08-29 22:40:05 Training - Training image download completed. Training in progress.\n",
- "2019-08-29 22:40:05 Uploading - Uploading generated training model\n",
- "2019-08-29 22:40:05 Failed - Training job failed\u001b[31m[2019-08-29 22:39:24.434 ip-10-0-174-72.us-west-2.compute.internal:1 INFO s3_trial.py:27] Loading trial base-trial at path s3://sagemaker-us-west-2-072677473360/tensors-tensorflow-custom-rule-tornasole-2019-08-29-22-32-10-697\u001b[0m\n",
- "\u001b[31m[2019-08-29 22:39:56.413 ip-10-0-174-72.us-west-2.compute.internal:1 INFO rule_invoker.py:76] Started execution of rule CustomGradientRule at step 0\u001b[0m\n",
- "\u001b[31mException during rule execution: Customer Error: RuleEvaluationConditionMet: Evaluation of the rule CustomGradientRule at step 0 resulted in the condition being met\u001b[0m\n",
- "\u001b[31mTraceback (most recent call last):\n",
- " File \"train.py\", line 214, in execute\n",
- " exec(_SYMBOLIC_INVOKE_RULE.format(self.start_step, self.end_step), globals(), exec_local)\n",
- " File \"\", line 2, in \n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 84, in invoke_rule\n",
- " raise e\n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 79, in invoke_rule\n",
- " rule_obj.invoke(step)\n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule.py\", line 56, in invoke\n",
- " raise RuleEvaluationConditionMet(self.rule_name, step)\u001b[0m\n",
- "\u001b[31msmdebug.exceptions.RuleEvaluationConditionMet: Evaluation of the rule CustomGradientRule at step 0 resulted in the condition being met\n",
- "\n",
- "\u001b[0m\n"
- ]
- },
- {
- "ename": "UnexpectedStatusException",
- "evalue": "Error for Training job CustomGradientRule-75136a67d770aaea32adbb833fb2ee4f: Failed. Reason: ClientError: RuleEvaluationConditionMet: Evaluation of the rule CustomGradientRule at step 0 resulted in the condition being met\nTraceback (most recent call last):\n File \"train.py\", line 214, in execute\n exec(_SYMBOLIC_INVOKE_RULE.format(self.start_step, self.end_step), globals(), exec_local)\n File \"\", line 2, in \n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 84, in invoke_rule\n raise e\n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 79, in invoke_rule\n rule_obj.invoke(step)\n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule.py\", line 56, in invoke\n raise RuleEvaluationConditionMet(self.rule_name, step)\nsmdebug.exceptions.RuleEvaluationConditionMet: Evaluation of the rule CustomGradientRule at step 0 resulted in the condition being met\n\n",
- "output_type": "error",
- "traceback": [
- "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[0;31mUnexpectedStatusException\u001b[0m Traceback (most recent call last)",
- "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msagemaker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mestimator\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mEstimator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0mrule_job_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrule_description\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'RuleExecutionJobName'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mexploding_tensor\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mEstimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mattach\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrule_job_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
- "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/estimator.py\u001b[0m in \u001b[0;36mattach\u001b[0;34m(cls, training_job_name, sagemaker_session, model_channel_name)\u001b[0m\n\u001b[1;32m 460\u001b[0m )\n\u001b[1;32m 461\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_current_job_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlatest_training_job\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 462\u001b[0;31m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlatest_training_job\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 463\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 464\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/estimator.py\u001b[0m in \u001b[0;36mwait\u001b[0;34m(self, logs)\u001b[0m\n\u001b[1;32m 1012\u001b[0m \"\"\"\n\u001b[1;32m 1013\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mlogs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1014\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msagemaker_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlogs_for_job\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwait\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1015\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1016\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msagemaker_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait_for_job\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mlogs_for_job\u001b[0;34m(self, job_name, wait, poll)\u001b[0m\n\u001b[1;32m 1479\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1480\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mwait\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1481\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_job_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdescription\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"TrainingJobStatus\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1482\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mdot\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1483\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36m_check_job_status\u001b[0;34m(self, job, desc, status_key_name)\u001b[0m\n\u001b[1;32m 1092\u001b[0m ),\n\u001b[1;32m 1093\u001b[0m \u001b[0mallowed_statuses\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"Completed\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"Stopped\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1094\u001b[0;31m \u001b[0mactual_status\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mstatus\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1095\u001b[0m )\n\u001b[1;32m 1096\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;31mUnexpectedStatusException\u001b[0m: Error for Training job CustomGradientRule-75136a67d770aaea32adbb833fb2ee4f: Failed. Reason: ClientError: RuleEvaluationConditionMet: Evaluation of the rule CustomGradientRule at step 0 resulted in the condition being met\nTraceback (most recent call last):\n File \"train.py\", line 214, in execute\n exec(_SYMBOLIC_INVOKE_RULE.format(self.start_step, self.end_step), globals(), exec_local)\n File \"\", line 2, in \n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 84, in invoke_rule\n raise e\n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 79, in invoke_rule\n rule_obj.invoke(step)\n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule.py\", line 56, in invoke\n raise RuleEvaluationConditionMet(self.rule_name, step)\nsmdebug.exceptions.RuleEvaluationConditionMet: Evaluation of the rule CustomGradientRule at step 0 resulted in the condition being met\n\n"
- ]
- }
- ],
- "source": [
- "from sagemaker.estimator import Estimator\n",
- "rule_job_name = rule_description[0]['RuleExecutionJobName']\n",
- "exploding_tensor = Estimator.attach(rule_job_name)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "pycharm": {
- "name": "#%% md\n"
- }
- },
- "source": [
- "## Tornasole Rules Explained in depth\n",
- "Let us now walk through some of Tornasole's concepts which will be helpful to understand how rules are executed in SageMaker\n",
- "and how custom rules work. \n",
- "\n",
- "### Trial\n",
- "A Trial in Tornasole's context refers to a training job. \n",
- "It is identified by the path where the saved tensors for the job are stored. \n",
- "\n",
- "### Rules\n",
- "Rules are the medium by which Tornasole executes a certain piece of code regularly on different steps of the job.\n",
- "They can be used to assert certain conditions during training, and raise Cloudwatch Events based on them that you can\n",
- "use to process in any way you like. \n",
- "\n",
- "These are defined by the class `smdebug.rules.Rule`. A rule takes a `base_trial` which refers to the job whose run invokes the rule execution. \n",
- "A rule can optionally look at other jobs as well, passed using the argument `other_trials`.\n",
- "\n",
- "Tornasole comes with a set of **First Party rules** (1P rules).\n",
- "You can also write your own rules looking at these 1P rules for inspiration.\n",
- "Refer [Developer Guide for Rules.md](../../DeveloperGuide_Rules.md) for more on the \n",
- "APIs you can use to write your own rules as well as descriptions for the 1P rules that we provide.\n",
- "\n",
- "### Storage\n",
- "The tensors saved by Tornasole are, by default, stored in the S3 output path of the training job, under the folder **`/tensors-`**. \n",
- "This is done to ensure that we don't end up accidentally overwriting the tensors from a training job with the others. \n",
- "Rules evaluation require separation of the tensors paths to be evaluated correctly.\n",
- "If you don't provide an S3 output path to the estimator, SageMaker creates one for you as: **`s3://sagemaker--/`**\n",
- "\n",
- "### Using Tornasole Rules in SageMaker \n",
- "Here we will talk about how to use SageMaker to evaluate these rules on the training jobs. \n",
- "The new parameters in Sagemaker Estimator to look out for are\n",
- "\n",
- "- `debug` :(bool)\n",
- "This indicates that debugging should be enabled for the training job. \n",
- "Setting this as `True` would make Tornasole available for use with the job\n",
- "\n",
- "- `rules_specification`: (list[*dict*])\n",
- "You can specify any number of rules to monitor your SageMaker training job. \n",
- "This parameter takes a list of python dictionaries, one for each rule you want to enable. \n",
- "Each `dict` is of the following form:\n",
- "```\n",
- "{\n",
- " \"RuleName\": \n",
- " # The name of the class implementing the Tornasole Rule interface. (required)\n",
- "\n",
- " \"SourceS3Uri\": \n",
- " # S3 URI of the rule script containing the class in 'RuleName'. \n",
- " # This is not required if you want to use one of the\n",
- " # First Party rules provided to you by Amazon. \n",
- " # In such a case you can leave it empty or not pass it. If you want to run a custom rule \n",
- " # defined by you, you will need to define the custom rule class in a python \n",
- " # file and provide it to SageMaker as a S3 URI. \n",
- " # SageMaker will fetch this file and try to look for the rule class \n",
- " # identified by RuleName in this file.\n",
- " \n",
- " \"InstanceType\": \n",
- " # The ML instance type which should be used to run the rule evaluation job\n",
- " \n",
- " \"VolumeSizeInGB\": \n",
- " # The volume size to store the runtime artifacts from the rule evaluation \n",
- " \n",
- " \"RuntimeConfigurations\": {\n",
- " # Map defining the parameters required to instantiate the Rule class and\n",
- " # parameters regarding invokation of the rule (start-step and end-step)\n",
- " # This can be any parameter taken by the rule. \n",
- " # Every value here needs to be a string. \n",
- " # So when you write custom rules, ensure that you can parse each argument from a string.\n",
- " #\n",
- " # PARAMS CAN BE\n",
- " #\n",
- " # STANDARD PARAMS FOR RULE EXECUTION\n",
- " # \"start-step\": \n",
- " # \"end-step\": \n",
- " # \"other-trials-paths\": (';' separated list of s3 paths as a string)\n",
- " # \"logging-level\": (can be one of \"CRITICAL\", \"FATAL\", \"ERROR\", \n",
- " # \"WARNING\", \"WARN\", \"DEBUG\", \"NOTSET\")\n",
- " #\n",
- " # ANY PARAMETER TAKEN BY THE RULE other than `base_trial` and `other_trials` \n",
- " # \"parameter\" : \"value\"\n",
- " # : \n",
- " }\n",
- "}\n",
- "```\n",
- "\n",
- "\n",
- "### CloudWatch Event Integration for Rules\n",
- "When the status of training job or rule execution job change (i.e. starting, failed), TrainingJobStatus [CloudWatch events](https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html) are emitted. \n",
- "\n",
- "You can configure a CloudWatch event rule to receive and process these events by setting up a target (Lambda function, SNS) as follows:\n",
- "\n",
- "- Configure the [SageMaker TrainingJobStatus CW event](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#sagemaker_event_types) to include rule job statuses associated with the training job\n",
- "- Configure the CW event to be emitted when a RuleStatus changes\n",
- "- Create a CloudWatch event rule that monitors the Training Job customer started\n",
- "- Set a Target (Lambda funtion, SQS) for the CloudWatch event rule that processes the event, and triggers an alarm for the customer based on the RuleStatus. \n",
- "\n",
- "Refer [this page](https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html) for more details. \n",
- "\n",
- "### Writing a custom rule\n",
- "\n",
- "Implementing a custom rule involves implementing the Rule interface that Tornasole provides.\n",
- "Let us go through the exercise of writing a rule which checks whether gradients are very high.\n",
- "\n",
- "#### Constructor\n",
- "Creating a rule involves first inheriting from the base Rule class Tornasole provides: `smdebug.rules.Rule`\n",
- "\n",
- "Every rule is required to take the argument `base_trial` which represents the Trial object for the job whose execution \n",
- "invokes this rule. In addition to this you might want to pass `other_trials` which represents\n",
- "list of Trial objects for other jobs if you want your custom rule to look at other jobs for some comparision. \n",
- "For this rule here we do not need to look at any other trials, so we set `other_trials` to None.\n",
- "\n",
- "```python\n",
- "from smdebug.rules import Rule\n",
- "\n",
- "class CustomGradientRule(Rule):\n",
- " def __init__(self, base_trial, threshold=10.0):\n",
- " super().__init__(base_trial, other_trials=None)\n",
- " self.threshold = float(threshold)\n",
- "```\n",
- "\n",
- "Please note that apart from `base_trial` and `other_trials` (if required), we require all \n",
- "arguments of the rule constructor to take a string as value. You can parse them to the type\n",
- "that you want from the string. This means if you want to pass\n",
- "a list of strings, you might want to pass them as a comma separated string. This restriction is\n",
- "being enforced so as to let you create and invoke rules from json using Sagemaker's APIs. \n",
- "\n",
- "#### Function to invoke at a given step\n",
- "When a rule is executed, it is invoked at each step. We need to now define what to do when the rule is invoked at a given step, `step`.\n",
- "In this function you can implement the core logic of what you want to do with your selection of tensors. If your custom rule \n",
- "has access to other trials, you can access tensors from other trials as well.\n",
- "\n",
- "This function should return a boolean value `True` or `False`. When `True` is returned,\n",
- "SageMaker will raise the exception `RuleEvaluationConditionMet`. This will also create a CloudWatch Event which can be used to configure your chosen action. \n",
- "\n",
- "The invoke function for `CustomGradientRule` to check whether tensors have large gradients can look like below:\n",
- "```python\n",
- " def invoke_at_step(self, step):\n",
- " for tensor in self.base_trial.tensors_in_collection('gradients'):\n",
- " abs_mean = tensor.reduction_value(step, 'mean', abs=True)\n",
- " if abs_mean > self.threshold:\n",
- " return True\n",
- " return False\n",
- "```\n",
- "Here, we can access the names of tensors in `gradients` collection by using the method `tensors_in_collection`. \n",
- "You can see the full API that Trial provides to get tensors in our [Developer Guide For Rules](../../DeveloperGuide_Rules.md).\n",
- "\n",
- "#### Optional: RequiredTensors\n",
- "RequiredTensors is an optional construct that allows Tornasole to bulk-fetch all tensors that you need to \n",
- "execute the rule. This helps the rule invocation be more performant so it does not fetch tensor values from S3 one by one. \n",
- "\n",
- "##### RequiredTensors API \n",
- "This is a class whose object is provided as a member of the rule class, so you can access it as `self.req_tensors`. \n",
- "Its full API is described in our [Developer Guide For Rules](../../DeveloperGuide_Rules.md). \n",
- "In short, it has the following methods:\n",
- "```python\n",
- "# Add name of required tensor for a particular trial at given steps \n",
- "self.req_tensors.add(name=tname, steps=[step_num], trial=None, should_match_regex=False)\n",
- "\n",
- "# If required tensors were added inside `set_required_tensors`, during rule invocation it is\n",
- "# automatically used to fetch all tensors at once by calling `req_tensors.fetch()`\n",
- "# If required tensors were added elsewhere, or later, you can call the `req_tensors.fetch()` method \n",
- "# yourself to fetch all tensors at once.\n",
- "self.req_tensors.fetch()\n",
- "\n",
- "# This method returns the names of the required tensors for a given trial\n",
- "self.req_tensors.get_names(trial=None)\n",
- "\n",
- "# This method returns the steps for which the tensor is required to execute the rule at this step.\n",
- "self.req_tensors.get_tensor_steps(trial=None)\n",
- "\n",
- "# This method returns the list of required tensors for a given trial as `Tensor` objects\n",
- "self.req_tensors.get(trial=None)\n",
- "``` \n",
- "\n",
- "##### Declare required tensors\n",
- "To use this construct, you need to implement a method which lets Tornasole know what tensors you are interested in for invocation at a given step. \n",
- "This is the `set_required_tensors` method.\n",
- "\n",
- "```python\n",
- "def set_required_tensors(self, step):\n",
- " for tname in self.base_trial.tensors_in_collection('gradients'):\n",
- " self.req_tensors.add(tname, steps=[step])\n",
- "```\n",
- "##### Accessing required tensors\n",
- "Since we defined required tensors in the `set_required_tensors` method, these will have been\n",
- "pre-fetched when invoking the rule at a given step. You can continue to access the tensors as before.\n",
- "\n",
- "If you do not want to determine which tensors you want to process again, you can also just call\n",
- "self.req_tensors.get() to get them. In that case, the function would look as below: \n",
- "\n",
- "```python\n",
- "def invoke_at_step(self, step):\n",
- " for tensor in self.req_tensors.get():\n",
- " abs_mean = tensor.reduction_value(step, 'mean', abs=True)\n",
- " if abs_mean > self.threshold:\n",
- " return True\n",
- " return False\n",
- "```\n",
- "\n",
- "### Executing the custom rule\n",
- "\n",
- "You need to now provide Sagemaker the S3 location of the file which defines your custom rule classes as the value for the field `SourceS3Uri`. \n",
- "\n",
- "From above, our rule constructor takes the arguments `base_trial` and `threshold`. The `base_trial` argument will automatically be passed by SageMaker Rule Executor. The other arguments need to be passed through the RuntimeConfigurations dictionary as a mapping from string to string. \n",
- "\n",
- "If the custom rule took `other_trials`, which represents list of Trial objects for other jobs that the rule is interested in, that can be passed by passing the argument `other-trials-paths` which needs to be in the form of `s3_path_other_trial_1;s3_path_other_trial_2`.\n",
- "\n",
- "Note that the custom rules can only have arguments which expect a string as the value except the two arguments specifying trials to the Rule (`base_trial` and `other_trials`). \n",
- "\n",
- "Here's an example:\n",
- "```\n",
- "rules_specification = [\n",
- " {\n",
- " \"RuleName\": \"CustomGradientRule\",\n",
- " \"SourceS3Uri\": \"s3://tornasole-external-preview-use1/rules/scripts/my_custom_rule.py\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"threshold\" : \"20.0\"\n",
- " }\n",
- " }\n",
- "]\n",
- "```"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "conda_tensorflow_p36",
- "language": "python",
- "name": "conda_tensorflow_p36"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.5"
- },
- "pycharm": {
- "stem_cell": {
- "cell_type": "raw",
- "metadata": {
- "collapsed": false
- },
- "source": []
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/examples/rules/scripts/my_custom_rule.py b/examples/rules/scripts/my_custom_rule.py
deleted file mode 100644
index 4c570a46f..000000000
--- a/examples/rules/scripts/my_custom_rule.py
+++ /dev/null
@@ -1,19 +0,0 @@
-# First Party
-from smdebug.rules.rule import Rule
-
-
-class CustomGradientRule(Rule):
- def __init__(self, base_trial, threshold=10.0):
- super().__init__(base_trial)
- self.threshold = float(threshold)
-
- def set_required_tensors(self, step):
- for tname in self.base_trial.tensor_names(collection="gradients"):
- self.req_tensors.add(tname, steps=[step])
-
- def invoke_at_step(self, step):
- for t in self.req_tensors.get():
- abs_mean = t.reduction_value(step, "mean", abs=True)
- if abs_mean > self.threshold:
- return True
- return False
diff --git a/examples/tensorflow/README.md b/examples/tensorflow/README.md
new file mode 100644
index 000000000..6a93b72df
--- /dev/null
+++ b/examples/tensorflow/README.md
@@ -0,0 +1,22 @@
+# Examples
+## Example notebooks
+Please refer to the example notebooks in [Amazon SageMaker Examples repository](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger)
+
+## Example scripts
+The above notebooks come with example scripts which can be used through SageMaker. Some more example scripts are here in [scripts/](scripts/)
+
+## Example configurations for saving tensors through SageMaker pySDK
+Example configurations for saving tensors through the hook are available at [docs/sagemaker.md](../docs/sagemaker.md)
+
+## Example configurations for running rules through SageMaker pySDK
+Example configurations for saving tensors through the hook are available at [docs/sagemaker.md](../docs/sagemaker.md)
+
+## Example for running rule locally
+
+```
+from smdebug.rules import invoke_rule
+from smdebug.trials import create_trial
+trial = create_trial('s3://bucket/prefix')
+rule_obj = CustomRule(trial, param=value)
+invoke_rule(rule_obj, start_step=0, end_step=10)
+```
diff --git a/examples/tensorflow/local/mnist.py b/examples/tensorflow/local/mnist.py
new file mode 100644
index 000000000..00bb26185
--- /dev/null
+++ b/examples/tensorflow/local/mnist.py
@@ -0,0 +1,153 @@
+"""
+This script is a simple MNIST training script which uses Tensorflow's Estimator interface.
+It has been orchestrated with SageMaker Debugger hook to allow saving tensors during training.
+Here, the hook has been created using its constructor to allow running this locally for your experimentation.
+When you want to run this script in SageMaker, it is recommended to create the hook from json file.
+Please see scripts in either 'sagemaker_byoc' or 'sagemaker_official_container' folder based on your use case.
+"""
+
+# Standard Library
+import argparse
+import logging
+import random
+
+# Third Party
+import numpy as np
+import tensorflow as tf
+
+# First Party
+import smdebug.tensorflow as smd
+
+logging.getLogger().setLevel(logging.INFO)
+
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--lr", type=float, default=0.001)
+ parser.add_argument("--random_seed", type=bool, default=False)
+ parser.add_argument("--out_dir", type=str)
+ parser.add_argument("--save_interval", type=int, default=500)
+ parser.add_argument("--num_epochs", type=int, default=5, help="Number of epochs to train for")
+ parser.add_argument(
+ "--num_steps",
+ type=int,
+ help="Number of steps to train for. If this" "is passed, it overrides num_epochs",
+ )
+ parser.add_argument(
+ "--num_eval_steps",
+ type=int,
+ help="Number of steps to evaluate for. If this"
+ "is passed, it doesnt evaluate over the full eval set",
+ )
+ parser.add_argument("--model_dir", type=str, default="/tmp/mnist_model")
+ args = parser.parse_args()
+
+ if args.random_seed:
+ tf.set_random_seed(2)
+ np.random.seed(2)
+ random.seed(12)
+
+ hook = smd.EstimatorHook(
+ out_dir=args.out_dir,
+ include_collections=["weights", "gradients"],
+ save_config=smd.SaveConfig(save_interval=args.save_interval),
+ )
+
+ def cnn_model_fn(features, labels, mode):
+ """Model function for CNN."""
+ # Input Layer
+ input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])
+
+ # Convolutional Layer #1
+ conv1 = tf.layers.conv2d(
+ inputs=input_layer,
+ filters=32,
+ kernel_size=[5, 5],
+ padding="same",
+ activation=tf.nn.relu,
+ )
+
+ # Pooling Layer #1
+ pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)
+
+ # Convolutional Layer #2 and Pooling Layer #2
+ conv2 = tf.layers.conv2d(
+ inputs=pool1, filters=64, kernel_size=[5, 5], padding="same", activation=tf.nn.relu
+ )
+ pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)
+
+ # Dense Layer
+ pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])
+ dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
+ dropout = tf.layers.dropout(
+ inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN
+ )
+
+ # Logits Layer
+ logits = tf.layers.dense(inputs=dropout, units=10)
+
+ predictions = {
+ # Generate predictions (for PREDICT and EVAL mode)
+ "classes": tf.argmax(input=logits, axis=1),
+ # Add `softmax_tensor` to the graph. It is used for PREDICT and by the
+ # `logging_hook`.
+ "probabilities": tf.nn.softmax(logits, name="softmax_tensor"),
+ }
+
+ if mode == tf.estimator.ModeKeys.PREDICT:
+ return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)
+
+ # Calculate Loss (for both TRAIN and EVAL modes)
+ loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)
+
+ # Configure the Training Op (for TRAIN mode)
+ if mode == tf.estimator.ModeKeys.TRAIN:
+ optimizer = tf.train.GradientDescentOptimizer(learning_rate=args.lr)
+
+ # SMD: Wrap your optimizer as follows to help SageMaker Debugger identify gradients
+ # This does not change your optimization logic, it returns back the same optimizer
+ optimizer = hook.wrap_optimizer(optimizer)
+
+ train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
+ return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)
+
+ # Add evaluation metrics (for EVAL mode)
+ eval_metric_ops = {
+ "accuracy": tf.metrics.accuracy(labels=labels, predictions=predictions["classes"])
+ }
+ return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)
+
+ # Load training and eval data
+ ((train_data, train_labels), (eval_data, eval_labels)) = tf.keras.datasets.mnist.load_data()
+
+ train_data = train_data / np.float32(255)
+ train_labels = train_labels.astype(np.int32) # not required
+
+ eval_data = eval_data / np.float32(255)
+ eval_labels = eval_labels.astype(np.int32) # not required
+
+ mnist_classifier = tf.estimator.Estimator(model_fn=cnn_model_fn, model_dir=args.model_dir)
+
+ train_input_fn = tf.estimator.inputs.numpy_input_fn(
+ x={"x": train_data},
+ y=train_labels,
+ batch_size=128,
+ num_epochs=args.num_epochs,
+ shuffle=True,
+ )
+
+ eval_input_fn = tf.estimator.inputs.numpy_input_fn(
+ x={"x": eval_data}, y=eval_labels, num_epochs=1, shuffle=False
+ )
+
+ # Set training mode so SMDebug can classify the steps into training mode
+ hook.set_mode(smd.modes.TRAIN)
+ mnist_classifier.train(input_fn=train_input_fn, steps=args.num_steps, hooks=[hook])
+
+ # Set eval mode so SMDebug can classify the steps into eval mode
+ hook.set_mode(smd.modes.EVAL)
+ mnist_classifier.evaluate(input_fn=eval_input_fn, steps=args.num_eval_steps, hooks=[hook])
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/tensorflow/local/simple.py b/examples/tensorflow/local/simple.py
new file mode 100644
index 000000000..c0ca691e1
--- /dev/null
+++ b/examples/tensorflow/local/simple.py
@@ -0,0 +1,97 @@
+"""
+This script is a simple training script which uses Tensorflow's MonitoredSession interface.
+It has been orchestrated with SageMaker Debugger hook to allow saving tensors during training.
+Here, the hook has been created using its constructor to allow running this locally for your experimentation.
+When you want to run this script in SageMaker, it is recommended to create the hook from json file.
+Please see scripts in either 'sagemaker_byoc' or 'sagemaker_official_container' folder based on your use case.
+"""
+
+# Standard Library
+import argparse
+import random
+
+# Third Party
+import numpy as np
+import tensorflow as tf
+
+# First Party
+import smdebug.tensorflow as smd
+
+
+def str2bool(v):
+ if isinstance(v, bool):
+ return v
+ if v.lower() in ("yes", "true", "t", "y", "1"):
+ return True
+ elif v.lower() in ("no", "false", "f", "n", "0"):
+ return False
+ else:
+ raise argparse.ArgumentTypeError("Boolean value expected.")
+
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--model_dir", type=str, help="S3 path for the model")
+ parser.add_argument("--lr", type=float, help="Learning Rate", default=0.001)
+ parser.add_argument("--steps", type=int, help="Number of steps to run", default=100)
+ parser.add_argument("--scale", type=float, help="Scaling factor for inputs", default=1.0)
+ parser.add_argument("--random_seed", type=bool, default=False)
+ parser.add_argument("--out_dir", type=str)
+ parser.add_argument("--save_interval", type=int, default=500)
+ args = parser.parse_args()
+
+ # these random seeds are only intended for test purpose.
+ # for now, 2,2,12 could promise no assert failure when running tests
+ # if you wish to change the number, notice that certain steps' tensor value may be capable of variation
+ if args.random_seed:
+ tf.set_random_seed(2)
+ np.random.seed(2)
+ random.seed(12)
+
+ hook = smd.EstimatorHook(
+ out_dir=args.out_dir,
+ include_collections=["weights", "gradients"],
+ save_config=smd.SaveConfig(save_interval=args.save_interval),
+ )
+
+ # Network definition
+ # Note the use of name scopes
+ with tf.name_scope("foobar"):
+ x = tf.placeholder(shape=(None, 2), dtype=tf.float32)
+ w = tf.Variable(initial_value=[[10.0], [10.0]], name="weight1")
+ with tf.name_scope("foobaz"):
+ w0 = [[1], [1.0]]
+ y = tf.matmul(x, w0)
+ loss = tf.reduce_mean((tf.matmul(x, w) - y) ** 2, name="loss")
+
+ hook.add_to_collection("losses", loss)
+
+ global_step = tf.Variable(17, name="global_step", trainable=False)
+ increment_global_step_op = tf.assign(global_step, global_step + 1)
+
+ optimizer = tf.train.AdamOptimizer(args.lr)
+
+ # Wrap the optimizer with wrap_optimizer so smdebug can find gradients to save
+ optimizer = hook.wrap_optimizer(optimizer)
+
+ # use this wrapped optimizer to minimize loss
+ optimizer_op = optimizer.minimize(loss, global_step=increment_global_step_op)
+
+ # pass the hook to hooks parameter of monitored session
+ sess = tf.train.MonitoredSession(hooks=[hook])
+
+ # use this session for running the tensorflow model
+ hook.set_mode(smd.modes.TRAIN)
+ for i in range(args.steps):
+ x_ = np.random.random((10, 2)) * args.scale
+ _loss, opt, gstep = sess.run([loss, optimizer_op, increment_global_step_op], {x: x_})
+ print(f"Step={i}, Loss={_loss}")
+
+ hook.set_mode(smd.modes.EVAL)
+ for i in range(args.steps):
+ x_ = np.random.random((10, 2)) * args.scale
+ sess.run([loss, increment_global_step_op], {x: x_})
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/tensorflow/local/tf_keras_resnet.py b/examples/tensorflow/local/tf_keras_resnet.py
new file mode 100644
index 000000000..1a8b0f0d0
--- /dev/null
+++ b/examples/tensorflow/local/tf_keras_resnet.py
@@ -0,0 +1,76 @@
+"""
+This script is a ResNet training script which uses Tensorflow's Keras interface.
+It has been orchestrated with SageMaker Debugger hook to allow saving tensors during training.
+Here, the hook has been created using its constructor to allow running this locally for your experimentation.
+When you want to run this script in SageMaker, it is recommended to create the hook from json file.
+Please see scripts in either 'sagemaker_byoc' or 'sagemaker_official_container' folder based on your use case.
+"""
+
+# Standard Library
+import argparse
+
+# Third Party
+import numpy as np
+import tensorflow as tf
+from tensorflow.keras.applications.resnet50 import ResNet50
+from tensorflow.keras.datasets import cifar10
+from tensorflow.keras.utils import to_categorical
+
+# First Party
+import smdebug.tensorflow as smd
+
+
+def train(batch_size, epoch, model, hook):
+ (X_train, y_train), (X_valid, y_valid) = cifar10.load_data()
+
+ Y_train = to_categorical(y_train, 10)
+ Y_valid = to_categorical(y_valid, 10)
+
+ X_train = X_train.astype("float32")
+ X_valid = X_valid.astype("float32")
+
+ mean_image = np.mean(X_train, axis=0)
+ X_train -= mean_image
+ X_valid -= mean_image
+ X_train /= 128.0
+ X_valid /= 128.0
+
+ model.fit(
+ X_train,
+ Y_train,
+ batch_size=batch_size,
+ epochs=epoch,
+ validation_data=(X_valid, Y_valid),
+ shuffle=True,
+ callbacks=[hook],
+ )
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Train resnet50 cifar10")
+ parser.add_argument("--batch_size", type=int, default=32)
+ parser.add_argument("--epoch", type=int, default=3)
+ parser.add_argument("--model_dir", type=str, default="./model_keras_resnet")
+ parser.add_argument("--out_dir", type=str)
+ parser.add_argument("--save_interval", type=int, default=500)
+ opt = parser.parse_args()
+
+ model = ResNet50(weights=None, input_shape=(32, 32, 3), classes=10)
+
+ hook = smd.KerasHook(
+ out_dir=opt.out_dir,
+ include_collections=["weights", "gradients", "losses"],
+ save_config=smd.SaveConfig(save_interval=opt.save_interval),
+ )
+
+ optimizer = tf.keras.optimizers.Adam()
+ # wrap the optimizer so the hook can identify the gradients
+ optimizer = hook.wrap_optimizer(optimizer)
+ model.compile(loss="categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
+
+ # start the training.
+ train(opt.batch_size, opt.epoch, model, hook)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/tensorflow/notebooks/keras-sentiment/Loss_Accuracy.ipynb b/examples/tensorflow/notebooks/keras-sentiment/Loss_Accuracy.ipynb
deleted file mode 100644
index d52c648c3..000000000
--- a/examples/tensorflow/notebooks/keras-sentiment/Loss_Accuracy.ipynb
+++ /dev/null
@@ -1,199 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Using TensorFlow backend.\n"
- ]
- }
- ],
- "source": [
- "import matplotlib.pyplot as plt\n",
- "import numpy as np\n",
- "from keras.datasets import imdb\n",
- "from smdebug.trials import LocalTrial"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "WARNING: Logging before flag parsing goes to stderr.\n",
- "I0730 09:44:29.710710 4704150976 local_trial.py:20] Loading trial sentiment at path ts_output/\n",
- "I0730 09:44:29.720427 4704150976 local_trial.py:58] Loaded 4 collections\n"
- ]
- }
- ],
- "source": [
- "lt = LocalTrial( 'sentiment', 'ts_output/', parallel=True)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "['batch',\n",
- " 'size',\n",
- " 'loss',\n",
- " 'acc',\n",
- " 'mean_squared_error',\n",
- " 'embedding_1',\n",
- " 'conv1d_1_0',\n",
- " 'conv1d_1_1',\n",
- " 'dense_1_0',\n",
- " 'dense_1_1',\n",
- " 'dense_2_0',\n",
- " 'dense_2_1',\n",
- " 'val_loss',\n",
- " 'val_acc',\n",
- " 'val_mean_squared_error']"
- ]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "lt.tensors()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "def tplt( trial, tname ): \n",
- " t = lt.tensor(tname)\n",
- " steps = t.steps()\n",
- " _t = [t.value(s) for s in steps]\n",
- " plt.plot( steps, _t, label=tname)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJztvXl4W+WZ9/95JEuW991OYiexnThAEhJIAkmAsjTA0E7btJTOD6YLbWmZDnSj7dtpZzrTTtt523euvi3QBd60Qwdoh6W0BdpSKCFhKeCQhbAkkNhxNjuJF9mxLdtaLD2/P845sjbbsiNv0v25Ll22jh4dPcfH+p77fJ/7uR+ltUYQBEFIL2wz3QFBEAQh9Yi4C4IgpCEi7oIgCGmIiLsgCEIaIuIuCIKQhoi4C4IgpCEi7oIgCGmIiLsgCEIaIuIuCIKQhmTN1AeXl5fr2tramfp4QRCEOcnu3bu7tNYV47WbMXGvra1l165dM/XxgiAIcxKl1NFk2oktIwiCkIaIuAuCIKQhIu6CIAhpyIx57okIBAK0trbi9XpnuiuzEpfLRU1NDQ6HY6a7IgjCLGdWiXtraysFBQXU1tailJrp7swqtNa43W5aW1upq6ub6e4IgjDLGdeWUUrdo5TqUEq9OcrrSil1p1KqWSn1ulJqzWQ74/V6KSsrE2FPgFKKsrIyuasRBCEpkvHc/xu4ZozX3wU0mI+bgbvOpEMi7KMjfxtBEJJlXFtGa/28Uqp2jCabgfu0sV5fo1KqWCk1X2t9MkV9FIQZocvjY0dLN3+7av5MdyVt+NPrJzlwqi9ue252Fp+8uA5nVuJ4c/uBDl492jPmvkvynNy4sRabbXJBUEe/l5ea3Ww+b8G4gVQgGOKXLx7G4x0GoDTPyY0XjW4nP72/nfqKPJZU5E+qb5MhFZ57NXA84nmruS1O3JVSN2NE9yxatCgFHy0IU8eDrxzjB385yIb6KynLz57p7sx5DncN8LkH9hDSEKmB1jLOdeV5/M2KeXHv6/L4uOVXexgKBBlNc619rKopZu3ikkn171cvH+XObc1UFmRz0dLyMds+vOs4//uJt6O2XVBXyooFRXFtD3cN8A/37+Lq5fO4+6NrJ9W3yZAKcU/050646rbWeguwBWDdunWyMrcwqznZa4xvtPYMibingB9va8KZZeOFr76TioKRv6dvOMjqf/8LO1q6E4r7z59vwTccZOuXLmNpZeLIt8vjY913t7LjsHvS4n6gvR+A27c2sXHJ6GN//uEQP9t+iPMXFfO7f7yIpg4PV//oeZo7PAnF/cfbmghpeOVIN6GQnvSdxURJRZ57K7Aw4nkNcCIF+50x3v/+97N27VpWrFjBli1bAHjyySdZs2YNq1evZtOmTQB4PB4+8YlPcO6557Jq1Sp++9vfzmS3hRTT0e8DoO300Az3ZO5zuGuAR19t46MbFkcJO0B2lp21i0tobHHHvc/t8XHfy0d53+oFowo7QHl+Nsuq8mls6Z50H5s6PLgcNl450s3LCfpi8cjuVtpOD/HFK5ehlKK2LI8sm+KgeXGIxDruRaW5dA/4aerwTLp/EyUVkfvjwGeVUg8C64HeVPjt//6Hfew/Ee/NnQnLFxTyzfeuGLfdPffcQ2lpKUNDQ1xwwQVs3ryZT3/60zz//PPU1dXR3W38A33nO9+hqKiIN954A4CenrE9QWFu0dFnRe6DM9yTuY8Vtd986ZKEr2+oK+OHWw9yetBPca4zvH3LC0bU/tl3Noz7GRvqy3hkdyuBYAiHfWJxq284yFH3IJ+6pI5H97YZ0Xt9fPTuHw7x0+3NnL+omEsbDOvGmWWjtjyPpvZ44baO+47rz+MDP3uJxhY3Z80rmFDfJksyqZAPAC8DZymlWpVSNymlPqOU+ozZ5AmgBWgGfg7cMmW9nSbuvPNOVq9ezYYNGzh+/Dhbtmzh0ksvDeeXl5aWArB161ZuvfXW8PtKSiZ3OyjMTqzIvbVHIvczwYpeP7I+Pmq3WF9fhtbwyuGRyNvt8XHfS+NH7eF91JUx6A/yRlvvpPoYDGlWVBdxy+VLeeVw4ug9Nmq3aKjMj4vKI4/7/EUlVBfnJLw7mSqSyZa5YZzXNXDrWG0mQzIR9lTw7LPPsnXrVl5++WVyc3O5/PLLWb16NQcOHIhrq7WW9MQpZkeLm1/89TB3f2Qt9nG8yt7BALf8z26+94FVLCrLPaPPDYU0ndMg7t5AkH/81W6+dNVZnFsT79fORvzDIW66dyfHupO7o+n3DhtR+2X1o7ZZvbCI7CwbjS3dXG367lteaMGbZNQOsL7eCLoaW9ysWWQEWj946gB/eD3eJc6yKf7zulWsXWy856AZdS+ryqe2LI+fPdvMP9y/m9I8Z9T7Ovp8UVG7RUNVAU/tO4U3EMTlsANw97OHoo57Q30Z2w90TJvvPqtmqM4Gent7KSkpITc3l7fffpvGxkZ8Ph/PPfcchw8fDtsypaWlXH311fzkJz/h9ttvBwxbRqL31PLnN0/x9P52TpweYmHp2IK980g3Lza72Xmk+4zFvXvQz3DIGPOfSlvmwKl+th/o5IK60jkj7r/d08oLTV1ctbyKPKc9qfdcuqyCygLXqK9bvvuOw0Zk6/b4uD8Jrz2S8vxsGirz2dHSzS2Xw5GuAe567hArq4uoi/l/eHLfKR7beyIs7k3t/dhtirryPLKz7Pzndav5/Z7WuM+wKcUnL6mLC+qWVeUT0tDSOcDyBYVorXnm7Q7+ZsW88HFvqC/lt3taaerwTIs1I+IewzXXXMPdd9/NqlWrOOuss9iwYQMVFRVs2bKFa6+9llAoRGVlJU8//TTf+MY3uPXWW1m5ciV2u51vfvObXHvttTN9CGmFNUjV2jO+uFu3xd0D/jP+3HbTb68uzqG1Z2jK7tKsPnf0+VK+76nAPxziJ9uaOW9hMVs+ujalf5MN9WX8aOtBegcD/PyFwwwFgnwuyag9ch+/22P47j/e1kyWTfHzj62Nu7B87J5XoiySpnYPi8tyyc4yLlaXLavgsmXjrocRpqHSEOumjn6WLyjkUOcAXR4fG+vLovoGTJvvLuIeQ3Z2Nn/+858Tvvaud70r6nl+fj733nvvdHQrY7HEz4iey8Zua14IugbOXCgtv/38RcX88fWT9AwG4m7RU4HV547+uVFW4rd7DM/5Pz6wMuUXuw2m7/7kvpPc9/IR3rsq+ag9ch/3Nx7lT6+f5NG9bXz8otqEdwwb6kv5zycP0OXxUZ6fzcGOfhom+FmR1JXnYbep8KCqdeHYECHuNSU5Yd/9xotqJ/1ZySIlf4VZy+lB/4R874MdhlB2e848crcyZSzvdqqsGevOpH0ORO6RUftEotpksXz37/zxLYYCQT6/aemE92H57t949E2ybIp/GMXnt0T3lcPd4UyZZVWTj6adWTZqy3LD57Oxxc28QheLI+wgpRTr60vZcbgbrad+mo+IuzBricw+GE/cQyFN8wRtmZ4BP95AMOFrltiev6g4qc+P7cup3uQi8bAtM42Re/cYxz0WVtT+hSsbpsSiys6ys2ZRCR7fsBm1T1xsy/OzWVqZj8c3zIfXLx7V5z+3uohcp53GFnc4U2aidwmxLKsqoLnDg9aaxpZuNtSXxv2dNtSXTVu+u4i7MGuxoqAFRa5xI+fWniG8gRAAXUmIeyikefedL/D9P7+d8PWOfi8luQ7qzVogE4ncn9x3iou+/wyvt54es92Ab5jWniGybIr2Pt+0RHP93gBX/fA5fvBUfPbXeNz70hFW1RRx+RRE7RaXNJRjU0wqag/vY2k5LoeNz4yRneOw21hXW0pjizsiU+bMfPCGynyOuAfYf7KPLo8vypKx2Bjhu081Iu7CrKWp3UOe084FdaXjRs5NpiWzsDSH7iQ894Md/Zzs9fL8wc6Er7f3+agscFGU46DAlTWhyP2146cJabhja9OY7Q51GqKyZlEJ/uEQfUPDSX/GZLn3pSO4B/y83jqxXPDewQBvn+rn6uVVU5r+e9MldfzltssmFbVbfOVvzuLJL1xKZeHo2Tlg+O4H2z00trixKaivyJv0Z4KRDhnS8MArx8z9x4t7TUkOP/vwGt61cuqL0Ym4C7OWpo5+llbms7Akl1N9XoaDoVHbWtHXhrqypDz3xkNG5NTSNRDOjImko99HZaEx4aamJHdC4m7dcj/zdseY0bvV50vMnOn2KbZm+r1GFgoYF7eJ3Cm8etyYfb1mknVbksXlsJ+xPZKfnUVt+fhCvb7OEN/HXm2jtiwvnCkzWRqqjH7/fk9bnN9uoZTi3efOH3UyVyoRcRdmLQfbPTRUFVBTkkMwpDmVQIQtmjr6qSrMprY8jwF/cFxPubGlO1xeNtEtckeflyoz8qspyZmQLXOwvZ93nl1Jca5jzOi9qaMfp93GulpDMBNdZFLJvS8doXcowLXnV3N6MEDXBAae9xztwaZgdU3xFPZwellVU0SOw86APxgW5jPBypgZ8AdZn8Bvn25E3IVZiZUps6wqn5oSIwIaK3puavewrKqAMjNd0T2G7x4KaXYcdvOec+dT4MqKKzZlzU6tLLAi9xzazFz38bB89PMXFvOpS+rGjN6b2j3UV+SxoCgHmNpcdytq33R2JdeuqTE/P77Q1WjsOXaac+YXkpedPtnTjogLa8MZ2EAW2Vn2cLSeyJKZbkTcz4D8/OkrvJ+uPPpqW8ICcZa10VBpRO4QLe5b97eH65BYmTINlQXhXPRIa6a5o59Hdo/MNjzY0U/PYICNS8q4sLaUHTGRuzU7dSRyz2XAH+T0YGDc47F89IaqfG68qJainNGj94Pt/TRUFYTtn0hb5mTvEA+a3u1E6Bnw819/PUwoFH0hsqL2L1zZEI5Sk83YCIY0rx7rmXQp3dmMJcKpiNwBlpkXCRF3IaM52N7PbQ/v5YdPH4x7zZoM0lCVz/xiF0qNZKyEQpqv/vZ1PvfAHryBIG2nhxgKGLfWZfmGuEdOZLrnxSN85Tev8YY5iLjDjNQ31Jexob6Mlq6BcF47jETQkZE7JJlrH+53AQUuB9dfsJDtBzrwDUfbRIN+I8JvqMwn15lFQXZWVOT+q8ajfO13b4Tz/JPlgZ3H+M4f97MrZtWix/ae4KIlZayqKaayIJtCV1bCErWJOHCqnwF/MJzzn05cs3IeZ1UVcGFdaUr2d/WKKi5dVkHtGZa/SAWz9x7rz1+DU2+kdp/zzoV3fX/Ul//pn/6JxYsXc8stRmHLb33rWyileP755+np6SEQCPDd736XzZs3j/tRHo+HzZs3J3zffffdxw9+8AOUUqxatYr777+f9vZ2PvOZz9DS0gLAXXfdxUUXXZSCg5693PlME1obNWFiiykdbO8n12lnQVEONpuiqsAVFtemDk84l/2hncdZWGqI77KqfMryDEGOjNyt993xzEF+ceMFNLa4qS7OYWFp7siU8MPdvG/1AmAkgq6M8NyN/QyOW//F8tEXm6USVlQXhWuOnDO/MNzOyslfZkaMlYXZUbnu1kWi7fTQhAbfLItpR4s7LFhdHh9NHR4+uNawY5RSNFQVJB257zlmXCjSMXJfUpHPU7ddmrL9XbumJmx7zTQSuUdw/fXX89BDD4WfP/zww3ziE5/g97//PXv27GH79u18+ctfTsp7dblcCd+3b98+/uM//oNt27bx2muvcccddwDw+c9/nssuu4zXXnuNPXv2sGLFzFTFnC4OtvfzpzdOsqQij96hAG/FrKvZZE4HtwQ/clDTGgBdUmFU73uzzXjv0soCSs3IPXIiU2vPIFk2xda3DP97x+HusKgvX1BIQXZW1KBqZ1zkPr7nH+636aNnmfXELfGOjZKbIiJ8gKpCV1Tk3hSuqZP8QG4gGGLXEUPcGw+PHI91p7I+IjpdVpVPU3tyGTN7jvZQnp8dvsgJc4PZG7mPEWFPFeeffz4dHR2cOHGCzs5OSkpKmD9/PrfddhvPP/88NpuNtrY22tvbmTcvfjmwSLTW/PM//3Pc+7Zt28Z1111HebmR/mbVht+2bRv33XcfAHa7naKiuVEhcLLc+UwTuQ47P75hDe++8wUaW7qjlihravdwacRkmZqSnLDVYEXe39m8kr//xQ5+/kILVYXZFOU40FrjsKuwLaO1pq1niA+tq+GJN07xld+8RveAnw3mNHW7TXFhXWmUuFtZK5YXPpLrPr7QHmzv5/wI+8LKoGiOiZIPxkT4lQXZ7DYjZG8gGC6nO5EUzDfaehn0B6kuzmH30R58w0Gys4xZmHlOOyurR/6+DZUFPDB4nC6Pf9w7gz3Heli7uHjGsz+EiSGRewzXXXcdjzzyCA899BDXX389v/71r+ns7GT37t3s3buXqqoqvN7xU9ZGe5/UgB+J2m+8qJblCwpZXJYbJa69gwE6+n1RhZxqSnI52eslEAyFI29rQLTfOxzOdlBKUZrnDNsynR4fvuEQ58wv5FOX1I3kw8dU62vpHPHdO/p9FOc6ovKek8l1j/TRLawMitjIvTkmwrcid601hzo9WOOhE4ncrb/hLVcswRsIhScqNba4WVdbGrU60cig6ti++1H3AEfcg2lpyaQ7Iu4xXH/99Tz44IM88sgjXHfddfT29lJZWYnD4WD79u0cPXo0qf2M9r5Nmzbx8MMP43YbX0Rryb5NmzZx1113ARAMBunrS+0Sg7OJLc+3kOuw86l3GNPD19eV8srh7nCGx74ThihFZjBUm7nuf23qCkfeSim+eGVDXNuyvOywLWMJck1JDjdeXEuhKyvst1tYQv9CUxdgRO5VMTVJakpyONTpGdPGiPXRLZZVxvvbB9r7oybrVBRk4zNnqVqWTZ7TPuoFxRsI8nd3v8xje9vC2xpbullWlc+7zdmPjYfcYb89NnvDmmqfaGm4J944ybrvbmXtd57mb+/8Kw67YvN51aMetzA7EXGPYcWKFfT391NdXc38+fP58Ic/zK5du1i3bh2//vWvOfvss5Paz2jvW7FiBf/yL//CZZddxurVq/nSl74EwB133MH27ds599xzWbt2Lfv27ZuyY5xpjnQNsHphcThtcUN9WZTvfs+LhynOdXBhXXS5VCCc0miJ1cYlZXx78wo+trE23LYs3xmuLzMi7rkUuhzcecP5fPf9K6P6s3xBIfUVefz8hRZCIR01O9XinWdXcsQ9yHOjlCuAeB/doqEqn6PuwXDGzKleL609Q5y3cGRCkJV22dHvpamjnyybYkN92aji/tDO47xypJvvPfE2vuFg2G/fUF9GSZ6Ts+cVsONwdzhd1LKhLKyMmdjIPRAM8f0/v01etp13nTuP95+/gO9duyrcP2HuMHs99xnEWvAaoLy8nJdffjlhO49n9GyDsd534403cuONN0Ztq6qq4rHHHptEb+ceHt8wi/NHIuf14WJK3YRCsPWtDr5y9TLyIybMWIOaT+9vj4q8lVJRwg5QmufkiHsAGLE1qouNi8PlZ1XG9cduU3xhUwNfeHAvT+07RUeflyUV0cuofXBNDT/Z1sztW5u4bFlFQmst1ke3aKgqIBjS4YwZa7WhyGjaGrxt7/NxsN1DbXkedeV5vHioK87K8waC/OzZZuYXuTjZ6+XhncdZUV3EoD8Y3ueG+jIe3HmMhaU5cX679XdrqCoI21QWv3+1jWPdg/ziY+u4cnlV3DEKcweJ3IVpp987HDXTsbo4h0Wlhu9+xzMHKcpxxC1msKDYiBz9wVC4ZvdolOVlhz331p4hSnId486sfM+qBdRX5HH71iY6PfGRuzPLxq1XLGXv8dOjRu+xmTIWlgdvWTONLW4KXFlRqZFWZNze5zUnZOVTU5KDNxCKm2370M7jtPf5+L8fWs26xSX8dPuhcAE0K/1xQ30p3kCI3+1pY22M324RmzETCBr12ldWF7LpnPiLoDC3EHE/Q9544w3OO++8qMf69etnuluzGo9vmIIYsd1QX8rzBzvZ+lYHn35HHQUuR9Tr2Vl2qkzBHW/2X1m+M1xfpq1nKBz1j4Xdpvj8Oxs40N5PIKipSpBBct3aGqqLc7jjmaaE3rtV6CyW+gprlR5rIYdu1teVRi34bV1MjvcMctQ9YNbUiU/BtKL2C+tK2bikjC9euYxTfV7ufu4QDZX5lOcb+7EsLd9wKM6SsVhaWUDPYCB88XjUjNq/uGlZxg/6pwOzTtyno6Z1Kjn33HPZu3dv1GPHjh1T8lnT+bfRWsel76Vqvx7fMPmuWHEvwzccShi1W1hit3EccS+NqC/T2jOYdH72e1cvoN6sJpioXKwzy8YtVyzh1WPx0fugf5jj3UMJa4JbGTNN7R7a+7wc7hoIVyS0sGapvnzITUgbUXV1xOQpCytq/6K5YMbFS8tYt7gEbyAUddErNX13GP1iaA38Prb3BH9t6uIn2yVqTydmlbi7XC7cbvecE/jpQGuN2+3G5Zqega2dR3q48ofP8dS+Uyndr284RDCkyc+Ojsw3Likjy6a4+dL6uKjdYllVAfXleeOKdbh4mMdHa89Q0uJutym+eNUyAGrLEpeM/dDahSwocsVF71Ya4tmjLHzcUJnPwY7+hGtrWlQWZodngzZUFkSI+0jkfu/LR1i3uCR8gVNKcZvZ50tjFtG4bFkFJbkOzq1OPGfi7HmF2G2K7/xxPx/5rx0cdQ9y25UStacLs2pAtaamhtbWVjo7R89IyGRcLhc1NdMztfm4OYnm9q1NXHVOVVRpgDOh32ssSJGfHV07e35RDtu/cnl44DMR//qec/AGQuOKj1Vf5mC7B99wKClbxuJ9qxdw/sLiqFTJSIzofSnfePRNXmjq4tJlFWitufOZZqqLcxIO2IJxYdr6VgfPH+yiIDuL5QsK49pUFrg41DmA3aaoK8/DmWWjKMcRjtzb+7y0dA7w93+7KOpvcPHScl746hVxF7HbrlrGTZfUJfTbwUi//Mttl4bTRnMc8QOvwtxlVom7w+Ggrq5uprshMDJ9/62TffxlfzvXrBx7Rm6yeHymuLvi//VGE1SLXGcWuc7xP6PUrC/z2nGj1O5Ep82P148PravhZ9ubuX3rQd7RUM5zBzvZe/w037v23HCN+FiWVuYTDGn+9MYJLl5SHuW3W1hjCrVlueH9GGUXjMh9rKg/UZ9dDjsux9gLUCypyGfJ1K2aJ8wgs8qWEWYP7gE/DruivjyPO55piishO1k84cg9sfWSCizP3aqjPpHIPRmys+zccsVS9hw7zQtNXdy+tYnq4hw+OEbBKMuLj/XGI7F8/kjfPlrcu+OybARhNETchYS4PT7K8rL53Kal4eg9FYQj9ylc9KHQlYXDrnjrpJGdUj0FBa8+tK6GBUUuvvyb19h7/DS3XrF01KgdjBozVrA+qribGToNUeKeS2vPIFpro9JjbWnCqF8QYklK3JVS1yilDiilmpVSX0vw+mKl1DNKqdeVUs8qpWZHzctZztP72/lLigcsY+ny+LjzmSaCE4y8uwf8lOY5ee+qBeHoPRUD3Za4FySwZVKFVV/GHwxRkuuYkguJFb139vuoLs7hurVj/8u7HHZqy/JG9dthJHKPrqlj5Lq/dbKflq6BWbEIhDA3GFfclVJ24KfAu4DlwA1KqeUxzX4A3Ke1XgV8G/heqjuabvQOBvjSQ3u545nR19hMBY++2sYPnz7Im20TW+3ePeCnLN9Jlt3G9Rcu5K2TfUmtRDQeHp+xj6lers3y3VNtyUTyoXU1vKOhnH/523PGjNotPri2hhsvqh018l67uIR1i0uiJmlZ/f/dnuiyC4IwHsl8wy4EmrXWLQBKqQeBzcD+iDbLgdvM37cDj6ayk+nIf714mH7fcFTd8anAylVv6vCwemHyixt3D/jD60FasyfdA35K8pIY0RyDEc99asW93MyYmcoa5NlZdu6/KfkJa7desXTM16uLc3jkH6MXaLH6/+jetjGjfkGIJZlvWDVwPOJ5KxD7H/0a8EHgDuADQIFSqkxrHbU4pVLqZuBmgEWLFk22z3Oe3sEAv/zrYQDcHv+UlgG2Ss1OZDFkGPHcgfBPt8eXcAbmROifBlsGRgZV5/oCE9Z4QZfHz6azKyfvt2sNwQAEBs3HEPgHjJ+R2wKD4B+M37biA1CXuhWLhKknmW9Yov+mWPP1K8BPlFIfB54H2oDhuDdpvQXYArBu3bqMnalkRe0fOL+a37/aZkzHH2XizpmgtQ7XM0l2STUwprgP+IPhfPHwotMpuMsY8A2TZVNkJ2FjnAkj4j7za1kmRSiYQHSHKAwM8B7X64T8g/xdTim8sjeBEA9BYCDBthgh18Hx+xGFAkcuOHJgwfki7nOMZMS9FVgY8bwGOBHZQGt9ArgWQCmVD3xQaz0xkzdD6B0K8MsXD3PNinlcvLSc37/aRveAf0rEvb3PR793GJuKX+ZtLCwRtwTSsjhiC1hNBo9ZNGyqZ0FaNVZSErlrDcPeiIh2NDEda9sokbK1LTj6Qtg/AXACb5kPiyyXIbyO3BERduaBqxAK5o1sc+SC0/o9L8G23JGHM2JfWS6Q2apzlmTEfSfQoJSqw4jIrwf+PrKBUqoc6NZah4CvA/ekuqPpwoOvHKPfO8znNzWEl3NzD/hZPMp09zPBEvSNS8p4sdnNgG844UDmp+7dyWVnVfLRDYuN/pgVFa1p/CXh6fzR4v6lh/dSkuvkX98TO74+Ov2+4dT77cFAnHCeFTjMxbb9nNXrhTf02FZERKQcFmN/jC0Rd7M6DrasCCE1Rdf6PafE3BYhpAnbGj+/v/Uoja1eHvncJrJceSPvsY09QUnIbMb9lmmth5VSnwWeAuzAPVrrfUqpbwO7tNaPA5cD31NKaQxb5tYp7POc5oh7kPL8bJYvKCQQDAHxopkqLCvm3efO58VmN4c6PayqiR5UDYU02w90YlNqRNzN9UctW8Zht1HoyqJ7IDq6/GtTF10eHx/ZsJi68nEuTgEveE5R0NfM2qzTcNgxQYEdRXQDgxCKcwC5ErjSCTyZqDNq9Mg1tzxCdGMj35j2ibZZ4mxP3Z3Y5vesZkOfl6xyKeglJE9SIZTW+gngiZht/xbx+yPAI6ntWnrS5w1QmGP82S3xjBXNVNHU3k9pnjNcgfBge7y4uwf8BEM6qjjViC0zUvYW2u4XAAAe6klEQVS2PD87ypYJhjRdHh8hDT/Z1sz//bvVY3fm6Ivwq2v5d+v5vWO0Hc1uyC6A/KrobVa0m9BuiN02N+2Gc+YXyqxUYcLMqtoymUDfUIBC018PZ6FMUTpkk7noQ21ZLg67SrgYcke/YQ1FlpW1xN26+IDhv0feYbhNYS/Pz+bRvW187p1LqR0req9aAe+/i//zzFGyXPl8+d3njSLEYjcIQiqQ8gPTTJ93mMIcQ9xznHZyHPbwqkGpRGvNwfZ+GqryybLbqC/PT7gYckefL9yvPq8xwajLY9SViVxQozTPGZUt026+74tXNpBlU/x4W/PYHSqYB+f9PU+xkZaSi6HuHVCzFirPgZJayK+A7HwRdkFIERK5p4DWnkFcDns4Q2Ms+ocCLIzI4CjLd044cnd7fPR5h8f0uTv6jUwZqwhVQ1U+r5mFtKLbecO/t/UMUTjfQfeAj9I8Z1RGS1m+kz3HTse9b2V1ER/ZsJj/fukIG5eUkeeMFufiXCcbl4zMqvR441dhEgQh9ci3LAXcfN9uFpXmcvdH147b1vDcRwbbyvImJu5aaz7zq9209/l4/qtXjNrOypSxJh0tqyrgj6+fZNA/TK5z5LRbETgYi0KcM7/QrCsTfaEqy8umZ9BPKKSx2VT4fZUF2fzDZfU88MoxvvKb1xL25U+fv4QVC4w64Z6pyJYRBCEO+ZadIVprDnV66PclV3elzzsc9tzBsDs6PckPqL7Y7GbnEWO1nn5vYNT8eGtV+3Dkbor8oY4Bzq0ZWZCho9+L027DHwyFffcujz+c2x7Zz2BI0zsUoCTPGY7cKwqycdhtPPu/Lo+b5DTgC/Khu1/iqX3trFhQRDCkGfQHE9ZyFwQhtYjnfoZ0efz4hkO09gwx6I9PyYvEGwjiHw6Fs2XAyEhJNhVSa83tWw+GEz3GWuO0ucPIlLGsIquMbOxkpvY+H3XleeQ47OGMGasiZCRlMROZ2vt8lOc7w6v8VBa4OHteYdTDKIRVGq58OeCfnroygiCIuJ8xVrSrtREVj4U1YBkZuZebnnsy5XRfOuRm19EePnmxsVpVogFSi4Ptnqg6MIvNjJmDMRkzHf0+KguzzUUhjGNJKO6mTWNF5539XioKxl/P9eoVVbx9qp/j3YPTVjRMEAQR9zMmMj98vCn+fUOGuEV67qV5TvzDIQb8Y9f9sKL2eYUuvnL1WWRn2RKmNlptm9r7w6vbgzERqb48n+aYC0JHn5eqQld4xR9vIIjHNxw3OFwaseg0GJG7tSzcWFy1vAqAv+xvH3OJPUEQUouI+xliibvdpsYtzjUSuUenGALjpkO+fMjw2m+9Ygk5TjtLKvLDvnosHf1GNk1DZUHU9oaq/KjIPRTSdPYbIm2s+DMUV1fGIt6W8YZXDhqLxWV5nD2vgL/sOxWxOLaIuyBMNSLuZ0jb6UGKcx0srcgft6xu35Ah7pGDoJZodo0zS/WJN0+Sn53Fh9YZNdyWVY3+eXvNhaFXxNT+bqgsiBob6B70MxzSVBYYkXvvUICjbsOaiRX3ktyRypDW7FSrzvt4XLW8ip1HusO2j4i7IEw9Iu5nSGvPEDUlOTRU5ScRuRuiWhQxoBr2sseJ3BtburmgtiS8mn1DVQEner30e+OzdBpb3LgctrhSA8uq8qPGBqzCZVbkDiOLSsdmyzizjPoybo8vPDs1mcgd4Orl8whpeGyvUUxUbBlBmHpE3M+Q1p4haopzaags4HjPIENjeOdW5B6bCglj10rv7PfR3OGJWmLNSm1MlDHT2NLNusWlcUu/NZgevDU20NFv3C1UmJE7EJ7oFJvnDlBm1pex3leZZOS+srqQ+UUutr3dAUjkLgjTgYj7GaC1prVnkJqSnJGouHP06D3suefE2zJjTWTacdhY0Gp9hLhb+euxdwunB/28faqP9XWlxLK4LM+sMWO8pyMqcjfF/bhRhj/WlrG2dQ/4wxF/spG7UorLz6oIPy/ITn3tekEQohFxPwPcA368gVDYloGxM2b6hoZx2m1RqxDlOrPIcdjDWSiJaGxxk+e0szLCQ19YmmtkzMR83o7D3WgNG5bEL6TsCNeYMSP3Pityz6Y0z0mOw07b6SEcdhU16GthFQ+zIvdkPXeIvqDlZUv9GEGYakTczwArU6amJDccFY+WwQIj5X5jVyGKLcoVS2NLNxfUlZJlHzlddptKmDEz4rcXxe4GgKURYwPt/V5Kch1kZ9lRSoWj99i6MhZWTr4VuSdTS8fCYTP67nLYoo5DEISpQb5lZ4CV/VFTmjOSRz5K7jlEl/uNZKziYYn8douGqvw4z72xpZu1i0vIzkocHS+LGBvo6IvOeBkR98SiXZrnpGfQEPeyPGecpz8WWXbjYpHnFL9dEKYDEfczwIrcq4sNUVxaNXruORjZMgU58eI+VuRu+e2JxH1ZVQFtp4fCk4Msv31DXXxbiwZzbKC5w0N7v4+KCN/cypgpS+C3G/3MJhjSNLV7ot6XDFk2Q9ytbB9BEKYWEfczoLXHyHG38taXjZMxY0Tu8ZFrWV72qJ57Ir/dIjZjZiy/3cKatdrU0U+nOTvVworcy/ITi7uVHvn2qf4J+e1A2IpxOeRfThCmA/mmnQFWjrtFwzgZM7Hlfi3Kxqgv09jSzbra0oQ+dWwxsMYWN9lZo/vtMJIxc+BUv1FXJkHknihTJnK7xzecdKaMhUTugjC9iLhHsO9EL10TKL/b2jMUtmRgJCoeLWOmb2g4oedemufENxxiMCbiH8tvB1hkZsw89eYpHt51nO1vd4zpt4ORMVNXnseOw90Mh3TiyH0ccYeJZcrAiLhnT8CnFwRh8sg3LYKb/nsX//tPbyXVdiTHPTe8bXFZHnlOe3iyTiyRi2NHMlKUK9p3/8NrxozOS5aWJ9yf3aZYvbCYZ97u4KuPvM4R9yBXnFU5bt8bqgrCM1EjI/C6ijwKsrPCOfSxlEUMtCZTNCwS684jxymRuyBMB5K6EMHpIT8vHXKjtU6YChhJZI67hcNu42MX1XL3c4f4Ykc/SyMKd4VruSeI3MvDE5l8LCrLDbe/+7lDbKgvjVpcI5b7Pnlh+G7DphTzi8aPqBsq8/mT6QBFzjItdDnY/a9X4bAnPvbIyD2Zcr+RWPt0jXFXIQhC6pDI3SQU0ngDIU71ecPFs8YiMsc9kk+/o54ch507n4leMDrR7FSL0pha6QAPvHKMjn4fX9i0bMx+uBx2akpyqSnJZUFxzrgXJSAqMo+NwJ1ZtlH34cyyUWAOCE84cg/nuYu4C8J0IOJu4h0e8bsbW9zjtg/nuEdE7mBEtx/bWMsfXj8RlYMeruWeMFsmugSBNxDkrmcPsb6uNGpx6VQRWed9oimNVl+TrStjETQHi7MlW0YQpgX5pplEDmYmI+5tVo57jLgDfPoddeQ47Px4W1N4W/8YkXu4vozpuVtR+xevHDtqnyxWxow1O3UilJmzUismMDsVwBcw/r4SuQvC9CDibmLlpmfZFI0t3eG0xFO9Xr708F56B6NL6x7vGaQoxzHKjNNsPraxlsdfO0GLmRZplftN1D7XmYXLYeNXjUf58C8a+dHTB6csaoeRjJnKCfrmYNyZlE5wdiqANxACIEfEXRCmBRF3kyEzsrywrjTKd79960F+t6eNrW+1R7V/o60vyt6I5cPrF6G1se4pjJT7LUqQLQPwsY21zC9y4QuEOGd+If/87nPO+JjG4qZL6vjIxsUTft8H19TwqXfUTfh9Q+HIXf7lBGE6SCpbRil1DXAHYAd+obX+fszri4B7gWKzzde01k+kuK9TimXLXHFWJS8dctPY4sZuUzyyuxUwrJoPrq0BDIvlzbZebrl8yaj7qynJIc9pD/vuiRbHjmSqxTyW/++CRZN63zUr503qfcMh404oV2rLCMK0MO43TSllB34KXAW0AjuVUo9rrfdHNPsG8LDW+i6l1HLgCaB2Cvo7ZVhLz62sLqI8P5vGFjevtZ7GZlOsqSmi8fCID7/raA/BkB51chEYNcyXVhWEJzQlWhw7k/jUO+ro7Pfy8YtqZ7orgpARJHOPfCHQrLVu0Vr7gQeBzTFtNGAVPykCTqSui9OD17QNcp121teX8uzBTn6zq5UbLljIe1cv4Hj3EG2njUHUxhY3DrtizaKSMfe5rHKkkFifNxBXyz2TKHQ5+N61q8iTVZgEYVpIRmmqgeMRz1vNbZF8C/iIUqoVI2r/XEp6N41YtkyO086G+jJODwawKcU/Xr6U9WaVxR1mFk1jSzera4rHnW3ZUJVPl8dHz4DfKBqWoJa7IAjCVJCMuCdSo9gKVzcA/621rgHeDdyvlIrbt1LqZqXULqXUrs7Ozon3dgoJi7vDzkbTbrn+woXMK3Jx9rwCinIcNLa48fiGebOtd0xLxqIhYim8Pm/iujKCIAhTQTL3yK3AwojnNcTbLjcB1wBorV9WSrmAciCqyIrWeguwBWDdunXxJRBnkEhbZmFpLr/8xAVcWGusQ2qzKdbXldLY0s2uI93j+u0WVkneg+399A0FEtZyFwRBmAqSidx3Ag1KqTqllBO4Hng8ps0xYBOAUuocwAXMrtB8HCJtGTCyZiL94Q31ZRzrHuR3e9oMv31x8bj7rC4eyZjp8yau5S4IgjAVjCvuWuth4LPAU8BbGFkx+5RS31ZKvc9s9mXg00qp14AHgI/rRMXJZzGWuI9W2MqK1P/w+glW1xQnldIXmTFjeO4SuQuCMD0kFUqaOetPxGz7t4jf9wMXp7Zr04s3EMTlsGGzJR7wtHz33qFAUpaMRUNlPs8e6ESp0XPcBUEQUk1m5uUlYNA/PGY0bvnukHg909FYZmbMuD2+hLXcBUEQpgIRd5NBf3DcuifXrJxHRUF2Un67RYNZ0z2kJXIXBGH6kFDSxBsIjpu3fu2aGq5dUzOh/TZE1J8Rz10QhOlCIneTQX+Q3ClYAs7KmIHEtdwFQRCmAhF3k2RsmcmglGKpme8utowgCNOFiLtJMrbMZLFmqsqAqiAI04WIu8lU2TIwMlNVIndBEKYLCSVNhvxBchxT8+d47+oFnDg9RF153pTsXxAEIRYRd5OhQJAc59TcyCwozuHfN6+ckn0LgiAkQmwZk/EmMQmCIMwlRNyBUEjjDYRk8WZBENIGEXfAOxxdEVIQBGGuI+LOSEXIqcqWEQRBmG5E3DEyZQCxZQRBSBtE3DEyZUBsGUEQ0gcRd8SWEQQh/RBxZ8SWcYktIwhCmpCR4t7aM8izB0bW7h4KDANInrsgCGlDRor7ludb+If7d2Mt8yq2jCAI6UZGivux7kF8wyH6fUbELtkygiCkGxkp7q09QwB0e/yAZMsIgpB+ZJy4a61p7RkEwD1giLvYMoIgpBsZJ+7uAT/eQAiAblPcw9kyWSLugiCkBxkn7pYlA+D2+ADDlnE5bNhsaqa6JQiCkFIyUNwHw7+P2DJS7lcQhPQiA8XdiNwddhVhy0i5X0EQ0ouMC1dbewYpynFQmJMVYcsMS6aMIAhpRVKRu1LqGqXUAaVUs1Lqawle/5FSaq/5OKiUOp36rqaG1p4hakpyKM3LjsqWkUwZQRDSiXEjd6WUHfgpcBXQCuxUSj2utd5vtdFa3xbR/nPA+VPQ15TQ2jPEkoo8hoOaU31ewMiWkboygiCkE8lE7hcCzVrrFq21H3gQ2DxG+xuAB1LRuVRj5bjXlORSmucc8dwDErkLgpBeJCPu1cDxiOet5rY4lFKLgTpg25l3LfV0mznuNSU5lOY7cXv8aK0Z9AdlQFUQhLQimQHVRMnfepS21wOPaK2DCXek1M3AzQCLFi1KqoOpxMqUqSnJZTio8QdDeHzDDPmDMqAqCEJakUzk3gosjHheA5wYpe31jGHJaK23aK3Xaa3XVVRUJN/LFDEi7jmU5jkBI5oXW0YQhHQjGXHfCTQopeqUUk4MAX88tpFS6iygBHg5tV1MHdYEpmrTlgHo8vgZ9A+LLSMIQloxrrhrrYeBzwJPAW8BD2ut9ymlvq2Uel9E0xuAB7VVJH0W0tozZOS4uxyUmZG72+PDGwiRIzNUBUFII5JSNK31E8ATMdv+Leb5t1LXranByJTJAaAsPxuAE6cNq0ZsGUEQ0omMKj9gTWACwpH7cdOHF1tGEIR0ImPE3chxH6KmJBcwFsPOddrDPrxkywiCkE5kjLhbWTHVxTnhbWX5TtrElhEEIQ3JGHGPTIO0KM3L5ni32DKCIKQfGSPuxyPSIC3K8pz0DgUAsWUEQUgvMkbcXz12GmeWjSUV+eFt1kQmQBbrEAQhrcgYcW9scbN2UUlU9cey/BFxF1tGEIR0IiPEvXcwwP6TfayvL43aXhYVuYu4C4KQPmSEuO880o3WsKG+LGp7aV52+Hep5y4IQjqREeLe2OLGmWXjvIXFUdslchcEIV3JDHE/7GbNouK46Fw8d0EQ0pW0F/feoQD7TvTFWTIwki2TnWXDZktUtl4QBGFukvbivvNwYr8doMz03MWSEQQh3Uh7cR/Nbwdj4lKu0y457oIgpB1pr2o7Dncn9NstSvOcZGel/TVOEIQMI61Vrc8bYN+J3oSWjEVZnlMid0EQ0o60VrUjXQOENCyfXzhqm79ZOQ9fIDSNvRIEQZh60lrcB/1BAPKyRz/MWy5fOl3dEQRBmDbS2pYZChjiLhUfBUHINNJb3M3IXVIdBUHINDJC3GX2qSAImUZai/ug2DKCIGQoaS3uQ/5hQBbiEAQh80hzcTdSHMWWEQQh00hrcR8MDOPMsmGXomCCIGQYaS3uQ/6gZMoIgpCRzDlxP+oe4I+vn0iq7ZA/KJaMIAgZSVLirpS6Ril1QCnVrJT62iht/k4ptV8ptU8p9T+p7eYIT755is/+z6v0eQPjth0MBCVTRhCEjGTcNBKllB34KXAV0ArsVEo9rrXeH9GmAfg6cLHWukcpVTlVHa4pyQWgrWeIwvmOMduKLSMIQqaSTOR+IdCstW7RWvuBB4HNMW0+DfxUa90DoLXuSG03R6gpyQGgtWdo3LZiywiCkKkkI+7VwPGI563mtkiWAcuUUi8qpRqVUtekqoOxjIj74LhtDVtGctwFQcg8klG+RHmEOsF+GoDLgRrgBaXUSq316agdKXUzcDPAokWLJtxZMBbXyHHYk4rcvf4g8wtdk/ocQRCEuUwykXsrsDDieQ0Qm67SCjymtQ5orQ8DBzDEPgqt9Rat9Tqt9bqKiopJdVgpRU1JTpKR+7AMqAqCkJEkI+47gQalVJ1SyglcDzwe0+ZR4AoApVQ5hk3TksqORmKIe5Keu4i7IAgZyLjirrUeBj4LPAW8BTystd6nlPq2Uup9ZrOnALdSaj+wHfhfWmv3VHW6piQ3aXHPlQFVQRAykKRGG7XWTwBPxGz7t4jfNfAl8zHl1JTk0DsUoM8boNCVOB1Say157oIgZCxzboYqROe6j4ZvOITWUu5XEITMZI6K+/i57rJQhyAImcycFPdqU9zbxsiYsRbqkBmqgiBkInNS3MvynLgctnEid2OhDpnEJAhCJjInxd3IdR87Y0YW6hAEIZOZk+IOZq776TFsmfASeyLugiBkHnNb3MeK3GVxbEEQMpg5LO65nB4M0D9KXXfJlhEEIZOZw+JuZsycThy9D/olW0YQhMxlDou7MZGptTuxuIstIwhCJjOHxX3suu5iywiCkMnMWXEfL9d9xJaRPHdBEDKPOSvu4+W6DwWCOLNs2G2J1hoRBEFIb+asuINhzRzrHs2WGRZLRhCEjGVOi/u6xSXsP9nHm229ca8N+oOSKSMIQsYyp8X9oxtrKXBlceczTXGvDUktd0EQMpg5Le5FOQ5uuqSOv+xvZ9+J6Oh9yB8UW0YQhIxlTos7wCcurksYvQ8FxJYRBCFzmfPiXpTj4JMX1/HUvujofdAflHK/giBkLHNe3AE+eUkdBdlZ/PLFI+Fthi2TFocnCIIwYdJC/YpyHKyoLuRI10B4m2HLSOQuCEJmkhbiDlBZ4KKj3xd+PugP4pIBVUEQMpS0Efeqwmza+7xorQFjEpMMqAqCkKmkjbhXFrjwDYfo8w6jtZZsGUEQMpr0EffCbAA6+rz4hkOENGLLCIKQsaSNuFcVugBo7/OFy/1K5C4IQqaSNuJeWWBG7v3e8EIdIu6CIGQqSYm7UuoapdQBpVSzUuprCV7/uFKqUym113x8KvVdHZvKiMjdquUutowgCJnKuIngSik78FPgKqAV2KmUelxrvT+m6UNa689OQR+TIj87izynnY5+L96ALNQhCEJmk0zkfiHQrLVu0Vr7gQeBzVPbrclRVeiiIyJyF1tGEIRMJRlxrwaORzxvNbfF8kGl1OtKqUeUUgtT0rsJUlGQTUe/l0H/MCC2jCAImUsy4p5onTod8/wPQK3WehWwFbg34Y6UulkptUsptauzs3NiPU2CqkIX7X2+CFtGxF0QhMwkGXFvBSIj8RrgRGQDrbVba23N/f85sDbRjrTWW7TW67TW6yoqKibT3zGpLDBmqYotIwhCppOMuO8EGpRSdUopJ3A98HhkA6XU/Iin7wPeSl0Xk6eq0JileqrPCyCLdQiCkLGMm06itR5WSn0WeAqwA/dorfcppb4N7NJaPw58Xin1PmAY6AY+PoV9HhVrlurRLmPRbFlmTxCETCWpXEGt9RPAEzHb/i3i968DX09t1yZOZYGR637EbZT+lchdEIRMJW1mqIJRGRLgqHsQp91Glj2tDk8QBCFp0kr9rFmqp/q8YskIgpDRpJW4W7NUQTJlBEHIbNJK3GEkehe/XRCETCb9xN2sDim2jCAImUz6ibtE7oIgCOkn7lUSuQuCIKSfuFsTmWRAVRCETCbtxL1KbBlBEIT0E3drlmqOLNQhCEIGk37iLraMIAhC+om72DKCIAhJFg6bS+RnZ/G1d53NO8+unOmuCIIgzBhpJ+4An7lsyUx3QRAEYUZJO1tGEARBEHEXBEFIS0TcBUEQ0hARd0EQhDRExF0QBCENEXEXBEFIQ0TcBUEQ0hARd0EQhDREaa1n5oOV6gSOTvLt5UBXCrszG0n3Y5Tjm/uk+zHO1uNbrLWuGK/RjIn7maCU2qW1XjfT/ZhK0v0Y5fjmPul+jHP9+MSWEQRBSENE3AVBENKQuSruW2a6A9NAuh+jHN/cJ92PcU4f35z03AVBEISxmauRuyAIgjAGc07clVLXKKUOKKWalVJfm+n+TAal1EKl1Hal1FtKqX1KqS+Y20uVUk8rpZrMnyXmdqWUutM85teVUmtm9giSQyllV0q9qpT6o/m8Tim1wzy+h5RSTnN7tvm82Xy9dib7nSxKqWKl1CNKqbfNc7kxnc6hUuo28//zTaXUA0op11w/h0qpe5RSHUqpNyO2TficKaVuNNs3KaVunIljGY85Je5KKTvwU+BdwHLgBqXU8pnt1aQYBr6stT4H2ADcah7H14BntNYNwDPmczCOt8F83AzcNf1dnhRfAN6KeP5/gB+Zx9cD3GRuvwno0VovBX5ktpsL3AE8qbU+G1iNcaxpcQ6VUtXA54F1WuuVgB24nrl/Dv8buCZm24TOmVKqFPgmsB64EPimdUGYVWit58wD2Ag8FfH868DXZ7pfKTiux4CrgAPAfHPbfOCA+fv/A26IaB9uN1sfQA3GF+WdwB8BhTEhJCv2XAJPARvN37PMdmqmj2Gc4ysEDsf2M13OIVANHAdKzXPyR+Bv0uEcArXAm5M9Z8ANwP+L2B7VbrY85lTkzsg/nEWruW3OYt6+ng/sAKq01icBzJ/WQrBz8bhvB74KhMznZcBprfWw+TzyGMLHZ77ea7afzdQDncAvTevpF0qpPNLkHGqt24AfAMeAkxjnZDfpdQ4tJnrO5sS5nGvirhJsm7PpPkqpfOC3wBe11n1jNU2wbdYet1LqPUCH1np35OYETXUSr81WsoA1wF1a6/OBAUZu5xMxp47RtBk2A3XAAiAPw6aIZS6fw/EY7ZjmxLHONXFvBRZGPK8BTsxQX84IpZQDQ9h/rbX+nbm5XSk133x9PtBhbp9rx30x8D6l1BHgQQxr5nagWCllLcoeeQzh4zNfLwK6p7PDk6AVaNVa7zCfP4Ih9ulyDq8EDmutO7XWAeB3wEWk1zm0mOg5mxPncq6J+06gwRyxd2IM8Dw+w32aMEopBfwX8JbW+ocRLz0OWCPvN2J48db2j5mj9xuAXus2cjaitf661rpGa12LcY62aa0/DGwHrjObxR6fddzXme1nXSQUidb6FHBcKXWWuWkTsJ80OYcYdswGpVSu+f9qHV/anMMIJnrOngKuVkqVmHc4V5vbZhczbfpPYjDk3cBB4BDwLzPdn0kewyUYt3GvA3vNx7sxPMpngCbzZ6nZXmFkCR0C3sDIYJjx40jyWC8H/mj+Xg+8AjQDvwGyze0u83mz+Xr9TPc7yWM7D9hlnsdHgZJ0OofAvwNvA28C9wPZc/0cAg9gjCEEMCLwmyZzzoBPmsfaDHxipo8r0UNmqAqCIKQhc82WEQRBEJJAxF0QBCENEXEXBEFIQ0TcBUEQ0hARd0EQhDRExF0QBCENEXEXBEFIQ0TcBUEQ0pD/H8XxIbt+srKcAAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "tplt( lt, 'acc')\n",
- "tplt( lt, 'val_acc')\n",
- "plt.legend()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsvXeYZGWZ9/95KoeuzmF6pnumJ2dmgAFEJIoIroKRoKL4srCroqK7pp8JUVfFffV1XQzoIu4uCiziMhINJMmTc2By94TOsaq74vP745xTXdVd1V09U10d5v5cV1/Tp85Tp54zNfOtu77P/dy30lojCIIgTC9sEz0BQRAEIf+IuAuCIExDRNwFQRCmISLugiAI0xARd0EQhGmIiLsgCMI0RMRdEARhGiLiLgiCMA0RcRcEQZiGOCbqhSsrK3VDQ8NEvbwgCMKUZMOGDW1a66rRxk2YuDc0NLB+/fqJenlBEIQpiVLqcC7jxJYRBEGYhoi4C4IgTENE3AVBEKYhOXnuSqkrgR8DduBXWuvvDTn/I+BS89AHVGutS/M5UUEQpgfRaJSmpiYGBgYmeiqTGo/HQ11dHU6n86SeP6q4K6XswN3A24AmYJ1Saq3Weqc1Rmv92ZTxnwLOPKnZCIIw7WlqaiIQCNDQ0IBSaqKnMynRWtPe3k5TUxNz5849qWvkYsucC+zTWh/QWkeAB4BrRhh/A/C7k5qNIAjTnoGBASoqKkTYR0ApRUVFxSl9u8lF3GcBjSnHTeZjmSY0B5gLPJPl/K1KqfVKqfWtra1jnasgCNMEEfbROdW/o1zEPdMrZOvNdz3wsNY6numk1voerfUarfWaqqpRc/BPiv5InN+9foSuUGRcri8IgjAVyGVBtQmoTzmuA45lGXs98MlTnVSuaK0ZiCbwuuwAvHqgnS/9fiuH2kM0doT4wpVLCjUVQRCmEEVFRfT19U30NMaVXMR9HbBQKTUXOIoh4B8cOkgptRgoA17J6wyz0B2KcvNv1rH+cCdVATczSzxsaepmToWP+VV+ntvTKuIuCMJpy6i2jNY6BtwGPA3sAh7SWu9QSt2plLo6ZegNwANa62yWTd5o7wtzwy9fZWtTN/9w8TwuWVSF22nnHy+ez1OfuYj3nV3HzuM9tPRIqpUgCNnRWvP5z3+eFStWsHLlSh588EEAjh8/zkUXXcTq1atZsWIFf/vb34jH49x0003JsT/60Y8mePYjk1Oeu9b6CeCJIY99fcjxHfmbVnZOdA/woV+9ytGufn750TVcvGi4d3/JomruemoPz+1t5do19RmuIgjCZOCbf9zBzmM9eb3mspnFfONdy3Ma+8gjj7B582a2bNlCW1sb55xzDhdddBG//e1vefvb385XvvIV4vE4oVCIzZs3c/ToUbZv3w5AV1dXXuedb6bcDtX/Wd/Iie4BfvOxczMKO8DS2gA1xW6e3yMZOYIgZOfFF1/khhtuwG63U1NTw8UXX8y6des455xz+PWvf80dd9zBtm3bCAQCzJs3jwMHDvCpT32Kp556iuLi4ome/ohMWFXIk+WTly7gXatm0lDpzzpGKcXFi6p4cvsJYvEEDvuU+wwThNOCXCPs8SKbi3zRRRfxwgsv8Pjjj3PjjTfy+c9/no985CNs2bKFp59+mrvvvpuHHnqIe++9t8Azzp0pp3o2mxpR2C0uXVxN70CMjUcm91cnQRAmjosuuogHH3yQeDxOa2srL7zwAueeey6HDx+murqaW265hZtvvpmNGzfS1tZGIpHgfe97H9/61rfYuHHjRE9/RKZc5J4rFyysxGFTPLenhXPnlmcdd8faHZw3t5yrVtYWcHaCIEwG3vOe9/DKK6+watUqlFLcddddzJgxg9/85jf84Ac/wOl0UlRUxH/+539y9OhRPvaxj5FIJAD47ne/O8GzHxlVgOSWjKxZs0aPd7OOa3/xCn0DMZ74zIUZz8fiCRZ/7Snec+Ys/vUDq8Z1LoIgGOzatYulS5dO9DSmBJn+rpRSG7TWa0Z77pSzZcbCJYurRkyJPN49QDyhCYZjBZ6ZIAjC+DKtxf2C+ZUArD/cmfH84fYQAMFIxmoJgiAIU5ZpLe5LagM47YotTZkXVY90mOIukbsgCNOMaS3uboedpbXFbG3sznhexF0QhOnKtBZ3gDPqSth2tJtEYvjCcaMl7hERd0EQphfTXtxX1ZXSF45xoG14BbjByF08d0EQphfTX9zrjVauWzJYM4fbgwD0iS0jCMI0Y9qL+/yqInwuO1uHLKp2h6L0DMQo9jiIxBJE44kJmqEgCJOZoqKirOcOHTrEihUrCjib3Jn24m63KVbMKmFLU3rkblkyS2qN4j8hsWYEQZhGTNvyA6msqivhN68cJhJL4HIYn2eWuC+rLeb1gx30RWKU+JwTOU1BOP148ktwYlt+rzljJVz1vaynv/jFLzJnzhw+8YlPAHDHHXeglOKFF16gs7OTaDTKt7/9ba655poxvezAwAAf//jHWb9+PQ6Hgx/+8Idceuml7Nixg4997GNEIhESiQS///3vmTlzJtdeey1NTU3E43G+9rWvcd11153SbQ/ltBD3M+pKicQOsre5lxWzSoBBcV9aGwAkHVIQTheuv/56br/99qS4P/TQQzz11FN89rOfpbi4mLa2Nt70pjdx9dVXj6lJ9d133w3Atm3b2L17N1dccQV79+7l5z//OZ/5zGf40Ic+RCQSIR6P88QTTzBz5kwef/xxALq7M6drnwqnhbivqjMXVZu6UsQ9SIXfRXXAA4i4C8KEMEKEPV6ceeaZtLS0cOzYMVpbWykrK6O2tpbPfvazvPDCC9hsNo4ePUpzczMzZszI+bovvvgin/rUpwBYsmQJc+bMYe/evZx//vl85zvfoampife+970sXLiQlStX8s///M988Ytf5J3vfCcXXpi5/tWpMO09d4D6ci9lPmfaZqYjHSFmV/jwmc21JR1SEE4f3v/+9/Pwww/z4IMPcv3113P//ffT2trKhg0b2Lx5MzU1NQwMjK1NZ7YijB/84AdZu3YtXq+Xt7/97TzzzDMsWrSIDRs2sHLlSr785S9z55135uO20jgtInelFCvrStPKEBzpCHHW7DL8buOvQNIhBeH04frrr+eWW26hra2N559/noceeojq6mqcTifPPvsshw8fHvM1L7roIu6//34uu+wy9u7dy5EjR1i8eDEHDhxg3rx5fPrTn+bAgQNs3bqVJUuWUF5ezoc//GGKioq477778n6POUXuSqkrlVJ7lFL7lFJfyjLmWqXUTqXUDqXUb/M7zVPnrNml7G3u5XB7kGg8wbGuAWaX+ygyxT0ku1QF4bRh+fLl9Pb2MmvWLGpra/nQhz7E+vXrWbNmDffffz9LliwZ8zU/8YlPEI/HWblyJddddx333XcfbrebBx98kBUrVrB69Wp2797NRz7yEbZt28a5557L6tWr+c53vsNXv/rVvN/jqPXclVJ2YC/wNqAJWAfcoLXemTJmIfAQcJnWulMpVa21bhnpuoWo555KS88AF/3gWd6xopbbL1/ERT94lrvefwaXLq7mnO/8hW9ds5wbz28o2HwE4XRF6rnnznjXcz8X2Ke1PqC1jgAPAENzhG4B7tZadwKMJuwTQXWxh4+e38AfNh/lz7uaAZhd7sPvNjz3PvHcBUGYRuTiuc8CGlOOm4DzhoxZBKCUegmwA3dorZ/KywzzyD9ePJ/7XzvCvz69B4A5FT68Tjs2JdkygiBkZ9u2bdx4441pj7ndbl577bUJmtHo5CLumRI9h3o5DmAhcAlQB/xNKbVCa522518pdStwK8Ds2bPHPNlTpczv4pYL5/Gjv+zFZbdRE/CglMLvckhlSEEoIFrrMeWQTzQrV65k8+bNBX3NU22Bmost0wTUpxzXAccyjHlUax3VWh8E9mCIfRpa63u01mu01muqqqpOds6nxM0XzqXc76Ku3IvNZvzj8rntErkLQoHweDy0t7efsnhNZ7TWtLe34/F4TvoauUTu64CFSqm5wFHgeuCDQ8b8L3ADcJ9SqhLDpjlw0rMaR4rcDv79hjMJxwYLhfndDslzF4QCUVdXR1NTE62trRM9lUmNx+Ohrq7upJ8/qrhrrWNKqduApzH89Hu11juUUncC67XWa81zVyildgJx4PNa6/aTntU48+YFlWnHRW6xZQShUDidTubOnTvR05j25LSJSWv9BPDEkMe+nvK7Bj5n/kw5/C6H2DKCIEwrTovyA6Phd9slFVIQhGmFiDuW5y6RuyAI0wcRdwxxl/IDgiBMJ0TcMRZUpXCYIAjTCRF3wOeyMxBNEJM+qoIgTBNE3GGwMmRUFlUFQZgeiLhDsqa7LKoKgjBdEHGHlG5MIu6CIEwPRNwZtGUk110QhOmCiDuDtkxIIndBEKYJIu6kRu4i7oIgTA9E3Enx3GUjkyAI0wQRdwYjdyn7KwjCdEHEHUmFFARh+iHiDniddtQ49lG98487+eGf947LtQVBEDKRUz336Y7NpvA5x6/s7wtvtFJZ5BqXawuCIGRCIneT8awM2RWK0B+VujWCIBQOEXeT8aoMqbWmKxRlICKLtYIgFA4RdxOf2z4unntfOEYsoemXomSCIBQQEXcTv8tBcByi665QFEDEXRCEgpKTuCulrlRK7VFK7VNKfSnD+ZuUUq1Kqc3mz9/nf6rjS9E4tdrrDEUAxJYRBKGgjJoto5SyA3cDbwOagHVKqbVa651Dhj6otb5tHOZYEMarj2qnRO6CIEwAuUTu5wL7tNYHtNYR4AHgmvGdVuHxu8cnFbLLjNxjCU1UOj0JglAgchH3WUBjynGT+dhQ3qeU2qqUelgpVZ+X2RUQv2t8UiEtzx0kehcEoXDkIu4qw2N6yPEfgQat9RnAX4DfZLyQUrcqpdYrpda3traObabjjJHnHieRGHprp4bluYP47oIgFI5cxL0JSI3E64BjqQO01u1a67B5+Evg7EwX0lrfo7Veo7VeU1VVdTLzHTeSxcPyHL1L5C4IwkSQi7ivAxYqpeYqpVzA9cDa1AFKqdqUw6uBXfmbYmHwuY2yv6E8R9ddqZG77FIVBKFAjJoto7WOKaVuA54G7MC9WusdSqk7gfVa67XAp5VSVwMxoAO4aRznPC6kNuyoyeN1OyVyFwRhAsipcJjW+gngiSGPfT3l9y8DX87v1AqL3zU+ZX+7QhHcDhvhWIJ+8dwFQSgQskPVxLJl8l1fpjMUpbbEA8CARO6CIBQIEXeTomST7Px77jNMcRdbRhCEQiHibuIfhybZsXiCnoEYM0u8AGLLCIJQMETcTWpLPBR7HDy8oQmt85Pr3t1vLKbWlkrkLghCYRFxN/G5HHzubYt4cV8bf9rZPOr4g21BegeiI46xMmVqzchdPHdBEAqFiHsKH37THBbVFPGtx3aOKMSxeIKr//1Ffvbc/hGv191v5LhbC6piywiCUChE3FNw2G3c8a7lNHX288sXDmQdZ0TtMRo7+0e8XmfQiNwri9w47UpsGUEQCoaI+xDevKCSq1bM4O7n9tHeF844ZufxHgDaejOft7DqypT5XHicdhF3QRAKhoh7Bj52wVwGogm2NHVlPL/zmCHurVnE38KqK1Pqd+J12sVzFwShYIi4Z2DxjAAAu0/0ZjxvRe6to0TuXf0R7DZFwO3A67KL5y4IQsEQcc9AidfJzBIPezKIu9Y6Gbl390cJx7ILdmcoSqnXiVIKr9gygiAUEBH3LCyaEcgo7i29YdqDEZaY0X1bX2TYGIuuUIRSnxPA9NylKqQgCIVBxD0Li2cE2N/aN6w1nmXJXLzIqEc/kjXTGYxS5nMBGJ57Fltm45FOPvDzl8WTFwQhb4i4Z2HJjADRuOZgWzDtccuSucgU95EyZrr6o5Ra4u7KbstsOtLFukOdHO0aObVSEAQhV0Tcs7C4phgYvqi683gP9eVe5lb6gZEzZlJtmZE8936z+1NnMLvFIwiCMBZE3LMwv9qP3abYO0Tcdx3rYVltMRVFRkQ+oi0TilCW6rlnsWWs7k8dIu6CIOQJEfcsuB125lb60yL3YDjGwfYgy2pLcDvslPqcWcV9IBpnIJpIsWVsWT11EXdBEPKNiPsILJ4RYE9zT/J494letIZlMw3LpqrInVXcrQ1MqQuq2W0ZU9xDk1vco/GEWEeCMEUQcR+BJTUBGjv6kzXerUyZpLgH3Fk9d6v0wFDPPVM54eAU8dx//dJB3vaj5/NWElkQhPFDxH0EFpm57G80G9bMzmM9yQ1OYIp7lsh9qLi7nXa0hnBseK67Fbm3T3JxP9gWoq0vkvdWhIIg5J+cxF0pdaVSao9Sap9S6ksjjHu/Ukorpdbkb4oTh7VRac+JXg61BXl86zHOnlOGUgowqj22ZYncuzPYMpC5prvluU/2yL3L/MCyql0KgjB5GVXclVJ24G7gKmAZcINSalmGcQHg08Br+Z7kRFFf5sPnsrPpSBf/8F8bsNkU37x6efJ8VcBNKBInmCGS7Rwq7i5D3DP57qHo1FhQteY32dcGBEHILXI/F9intT6gtY4ADwDXZBj3LeAuYCCP85tQbDbFwpoAD65v5I2WXn5yw5nUl/uS56uK3EDmdMhMnjtkbthh5blPtGiGIrFhO3JTse6pU8RdECY9uYj7LKAx5bjJfCyJUupMoF5r/Vge5zYpWFxTBMAXrlzChQur0s5VBUxxN62ZrlCE//unPXzmgU387vUjeJw2PKaoW39mityDYcuWmVi7470/fZkf/Xlv1vPWt5HJbh8JggCOHMaoDI8l0yWUUjbgR8BNo15IqVuBWwFmz56d2wwnmI9dMJdFNQFufsvcYeeS4m5G7g+tb+Qnz+yjvtxLXZmXD543eI+WLZPJc7cEvy8cIxyL43bY834foxGLJ9jb3Mv8qqKM57XWSVG3RF4QhMlLLuLeBNSnHNcBx1KOA8AK4DlzoXEGsFYpdbXWen3qhbTW9wD3AKxZs2ZK5NMtrS1maW1xxnNDxf3VAx3Mq/LzzD9dMmzsoC0z3PYIRWIUuR30hWN0BqPMKCm8uLf1RUho6MnS9LsvHCOWMN6ysUbu/ZE497xwgI9fMh+XQxK0BKEQ5PI/bR2wUCk1VynlAq4H1lontdbdWutKrXWD1roBeBUYJuzTkTKfC5syxD2e0Kw72MGb5lVkHOvNYsskEpqBaIK6Mi8A7cGRG4CMFyd6jKWSnoHMaY6pltFYPfefPb+fH/1lL797/cjJT1AQhDExqrhrrWPAbcDTwC7gIa31DqXUnUqpq8d7gpMZu01RYaZD7jzWQ284xnlzyzOO9bqMv+qhtowl9pa4T5TvfqLbEPfe/syvn7rYO1Zxj5mLtL1ZvhUIgpB/crFl0Fo/ATwx5LGvZxl7yalPa+pglSB49UA7QNbIPduCqpXjXldmZOFMVMZMS+8okbs5L5fdNuYPIMuKiWTYwCUIwvggBugpYpUgePVAO/Mq/dQUezKOy7aJKWSmQc4qNSL3jlGabo8XVuSezXO3fPaGSt+YI3dL3MMjpFkKgpBfRNxPkaqAmxPdA7x+qIPzskTtkLKJKZI5cq8t9aAUdExQJorluUdiiYwZPdYGpnmVRWMXd7tE7oJQaETcT5GqgJuW3jC9AzHeNC+z3w7gcYxsyxS5HZR6nROWQ97cM7j3rDeDNdMVimK3KeZU+OgMRsdUPMxtRe4i7oJQMETcTxFrlypk99vB2O3qdtiGibsVyftcDsr9rgkrQXCiewCzZE5Ga6YjFKHU66Tc7yISTyQ/lHJBPHdBKDwi7qdIpZnrPpLfbuF1DW+SbXnuPpd9QsW9uSfMbLO0Qk+GjJnOYIQyv4syv1ErZyzztAqtibgLQuEQcT9FrMj9vBEsGYtMDTusCNjrslPmmxhx7wvH6AvHWFhtVMHMlDHTGYpQ7nMlC6F1jWFtIG5ufhJxF4TCIeJ+isyp8GG3KS5ZXD3qWEPc0wUulLRl7FQUuSYkFdLKlFlk1tHJlI/eGYxS6nNS7jcKoY1lnklxl2wZQSgYOeW5C9mZWerl1S+/NVmKYCQyNclO2jJOB2U+F53BCFrrpJVRCKzF1IWmuPf0D4/cO0IRzpxdmuwJO5aFX4ncBaHwSOSeB3IRdjA99ywLql7Tc48ldNaNRADtfWFu/c/1yU1H+cCK3AdtmfTIXWtNV8jw3MstcT+ZyF3EXRAKhoh7AfE4h2fLhKJxnHaFy2Gj3D96VPzsnlb+tLOZ53a35m1eVo77vCo/dpsaZsv0hWNE45oyn5NirxOlTi5yl01MglA4RNwLiDeDLdMfiSd3r1qZKCP1Ut3S2AXAjmPdeZtXc88AxR4HPpeDgMcxzJaxyg2U+VzYbcrIxx/LgqqWyF0QCo2IewHxOIfbMsFwDJ/LWPooz8HP3pwU9568zetE9wAzzKbfxR7nMFvGsmCsbxZlvrEt/A7aMrnnxguCcGqIuBeQjKmQ0Tg+szSBJZ7ZhHMgGmfX8R5sCnYd7yGRyE9J/OaegWSOfrHXMWyHakeyZaAp7n5Xsll2Lki2jCAUHhH3AuJ1DRf3/kgcn3uIuGeJ3Hcc6yGW0Fy2pJpgJM6h9mDyXCyeIHySkfGJngFmmOIecDuHbWKyvkmkRe5jqAyZ9NyjIu6CUChE3AtIJs89FInhcxq2jM9lx+2wZbVlLL/dat+Xas187dHtXPeLV8c8p1g8QWtveNCW8Toy2DLGsWUblfmcJxW5Z+ofKwjC+CDiXkA8TjvhWCLNTumPxJMVI5VSI5Yg2NzYxYxiD29ZUIXTrpLiHokleGzLcbY0ddEXzp5GmYn2oNFeL2nLeJzDbJnOYASbgoDHXBsYY5kEa0E1U7VJQRDGBxH3ApJskp1inwQjg547MGIJgi1NXayuL8XlsLGwOpDMmHntYDu94Rhaw84xLrRaOe5JW8aTwZYJRYyWgjZjY1Wpz0U4lhj2LSQbVuQejWui4rsLQkEQcS8gg02yB0UxNXIHspYg6AxGONweYlV9KQDLZxaz81gPWmv+tKM5WTN929GxpUhaOe6ptkwwEk+2xgNT3E2/HRhzCYLUbxNizQhCYRBxLyCZmmSHIjH8rsEqEGU+F0faQ2n11QE2Nxl+++oUcW8PRjjRM8CfdzZz2ZJqaordbB+juFuvU11s7LIt9hjCnSrIHcEIZT5n8ngsJQja+8I8uulo8jjXaF8QhFNDxL2AeFzDW+2Fhtgy151TTygS5+/+7UVe3t+WfHxLYxdKwcq6EgCWzzL+fOD1Rk70DHDF8hpWzioZe+TePYDDpqj0G+Ju+eqpG5m6QtFkNUgYzJrJpQTBvz+7j/5onNsuXQCIuAtCochJ3JVSVyql9iil9imlvpTh/D8qpbYppTYrpV5USi3L/1SnPoO2jGF5xBOacCyRZstcsKCStbddQInXwYd/9RrfeHQ7O451s7mxi0XVAYrchvgurS1GKbj3xYPYbYrLllSzYlYJ+1v7Rl1UbewI8e3HdvLlR7bx1I4TVAfcST+92GtE6KkZMx3BSFLQgWQUP9ou1caOEP/96mGuXVPPilnFAGNq8iEIwskzqrgrpezA3cBVwDLghgzi/Vut9Uqt9WrgLuCHeZ/pNGCoLWP9mRq5AyysCbD2trfwgbPr+d3rjfzdv73Ic3taWVVfkhxT5HbQUOGnNxzjvLnllPpcrJxVMuqiqtaa2x/czH0vH+LPO08wEInzjpW1yfOWLWOJu9aazlAkacUAySh+NFvmh3/ei00pbr98EV7TehLPXRAKQy4lf88F9mmtDwAopR4ArgF2WgO01qlq4gfys3VymuF1GZ+llsCFzAjb6xr+NvjdDr7//jP40lVL+OPWY/x5ZzPvObMubcyymcUcbAtyxbIaAFaaVs22o92cOzdz85DHtx1nw+FOvvfelVx/7uxh54faMsFInGhcJxdRAUrM6H6kdMgdx7r5381HufWiecwo8XCkI2Tcu0TuglAQchH3WUBjynETcN7QQUqpTwKfA1zAZXmZ3TTDMyRbJtmow2nP+pwyv4uPnN/AR85vGHbuzPpSntx2nLctnwFAdbFnxEXVgWic7z25m6W1xXxgTX3GMSVDbBkrOk/13B12GyXe7BuZYvEEX35kG+U+F5+42PDarW8nErkLQmHIxXPP1DViWGSutb5baz0f+CLw1YwXUupWpdR6pdT61tb8laydKli2jLWgaom7351d3Efiw2+aw2OfupBZpd7kYyMtqt770kGaOvv52t8txW7L3AzEsmWsjUwdGcQdzI1MWTz3X790iK1N3dxx9XJKTH/e+mCzmpMIgjC+5BK5NwGpYV4dcGyE8Q8AP8t0Qmt9D3APwJo1a04768brGuq5Z7dlcsHjtLNsZnHaYytmlfDX3S0EwzH87sHrtveF+emz+7l8aQ1vXlCZ9ZpFSVvGEG4rl70sxZYBKPU5eWV/Ozf+x2v0hWOcPbuMWy+eRygc51//tIfLl9bwzjMGvXxfhkwhQRDGj1wi93XAQqXUXKWUC7geWJs6QCm1MOXw74A38jfF6cPQTUyp/VPzRXJR9Xj6ouqL+9roC8f49FsXjPh8u01R5B6sL3Og1ShOVl/uSxt3xbIZlHgd9IVjOO02fv3yIS78/rPceO9ruBw2vvOeFWmtAr3JyF3EXRAKwagho9Y6ppS6DXgasAP3aq13KKXuBNZrrdcCtymlLgeiQCfw0fGc9FTFMyRbJhg2W+yN4LmPleSialM35zQMLqruOdGLw6ZYMqM421OTFHsGy/5uP9pNTbGb6oAnbczHL5nPxy+Znzw+1Bbkp8/t4w+bjvLd956RrFVjMfRbiyAI40tOfoDW+gngiSGPfT3l98/keV7TErfDht2mknnoli2Tz8i9uthDdcA9zHffc6KX+VVFuByjf1kr9g7Wl9l+tJsVM0tGeQY0VPq56/2r+Jf3rMRhH/4abocNpSRbRhAKhexQLSBKKWaVemk00wIHF1RPznPPxhl1JWw1yxVY7GnuZdGMQE7PD3gMWyYUibG/tY8Vs0YXd4tMwg7GvfsylDwWBGF8EHEvMA2V/mSTDUvovHmM3AHOqCvlQFsw2ei6LxyjqbOfxTVFOT3fKvu763gvCc2YxH0kvC4gWkOYAAAgAElEQVQ7IbFlBKEg5DdkFEZlboWPDYc60FrnlOd+MpxRZyyqbj/aw/nzK9jb3AvA4hz8djBsmb0tvcmSwlbpgJOm8XVY+2nuS8Rx7gvAA7XgKgJ3Ebj84AoYf2Y8tn78xo/KnMIpCEI6Iu4FZm6ln2AkTmtfmFAkjstuy2plnCxn1BmVI7c2dRnifsIU95ox2DL9MbYf7abC70rWej9p7C6omE+4swlfrB86DkC4DyLmTzzXxh/KFPmizOI/6nFgyPMDYHeO/rKCMAURcS8wDZV+AA61hQhFYnm3ZMDYYFRX5mWruai6+0QvPpedujLvKM80MGyZKFubulk+qyQtpfGkmLkarr+fO+9+iRKvk//8P+emn49FTKEPDv4Z7k057jM/DLIc950Yfj5X7K4s3x6K5NuFMKURcS8wc5PiHhxW7jefpC6q7m3uZWFNIFn5cTSKvQ4S2liEfevS6rzNyeu0MZBpQdXhAkc5+DLXwxkziQREQ+liHwmmf1vIeJzygdLXMjm+XbiKjL8fQRgjIu4FZlapF4dNcbA9SP+4inspT2w7QUcwwp4TYxPpgFmCQGtySoPMFZ/LQUvvwOgDTxWbzRBNdxFQk59ryrcLYYoh4l5gHHYbsyt8HGwNEo7F8Z1k6YHROMNs6vHcnhbag5GcF1NhsL4M5C9TBozNWlM2FXI8vl3E+nP4NjHB3y4WXwWzzsrPPQsFRcR9AphbYaRDlnid4+K5w6Ao/8/6JiD3xVQwbBkwdqrm6tPngtc1hcU939hsg1F0lm8Xu473cOcfd3LvTefk9u8kr98uzMdK6kTcpygi7hNAQ6Wfl/a3saC6iMoi97i8RrHHybwqP68caAdgcY4bmGDQllmRj8XUFHwuu5QfGAPrDnXwyoF2DrYFhxWIy8h4fLuQ1gxTFtnENAE0VPoZiCY41BYaN88dYJWZElnud1FZlPuiXLFZGXJlHi0ZMGwZKRyWO91mSeWRmqKMKzYb2Mbv36cwvoi4TwDzzIyZvnBs3Dx3GBTnxTWBMUXgM0u9nD2njCvMJiD5wuuyE44lSCQkGsyFbrO+T3swPMEzEaYiYstMAFauO+S3aNhQrJ6rY7FkwKhe+fuPvznv80ntIZvvejrTkaS4901Q5C5MaSRynwBqiz24zeqM47WgCrB8ZgkrZhVz6ZL85aqfCtJqb2xY4j5htowwpZHwaQKw2RQNFX72NPfic47fW+Bx2nnsUxeO2/XHytAessLIDNoyIu7C2JHIfYJoqDQ6G42nLTPZsNYXTrfIffeJHt71kxdp7R2bdz5oy4jnLowdEfcJwvLdx9OWmWx4XcY/t9MtY2b9oU62He3msa0jtR4eTo/YMsIpIOI+QcytMMTd7z6NxN20oE43W6alxyi58PjW42N6nnjuwqkg4j5BzKsyGmcUuU+fkrODfVRj4/o6z+xu5ld/OzCurzEWmnsMW2X94U5OdOdWWycWTxA0PwTbxJYRTgIR9wlizZwy7nrfGVy0qHKip1IwktkykURertfeF+bDv3qN4939aY//z/omfvbc/ry8Rj5o7h2g3G9sIntye27Re4/ZoLzc76JnIEY0np+/M+H0ISdxV0pdqZTao5Tap5T6Uobzn1NK7VRKbVVK/VUpNSf/U51e2GyKa8+px+04nWwZ415DkfxE7q8e6ODFfW1sPJzeL7YjGKE9GMm7IGqtT6qqZXNPmLNml7FkRiBna8ayZKwS0Z1izQhjZFRxV0rZgbuBq4BlwA1KqWVDhm0C1mitzwAeBu7K90SFqY9lywzkKVvGah/YGUoXvq7Q+Gz+ee1gB+f9y195zazXkystPQPUFLt5x8rarNbMe3/6Er97/UjyeKi4t8lGJmGM5BK5nwvs01of0FpHgAeAa1IHaK2f1VqHzMNXgbr8TlOYDgxG7nESCc1nHtjEM7ubT/p6SXEfEtVaYp/v2vG7jvegNfzihXQ//xuPbuer/7st43MisQTtwQg1xR7esbIWGG7NhGNxNh7pYsPhzuRjlrjPqzLEXRZVhbGSi7jPAhpTjpvMx7JxM/DkqUxKmJ6klh94ZncLj24+xv9uGlt6YCqWuHekRO5a62TkPta88tE40mHEL8/sbmFfi/HaGw538ptXDvPs7taMz2k1F0Nrit0sqC7KaM1kmm9S3M3IXerLCGMlF3HPVHEqY+UnpdSHgTXAD7Kcv1UptV4ptb61NfN/BmH6YrMp3A4b/ZE49750EDCi4ZMhHItzqN0QW0scwfhWEDG99nyLe2NHiJklRumIX/3tIFprvv34TgBO9AwQz1AQrdlMg6w2m4xfvKiKTY1dacXTLPuoJYO4z60sShuTTxIJzS9fOMB5//IX3jA/KIXpQy7i3gTUpxzXAcPCLaXU5cBXgKu11hn/V2mt79Far9Far6mqqjqZ+QpTHJ/LzqYjXby8v53KIhf7W/tOyoM/2BZMimmqZZH6e8sYxD0WT/DdJ3fxvSd3Zx1zpCPE8lklvPesOh7ZdJT7Xj7EpiNdnDW7lHgi82KrleNeEzDEvabYQzyhk+KdOufUDyNrA9Psch92m8q7LdMRjHDzb9bxnSd20dwT5vFtY8vBFyY/uYj7OmChUmquUsoFXA+sTR2glDoT+AWGsLfkf5rCdMHncvD6oQ68Tjv/dMViEnrQXhkLe04Yz6kpdqctqKZG8blG7l2hCB/99ev84vkDWXeRaq050hFidrmPv79wLpFYgm/+cSfLaov55KULADjW1T/seVaOe02x0ZSlwqyrn2qzWLZSRzCc/MDq6Y/idtjwuuyU+Zx5rS8Tiyd4z09f4qV97XzrmuWsri/l+b3yTXq6Maq4a61jwG3A08Au4CGt9Q6l1J1KqavNYT8AioD/UUptVkqtzXI54TTH4zT+yb3v7FmcP68CgJ3Hxm7NvNHch92mWDOnPE3cU3/PZUH1aFc/7777JdYd7GRxTSAtok6ltS/MQDTB7HIf86uKuNxsOP7Vv1tKfblRJ+hY1/DXa+4ZwGlXlPkMUbc6b6Vmv3SYvnxCD4p+d3+UEq+xwa3C76Yjj577sa4BDreH+No7l3Lj+Q1cvKiKLY1ddIVk0XY6kVOeu9b6Ca31Iq31fK31d8zHvq61Xmv+frnWukZrvdr8uXrkKwqnK1bxsJvePJfZ5T78LntOvvv3ntzN3c/uSx7vbe6locJHTbGHzuCgIFviXlviSYvco/EE7/3pSzy3J/2L5e9eO0JjZz+/veU8rlwxg96BWEbvvNFcTJ1tCvk33rWcH1+/mjcvqKS2xLBcskXu1QEPNpuxdJWM3FPFPeXbRkvPoLgXm+Je7nfl1XNv7DTuZb65S/rixVUkNLy4ry1vryFMPLJDVSgoS2YEuHrVTBZUF2GzKZbUFrPr+Oi2zOPbjvGz5/Yn69K80dLHopoAZT4nfeEYkZixiGqlRS6sCaR57ofbQ2w80sUzu9PFfW9zL3Mr/axpKKfUZ4hpT4bo3cqUsaL0+nIf16w2ksYCHicBjyOjuLf0DlBdPNgnt8Jv/J5my6T8bmXXpEbu5UWuvHrujUPuZVVdKSVeJ8/vEWtmOiHiLhSUH3xgFT++fnXyeFltsZk/nr31ntaa1t4wfeEYf9p5goFonEPtQRbWBCgzt/VblkKnGQUvrC6itTecvO6htiAw6NVb7GvpY2G1EcFaYtqVSdzbDeGuK/NmnOOsUi9Hs9gy1mIqGFG4UkNsmWAkaVdZ3zZSxb3S78qr597YGcJuU8lvHHab4i0LK3nhjdYR3wdhaiHiLhSc1H6uS2uL6Q3HaOocHvVa9IVjDESNyPz3G4+yr6UPrY3esFbNFmtRsisUodjjoLbEQziWoDdslDo41G6I+xstfcnrGumUwaS4W5F7Jt/9SEeIGcWeZMORocws9Wa1ZWpSIne7TVHuc6UVA+sIRlhYbbRCzCTu5X433f3RvJVTaOrsp7bEg8M++N//4oVVNPeE2SMpkdMGEXdhQllaa4jajhEWVS3Bm1Ph48U3WpPe8KKaoqQgW7ZFZyhKud9FVcAQVMvDtsS9IxhJCuvBtiAJDfOHRu4ZFhYbO0NJvz0TM0s9HBtSwGwgGqe7P5rMcbeoKHKlNeDoCEaYWeoh4HZkFnfTp89XfZnGjhD1Zen3cqFZwO4FyZqZNoi4CxPKkhnF2NTIm5ks7/yWC+eR0PDz5/fjtCsaKv3JyN1aVO0MRSj1uagys1IssTzUZlgRAHtNa2afGcVbUXOJ17hWpsi9sSOU9KgzUVvipSsUTSuK1pJMgxwi7n53+oJqMEq5301VwE1rr5EO2TsQSy6oVvqt9Mk8iXtnP/Xl6fZSbYmXxTUBSYmcRoi4CxOK12WnodLPzhHE3RLo8+aWc+bsUrpCUeZW+nHabZSbKYadSc89QpnPmVzEtNIhD7YFOW9uOTCYV/9Gcx82NVi/xYqUh4r7QDTOiZ6BESP3WaWGWKamQzabr51qy4AZuZtCnUhoOkMRyv3OpLj3meV+S1KyZSD3XarP7G5m0VeeZPFXn2TVN//EJ3+7Me1eWnvDwyJ3gIsWVbLuYCcD0TjReIJP/nYju0+c3A5iYeIRcRcmHGtRNRuWuFcF3LzvLKMm3aIaI9ou9aVbFp3BKGU+F1VFnuRzB6JxjnX3c05DOSVeJ3vNiH1fSx+zy31JH33QlkkX96Nd/WgNsysyL6aC4blDejqkVXpgaOReWeROWkM9A1HiCT0YufeFkx8uyTz3DBufRuLPO1twOWzcdEEDi2sCPLntOEFz7aGpMz1TJpVz51YQiSfYfrSbE90DPL71uNg0UxgRd2HCWVpbTFNn/4gbiJx2RYnXybvOmEnA7eDM2WUAuBw2ityOtAXVUp+LYq8Dl8NGa2+Yps4QWhvlcxfVFCVtmTdaellgWjLWtXwu+7B5HBmS456JmaXDc92Tu1MDQ8XdRe9AjHAsnlwrsCL3lp6B4eJupk/mmg656UgnZ80p48tXLeUfLjasLOubUWOHMb+htgzAqvoSADY3diX73HYEM78nwuRHxF2YcJbNLAbgxTcyb6Jp6QlTVeRGKUWJz8nfvngpHz1/sB9Mmd9JVyhKOBYnGIlT7neilKKqyLA5DrYZ4txQ6WdRTYC9zb1E4wkOtgVZWFOU9lqlXuewyH1oXngmaoo92FS6uLf0DOB22Cj2OtLGVhQNivWguLupDngIRuLJzlLFHuN5JV4ndpvKyZbpC8fY09zLWbNLAVg5yxDsrU3dxr2YkXtdBlumOuBhVqnXFHcj0pcmIVMXEXdhwjm3oZz5VX5uf3ATD647Mux8a1+YqhRro9TnSkvjK/cZm3wsUbasmupiw+awctznVhji3jMQY93BDqJxnUyDtCj2OodH7u0hPE5bcpE2E067jeqAJy3XvblngJpiT1rqJ0CF6aG39UaS3nu5bzDDZ1+rYRuVmJlANpvKub7MlsYutCb5zaa62MOMYg/bmoxuVU2d/bgc2e9ldX0pmxu7kpvF8plfP97csXYHr46xkcp0RsRdmHD8bgePfOIC3jSvgi/+fhv/8sSutPOtveERhbXU56IzFEkuqlp1XKqK3LT0hDnYHqTU56TE50x69Y+ZVRAXptgyxrWcdPenC5pVMGyoSA9lZqknrZ/r0Bx3CytybwuGk5FxeVGKuJtrApYtA7nXl9loNvxYXV+afGxlXclg5N4Roq7MmyyHMJRV9SU0dfYn9x0M7XI1WYnEEtz38iGe3nFioqcyaRBxFyYFJV4nv77pHK5dU8c9LxzgoBltgynugeziXu43xd30h8vMiNeK3A+3B2moMDJiFpk2zNPbDRGYX+1Pu1ap15XRcx/Jb7cYupGpuXdgWI47GJ47GNkvqZF79Qjinmt9mY1HOllYXZT23FV1JRxoC9IzEKWxc3iOeyqr642I/xUzAp4qHaC6+q0F9akx30Ig4i5MGhx2GzecOxsYFLhYPEF7cGRxL/O56AxGk5uPrJIEVUUeOoIR3mjuS/YirShyU2Fu568r8yYLmVmUDPHctdaj5rhbzCr1cqx7INmIo6UnPGwx1ZoDQHufEbl7nXa8LnvyHve39OG0q2TnKjAi+6EWSXPPAJ97cDMvm5u6tNZsauzizNmlaeNW1hnH249209gxPMc9lRWzirHbFC/vN645VcS923zPOkKyAGwh4i5MKuaZlQr3m75zRzCC1owi7kbxMCv1MGnLWLtUe8PMqRgUZ8uaGeq3g2XLpDfSCEbiI0a7FjNLvcmeqX3hGH3hWEZbxu+y43HaaDcXVK089jKfC7tNEYzEKfE602ygBVVFHGwL8vH/3sDRrn7+9kYr7/jx33hk01H+vz9sSy4Qd4WinGX67RbWourL+9rp7o+OeC8+l4NFNYFkpk93f5RYnsoejCdWPSCJ3AdxjD5EEApHiddJZZGb/Wbkbu1OrR5J3E1xtKwcqyRB6nOsyB0Ma+aVA+0syCDuxV4n4ViCgWgcj9Oe3JQ0K0vBsFRSS//+1aw+aX2QpKKUosJv5Lq3p4i73aao8Lto6Q0nd6dafOLS+Tjtin9/dh/P7mkhHEuwqDrAxy+Zz7cf38XDG5pwmovMZ81JF/dyv4u6Mi9PmOsMo30LWV1fmrbvoDMUHfHDdTJgfduaKt80CoGIuzDpmF/l54Ap1FYJ3NE8d4ADbUG8TntyU1LqcyzPHWDRDCtyHy68qcXDPE47R00P3dqBOhLWRqb7Xj7EHzYd5QNn13HJ4sztJCuLDA/d2J3qSj5eXeympTec5pkDuB12brtsIe8+cxZ3PbWHYq+Dr7xjGR6njce2Huff/voGFyyoJOB2sKBq+IfWGXUlPLHNWGcY7VvI6voSfvf64HFnKDLpxT11h7JgILaMMOmYV1WUtGVaTXtg5GwZQwgPtAaTi6lAWh31VHE/b65Ru31NQ3qEC8N3qVrZLzNzEHfrA+APm46yqr6Ub717RdYMm4oiN+3BMO196eJu3Wexx5nxeXVlPv7thjP59rtX4nXZUUrx+bcv5nj3AL/f2MTq2aUZM2HOqBv04Ufy3GFwUdViPJpz5xvLcw9F4ifVk3c6IuIuTDrmV/npCkXpCEbGFLkf7epPWjQwuLOzzEyDtFhQHWDz169I+vuplA4pHnasqx+3w5b2oZGNUp8Tn8tOZZGbX3z47KzlgY25uWjrHR65W/c5NHIfiQsWVPLm+RVp+e1DOcP03QNux6jXXlBdhN81OPepEA13paSvToX5FgIRd2HSMT9lUbW1N0zA4xhRKK3iYTC4mApGOYFyv4uGSn+mp2VkaNnfY10DzCr1jprjDoaX/sNrV3P/35/HjJLhWTKpVBQZaZqhSDzdljGza8Yi7gBfuHIJLoeNi83SvUNZbop7XQ75+nabYmVdCT5T4KfCRqbUDCfx3Q3EcxcmHZa4HzDFfaTFVBjckWr8ni6K5zSUZfTWs18rvRvT0a7+nCwZiytXzMhpXGWRK9mr9VQjdzAWQXd88+3JRdWhlHidLK4JZFxEzsQXr1zCie4BPn7/ximRgZIq7p15rodz/2uH2XOilzuvWZHX6443OUXuSqkrlVJ7lFL7lFJfynD+IqXURqVUTCn1/vxPUzidmFXmxeWwsb81SEvvwKiLeVbxMEiP3AF+ceMa/vnti3N+bStLxeqjery7P5kFk08qU9YQ8iHuQFZht7jv/5zDnVcvz+laZ84u46qVtQQ8jikRCXf1R5K1eDrybMs8se04a7ccy+s1C8Go4q6UsgN3A1cBy4AblFLLhgw7AtwE/DbfExROP+w2xdwKfzJyr8qwEWgoZX6n+adrlJEjE3A7sCkjEozEErT0hscUueeKVcYX8ifuo1Fb4h3z30+5P7/NuceLrlCUueY3vkydtIYSDMdyXnht7OhPFqabSuQSuZ8L7NNaH9BaR4AHgGtSB2itD2mttwKTf7eDMCWYX+1nf2tw1LoyFpbvnsvC50jYbEZp4e7+KM09A2idWxrkWLEWeyFd3BfVBFhVXzpsl+lEYZV2mOx0haI0mBvVcvkwuvk36/jKH7aPOi4WTyTTYa2+AlOFXMR9FtCYctxkPiYI48a8yiIOtwcJRuI55ViX+gZ3eZ4qJV4nXf3RZJ2Y2tLxsGVSIveUOZd4nTz6yQtYmGHz00RQ7sutps1E0xWKUOF3U+J1jrpGoLVmx9Ee9rWM3gz8ePdAcm3E2rU7VchF3DMtreuTeTGl1K1KqfVKqfWtrdLhRcjO/Go/5v+pURdUYTD6HbqgejKU+IziYcfGkOM+VspSdqWOhwWTL6ZC5B6JJQhG4pT5nIaNNEp9ma5QlN5wLCexthq1ALT2DowwcvKRi7g3AfUpx3XASa0uaK3v0Vqv0VqvqarKvHNPEMCI3C1yidzL8hy5d4ciydIDM0vyL+5Ou41Sn5MynzNr+d3JgOW5a31S8VxBsPYkWH+fo0XulmC39oWTUfloY2GwFMZUIRdxXwcsVErNVUq5gOuBteM7LeF0x2paDbmKe3oz6VOh1PTcj3b1U+534XVlz7E/FSqL3Hn5MBpPyv0uwrFEsu3eZMRaQC3xuXJaALYEO57Qo/albewI4bAp7DaVLEw3VRhV3LXWMeA24GlgF/CQ1nqHUupOpdTVAEqpc5RSTcAHgF8opXaM56SF6U/A40zaMbmI+5qGclbVl6aVHDhZLM/9eNf4pEFazK30j2mD1URg2UeTOWPG2pNQ5nMa5Z9HsZFSo/Hm7pHF/UhHiFll3mTjl6lETpuYtNZPAE8MeezrKb+vw7BrBCFvzK8qMqom5hDdnj+/gkc/eUFeXrfU56SnP0pTZ/+4iu+Prls9btfOF9bffUcwklNN+4kg2V7R60qzkbLtxD3SniLuPQOspCTrtRvNRi3d/dFpacsIwoSwpqGMRTWBgnvSJV4nCW2UEB6PNEiLIrcjuflqslJuZvXke2NQPrFsmVKfkzLTRuofIYf9SEco+b42j7JIesRs1FIdcGcV97a+MLf9dvLt5BVxFyYtt1++KG/R+FiwsldiCT2utsxUIBm5T+J0SCtyLzEXVGFkG+lIR4iz5pSh1MjpjT0DUTpDUWaX+6gKeGjJ4rn/aUczj209zrN7Wk7hLvKPiLswabHbFC5H4f+JpqYmjkca5FTCitwnczpkV38Eu00RcDuSC9TZ6stEYgmOd/czt9JPZZE7q2CDYckAzC73UVPspj0YIZqhK9UGsyn5lsauU72VvCLiLghDSC1EdrqLe8DtwGlXwyLhREKz/Wj3pEiR7ApFKTXbElrZUtk+jI529ZPQg4I9UgZMY4exz2F2uS9ZrbOtb3ikv/GIIe6bm7qzXuv3G5qS4wqFiLsgDCF1I9R4eu5TAaUUZT5XsoWhxV1P7+GdP3mR/3718ATNbJCu/miyXn/ZKOJ+JCUan1HsGdGWsSL3+jJfMnNraMZMe1+Yg21BitwOdh3ryVh/Jp7QfOV/t/HTZ/eN8c5ODRF3QRiCZcs4bGrSt5crBFevmsmT20/wX6aQ/3VXMz9/fj9+l53vPbk7WXtlougKRZJ2TGp2TyZSxb262EPLCAuqRzpCFHsclPicyRTboZH+xiOGFXPtmnoi8QS7jw8vaXCkI8RANMHOYz3Dzo0nIu6CMARL3GuKPdgn8e7RQvGlq5bw1iXVfOPR7fzXK4f43ENbWD6zmEdvu4CEhq/8YduE2jOWLQNGyWabIpm58vzeVt7y/WdoN+2Uxo4QboeN6oCbmoCHtr7MPjoYojzbLEZm2TJDM2Y2HO7EaVd8+E2zAdjSNNx33202Gz/WPVDQjBoRd0EYgsdpx+2wnfaWjIXDbuMnHzyT5TNL+NqjO0hozU8/dBYLqgN84crFPLenlT9sOjph8+sKDdoydpui1OdKpm4+uukoTZ39PLb1OGDkuNeX+7DZFDVmNJ6t2qOV4w5GoTelhov7xiOdLJ9ZwtxKP1UBN5szLKruPjEYze86XrjoXcRdEDJQFXBP2k07E4HP5eA/blrDpYur+LcbzmSO2XD8o+c3cPacMu5Yu6OgwpVKd3802fsWMOvLRNFa88IbbQA8Yn74HEkR7JpiIxo/kWFRNZHQNHX2J/8NOOw2KvzutOJh0XiCLY1dnDW7DKUUq+pKM2bM7D7RQ4W5FrBTxF0QJpZ7blzDF67MvYPT6UB1wMOvP3Yuly6uTj5msyn+33Wr8bsdfOhXr7HHjFJ3HOvm5vvW8cDrR3K69pH20LBF21yIxBL0hWNpdfytXaq7jvfS1hdmaW0xWxq7ONDalybulo+eKR2yuXeASDyRHGvcvzttAXbnsR7CsQRnzzGakq+uL2F/a5CegfQ0zN0nejlvXjk1xW4Rd0GYaJbNLE5GdsLI1Jf7+N0tb8JpV3zwl6/y+f/Zwrt+8iJ/3d3Ctx/flfS7s6G15ubfrONDv3yVSGxs/X5SK0JaWPVlXnjDKCv+/fetxKbg3pcO0heODYvcM2XMWCUK6stSxL3YnbYAa+W3nzXHaKyyqt74c1tKSmQwHONIR4glM4pZVltc0EVVEXdBEE6Zhko/v7vlTdhtikc2HeUj5zfwyCfeTCgS4yfPjJwCuPFIJ2+09HGse4D/3Tw27767f7AipEWZz4jc//ZGK4trApxRV8oFCyp5aF0TQFLcy30unPb0ao/doShP7zjBf7x4MG0sQE3Ak5YKueFIJ7NKvdSaJaHPmGWIe6rvvre5F61h8YwAS2uL2dfSV7B2fZO7sIUgCFOGeVVFPPbpt9AfiSc9+evOqef+1w7zsQsako8N5XevN+J32akv9/Gz5/bzvrPqcs5SskoPpNoyZaYts+5gJx998xwA3r16Fn8z/XcrA8ZmU1QHBnPd32ju5Zq7XyIUieNx2rhy+Yy0dZfqYjdtZg14u02x8XAnaxrKk+dLfE7mVfrTfHdrMXXpjGKi8QSxhOaN5j5WzMperCxfSOQuCELeqA540kT89ssXYbcpfvD0HlWdB3MAAAoSSURBVMBYhOwLx5LnewaiPL71OFevnsmn37qQg21Bntx+fMTX+MvOZr7z+E601nSmVIS0KPc7iSU0kXiCCxcaTYGuXDEDr9Ooy5/Nanl4QxORWILf3nIeW75xBT+/8ey0D5nqgJuENjYubTrSyfHuAc6bOyjuYFgzmxu7kqmhe0704nfZqSvzsqy2GCjcoqqIuyAI40ZNsYe/f8s8Htt6nMv+9TmWfu0pVn3zTzy8wbBI1m4+Rn80zvXnzObK5TOYX+Xn7mf3Z82b7wpF+PzDW/jl3w6y/nBnWkVIC2tDk9th41xTfP1uB+9YWcucCl9a85WagMdshK55bOtx3rKwkjfPr8TtGN6gpSol1/3uZ/dT6nPynjPT20mfP7+Clt4wL+9vB4zUx0UzjMqmcyr8+Fz2gvnuIu6CIIwr/3DxPC5bUs2imgC3XjSPcxrK+MLDW1i75RgPrmtkaW0xZ9SVYLMpPn7JAnYd7+HpHc0Zr/Wvf9pDd3+UgMfBL57fn1xQLRmSLQNw3rwKPM5Bkf72u1fw8D++Oe16Rn2ZMJsauzja1c87z5iZ9T6svPgX3mjlL7ua+dib5+IfUrL5mtUzqS3x8P/+shetNXuae1kyw4jY7TbFkhmBgkXu4rkLgjCuBDxO7r3pnORxfyTOR3/9Orc/sImEhm9evTzZWOOa1TP5+fP7+cwDm7jr/WdwzerByHhbUzf3v3aEj57fQInXyY//+gZ2swVeIEVkrfoyFy2sTJuH12Uf1jKxuthDd3+Uhzc04bLbuGJ5Tdb7qDaza+5+Zh9+lz3p56fidtj5xCXz+dqjO3hk41G6QlGW1gaS55fNLObRzcdGbCaSLyRyFwShoHhddu696RxW15dS5Hbw7hQBd9ptPPQP57OqvpTPPLCZu57azZ4TvRxpD/G1R7dT4Xfz2bct4iPnz8HjtPH0juZkRUiLlbNKuP3yhXzg7PpR52KlQz6ysYmLF1dR7HFmHVtVZETuwUicD58/J616aCrXnlPPjGIPd/zR6Da6uCZF3GtL6B2I0dQ5/vV4JHIXBKHgFLkdPHDr+XSFImmWChi2yn/ffB5ff3Q7P31uPz99bn/y3P/9wKpk7Z8PnF3Pf716eNjznXYbt1++KKd5WFbLQDTBO8+oHXGsy2GjzOckFInz92+Zl3Wc22HnE5fO5+uPGuJu2TJgRO5gLKqO9w5oEXdBECYEl8OWtDoynfvue1fy/rPraO4JE4rEKPO5eOvSwd2xf3/hXO5/7XByAfVksCJ3j9PG5UuzWzIWVyybQV2Zd9Rqodeuqeenz+5HqfT1gMU1AeZV+bMWK8snOYm7UupK4MeAHfiV1vp7Q867gf8Ezgbageu01ofyO1VBEE4nlFJpeeRDmVPh57ZLF1DszW6ljIYl7pctqR62OJqJ77//jJyu63HauftDZ9IzEEt73Ouy88w/XTLmeZ4Mo96NUsoO3A28DWgC1iml1mqtd6YMuxno1FovUEpdD3wfuG48JiwIgmDxuStOrf5PscfB5962aMSF1JPl7DnZP5gKQS4LqucC+7TWB7TWEeAB4JohY64BfmP+/jDwVjXeS8GCIAiniFKKT791YZovPl3IRdxnAY0px03mYxnHaK1jQDdQMfRCSqlblVLrlVLrW1tbT27GgiAIwqjkIu6ZIvCh28dyGYPW+h6t9Rqt9Zqqqqpc5icIgiCcBLmIexOQmjBaBxzLNkYp5QBKgI58TFAQBEEYO7mI+zpgoVJqrlLKBVwPrB0yZi3wUfP39wPP6IlsqigIgnCaM2q2jNY6ppS6DXgaIxXyXq31DqXUncB6rfVa4D+A/1JK7cOI2K8fz0kLgiAII5NTnrvW+gngiSGPfT3l9wHgA/mdmiAIgnCySG0ZQRCEaYiIuyAIwjRETdS6p1KqFTh8kk+vBNryOJ3Jhtzf1Ebub2oz2e9vjtZ61FzyCRP3U0EptV5rvWai5zFeyP1NbeT+pjbT5f7ElhEEQZiGiLgLgiBMQ6aquN8z0RMYZ+T+pjZyf1ObaXF/U9JzFwRBEEZmqkbugiAIwghMOXFXSl2plNqjlNqnlPrSRM9nrCil6pVSzyqldimldiilPmM+Xq6U+rNS6g3zzzLzcaWU+jfzfrcqpc6a2DvIDaWUXSm1SSn1mHk8Vyn1mnl/D5p1ilBKuc3jfeb5homcdy4opUqVUg8rpXab7+P50+n9U0p91vy3uV0p9TullGcqv39KqXuVUi1Kqe0pj435/VJKfdQc/4ZS6qOZXmsyMaXEPaUr1FXAMuAGpdSyiZ3VmIkB/6S1Xgq8CfikeQ9fAv6qtV4I/NU8BuNeF5o/twI/K/yUT4rPALtSjr8P/Mi8v06M7l2Q0sUL+JE5brLzY+AprfUSYBXGfU6L908pNQv4NLBGa70Co56U1V1tqr5/9wFXDnlsTO+XUqoc+AZwHkYDo29YHwiTFq31lPkBzgeeTjn+MvDliZ7XKd7ToxgtDPcAteZjtcAe8/dfADekjE+Om6w/GGWh/wpcBjyGUe+/DXAMfR8xCtKdb/7uMMepib6HEe6tGDg4dI7T5f1jsPFOufl+PAa8faq/f0ADsP1k3y/gBuAXKY+njZuMP1Mqcie3rlBTBvMr7JnAa0CN1vo4gPmn1eZ9Kt7z/wO+AFgt3iuALm106YL0e8ipi9ckYh7QCvzatJ1+pZTyM03eP631UeBfgSPAcYz3YwPT5/2zGOv7NaXeR5hitgw5dnyaCiilioDfA7drrXtGGprhsUl7z0qpdwItWusNqQ9nGKpzODcZcQBnAT/TWp8JBBn8Sp+JKXV/ptVwDTAXmAn4MayKoUzV9280st3PlLvPqSbuuXSFmvQopZwYwn6/1voR8+FmpVSteb4WaDEfn2r3fAFwtVLqEEYz9cswIvlSs0sXpN/DVOvi1QQ0aa1fM48fxhD76fL+XQ4c1Fq3aq2jwCPAm5k+75/FWN+vqfY+Tjlxz6Ur1KRGKaUwmpvs+v/bu0OXCIIojuPfSSe2MxvkitV4wSAIF+5fEBT1rxCT/4D/hMFguWAxqF0MoiKie8liNRueYd7CgsF1y9w+fh9Y7nZ2wzze8riZ2WPM7LRxqbmb1R55Lr5u3/VV/DHwVQ8nF5GZHZnZqpmtkfNzY2Y7wC15ly74HV9vdvEys0/gI6W07k3bwAtB8keejhmnlJb9Wa3jC5G/hv/m6wqYpJSGPrqZeNviKj3p32FhZAq8AXPguHR/OvR/kzycewQe/JiS5ymvgXf/XPH7E/kNoTnwRH6LoXgcLWPdAi79+wi4AyrgAhh4+5KfV359VLrfLeLaAO49hzNgGCl/wAnwCjwDZ8Cgz/kDzsnrB9/kX+CHXfIFHHicFbBfOq6/Dv1DVUQkoL5Ny4iISAsq7iIiAam4i4gEpOIuIhKQiruISEAq7iIiAam4i4gEpOIuIhLQD3b3Ib4szY25AAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "tplt( lt, 'loss')\n",
- "tplt( lt, 'val_loss')\n",
- "plt.legend()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsvXl8nGW5//++Z59ksu9N0yZNN7oXugBlFS0gCCJFUWQVARFEBb/qOUfZ9JwjoiKLoOcny0E8BUEQFQTEQqEsbUr3Qrc0bbZm3yaZfe7fH888T2aSmWTSZmtyv1+vvDrzzD3P3JO2n+d6Pvd1X5eQUqJQKBSKyYFprCegUCgUitFDib5CoVBMIpToKxQKxSRCib5CoVBMIpToKxQKxSRCib5CoVBMIpToKxQKxSRCib5CoVBMIpToKxQKxSTCMtYT6Etubq4sLS0d62koFArFccXmzZubpZR5g40bd6JfWlpKRUXFWE9DoVAojiuEEIeSGafsHYVCoZhEKNFXKBSKSYQSfYVCoZhEjDtPX6E4XggEAtTU1OD1esd6KopJhMPhYOrUqVit1qN6vxJ9heIoqampIS0tjdLSUoQQYz0dxSRASklLSws1NTWUlZUd1TmSsneEEOcJIfYIIfYLIX4Q5/XvCiF2CyG2CyHeFEJMj3otJITYGvl5+ahmqVCMQ7xeLzk5OUrwFaOGEIKcnJxjurscNNIXQpiBR4DPADXAJiHEy1LK3VHDtgDLpJQ9QohvAPcBX4q85pFSLjnqGSoU4xgl+IrR5lj/zSUT6a8A9kspK6WUfmAtcHH0ACnlOillT+TpB8DUY5rVCBAOS/6ytZamLt9YT0WhUCjGjGREvxiojnpeEzmWiK8Br0Y9dwghKoQQHwghPh/vDUKIGyJjKpqampKY0tAIhMJ87/nt3LZ2K795a/+wn1+hUCiOF5JZyI13LxG3m7oQ4qvAMuDMqMPTpJR1QogZwL+EEDuklAdiTibl74DfASxbtuyYO7XXtXv444eHmZnvYm5RGve/tod/ftyIy25hR03HsZ5eoVAcB+i7+3Nzc8d6KuOKZES/BiiJej4VqOs7SAjxaeDfgTOllIaHIqWsi/xZKYR4C1gKHOj7/uHCHwzzjWc+Ylt1e9Tc4N7PL6Cyyc3ajdUEQ2EsZrVFQaE43ggGg1gso5d0GAqFMJvNCZ8nYrTnORSSmdUmYJYQogyoBS4HvhI9QAixFPgtcJ6UsjHqeBbQI6X0CSFygVVoi7wjxv2v72FbdTsPfXkpswvS2FHbwbTsFFaUZfPilhqe2FDFgaZu5hSmjeQ0FJOMu/+6i911ncN6znlT0rnzc/MHHFNVVcV5553HaaedxgcffMDixYu59tprufPOO2lsbOSZZ55h/vz53HrrrezYsYNgMMhdd93FxRdfTFVVFVdeeSXd3d0APPzww5x66qm89dZb3HXXXeTm5rJz505OOukk/vCHPyRcQPzBD37Ayy+/jMViYfXq1dx///0cPHiQr3zlKwSDQc477zx+9atf4Xa7eeutt7j//vv529/+BsAtt9zCsmXLuOaaa7jnnnv461//isfj4dRTT+W3v/0tQgjOOussTj31VDZs2MBFF13EVVddxU033cThw4cBeOCBB1i1ahUtLS18+ctfpqmpiRUrViDlwKbBH/7wBx588EH8fj8rV67kN7/5DWazGZfLxXe/+11ee+01fvGLX/DVr36V6667jtdff51bbrmFuXPnctNNN9HT00N5eTmPP/44WVlZ/eZ5++23D/WvfFQYNNyVUgaBW4DXgI+B56SUu4QQ9wghLooM+zngAv7UJzXzBKBCCLENWAf8d5+sn2Fl3Z5Gfre+kq+ePI3PLZ7CnMI01pw0lRVl2QAsLM4AYEetsngUE4f9+/dz2223sX37dj755BP++Mc/8u6773L//ffzn//5n/z0pz/lU5/6FJs2bWLdunV873vfo7u7m/z8fN544w0++ugjnn32Wb71rW8Z59yyZQsPPPAAu3fvprKykg0bNsT97NbWVl588UV27drF9u3b+Y//+A8AbrvtNr7xjW+wadMmCgsLk/oet9xyC5s2bWLnzp14PB7jwgDQ3t7O22+/ze23385tt93Gd77zHTZt2sQLL7zA9ddfD8Ddd9/NaaedxpYtW7jooouMi0I8Pv74Y5599lk2bNjA1q1bMZvNPPPMMwB0d3ezYMECPvzwQ0477TRA2xD17rvvcvnll3PVVVfxs5/9jO3bt7Nw4ULuvvvuuPMcryR1/yGlfAV4pc+xH0c9/nSC970HLDyWCSZLQ6eX25/bxtzCNP7jgnlxx5Tluki1mdlR086ak8ZdgpHiOGawiHwkKSsrY+FC7b/Z/PnzOeeccxBCsHDhQqqqqqipqeHll1/m/vvvB7T9BYcPH2bKlCnccssthujt3bvXOOeKFSuYOlX7P7JkyRKqqqoMAYwmPT0dh8PB9ddfzwUXXMCFF14IwIYNG3jhhRcAuPLKK/n+978/6PdYt24d9913Hz09PbS2tjJ//nw+97nPAfClL33JGPfPf/6T3bt7Y8fOzk66urpYv349f/7znwG44IILyMrKSvhZb775Jps3b2b58uUAeDwe8vPzATCbzVx66aUx4/XP7+jooL29nTPP1JYtr776ai677LJ+48Yz49N0OgpSbGbOmZvPjWfOwGGN77mZTYL5xRlsV5G+YgJht9uNxyaTyXhuMpkIBoOYzWZeeOEF5syZE/O+u+66i4KCArZt20Y4HMbhcMQ9p9lsJhgMxv1si8XCxo0befPNN1m7di0PP/ww//rXv4D4+eQWi4VwOGw81zcZeb1ebr75ZioqKigpKeGuu+6K2YCUmppqPA6Hw7z//vs4nc5+5082h11KydVXX81//dd/9XvN4XD08+2jP38gkh03lkyY1cw0h5WfX7aYmfkDe/ULizPYXddJMBQecJxCMVE499xzeeihhwyPe8uWLYAWtRYVFWEymXj66acJhUJDPrfb7aajo4PPfvazPPDAA2zduhWAVatWsXbtWgDDNgGYPn06u3fvxufz0dHRwZtvvgn0in9ubi5ut5vnn38+4WeuXr2ahx9+2Hiuf+YZZ5xhfNarr75KW1tbwnOcc845PP/88zQ2akuQra2tHDo0eDn6jIwMsrKyeOeddwB4+umnjaj/eGHCiH6yLJqagS8YZl+je6ynolCMCj/60Y8IBAIsWrSIBQsW8KMf/QiAm2++maeeeoqTTz6ZvXv3HlWU2tXVxYUXXsiiRYs488wz+dWvfgXAr3/9ax555BGWL19OR0fvnXVJSQlf/OIXWbRoEVdccQVLly4FIDMzk69//essXLiQz3/+84btEo8HH3yQiooKFi1axLx583jssccAuPPOO1m/fj0nnngir7/+OtOmTUt4jnnz5vGTn/yE1atXs2jRIj7zmc9QX1+f1Hd+6qmn+N73vseiRYvYunUrP/7xjwd/0zhCDLbCPdosW7ZMjmTnrMomN5/6xdvcd+kivri8ZPA3KBQJ+PjjjznhhBPGehrHBS6XC7dbBVrDRbx/e0KIzVLKZYO9d9JF+qU5qdomrQF8/cZOLxc+9A41bT0JxygUCsXxyIRZyE0Wk0mwoDh9wMXc7TUd7Kzt5JP6LqZmpYzi7BSK8csll1zCwYMHY4797Gc/49xzzx30vWMV5be0tHDOOef0O/7mm2+Sk5MzBjMaeyad6AMsmprJk+9VEQiFscbZmdsYKcrmCQx9YUuhmKi8+OKLYz2FIZOTk2Ms9Co0Jp29A7CgOAN/MMy+hvjRR0OnlkmgRF+hUEw0JqXoz8p3AXCgKb7oN3ZFcoeV6CsUignGpBT9stxUhIDKpu64rzd2RuwdvxJ9hUIxsZiUou+wminOdCaM9Bu6lL2jUCgmJpNS9AFm5LmobE7k6auFXMXEw+VyjfUUjlsm0u9u8op+bioHm7r7lV8NhsK0uDXR9yp7R6GYsBxN2YmjRUoZU3NoKJ8/3POclCmbAOV5qXT7QzR0+ijM6C001dLtJxy5DqhIX5E0r/4AjuwY3nMWLoTz/zvhy9///veZPn06N998M6AVUBNCsH79etra2ggEAvzkJz/h4osvTngOnbfeeos777yTgoICtm7dyhe+8AUWLlzIr3/9azweDy+99BLl5eU0NTXFrWW/ceNGvv3tb+PxeHA6nTzxxBPMmTOHJ598kpdffpmenh4OHDjAJZdcwn33xW+pEQqF+NrXvkZFRQVCCK677jq+853vsHnzZq677jpSUlI47bTTePXVV9m5cydPPvkkFRUVRh2eCy+8kDvuuIOzzjrLKOvs8XhYs2aNUf64tLQ0pjb+8uXL+eY3v0lTUxMpKSn8z//8D3Pnzu3XD2Awfv7zn/Pcc8/h8/m45JJLuPvuu6mqquL888/n7LPP5v333+ell15i/vz5MbX6fT4fd9xxB8FgkOXLl/Poo49it9v7zfPyyy8fdA7JMmkj/fK8+Bk8+iIugCegirIpxi+XX345zz77rPH8ueee49prr+XFF1/ko48+Yt26ddx+++2DNhPR2bZtG7/+9a/ZsWMHTz/9NHv37mXjxo1cf/31PPTQQwAJa9nPnTuX9evXs2XLFu655x7+7d/+zTjv1q1befbZZ9mxYwfPPvss1dXVcT9/69at1NbWsnPnTnbs2MG1114LwLXXXsuDDz7I+++/n/Tv5qc//SkVFRVs376dt99+m+3btxuvRdfGv+GGG3jooYfYvHkz999/v3EBHUo/gNdff519+/axceNGtm7dyubNm1m/fj0Ae/bs4aqrrmLLli1Mnz49pla/3jxG/90Eg0EeffTRuPMcTiZtpD8jIvqVTW5Wzeztoann6AsBHn/8crIKRT8GiMhHiqVLl9LY2EhdXR1NTU1kZWVRVFTEd77zHdavX4/JZKK2tpaGhoakGpksX76coqIiAMrLy1m9ejUACxcuZN26dUDiWvYdHR1cffXV7Nu3DyEEgUDAGHPOOeeQkaE1MJo3bx6HDh2ipKR/3asZM2ZQWVnJrbfeygUXXMDq1av71a+/8sorefXVVwf9Ls899xy/+93vCAaD1NfXs3v3bhYtWgT01rx3u9289957MfXwfT4t6BtKP4DXX3+d119/3Sge53a72bdvH9OmTWP69OmcfPLJxtjoWv179uyhrKyM2bNnA1pt/kceeYRvf/vbMfMcbiat6Bek20m1mTnQJ21Tz9yZkuFU9o5i3LNmzRqef/55jhw5wuWXX84zzzxDU1MTmzdvxmq1UlpaGlOXfiAGq8sPiWvZ33rrrZx99tm8+OKLVFVVcdZZZ8U970C1+bOysti2bRuvvfYajzzyCM899xy//OUvE9bIT1Sb/+DBg9x///1s2rSJrKwsrrnmmri1+cPhMJmZmQl37A6lNv8Pf/hDbrzxxpjjVVVV/SqXRtfqH+wObKRq809ae0cIwYw8V1x7RwiYmuVUefqKcc/ll1/O2rVref7551mzZg0dHR3k5+djtVpZt25dUjXih0KiWvYdHR0UFxcD8OSTTx7VuZubmwmHw1x66aXce++9fPTRR2RmZpKRkcG7774LxNbmLy0tZevWrYTDYaqrq9m4cSOg3X2kpqaSkZFBQ0NDwjuD9PR0ysrK+NOf/gRoIrxt2zYgcT+AeJx77rk8/vjjRn2h2tpao07/QMydO5eqqir2798PjF5t/kkr+gAz8lL7bdBq7PKSk2ojzWFVnr5i3DN//ny6urooLi6mqKiIK664goqKCpYtW8YzzzzD3Llzh/XzEtWy/3//7//xwx/+kFWrVh11tkltbS1nnXUWS5Ys4ZprrjG6Wj3xxBN885vf5JRTTom5w1i1apXRKvKOO+7gxBNPBGDx4sUsXbqU+fPnc91117Fq1aqEn/nMM8/w+9//nsWLFzN//nz+8pe/AIn7AcRj9erVfOUrX+GUU05h4cKFrFmzhq6urkG/r8Ph4IknnuCyyy5j4cKFmEwmbrrppkHfd6xMunr60fz6n/t44M29fHzPeUaLxa89uYn6Di8z813sqO1g3R1njcpcFMcfqp7+6FNVVcWFF17Izp07x3oqY4qqp3+UlOenIiUcbO6N9hu6vOSn23FazcreUSgUE45Ju5ALMCO3N23zhKJ0QNuNO78oA6fNrBZyFROOHTt2cOWVV8Ycs9vtfPjhh6M6j5UrVxqZMjpPP/00CxcuHPB9paWlYxblj5ff3bEyqUW/LFdbHdd9fX03bkG6HX9IKtFXDIqUMuksj/HAwoULx0V9+eNNKGH8/O6O1ZKf1PaO06YVXquMZPDou3Hz0h04rWb8wTCh8Pha81CMHxwOBy0tLcf8n1ChSBYpJS0tLTgcjsEHJ2BSR/qgZfDoufr6xqyCNLuxMcsbCJFqn/S/JkUcpk6dSk1NDU1NTWM9FcUkwuFwMHXq1KN+/6RXsyUlmfzmrQPUd3iMEgwF6Y6Y7llK9BXxsFqtlJWVjfU0FIohMantHYDLTiohLCXPbqo2duPmp9uNFE6VwaNQKCYSk170p+WkcMasPNZurKa+3YsQkOuy47RFRF8t5ioUignEpBd9gCtWTuNIp5cXPqohJ9WO1WzCqSJ9hUIxAVGiD3xqbj6F6Q7qO7zkp2nFoQzRV5G+QqGYQCQl+kKI84QQe4QQ+4UQP4jz+neFELuFENuFEG8KIaZHvXa1EGJf5Ofq4Zz8cGExm/jScq3Ua0G6JvoOZe8oFIoJyKCiL4QwA48A5wPzgC8LIeb1GbYFWCalXAQ8D9wXeW82cCewElgB3CmEyBq+6Q8fl68owSQwumjpkb5qmahQKCYSyUT6K4D9UspKKaUfWAvE9F+TUq6TUvZEnn4A6Emk5wJvSClbpZRtwBvA4L3HxoCiDCePffUkvn76DABSVKSvUCgmIMkkoBcD0f3NatAi90R8DdALWMd7b/FQJjiarJ7f211IefoKhWIikozoxyssEnffuRDiq8AyQO8EkNR7hRA3ADcATJs2LYkpjTyGp6/sHYVCMYFIxt6pAaIbWk4F6voOEkJ8Gvh34CIppW8o75VS/k5KuUxKuSwvLy/ZuY8ohqevIn2FQjGBSEb0NwGzhBBlQggbcDnwcvQAIcRS4Ldogh/dJ+w1YLUQIiuygLs6cmzcYzWbsJiEsncUCsWEYlB7R0oZFELcgibWZuBxKeUuIcQ9QIWU8mXg54AL+FOkzOxhKeVFUspWIcS9aBcOgHuklK0j8k1GAKfVTI+ydxQKxQQiqUpiUspXgFf6HPtx1ONPD/Dex4HHj3aCY4nDZlb2jkKhmFCoHbkDoFomKhSKiYYS/QFwWlXLRIVCMbFQoj8ADpsZTyA81tNQKBSKYUOJ/gA4rSZVhkGhUEwolOgPgLJ3FArFREOJ/gA4bUr0FQrFxEKJ/gA4rRaVvaNQKCYUSvQHwGkzqTx9hUIxoVCiPwBqR65CoZhoKNEfAH0hV8q4RUUVCoXiuEOJ/gDo5ZV9QZWrr1AoJgZK9AfAaKSiLB6FQjFBUKI/AKp7lkKhmGgo0R8Ap+qTq1AoJhhK9AfAoewdhUIxwVCiPwCqZaJCoZhoKNEfgJG0d3zBEOGwSgVVKBSjixL9ARip7B0pJWfe9xbPbDw8rOdVKBSKwVCiPwCOEcre6fAEONLp5XBL97CeV6FQKAZDif4AGPbOMEf6zW4fAF7VoEWhUIwySvQHIGWEIv1mt39EzqtQKBSDoUR/AEZqIVeP9JXoKxSK0UaJ/gDYLdqvZ7hbJjZ3+UbkvAqFQjEYSvQHQAgxIi0TW7qVvaNQKMYGJfqDMBItE3sXcpXoKxSK0UWJ/iA4rWY8/uHNsuldyFXZOwqFYnRRoj8IDuvwt0xUkb5CoRgrlOgPwkjaO6qQm0KhGG2U6A+CZu8M80KuytNXKBRjhBL9QXAMc/ZOjz9Ijz+E2SSUvaNQKEYdJfqDMNyRvh7lF6Y78AXDqtKmQqEYVZISfSHEeUKIPUKI/UKIH8R5/QwhxEdCiKAQYk2f10JCiK2Rn5eHa+KjxVA8/X990sDbe5sGHNMU8fNLsp0AeIMq2lcoFKOHZbABQggz8AjwGaAG2CSEeFlKuTtq2GHgGuCOOKfwSCmXDMNcx4ShbM76z1c+IcNp5czZeQnH6Ltxp2alAK14/CFSbIP+NSgUCsWwkEykvwLYL6WslFL6gbXAxdEDpJRVUsrtwIRLPHfakrN3fMEQB5u7aYvstk2Evhu3JCsFAG9wwv3KFArFOCYZ0S8GqqOe10SOJYtDCFEhhPhACPH5eAOEEDdExlQ0NQ1sj4w2ZbmpuH1BKqpaBxxX2dRNKCwNUU+EHukXZ2n2jkrbVCgUo0kyoi/iHBvK6uM0KeUy4CvAA0KI8n4nk/J3UsplUspleXmJrZGxYM1JU8l12fjVP/cOOG5vQxegNUgJhhJH7y3dftIcFjKcVkBt0FIoFKNLMqJfA5REPZ8K1CX7AVLKusiflcBbwNIhzG/MSbFZuOnMcjbsb2HjwcTR/p4jXcbjdk8g4bgmt488l723FaMSfYVCMYokI/qbgFlCiDIhhA24HEgqC0cIkSWEsEce5wKrgN0Dv2v8ccXK6eS67PzqjcTRvh7pAwP6+s1dPnJddpw27Vev7B2FQjGaDCr6UsogcAvwGvAx8JyUcpcQ4h4hxEUAQojlQoga4DLgt0KIXZG3nwBUCCG2AeuA/+6T9XNc4LSZ+cZZ5bxf2cIHlS1xx+xp6CIn1QYwoK/f0u0nx2Uz+u8qe0ehUIwmSeXpSylfkVLOllKWSyl/Gjn2Yynly5HHm6SUU6WUqVLKHCnl/Mjx96SUC6WUiyN//n7kvsrIcsXKaeS6bDz9waF+r3X7glS3elg5IxsYJNJ3a5H+SDVdVygUioFQO3KTxGE1s6Ismx01Hf1e29foBuCUGTkAtPbEF/1AKEx7T4Acl83w9FWkr1AoRhMl+kNg/pQMDrf20OmNXajdG1nEXRkR/USRfmvkeG70Qq7y9BUKxSiiRH8IzJuSDsDHdZ0xx/c0dOGwmijPc+GyWxJ6+k2RHH1tIVe3d9TmLIVCMXoo0R8C84s00d9dHyv6exu6mJWfhtkkyE61JYz0W4xI39bbdF3ZOwqFYhRRoj8E8tMd5Lrs7Oob6R/pYnZBGgBZqTZae+Ln6TdHRfpCiBHpyqVQKBQDoUR/iMybkh4j+u09fhq7fMwpdAGQnWIdINKPiH6aHRhaMTeFQqEYDpToD5H5U9LZ39iFP1IobW+DlrkTE+knEP1mtx+7xURqxM8fia5cCoVCMRBK9IfI/CnpBELS2IG7o1ZL4dRFP2cg0Y/sxhVCK2fkGIH+uwqFQjEQSvSHyDx9Mbeuk3BY8scPDzGvKJ2iDAegRfqeQKhfBC+lZEt1OzPyUo1jTqsZr8reUSgUo4gS/SFSmpNKqs3M7vpO3vykkQNN3dx45gwjes9O0UoxtPXZoPXJkS4ONndz/oIi45jDak64kNva7VeLvAqFYthRoj9ETCbBCUXp7Krr4HfrD1Cc6eSChb1CnhWpv9PX4nllRz0mAefOLzCODbSQu+ax9/jF63tG4BsoFIrJjBL9o2DelHQ+OtzOpqo2rj+9DIu599eYE0f0pZT8fUc9J8/IIcdlN447BljIrWv3sLO2M+5rxzObD7Wxrbp9rKehUExalOgfBfOnpBMKSzKcVr64rCTmNT3Sj7Z39ja4qWzq5rNRdwSgVe+MZ+EEQ2G8gTBVLd0jMPux5dJH3+PiRzaM9TQUikmLEv2jYEFxBgBXnjydVHtsU3Pd04+O9P9uWDuFMWMdlvibs7p92rH6Dq9K6VQoFMOKZfAhir7MK0rnsa+exFlz+rd2zHBaMYlY0X91Rz0ryrLJS7PHjHUmSNl0+4PG46qWbk6IZAwpFArFsaIi/aNACMF5CwqNmvjRmEyCrJTeXP19DV3sa3T3s3Yg8UJuty9K9JvHxuLx+ENGgTiFQjFxUKI/AmSl2gxP/58fNwL9rR3QUzbDhMOxfea7vL2if3CMfP0H3tzLmsfeS/j69pp27n9NZRcpFMcbSvRHgOyoSP+dfU3MLUyjIN3Rb5xeXtkXjN2gNR4i/cqmbmraPEgp477+0pY6Hl63X605KBTHGUr0R4DsSCmGHn+Qiqo2zpjd3/sHbSEX+pdX1kU/zWHh4BiJfmOnl1BY0p1A1Bu6vIDW/vFoSHQxUSgUI4sS/RFAK7oW4MODrfhDYU6flRt3XG8jlVhhdUdEf8GUDA4294zsZBNwpFMT9Q5P/DLRjZ3HJvr+kCo/oVCMBUr0R4DsVCttPX7e2duMzWJieWl23HGJmqPror9wagbNbh9d3vjCO1KEwtJYxG1P0O/3iCH6iZvAD4TXr0RfoRgLlOiPANmpdkJhyas761lZlh03ywdI2CdXt3fmR9ozHmoZ3Wi/xe1DX1uOF+lLKWno9BljjwZVXVShGBuU6I8A2alWQNtcddrM+NYO9No7fT19ty+E1SyYU6iVa64cZV9fF3SAzjii3+EJGP0EjtbeUaKvUIwNSvRHgKzIrlyA02fFX8SFXnunb3nlbl8Ql93C9GytDPNoZ/A0RKwbiB/pR18Ujtbe6YnagKZQKEYPJfojQHak/k6uy87cSLQeD+cAnn6q3YLTZqYowzH6ot/VK/rtcfr9Rl8UjjbSV2WjFYqxQZVhGAF00T9tZg4mk0g4bqCFXFekpk9Zbuqob9Bq6PQhBJiFiBvp64u4Bel2Wo4y0veohVyFYkxQkf4IkJ/mYEVpdr8KnH0xPP04C7m66Jfmph5Vrn5tu4fads+Q3wdaOmauy06G0xpX9PV0zXlF6UOK9KN3HitPX6EYG5TojwA2i4nnbjqFUwdYxIXE9k53xN4BKMtJpb0nkDB1sssb4Bt/2Mz7B1qMYy1uH5c8soFvr91yVPNv6PRSkJ5Y9Bs6fWQ4rRRnOYck+sEo0VeevkIxNijRH0Mc1vg7crv6RPoAB5rccc/xj51HeHXnEW58uoLKJjdSSr73/HYau3xGH9+h0tDpoyDNQUZKItH3UpjuICfVTltPgGCSG60Ot/besShPX6EYG5TojyEOy0CRvvbakpJMUmxmHv7X/rilC/7uWf/5AAAgAElEQVS6vZ7CdAcWs4mvPVXBQ//az78+aWRxSSbd/lBSFo/HH4q5ODR0eslPdySO9Lt85KfbyY2Uiu7bGjIRT79/KOYzFQrF6JOU6AshzhNC7BFC7BdC/CDO62cIIT4SQgSFEGv6vHa1EGJf5Ofq4Zr4RMBkEtgtpjiiH8Jl13L989Ls3LF6Duv2NPHytrqYcS1uHxv2N/OFE4v57ZUnUdPWwy/f2Mun5ubzowtOAGBvQ9eAc/D4Q5x+3zoeW38AAH8wTEu3n8KBRL/DS0G6g9zIgnUyaZtd3gDPb67h/AVatVFPQC3kKhRjwaCiL4QwA48A5wPzgC8LIeb1GXYYuAb4Y5/3ZgN3AiuBFcCdQoisY5/2xMFpM8cs5IbDkm5/EJe9dxfv1aeWsrgkk3v+upu2qKj6H7uOEApLPrd4CstLs7n/ssWsLMvm52sWMTuSKrpnENF/ffcRmt0+3t3XDEBTxKNP5OmHwpImt4+CqEg/GV//xS21dPtD3HDGDAA8ytNXKMaEZCL9FcB+KWWllNIPrAUujh4gpaySUm4H+oZv5wJvSClbpZRtwBvAecMw7wmDwxLbSKUnEEJKYtowmk2C//7CQjo8Ae75227D5vnrtjpm5ruMvQAXLynm2RtPIcdlJ91hpTjTyd4jA4v+85trANhW3U4oLI0c/IJ0B5kR0Y+2flq6fYTCMuLp24xjAyGl5Kn3qlg8NYOl07ISNo9RKBQjTzKiXwxURz2viRxLhmN576RAa47ee63U6+707b17QlE6N589kxe31HL3X3dT3+Hhw4OtXLioCCHi7wWYXeBiT0P8BWCAIx1eNuxvZkZuKt3+EPsau4x0zPx0O+lOK1JqC8s6jZHduPnpjt5Iv2tge2fD/hYONHVz9amlxndWoq9QjA3JiH48RUk2JSSp9wohbhBCVAghKpqampI89cTA0SfqdUfV0u/Ldz49i6+fXsaT71Xxpd9+gJRw4aIpCc89uzCNA43uhNk1L22tJSzhR5/T3Loth9uNEgsFEU8fYuvvRN8JpNkt2MwmmgeJ9J96v4qcVJvRMtJpNavNWQrFGJGM6NcA0buMpgJ1CcYe1XullL+TUi6TUi7Ly0tcq2Yi4rSaYtIXjUjf1l/0hRD822dP4HvnzuFwaw/zitKZme9KeO45BWn4Q2Gq4uzolVLywuYaTpqexVmz88hKsbLlcBsNnV4sJkF2is0Q/WhfP3o3rhCCXJdtwEi/rt3Dmx838KXlJcYOZC3SV56+QjEWJCP6m4BZQogyIYQNuBx4OcnzvwasFkJkRRZwV0eOKSI4beaY9EW3N769oyOE4Jtnz+Txa5Zx35pFA557dkFkMfdIf4tnR20H+xrdfOHEYoQQLCnJNCL9/DQ7JpMgM1I4Lrr+jl6iIdelWTs5LruxkOsNhPj22i3squswxq/deBgJfHnFtN7vbDWrlE2FYowYVPSllEHgFjSx/hh4Tkq5SwhxjxDiIgAhxHIhRA1wGfBbIcSuyHtbgXvRLhybgHsixxQR+i7kDmTvRPOpuQUsKM4YcMzMfBcmET+D58UttdgsJsMeWjoti32NbvY3dpEf6ecbL9LXSzRYzdo/nVyXzVjI3XiwlZe21nH7c9sIhMIEQmHWbqrm7Dn5lGSnGOdQC7kKxdiRVME1KeUrwCt9jv046vEmNOsm3nsfBx4/hjlOaBw2c6y94x840h/Sua1mSnNS42bw7K7rZFFxhiHsS6dlArCtpoNz5xcA8UVfL9Ggk+Oy83G9dv4PKrVSEJ8c6eKJDQeZmpVCY5ePr57cG+WD9p07EpSVUCgUI4uqsjnGOK2x2Ttun3YBSLXH77Y1VOYUprEnjuhXt/Zw8owc4/nikkyEACm1RVpIJPo+ijIcxvNcl52Wbh9SSt6vbOHEaZlkp9r51Rv7KMtNpTjTyZmz82M+O8Vq5oiK9BWKMUGVYRhj+loduqfvGoZIHzRfv6qlO+Zuwh8MU9/pZWqU5ZLusDIzT1sU1kXfYTVhs5j6Rfq6/QOavRMISeo7vGyv6eCU8hzuvng+QsDu+k6+snIa5j7lpVXKpkIxdijRH2P6LuR2+4KYRG8FzmNlTmEaYQn7G3sXc+vaPUgJJVnOmLG6xaOLvhAisitXs2KiSzTo6Au6r+7UdgefMiOX4kwnPzh/LukOS9zy0g6VsqlQjBlK9McYR6T2jr7LVu+alWjD1VDpzeDptXgOt2qN1qdFRfqgLeYCMZ59dCmG6BINOjkuLcPnb9vrsJoFJ03XznHVKaVs+fFq8tJ6x+po2TsqZVOhGAuUpz/GOCKNVHzBMA6rOaaBynBQmpOCzWyKKbxW3aaJfkkf0T9/QSG76zoN4YZY0T8Uyfefktl7h6BH+lsOt7O8NMtoDAP0s3V0UiL2jpRy2C5uCoUiOVSkP8Y4jebomsXjjmqgMhxYzCbK810xaZvVrR6sZmHYODqZKTbu/fwCUqI2hmVGif7OWi3/PjpVVI/0AU6JWhgeCKfNTFiCP8k6/AqFYvhQoj/G9O2e5R7mSB9gbmFaTNpmdWsPxZnOhJF4NBlOq7E5a1tNB1OznEYPYIDsFBt6sH5yeXKir+/M9SpfX6EYdZTojzG6HaIv5g63vQOar1/X4aXTq4l3dVtPP2snEelRkf72mnYWTY3dEGYxm8hKsWGzmDhxWnJVs/ULXY8qxaBQjDrK0x9j7JHuWT2G6IfiLn4eC3MKtVTMvUe6WFaaTXVrDwsixc8GI8NppcsbpNnto7rVw1dWTO83pihDK87mSCbj6A9ruKR6KyttFrKeKYDUdLCngz0NbC7tz3g/NlfvOLsLrCmg1gMUiiGjRH+M0TNhats9LCjOGHZPH6IyeBq6mFOYRltPgJKs5CL9zBRtg9aG/VqTlcVT+5d+ePDLS5MTfIDys2kOZbJ7fzX5dieOoAd6qsDXCb4u7SecxB2AMEUuBn0vEPqFIz3xRcQWNcaeBhbb4J+nUEwQlOiPMXML0zEJ+Li+k3PnF+L2BUkbZtEvznTislvYe6SL6hKtZ25JtnOQd2nou3LfiXTWmh+n3k95XuJKn/045ZtU5jRx68cbeeHTp3DS9OzY16WEoC9yAegEv7v3YuBzx14cjNc6tde8HdBRE/ueZKqAm21RF4R4F5B4F5GoO4/o46bh2V+hUIwUSvTHGKfNTGluKrvrOpFSRpqiD+9fixAi0lCly0jX7Jujnwhd9N/dpzVb0Z8fC4anH6/SphBgdWg/rmMssx0OQ6A7crHQLxRdUReRPj/RFwv3EWiJeh4cvME8oNlOA9lScS8ucS4itlRlXylGBCX644ATitLZXtOOLxgmGJbDLvqg7cz9x84jVEc2ZiVr7+gif6TTy8VLEjdsGQopfRavRwyTqVdUSW4NIyGhYNQFI/ruo+/dSJwLSfuh3rsRX2fS9lXI6sJvTsHpyjy2i4hleNeIFMc3SvTHAfOK0vn79nrqO7QGJYOVVT4aZhek8X8bq9lyuB2X3WJ49YMRPW7R1MxhmYujT5rqcYHZAs4s7edY0O0rfx+ryufudxF5d0clHe2tXFSapr3m7YTOutiLSjL2lck6yNqGvr7R5yJSdoZa75iAKNEfB8wrSgegokprNRCva9axMieymPv23iZKslOS3gmb7owW/YHr9yeLnqbqPZ5Ef7iItq9Scwcc+njVRt6ub2L1F86Lv1AeDkOgJ4F1Fb3e0dX/bsTdCL4DvccDPf3P/8NaJfoTECX644B5U3TRbwOGp5Z+X+YUaqLv9gX7FVobCN3eMQmYH5nnsTKgpz+MNHR6yXNpXcCOR9oj+yMaOr1Mz0ntP8BkikTnLobPvoq6ONjifKbiuEdtzhoH5KfZyU61sSkS6Q/35izQmp3kRkomJLsxC7R9BA6riVn5aTHlGY4Fw9MfwUi/vcfP6fet45Wd9SP2GSON3mjmSMT2G1F0+yqzBArmwbSVaiF5gqJEfxwghGBeUTqVzVpBM9cIePrQm68/lEhfG5/CyhnZgw9MErtF+2fnHcFIv7HLhz8Y5lBLHNviOEHfCa03o1cohgMl+uOEE4rSjMeuYeqa1Rdd9KflJB/pA6y94WR+eP4JwzYPIcSI98ntipScaHEfn20Zw2HZK/qjEekrJg1K9McJ86L88pHw9KHXky/LHcJmKjRrKLpk8nDgtJlH1NPv9GhpkW3joBdvty/Iqf/1Jv/c3ZBwjJTS6KkA0OULEo48VZG+YjhRoj9OOKFo5EX/80uLee7GUyjLHfsFupGO9PXici3dYy/6h1p6qOvw8urOI3FfD4clp/73v/hTRY1xrKOnt0WlivQVw4kS/XFCeZ4Lm1n76xiJlE0Aq9nEirLh8+aPBafNPKIpm50Ra6S12zdin5Esde3abt6NVS1xX+/wBKjv8LK7vjPmGGhrqSMV6Td2enn0rQOEwknk+ismDEr0xwlWs4lZBS5SbOak6twf7zitI2zvRBrMt3UHBhk58tR1aKJf3eqhvqN/OQfdgmp2916g2iN9iadlp4xYpP9fr37Cz/7xCdtq2kfk/IrxiRL9ccTSaZn9ullNVLQ+uaNh74x9pF/b3iv0Gw+29nu9raf/orPeuGZOQRqNXb5hj8YPNLn5y9ZaADZH9ocoJgdK9McRPzj/BP7v6yeP9TRGhZG3d7RI3xsI0zPCTdg7egL8/t2DCT+nvt1rVDqNJ/rtkUg/+gKlb8yaW5hGKCxj7gKGg4f/tR+7xUx+mp2KQ/3npJi4qB254wiX3TIiG7PGI06rmfpRWMgFaO32D9vGsr7sqOng5j9uprrVQ6bTyqUnTe03pq7dQ0m2k1lWV9KRvr4xa06htsB/pMM7bHeBepT/9dNn0NTlY/2+JtWkfhKhIn3FmHC0KZstbl9Skbu+kAua6A83Ukqe+fAQlz76HsGQZr0kWnCta/cwJcPJirJs9jW6aekTteuRfmuP37BxOjwBUmxmowT2cC7m6lH+18+YwUmlWTS7/RxuPX43sSmGhhJ9xZjgsB6dvfOV//mQO/+ya9BxXd7eXsNDSdt8dUc9T71XNeAYbyDEHX/azr+/uJOTy3P4+7dOJzvVZmTpRBMMhWno8jEl08nKSObUpj4eur6QK2Xv4/aeABlOKwUZWlnk4VrM7egJ8JettVyxchq5LjsnTdeqhlYoX3/SoERfMSak2Ia+kCul5FBrN//6pJHwIAubnd4ApblalNyWpOj/Y2c93/zjRzyybn/CMXXtHr7wm/f485YabjtnFk9es5zsVBuF6Y64wqwvwk7JdLKwOBO7xdTP4mmPysnXLZ52jyb6ual2LCYxbJH+4dYewhKWRy5As/PTSHNYqDikRH+yoERfMSbom7Oid6EORo8/hDcQpqXbz56GLuP4x/WdPPTmvpixnZ6gUZkyGXvn3X3NfOv/tiLpzZGPx4Nv7qOy2c3jVy/nO5+ZbVTwLMpwGP0QotGj/ymZDmwWEydOy+qXrx8r+pr109ETIDPFiskkKEhwQTka9M5pUyP1l0wmwYnTsvhIif6kQYm+Ykxw2syEJfiC4aTfE73QqTdqB/jlG3v5xRt7cft6vf5Ob4DiTCdWsxjU3qlq7uaGpysoy03lhjNm4AuGE1pPG6taWVWey9lz82OOF2Y44kbjdRGxnpKpiezysmx213XGzLWtx096pMhec7ce6fvJdNp6zz1Mol9jiH5v/aVl07PY29g14MVOMXFISvSFEOcJIfYIIfYLIX4Q53W7EOLZyOsfCiFKI8dLhRAeIcTWyM9jwzt9xfGK3hRkKL5+dErjuxHRb+/x89aeRgCau3zGOf3BMBlOK1kptkHtnQ0Hmunxh/jNV09kenZq5Lz9BbDF7aOyqZtlpf13NRdlOGjt9vf7PnqkX5ShZd7MKUgjLDHaVoKWvTMz32V8Bmh3G3rXssL0+BeURLyzrwlfMP7vtabNQ7rDEtPr+KTSLKSEjw6raH8yMKjoCyHMwCPA+cA84MtCiHl9hn0NaJNSzgR+Bfws6rUDUsolkZ+bhmneiuOco6mpr0f6S0oy+bCyFX8wzCs7jhCIZM/ouexdkd246Q4L2am2mEj/0bcO8PPXPok5b3WrB6tZUJqTagitviM2Gt33Xlbav2ViYYYWyTf0Eee6dk1k0xzaeQvS7f3Gtff4Kc1NxWwSvZ5+ZCFXO7cW6Sdjhe050sWVv9/IL9/YG/f1mjZPTJQP2u/TbBJqk9YkIZlIfwWwX0pZKaX0A2uBi/uMuRh4KvL4eeAcoZJ+FQOgF5Xb2+BO+j26N3/R4il4AiG2HG7jpS21Rieu5ohg6jn6aQ4r2am2GE//pS21vPhRbcx5q9t6KM50YjYJMiNCGy/S33yoDZvZxMLi/m0j9Ui+r69f1+4xrB3AyLVv7Oy9a2nr8ZOTaotcoHx4AyF8wTAZUZG+JxAySksMxCdHtPo9T7xbFXM3oVPT1mP4+TopNgvzitLVJq1JQjKiXwxURz2viRyLO0ZKGQQ6gJzIa2VCiC1CiLeFEKcf43wVE4Sz5uRRlpvK7c9ti5vqGI/miL1z4aIiTAKerahmY1UrayIbovRIX8/RT3dqkb5u7wRDYQ42d1PX4Y2xYWpae4xuYrrQxhP9TVWtLJqaEbdfbWFE9Pt673Xt3hjRz+8T6XsD2uJ0ZoqNnFQbzW6/8dnRnn68c8djf6Mbs0lgMsF//yP2jkZKSXVr/0gf4KTpWWyv6VDF1yYByYh+vIi977+MRGPqgWlSyqXAd4E/CiH6NVoVQtwghKgQQlQ0NTUlMSXF8U66w8r/XHUS3kCIG5/ejMcfYvOhVn7+2ifsqOmI+54Wt59Um5n8dAeLSzL5cyRiv3ZVKRDP3rHG2Ds1bR78IW3hODoKro6yPDJTNKHt6GPveAMhdtZ2cFIcawe0aBziRPodHqZk9u6ktVvMZKVYDY9ez8vPSrGR67LT4vYZC6qGp6+LfhK+/r4GN9NzUrjxjHL+vr2ezVHRe2u3H08gREl2/85pS0oy6fGH2BuVFaWYmCQj+jVASdTzqUBdojFCCAuQAbRKKX1SyhYAKeVm4AAwu+8HSCl/J6VcJqVclpeXN/RvoTgumZmfxgNfWsLOug5OvPcNLn30fR5Zd4Af/WVnXP+6tdtPdqTP72kzcwEtQp2R5yIrxdob6Xv1SF8T/Q5PgEAozP7GXiupKtJGsdsXpLXbbwhhIntnW3U7gZBk+fT4palT7RbSHRaORFXR7PEHae8JxET6oFk8DRF7R68CmpViJcelR/rahcDw9NP1SH/wO6J9jV3Myndx45kzKEi3c8/fPjZ+lzVt2vvjRfpLp2UCsOWwqrg50UlG9DcBs4QQZUIIG3A58HKfMS8DV0cerwH+JaWUQoi8yEIwQogZwCygcnimrpgIfHpeAXdfNJ+z5uTxwJeW8P3z5rK1up3NcfLGm90+clI1e+T0WVpw8PmlmtOY67LT3BXx9CPF1tIcFnJStYtEe0+AA01Roh/pR6znrZdEhDDFZsZqFkbBMx19EVffwRqPogxnTKRf1649Lo4j+o1d3si8tDlr9o4W6eufrYt+gSH6Axdd8wfDVLX0GE3sbzqznG3V7Uaf4F7R7x/pT8tOITvVxtbqNtp7/Lz5ceIuX4rjm0GrUEkpg0KIW4DXADPwuJRylxDiHqBCSvky8HvgaSHEfqAV7cIAcAZwjxAiCISAm6SUarVIEcNVp5Ry1SmlgBYdP/b2Af7nncp+qZEtbr+xYLq8NIunrlvBqnJt6SjXZe8f6TusZEVEv7Xbz/5GN3lpdgKhMFUtEdFv1YRQ9/SFEGQ4bf0i/YqqVmbmu4zzxaNvrn7vxqy+om83Flz1YmtZqVqk3+0PGd69bu/YLCZKsp2RTV2zEn5+VUs3obBkVoGW/nlquXY3VHGojdLc1Kgc/f6iL4RgSUkmWw63880/fsSG/S1s/o9Pk+OyJ/w8xfFJUnn6UspXpJSzpZTlUsqfRo79OCL4SCm9UsrLpJQzpZQrpJSVkeMvSCnnSykXSylPlFL+deS+imIikGKzcOXJ03l9dwMHI9G4Tmu3n5yIvSOE4MzZeVgi3cZy0+yGd9/lDWA2CVJsZrIjIt3S7eNAk5vyvFSm56Qa0a/u7ZdECWFmijXG0w+HJZsPtbFsgCgf+u/K7Zujr1OQ7qCpy0cwFO7j6WtzrYzckejrCwBfXjGNDftbjIsFgNsXZF+UB78vkgml5/zPyneR5rAYvn51Ww+ZKVYjfbQvS0sy2d/kZlu1tqYyHvoLK4YftSNXMe646tTpWE0mHn/3oHFMSklLt4/s1PiRZ06qzdic1ekJku6wIIQw7CA90i/Pc1GWk9Ib6bf1xFwcQPP1oyP9/U1uOr3BuJuyoinMcNDs9uGP7DKu6/BiEvQriVyQ7iAstUJw0f69PtfK5m7MJkFqVDP6r6yYhtNqNn4n4bDkpqc3c9HDG4zdvfsauxBCa70JvSUWdKtMy9HvH+XrLJmWiZQY52vqUqI/EVGirxh35Kc5+PzSKfxpc7WRbtnlCxIISSMa7ktemp0uXxBvIESnN0B6xA/PStX+3NugCffMfBfTc1Kpa/fgC4aobvVQkpUSU0s+MyVW9CubtAvEnIK0AeddlOFASgy/vq7dQ0G6A6s59r+ZfhFo6PTS3hPAaTXjsJqNu5gDjW4yndY+c7Jx6UnFvLS1jma3j2c+PMS7+5vxBEKs+0Tbkbyv0c207JSYlNJl07PY2+CmoyegiX5m/0VcncUlmUTvrhnuxi2K8YESfcW45IvLSvAGwmyq0qwJfadqTgLR1y8GzW4fnZ4AaZFaNlkRi2RTpLJleZ6L0tyUSCkEDzVtPf1SGDOctpg6NHpOvV7mOBH6rlzdk69p6+ln7UDvrtwjHV7aegJkRbz73Ih/XtfhNfYLRHPtqjL8wTD/9con/Ocrn3D6rFxyXXb+sfMIAPsb3MzKj70w6QvPHx1ui7sxK5p0h9W4SwAl+hMVJfqKccmsSFRdGfH19Zo0iewdXTCb3X46vUHSI7611Wwiw2llS7VmceiRPmgZPNWtPf1SGLVIv9faONLpxWIS5Cb4bJ3oXbnNbh8VVW1xLSEj0u/y0d7jN7z76AtaprO/6JfnuTh7Th4vfFSDxSy4b80iVs8vYN2eRty+IJXNbmMRV2fJNK3Ewuu7j+ANhI0F60QsLck0HivRn5go0VeMSzKcVvLS7ByI5Nbri7Q5CbJnDNHv8tHlDRiiD5CdasMbCJNiM1OU4aAsIvpbq9vp9of6CWGm00q3P2R48w0dXvLT7EYZ5URE75z980c1BMOSLy4r6Tcu12XHJKCx00tbj9+woFJsFqOkRPQibjQ3nlmOzWzi3osXUJTh5PwFhfT4Q/zhg0MEQpJZ+bGir5dY+Pv2eiB+5k40S6ZFib7y9CckSvQV45byvFQjt35QeydNE/2Wbp+2kOvszUbWF2nL81wIIchMsZLusPDOPm33d0kfIdRTJXWL50inl4I4Nk1f0uwWUm1m6jo8rN1UzfLSLCOTJhqzSZCXZjc8/WiB179fRpxIH+DkGTls+fFnjP0JJ8/IIcNp5f97R9v+0tfeAc3i0ev2xNuYFc3Skt4MJRXpT0yU6CvGLeV5Lg40dSOlpLVbt3fii75+B6DZO4GYtETd1y/P0yJ8IQSlualsr9VSE/tG+hl9SjEc6fQau2IHQghBYYaD13YeobKpmy8tn5ZwbEG6gyOdPi3Sj/Lv9bz4RKIPvcXqQLOvPn1CgVFsrjw/td/46A1lg0X6swtclOVq52gegd7CirFHib5i3FKe56LDE6Cl20+z20+aw4Ld0r/YGWj1+dPsFo50eOnxh2LsHf2CEB11T89JRa/0EM/egd5SDA0d3n5pl4koynBS1+ElzW7hswsLE47TumF56PAEjIsSQG5krplxFnITcf4C7XOmZjlJsfXfb6mLfnaqLeaCEQ+L2cS6O87iC0uLjRRYxcRCib5i3DIjEplXNnXT0u1P6Ofr5KbZjQ1dMfaOq9fe0SnL0YQ+K8VqNFDXyYyqtNnlDdDtDxl+/WDo4z63ZEpcAdYpSNfmGpax/r2+NhFvITcRp83KJdVm7ufn60zJdDIlwzFolB+NVgfIN6R2lorjg0HLMCgUY4Uu0gea3LR2+wYtCZDrshm7WWMWclPiR/rQP8qH3pLG7Z6Aka6ZjL0DvRk8ly/vv4AbTUGaw2j+EmvvRDz9IUT6DquZR644kby0xL+fH372BKzm5Ftc5Lrs+IJh3L5gwh2844Gv/28Fp5bncO2qsrGeynGDEn3FuKU404ndYuJAo5sWt3/QdMNcl51Nke5P6VGR8vkLC+n0BmIi/dJc7VwlcRY2e2vq+40iZ8naO19aXsKUTGfcRivRRJ8vK2YhV4/0B76r6ctZc/IHfP1zi6cM6XzRKbDjVfRDYcm6TxoJhsKDir6Ukr9ur+dTc/P73dlNNpS9oxi3mEyCGXkuDjS5aXb7E+7G1YnO7NE3Z4GWsXL76jkxKZd6pD81Tm35NLsFk9Cyd/QCavE2WcVjalYKX14xjcEax0VnA0VH9fp3TB+CvTMS6NlQ4zmDp9ntIxiWHIrTIawv+xvdfOv/tvDSltpBxw5GY5eXG5+uGLT38nhFib5iXFOel8q+RjdtPf6EmTs6uVH2T/og0WlOqo0fnj83bh69ySRIj9TfMeydJEU/WfRduRAb6Z85O48bzpgx6J3CSGPscB7Hi7m1kYJ2Na0ewoN0/NJ7KejlpY+F9XubeW1XAxsONB/zucYCJfqKcU15nouaNg+hsDQKkiUiRvSdA9/CCyG48czyGMsnmkynlXZPgCMdXjKc1rgtEo+FgrRoe6f3ApWZYuPfPnsCNsvY/tfMc43/SL8+0q/AHwrT0DVwVzF9Z3dtkq05B2Jfo1bZdCj9nccTSvQV45ryqMXXRBuzdGJF/9jskYwUm+bpJ5mjP1QyU6zYLCZMYvVXG+gAABEqSURBVPC7krEgO9WGEL3N5scj9VGdxPRS2YnQN/kl2495IPQS1vuO09aSSvQV45oZub2bjQaL9PPS9Fr74BogXTIZMp1WOjwBGpPcjTtUhBAUpNvJcFoHLe8wFljMJrJSbOM60o+O2g8P4uvrlVKHQ/T1PsLHaz9hJfqKcY2eqw/JR/ouu+WYhVQvr6xF+iPTPaogzRHj5483clLHt+jXt3spzUnBbBIcHiDSl1IaqbwNnV4CofBRf2aPP0hNmweH1URVSw++YCjuOG8g/vHxgBJ9xbgmxWYxeswOtjlLT3ccDrsk02mltdtPU5dvROwdgEtOLObSk6aOyLmHA60F5fi1d+o6PJRkpzAl0zFgpN/SrVVenVuYRlj2lr4+GvQF4XPmFhAKy37d3QAOtXSz6K7X2XhwfHaGVaKvGPfo0f5A/WkBUm1mHFbTsKQ7ZqTYcPuChCUjYu8AXLFyOt88e+aInHs4yE2zj+tIv67dS3Gmk+nZqQOKvm7tnDZT6xk82GJuMNTbx6Ev+uLtZxcWxTyPZlNVG/5QmI0HWwb/EmOAEn3FuGfptCxKc1L6daDqixCCXJeddMexb76JLoMwUpH+eCfXZRu3KZu+YIhmt4+iDCcl2SmDiL4mzKfPzgMG9/X/9/1DXPbY+zH9iHX2NXRhM5s4e24eZpOIu5i7u057355xmt2jRF8x7rn1UzP527dOT2rsyrIclkQ1AjlaogueJbsbd6KR67LT7Q/h8Y8/f1q3aKZkOpiWnUJrt58ur1Yg71BLN89vrjHGVjZ3Y7OYWF6qFZ6rHSRXX3/v9kiD+Gj2NbqZkZdKis3C9JyUuIu5u+q09+2Jc9EYDyjRV4x7rGZT0lvnf/HFxfzwsycc82dGi/5wb8w6Xsgz2jf2imRjp5dfvrGX98Z4Y5Ju0UzJdDI9UjxPj/Yf+Oc+7vjTNqoifntlk5vSnBRSbBZyXbaY79OXj+s72V2vifXOuv6iv7ehy+jqNjs/zUjf1JFSsru+EyE0W0lvxDOeUKKvUMQhI1L7xmoWRsG2yYYuphc++C7ffXYr//biDk772ToefHMfX3uygp21/UURtAyXRFktfdnf2HVU2TT6xqwpmU6mRWoyVbf2EAiFefPjBgBe3lYHaOI7I9dljB9oV+6fP6rBYhLMLnCxqy42Uu/2aZk7ejXT2QUuqlq6YzJ1qls9dHmDnDIjh2BYGvsDxhNK9BWKOOiRfn6aY1zm0Y8GK2fk8OLNp3LJicW88XEDz1fUsGbZVF74xqlkp9r42lObYjZIHenwcu/fdnPSvf/k22u3Dnr++g4P5z3wDr98Y++Q56Z/blGGwyjEd6ilh40HW+n0BnHZLby8rY5AKMzh1h4jGaA405nQ0w+Gwry4pY6z5+azamYuu+s6CUWVd9AFfHakD/GsAi0bSF8ohl5r55JIZ7PxmMs/ucvNKRQJ0BdyJ6u1o7N0WhZLp2Xx4wvnEQiFjYqbv79mGWsefZ+rfr+RRVMzOdTSzfaaDkJSMjPPxas7j3CgyZ2wzAXA67saCIYlf3j/EN84q3xIqba17V5yUm04rGYcVjOZKVYOt/ZQ167l0H/707P4yd8/5rVdRwiGpTGPKZlO3trThJSyX1G8d/Y10+z2cemJU+n2BfEEqjjY7GZmpAWlnqlj2DuRP/c1djFvSjoAu+o6MZsE5y0o5Id/3sEnR7q4OOlvNTqoSF+hiIPernCyZu70xWE1x5RYnluYziNXnEhjl48N+5sxmwRXnjKdt+44iz9cvxKbxcTj7x4c8Jyv7TpCVoqVLl+QP354eEjzqe/wUJTZ+3czLZLB88buBs6YlccXTpyKxST49T/3AcRE+p5AiLZIVzQdKSXPb64hM8XKp+bmsyBS8G5nba/Fo2fuTI/cWZTlpmIxiZhoflddBzPzXKQ5rJTnudhzREX6CsVxgcVsojjTGbexuULjzNl5bP3xZ+KWkb5kSTEvfFTD7avnxK2O2tbt58ODrdx4xgy213Tw+LsHuXZVacJ2mH2pa/dQmtO7W7skO4U3djfgD4b5zmdmk51q4/RZuazb0wTAjKhIX39/dqqNX7y+h7Wbqmnr9hMMS646ZTo2i4nyvFTsFhM7azuMJvR7G7qYkZeKJZI6bLOYKM1NjcnV31XXaewHmFOYxuZDbUl9n9FERfoKRQL+eutp3Hx2+VhPY1yTqG/A104vwxsI88wHh+K+/uYnjYTCknPnF3LTmeU0dvmGVOu+vt1rCDjA9OwU/MEwJgHnnFAAwEVLtMYxuS6bceem7+6uadP6E/9ufSXFmU6+fsYM/uOCE7jtnFmAdtGfW5RuZPBIKdnb4DasHZ3ZBS7D+2/q8tHY5TOsnjmFadS2e+j0xt5VdPQEuOrxjazb05j09x1OlOgrFAnITrUlHXkqYpldkMaZs/N46v1DcTN5Xtt1hKIMB4umZrBqZg7zp6Tz2/WVMQun0Ww+1MbnHnqXvQ1ddHoDdPmCTOlj7wAsL8027iw+M68Qu8VkZO4AFGf1Rvp/3VaHLxjm3osX8P3z5nL96TNiWnIumJLOrtpOwmHJu/ubqW33sLIsO2Ze584vpLbdw2NvHzAWcedP0ayhObrn32cx99G3D7B+bxO3/nEL+xtjXwseQ12gZFGir1AoRoTrTy+j2e3jkkfe408V1UZqY48/yPq9TayeV4AQAiEEt5w9k8qmbh57+0C/87h9Qb797BZ21Hbw7y/uMDZXFWX0Rvq66K+eX2gcc9kt3HvxAm46a4ZxLCvFisNqorbdw58qqplbmMaC4vS4819QnEGXL8jh1h5+/toeijOdXLYstlbSRYuncOGiIn75xl7+b6O2LhEd6QN8EuXrH+nw8sSGg5w9Jw+H1cTX/3czHZ4Au+o6uOnpzdz27OBZT8eKEn2FQjEinDYzl5+vWUQwHP7/27v/2KrKO47j709/WJCipVALhWJhtDgtAbSUHy5uY9YyNug0s5ZoRqYLW6JR53SRbInOzGUmRpS4MM3mFs0GU0YcYdmYK8TNhanFuQ7lV/1dRFtjh8EZIuG7P87Telt621ta7D33fl/JSe95ztP2+fZpvvee55x7v9y2uZUlP93BfU8dYMsLhzh2/AT1CQl6WfXknuS5+43en3tz9x/30t71Edcsms7zr3exvjm6OJu4vLNgRjG31c+msU9SblxQztLzSnv2JTG1aCxPH+jk3+1HaKwpT7pEVR1esd/7l/20th/h5ksrTzrzk8Tdl89h8llj2P7Su5QXj+1ZSpo2YSyFBXm9LuY+0HyQE2bc1VDNhmsuor3rf9Sv+xtfWf8M/2h7j8+UFGI2cBWw4Uop6UtaJmm/pDZJt/dzvEDS78LxZyVVJBxbG9r3S6ofuaE759KZJK6sKWf7zZfw228tZH55EeubD/LDJ/dQdGY+tQlLJZL4yRVzKCsaw40bX+TIR9E6+M59HWx87k3WXDKTu1ZWU1tRzJ/2vAPQa3knPzeH6784K6Ui7mVFY2nrOEp+rnou0vananIheTliW+thZp1TyBUX9v+JqGePzWf9qnnk5qjniaI7pqrSwp5X+q92HuXxlre4euG5lBefyYKKYu7+2hxyBLfUVfHM7Uu5pa5q0PrKwzXo3TuScoGfAXVAO/C8pK1m9nJCt+uALjObJakJuAe4StL5QBNwAVAG/FVSlZml34d5OOdOC0ksmTWJJbMm8WrnUR7d9QZVpeN77oLpdtaYfNY3zefKn++i7r6nOWHw/ofHmF06nlvqqsjJET++vJrlD/wdI3rj3KmYFtb1684vHbDuckFeLlWl43n58AfcelkVuQO8Se+ic4t59NraXmcfEC3xbHnhEI0P7eJQ10cU5OX0+mTVxgXlNC44uU7z6ZTKLZu1QJuZvQogaRPQACQm/QbgzvB4M/CgoqerBmCTmR0DXpPUFn7erpEZvnMuTmaWFHLnyguSHp8/fQLrrprHtta3KR53BhPHFdBUW96zrFJVOp7b6mez+42uAZPwQMrCtYArawZPtsvnTGbK2WN6LUUlc3G4VTNRw7yptHUcJUdR6c+mBeWUjD89RXlSlUrSnwq8lbDfDixM1sfMjks6AkwM7f/s870nnU9JWgOsAZg+fXqqY3fOZaAVc8tYMbcs6fFvf354t9GumFvGseMnuKSyZNC+NyytHNbvWjRzIk98Z8mwfsZIS2VNv7+n075XGpL1SeV7MbOHzazGzGpKSgafCOecO1UVk8Zxa/3sUz5TiLtUkn47kHgeNA14O1kfSXnA2cD7KX6vc865T0kqSf95oFLSDElnEF2Y3dqnz1ZgdXj8dWCHRfcdbQWawt09M4BK4LmRGbpzzrmhGnRNP6zR3wBsB3KBR8zsJUl3AS1mthX4JfBYuFD7PtETA6Hf40QXfY8D1/udO845N3p0ut8IMFQ1NTXW0tIy2sNwzrlYkbTbzGoG6+fvyHXOuSziSd8557KIJ33nnMsinvSdcy6LpN2FXEmdQP+VF1IzCXhvhIaTTjIxrkyMCTyuuMmUuM41s0Hf3Zp2SX+4JLWkcgU7bjIxrkyMCTyuuMnUuJLx5R3nnMsinvSdcy6LZGLSf3i0B3CaZGJcmRgTeFxxk6lx9Svj1vSdc84ll4mv9J1zziWRMUl/sDq+6UxSuaSdkvZKeknSTaG9WNJTkg6GrxNCuyStD7G2SrpwdCNITlKupH9J2hb2Z4Q6ygdDXeUzQnvSOsvpSFKRpM2S9oV5Wxz3+ZL03fD/t0fSRklj4jpfkh6R1CFpT0LbkOdH0urQ/6Ck1f39rrjJiKSfUMf3y8D5wKpQnzcujgPfM7PPAouA68P4bweazawSaA77EMVZGbY1wIZPf8gpuwnYm7B/D7AuxNRFVF8ZEuosA+tCv3T2APBnMzsPmEsUY2znS9JU4EagxsyqiT5Rt7vedRzn69fAsj5tQ5ofScXAHUSVAmuBO7qfKGLNzGK/AYuB7Qn7a4G1oz2uYcTzB6JC9PuBKaFtCrA/PH4IWJXQv6dfOm1ERXOagaXANqJKau8BeX3njeijuxeHx3mhn0Y7hiRxnQW81nd8cZ4vPil5Whz+/tuA+jjPF1AB7DnV+QFWAQ8ltPfqF9ctI17p038d35Nq8cZBOE2eDzwLlJrZYYDw9ZzQLS7x3g98HzgR9icC/zWz42E/cdy96iwD3XWW09FMoBP4VVi6+oWkccR4vszsEHAv8CZwmOjvv5vMmK9uQ52ftJ+3U5EpST+lWrzpTlIh8HvgZjP7YKCu/bSlVbySvgp0mNnuxOZ+uloKx9JNHnAhsMHM5gMf8slSQX/SPrawbNEAzADKgHFEyx59xXG+BjOsGt9xkylJP/a1eCXlEyX835jZltD8rqQp4fgUoCO0xyHei4GVkl4HNhEt8dwPFIU6ytB73MnqLKejdqDdzJ4N+5uJngTiPF+XAq+ZWaeZfQxsAZaQGfPVbajzE4d5G7JMSfqp1PFNW5JEVHJyr5ndl3AosfbwaqK1/u72b4S7DhYBR7pPW9OFma01s2lmVkE0HzvM7GpgJ1EdZTg5pv7qLKcdM3sHeEvS7ND0JaKSoLGdL6JlnUWSzgz/j90xxX6+Egx1frYDl0maEM6ELgtt8TbaFxVGagOWAweAV4AfjPZ4hjj2zxGdNrYCL4ZtOdEaaTNwMHwtDv1FdLfSK8B/iO64GPU4BojvC8C28Hgm8BzQBjwBFIT2MWG/LRyfOdrjHiSmeUBLmLMngQlxny/gR8A+YA/wGFAQ1/kCNhJdm/iY6BX7dacyP8C1IcY24JujHddIbP6OXOecyyKZsrzjnHMuBZ70nXMui3jSd865LOJJ3znnsognfeecyyKe9J1zLot40nfOuSziSd8557LI/wHz6KCEYSaEJAAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "tplt( lt, 'mean_squared_error')\n",
- "tplt( lt, 'val_mean_squared_error')\n",
- "plt.legend()\n",
- "plt.show()"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.4"
- },
- "pycharm": {
- "stem_cell": {
- "cell_type": "raw",
- "source": [],
- "metadata": {
- "collapsed": false
- }
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/examples/tensorflow/notebooks/keras-sentiment/sentiment-analysis.ipynb b/examples/tensorflow/notebooks/keras-sentiment/sentiment-analysis.ipynb
deleted file mode 100644
index 91a3a18ad..000000000
--- a/examples/tensorflow/notebooks/keras-sentiment/sentiment-analysis.ipynb
+++ /dev/null
@@ -1,974 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Sentiment Analysis with TensorFlow\n",
- "\n",
- "A Convolutional Neural Net (CNN) is sometimes used in text classification tasks such as sentiment analysis. We'll use a CNN built with TensorFlow to perform sentiment analysis in Amazon SageMaker on the IMDB dataset, which consists of movie reviews labeled as having positive or negative sentiment. Three aspects of Amazon SageMaker will be demonstrated:\n",
- "\n",
- "- How to use Script Mode with a prebuilt TensorFlow container, along with a training script similar to one you would use outside SageMaker. \n",
- "- Local Mode training, which allows you to test your code on your notebook instance before creating a full scale training job.\n",
- "- Batch Transform for offline, asynchronous predictions on large batches of data. \n",
- "\n",
- "# Prepare Dataset\n",
- "\n",
- "We'll begin by loading the reviews dataset, and padding the reviews so all reviews have the same length. Each review is represented as an array of numbers, where each number represents an indexed word. Training data for both Local Mode and Hosted Training must be saved as files, so we'll also save the transformed data to files."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Using TensorFlow backend.\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "25000 train sequences\n",
- "25000 test sequences\n",
- "x_train shape: (25000, 400)\n",
- "x_test shape: (25000, 400)\n"
- ]
- }
- ],
- "source": [
- "import os\n",
- "from keras.preprocessing import sequence\n",
- "from keras.datasets import imdb\n",
- "\n",
- "max_features = 20000\n",
- "maxlen = 400\n",
- "\n",
- "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
- "print(len(x_train), 'train sequences')\n",
- "print(len(x_test), 'test sequences')\n",
- "\n",
- "x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
- "x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n",
- "print('x_train shape:', x_train.shape)\n",
- "print('x_test shape:', x_test.shape)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "data_dir = os.path.join(os.getcwd(), 'data')\n",
- "os.makedirs(data_dir, exist_ok=True)\n",
- "\n",
- "train_dir = os.path.join(os.getcwd(), 'data/train')\n",
- "os.makedirs(train_dir, exist_ok=True)\n",
- "\n",
- "test_dir = os.path.join(os.getcwd(), 'data/test')\n",
- "os.makedirs(test_dir, exist_ok=True)\n",
- "\n",
- "csv_test_dir = os.path.join(os.getcwd(), 'data/csv-test')\n",
- "os.makedirs(csv_test_dir, exist_ok=True)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "import numpy as np\n",
- "\n",
- "np.save(os.path.join(train_dir, 'x_train.npy'), x_train)\n",
- "np.save(os.path.join(train_dir, 'y_train.npy'), y_train)\n",
- "np.save(os.path.join(test_dir, 'x_test.npy'), x_test)\n",
- "np.save(os.path.join(test_dir, 'y_test.npy'), y_test)\n",
- "np.savetxt(os.path.join(csv_test_dir, 'csv-test.csv'), np.array(x_test[:100], dtype=np.int32), fmt='%d', delimiter=\",\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Local Mode Training\n",
- "\n",
- "Amazon SageMaker’s Local Mode training feature is a convenient way to make sure your code is working as expected before moving on to full scale, hosted training. With Local Mode, you can run quick tests with just a sample of training data, and/or a small number of epochs (passes over the full training set), while avoiding the time and expense of attempting full scale hosted training using possibly buggy code. \n",
- "\n",
- "To train in Local Mode, it is necessary to have docker-compose or nvidia-docker-compose (for GPU) installed in the notebook instance. Running following script will install docker-compose or nvidia-docker-compose and configure the notebook environment for you."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "/bin/bash: ./setup.sh: No such file or directory\r\n"
- ]
- }
- ],
- "source": [
- "!/bin/bash ./setup.sh"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The next step is to set up a TensorFlow Estimator for Local Mode training. A key parameters for the Estimator is the `train_instance_type`, which is the kind of hardware on which training will run. In the case of Local Mode, we simply set this parameter to `local_gpu` to invoke Local Mode training on the GPU, or to `local` if the instance has a CPU. Other parameters of note are the algorithm’s hyperparameters, which are passed in as a dictionary, and a Boolean parameter indicating that we are using Script Mode."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "WARNING: Logging before flag parsing goes to stderr.\n",
- "W0729 09:01:18.666472 4639487424 session.py:1106] Couldn't call 'get_role' to get Role ARN from role name olg to get Role path.\n"
- ]
- },
- {
- "ename": "ValueError",
- "evalue": "The current AWS identity is not a role: arn:aws:iam::722321484884:user/olg, therefore it cannot be used as a SageMaker execution role",
- "output_type": "error",
- "traceback": [
- "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
- "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0mtrain_instance_count\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0mhyperparameters\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mhyperparameters\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m \u001b[0mrole\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msagemaker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_execution_role\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 15\u001b[0m \u001b[0mbase_job_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'tf-keras-sentiment'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 16\u001b[0m \u001b[0mframework_version\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'1.13.1'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mget_execution_role\u001b[0;34m(sagemaker_session)\u001b[0m\n\u001b[1;32m 1310\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0marn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1311\u001b[0m \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'The current AWS identity is not a role: {}, therefore it cannot be used as a SageMaker execution role'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1312\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1313\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1314\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;31mValueError\u001b[0m: The current AWS identity is not a role: arn:aws:iam::722321484884:user/olg, therefore it cannot be used as a SageMaker execution role"
- ]
- }
- ],
- "source": [
- "import sagemaker\n",
- "from sagemaker.tensorflow import TensorFlow\n",
- "\n",
- "model_dir = '/opt/ml/model'\n",
- "train_instance_type = 'local'\n",
- "tornasole_s3 = 's3://' + sagemaker.Session().default_bucket() + \"/tornasole-parameters/\"\n",
- "hyperparameters = {'epochs': 1, 'batch_size': 128, \n",
- " 'tornasole-save-interval': 100, 'tornasole_outdir' : tornasole_s3 }\n",
- "local_estimator = TensorFlow(entry_point='sentiment_keras.py',\n",
- " model_dir=model_dir,\n",
- " train_instance_type=train_instance_type,\n",
- " train_instance_count=1,\n",
- " hyperparameters=hyperparameters,\n",
- " role=sagemaker.get_execution_role(),\n",
- " base_job_name='tf-keras-sentiment',\n",
- " framework_version='1.13.1',\n",
- " py_version='py3',\n",
- " image_name='072677473360.dkr.ecr.us-east-1.amazonaws.com/tornasole-preprod-tf-1.13.1-cpu:latest',\n",
- " script_mode=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we'll briefly train the model in Local Mode. Since this is just to make sure the code is working, we'll train for only one epoch. (Note that on a CPU-based notebook instance, this one epoch will take at least 3 or 4 minutes.) As you'll see from the logs below the cell when training is complete, even when trained for only one epoch, the accuracy of the model on training data is already at almost 80%. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 30,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Creating tmpsw39_nhj_algo-1-zwl3k_1 ... \n",
- "\u001b[1BAttaching to tmpsw39_nhj_algo-1-zwl3k_12mdone\u001b[0m\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m 2019-07-16 15:20:30,596 sagemaker-containers INFO Imported framework sagemaker_tensorflow_container.training\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m 2019-07-16 15:20:30,603 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m 2019-07-16 15:20:30,917 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m 2019-07-16 15:20:30,939 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m 2019-07-16 15:20:30,961 sagemaker-containers INFO No GPUs detected (normal if no gpus installed)\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m 2019-07-16 15:20:30,975 sagemaker-containers INFO Invoking user script\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Training Env:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m {\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"additional_framework_parameters\": {},\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"channel_input_dirs\": {\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"train\": \"/opt/ml/input/data/train\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"test\": \"/opt/ml/input/data/test\"\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m },\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"current_host\": \"algo-1-zwl3k\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"framework_module\": \"sagemaker_tensorflow_container.training:main\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"hosts\": [\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"algo-1-zwl3k\"\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m ],\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"hyperparameters\": {\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"epochs\": 1,\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"batch_size\": 128,\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"tornasole-save-interval\": 100,\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"tornasole_outdir\": \"s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"model_dir\": \"/opt/ml/model\"\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m },\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"input_config_dir\": \"/opt/ml/input/config\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"input_data_config\": {\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"train\": {\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"TrainingInputMode\": \"File\"\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m },\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"test\": {\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"TrainingInputMode\": \"File\"\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m }\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m },\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"input_dir\": \"/opt/ml/input\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"is_master\": true,\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"job_name\": \"tf-keras-sentiment-2019-07-16-15-20-27-160\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"log_level\": 20,\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"master_hostname\": \"algo-1-zwl3k\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"model_dir\": \"/opt/ml/model\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"module_dir\": \"s3://sagemaker-us-east-1-072677473360/tf-keras-sentiment-2019-07-16-15-20-27-160/source/sourcedir.tar.gz\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"module_name\": \"sentiment_keras\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"network_interface_name\": \"eth0\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"num_cpus\": 4,\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"num_gpus\": 0,\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"output_data_dir\": \"/opt/ml/output/data\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"output_dir\": \"/opt/ml/output\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"resource_config\": {\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"current_host\": \"algo-1-zwl3k\",\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"hosts\": [\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"algo-1-zwl3k\"\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m ]\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m },\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \"user_entry_point\": \"sentiment_keras.py\"\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m }\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Environment variables:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_HOSTS=[\"algo-1-zwl3k\"]\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_NETWORK_INTERFACE_NAME=eth0\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_HPS={\"batch_size\":128,\"epochs\":1,\"model_dir\":\"/opt/ml/model\",\"tornasole-save-interval\":100,\"tornasole_outdir\":\"s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\"}\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_USER_ENTRY_POINT=sentiment_keras.py\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_FRAMEWORK_PARAMS={}\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_RESOURCE_CONFIG={\"current_host\":\"algo-1-zwl3k\",\"hosts\":[\"algo-1-zwl3k\"]}\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_INPUT_DATA_CONFIG={\"test\":{\"TrainingInputMode\":\"File\"},\"train\":{\"TrainingInputMode\":\"File\"}}\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_OUTPUT_DATA_DIR=/opt/ml/output/data\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_CHANNELS=[\"test\",\"train\"]\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_CURRENT_HOST=algo-1-zwl3k\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_MODULE_NAME=sentiment_keras\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_LOG_LEVEL=20\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_FRAMEWORK_MODULE=sagemaker_tensorflow_container.training:main\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_INPUT_DIR=/opt/ml/input\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_INPUT_CONFIG_DIR=/opt/ml/input/config\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_OUTPUT_DIR=/opt/ml/output\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_NUM_CPUS=4\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_NUM_GPUS=0\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_MODEL_DIR=/opt/ml/model\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_MODULE_DIR=s3://sagemaker-us-east-1-072677473360/tf-keras-sentiment-2019-07-16-15-20-27-160/source/sourcedir.tar.gz\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_TRAINING_ENV={\"additional_framework_parameters\":{},\"channel_input_dirs\":{\"test\":\"/opt/ml/input/data/test\",\"train\":\"/opt/ml/input/data/train\"},\"current_host\":\"algo-1-zwl3k\",\"framework_module\":\"sagemaker_tensorflow_container.training:main\",\"hosts\":[\"algo-1-zwl3k\"],\"hyperparameters\":{\"batch_size\":128,\"epochs\":1,\"model_dir\":\"/opt/ml/model\",\"tornasole-save-interval\":100,\"tornasole_outdir\":\"s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\"},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"test\":{\"TrainingInputMode\":\"File\"},\"train\":{\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"is_master\":true,\"job_name\":\"tf-keras-sentiment-2019-07-16-15-20-27-160\",\"log_level\":20,\"master_hostname\":\"algo-1-zwl3k\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-us-east-1-072677473360/tf-keras-sentiment-2019-07-16-15-20-27-160/source/sourcedir.tar.gz\",\"module_name\":\"sentiment_keras\",\"network_interface_name\":\"eth0\",\"num_cpus\":4,\"num_gpus\":0,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_host\":\"algo-1-zwl3k\",\"hosts\":[\"algo-1-zwl3k\"]},\"user_entry_point\":\"sentiment_keras.py\"}\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_USER_ARGS=[\"--batch_size\",\"128\",\"--epochs\",\"1\",\"--model_dir\",\"/opt/ml/model\",\"--tornasole-save-interval\",\"100\",\"--tornasole_outdir\",\"s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\"]\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_CHANNEL_TRAIN=/opt/ml/input/data/train\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_CHANNEL_TEST=/opt/ml/input/data/test\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_HP_EPOCHS=1\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_HP_BATCH_SIZE=128\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_HP_TORNASOLE-SAVE-INTERVAL=100\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_HP_TORNASOLE_OUTDIR=s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m SM_HP_MODEL_DIR=/opt/ml/model\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m PYTHONPATH=/opt/ml/code:/usr/local/bin:/usr/lib/python36.zip:/usr/lib/python3.6:/usr/lib/python3.6/lib-dynload:/usr/local/lib/python3.6/dist-packages:/usr/lib/python3/dist-packages\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Invoking script with the following command:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m /usr/local/bin/python sentiment_keras.py --batch_size 128 --epochs 1 --model_dir /opt/ml/model --tornasole-save-interval 100 --tornasole_outdir s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m \n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Using TensorFlow backend.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m x train (25000, 400) y train (25000,)\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m [[ 0 0 0 ... 19 178 32]\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m [ 0 0 0 ... 16 145 95]\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m [ 0 0 0 ... 7 129 113]\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m ...\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m [595 13 258 ... 72 33 32]\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m [ 0 0 0 ... 28 126 110]\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m [ 0 0 0 ... 7 43 50]]\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m [1 0 0 1 0 0 1 0 1 0]\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m x test (25000, 400) y test (25000,)\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Instructions for updating:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Colocations handled automatically by placer.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Instructions for updating:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Colocations handled automatically by placer.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Instructions for updating:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Instructions for updating:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Instructions for updating:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Use tf.cast instead.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Instructions for updating:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Use tf.cast instead.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Instructions for updating:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Deprecated in favor of operator or tf.math.divide.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Instructions for updating:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Deprecated in favor of operator or tf.math.divide.\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Train on 25000 samples, validate on 25000 samples\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m Epoch 1/1\n",
- "25000/25000 [==============================] - 390s 16ms/step - loss: 0.4266 - acc: 0.7852 - val_loss: 0.2631 - val_acc: 0.8911\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m 2019-07-16 15:27:04,144 sagemaker_tensorflow_container.training WARNING Your model will NOT be servable with SageMaker TensorFlow Serving container.The model artifact was not saved in the TensorFlow SavedModel directory structure:\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m https://www.tensorflow.org/guide/saved_model#structure_of_a_savedmodel_directory\n",
- "\u001b[36malgo-1-zwl3k_1 |\u001b[0m 2019-07-16 15:27:04,144 sagemaker-containers INFO Reporting training SUCCESS\n",
- "\u001b[36mtmpsw39_nhj_algo-1-zwl3k_1 exited with code 0\n",
- "\u001b[0mAborting on container exit...\n",
- "===== Job Complete =====\n"
- ]
- }
- ],
- "source": [
- "inputs = {'train': f'file://{train_dir}',\n",
- " 'test': f'file://{test_dir}'}\n",
- "\n",
- "local_estimator.fit(inputs)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Hosted Training\n",
- "\n",
- "After we've confirmed our code seems to be working using Local Mode training, we can move on to use SageMaker's hosted training, which uses compute resources separate from your notebook instance. Hosted training spins up one or more instances (cluster) for training, and then tears the cluster down when training is complete. In general, hosted training is preferred for doing actual training, especially for large-scale, distributed training. Before starting hosted training, the data must be uploaded to S3. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 26,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{'train': 's3://sagemaker-us-east-1-072677473360/sagemaker-us-east-1-072677473360/data/train', 'test': 's3://sagemaker-us-east-1-072677473360/sagemaker-us-east-1-072677473360/data/test'}\n"
- ]
- }
- ],
- "source": [
- "s3_prefix = sagemaker.Session().default_bucket()\n",
- "\n",
- "traindata_s3_prefix = '{}/data/train'.format(s3_prefix)\n",
- "testdata_s3_prefix = '{}/data/test'.format(s3_prefix)\n",
- "\n",
- "train_s3 = sagemaker.Session().upload_data(path='./data/train/', key_prefix=traindata_s3_prefix)\n",
- "test_s3 = sagemaker.Session().upload_data(path='./data/test/', key_prefix=testdata_s3_prefix)\n",
- "\n",
- "inputs = {'train':train_s3, 'test': test_s3}\n",
- "print(inputs)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "With the training data now in S3, we're ready to set up an Estimator object for hosted training. It is similar to the Local Mode Estimator, except the `train_instance_type` has been set to a ML instance type instead of a local type for Local Mode. Additionally, we've set the number of epochs to a number greater than one for actual training, as opposed to just testing the code."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "metadata": {},
- "outputs": [],
- "source": [
- "train_instance_type = 'ml.p3.2xlarge'\n",
- "#hyperparameters = {'epochs': 10, 'batch_size': 128}\n",
- "hyperparameters = {'epochs': 1, 'batch_size': 128, \n",
- " 'tornasole-save-interval': 1, 'tornasole_outdir' : tornasole_s3 }\n",
- "\n",
- "estimator = TensorFlow(entry_point='sentiment_keras.py',\n",
- " model_dir=model_dir,\n",
- " train_instance_type=train_instance_type,\n",
- " train_instance_count=1,\n",
- " hyperparameters=hyperparameters,\n",
- " role=sagemaker.get_execution_role(),\n",
- " base_job_name='tf-keras-sentiment',\n",
- " framework_version='1.13.1',\n",
- " py_version='py3',\n",
- " image_name='072677473360.dkr.ecr.us-east-1.amazonaws.com/tornasole-preprod-tf-1.13.1-cpu:latest',\n",
- " script_mode=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "With the change in training instance type and increase in epochs, we simply call `fit` to start the actual hosted training. At the end of hosted training, you'll see from the logs below the cell that accuracy on the training set has greatly increased, and accuracy on the validation set is around 90%. The model may be overfitting now (less able to generalize to data it has not yet seen), even though we are employing dropout as a regularization technique. In a production situation, further investigation would be necessary."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 28,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "2019-07-16 00:36:04 Starting - Starting the training job...\n",
- "2019-07-16 00:36:09 Starting - Launching requested ML instances......\n",
- "2019-07-16 00:37:17 Starting - Preparing the instances for training......\n",
- "2019-07-16 00:38:15 Downloading - Downloading input data......\n",
- "2019-07-16 00:39:27 Training - Downloading the training image......\n",
- "2019-07-16 00:40:17 Training - Training image download completed. Training in progress.\n",
- "\u001b[31m2019-07-16 00:40:20,820 sagemaker-containers INFO Imported framework sagemaker_tensorflow_container.training\u001b[0m\n",
- "\u001b[31m2019-07-16 00:40:21,423 sagemaker-containers INFO Invoking user script\n",
- "\u001b[0m\n",
- "\u001b[31mTraining Env:\n",
- "\u001b[0m\n",
- "\u001b[31m{\n",
- " \"additional_framework_parameters\": {},\n",
- " \"channel_input_dirs\": {\n",
- " \"test\": \"/opt/ml/input/data/test\",\n",
- " \"train\": \"/opt/ml/input/data/train\"\n",
- " },\n",
- " \"current_host\": \"algo-1\",\n",
- " \"framework_module\": \"sagemaker_tensorflow_container.training:main\",\n",
- " \"hosts\": [\n",
- " \"algo-1\"\n",
- " ],\n",
- " \"hyperparameters\": {\n",
- " \"batch_size\": 128,\n",
- " \"tornasole_outdir\": \"s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\",\n",
- " \"model_dir\": \"/opt/ml/model\",\n",
- " \"epochs\": 1,\n",
- " \"tornasole-save-interval\": 1\n",
- " },\n",
- " \"input_config_dir\": \"/opt/ml/input/config\",\n",
- " \"input_data_config\": {\n",
- " \"test\": {\n",
- " \"TrainingInputMode\": \"File\",\n",
- " \"S3DistributionType\": \"FullyReplicated\",\n",
- " \"RecordWrapperType\": \"None\"\n",
- " },\n",
- " \"train\": {\n",
- " \"TrainingInputMode\": \"File\",\n",
- " \"S3DistributionType\": \"FullyReplicated\",\n",
- " \"RecordWrapperType\": \"None\"\n",
- " }\n",
- " },\n",
- " \"input_dir\": \"/opt/ml/input\",\n",
- " \"is_master\": true,\n",
- " \"job_name\": \"tf-keras-sentiment-2019-07-16-00-36-04-131\",\n",
- " \"log_level\": 20,\n",
- " \"master_hostname\": \"algo-1\",\n",
- " \"model_dir\": \"/opt/ml/model\",\n",
- " \"module_dir\": \"s3://sagemaker-us-east-1-072677473360/tf-keras-sentiment-2019-07-16-00-36-04-131/source/sourcedir.tar.gz\",\n",
- " \"module_name\": \"sentiment_keras\",\n",
- " \"network_interface_name\": \"eth0\",\n",
- " \"num_cpus\": 8,\n",
- " \"num_gpus\": 1,\n",
- " \"output_data_dir\": \"/opt/ml/output/data\",\n",
- " \"output_dir\": \"/opt/ml/output\",\n",
- " \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
- " \"resource_config\": {\n",
- " \"current_host\": \"algo-1\",\n",
- " \"hosts\": [\n",
- " \"algo-1\"\n",
- " ],\n",
- " \"network_interface_name\": \"eth0\"\n",
- " },\n",
- " \"user_entry_point\": \"sentiment_keras.py\"\u001b[0m\n",
- "\u001b[31m}\n",
- "\u001b[0m\n",
- "\u001b[31mEnvironment variables:\n",
- "\u001b[0m\n",
- "\u001b[31mSM_HOSTS=[\"algo-1\"]\u001b[0m\n",
- "\u001b[31mSM_NETWORK_INTERFACE_NAME=eth0\u001b[0m\n",
- "\u001b[31mSM_HPS={\"batch_size\":128,\"epochs\":1,\"model_dir\":\"/opt/ml/model\",\"tornasole-save-interval\":1,\"tornasole_outdir\":\"s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\"}\u001b[0m\n",
- "\u001b[31mSM_USER_ENTRY_POINT=sentiment_keras.py\u001b[0m\n",
- "\u001b[31mSM_FRAMEWORK_PARAMS={}\u001b[0m\n",
- "\u001b[31mSM_RESOURCE_CONFIG={\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\"],\"network_interface_name\":\"eth0\"}\u001b[0m\n",
- "\u001b[31mSM_INPUT_DATA_CONFIG={\"test\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"train\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}}\u001b[0m\n",
- "\u001b[31mSM_OUTPUT_DATA_DIR=/opt/ml/output/data\u001b[0m\n",
- "\u001b[31mSM_CHANNELS=[\"test\",\"train\"]\u001b[0m\n",
- "\u001b[31mSM_CURRENT_HOST=algo-1\u001b[0m\n",
- "\u001b[31mSM_MODULE_NAME=sentiment_keras\u001b[0m\n",
- "\u001b[31mSM_LOG_LEVEL=20\u001b[0m\n",
- "\u001b[31mSM_FRAMEWORK_MODULE=sagemaker_tensorflow_container.training:main\u001b[0m\n",
- "\u001b[31mSM_INPUT_DIR=/opt/ml/input\u001b[0m\n",
- "\u001b[31mSM_INPUT_CONFIG_DIR=/opt/ml/input/config\u001b[0m\n",
- "\u001b[31mSM_OUTPUT_DIR=/opt/ml/output\u001b[0m\n",
- "\u001b[31mSM_NUM_CPUS=8\u001b[0m\n",
- "\u001b[31mSM_NUM_GPUS=1\u001b[0m\n",
- "\u001b[31mSM_MODEL_DIR=/opt/ml/model\u001b[0m\n",
- "\u001b[31mSM_MODULE_DIR=s3://sagemaker-us-east-1-072677473360/tf-keras-sentiment-2019-07-16-00-36-04-131/source/sourcedir.tar.gz\u001b[0m\n",
- "\u001b[31mSM_TRAINING_ENV={\"additional_framework_parameters\":{},\"channel_input_dirs\":{\"test\":\"/opt/ml/input/data/test\",\"train\":\"/opt/ml/input/data/train\"},\"current_host\":\"algo-1\",\"framework_module\":\"sagemaker_tensorflow_container.training:main\",\"hosts\":[\"algo-1\"],\"hyperparameters\":{\"batch_size\":128,\"epochs\":1,\"model_dir\":\"/opt/ml/model\",\"tornasole-save-interval\":1,\"tornasole_outdir\":\"s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\"},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"test\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"train\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"is_master\":true,\"job_name\":\"tf-keras-sentiment-2019-07-16-00-36-04-131\",\"log_level\":20,\"master_hostname\":\"algo-1\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-us-east-1-072677473360/tf-keras-sentiment-2019-07-16-00-36-04-131/source/sourcedir.tar.gz\",\"module_name\":\"sentiment_keras\",\"network_interface_name\":\"eth0\",\"num_cpus\":8,\"num_gpus\":1,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\"],\"network_interface_name\":\"eth0\"},\"user_entry_point\":\"sentiment_keras.py\"}\u001b[0m\n",
- "\u001b[31mSM_USER_ARGS=[\"--batch_size\",\"128\",\"--epochs\",\"1\",\"--model_dir\",\"/opt/ml/model\",\"--tornasole-save-interval\",\"1\",\"--tornasole_outdir\",\"s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\"]\u001b[0m\n",
- "\u001b[31mSM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\u001b[0m\n",
- "\u001b[31mSM_CHANNEL_TEST=/opt/ml/input/data/test\u001b[0m\n",
- "\u001b[31mSM_CHANNEL_TRAIN=/opt/ml/input/data/train\u001b[0m\n",
- "\u001b[31mSM_HP_BATCH_SIZE=128\u001b[0m\n",
- "\u001b[31mSM_HP_TORNASOLE_OUTDIR=s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\u001b[0m\n",
- "\u001b[31mSM_HP_MODEL_DIR=/opt/ml/model\u001b[0m\n",
- "\u001b[31mSM_HP_EPOCHS=1\u001b[0m\n",
- "\u001b[31mSM_HP_TORNASOLE-SAVE-INTERVAL=1\u001b[0m\n",
- "\u001b[31mPYTHONPATH=/opt/ml/code:/usr/local/bin:/usr/lib/python36.zip:/usr/lib/python3.6:/usr/lib/python3.6/lib-dynload:/usr/local/lib/python3.6/dist-packages:/usr/lib/python3/dist-packages\n",
- "\u001b[0m\n",
- "\u001b[31mInvoking script with the following command:\n",
- "\u001b[0m\n",
- "\u001b[31m/usr/local/bin/python sentiment_keras.py --batch_size 128 --epochs 1 --model_dir /opt/ml/model --tornasole-save-interval 1 --tornasole_outdir s3://sagemaker-us-east-1-072677473360/tornasole-parameters/\n",
- "\n",
- "\u001b[0m\n",
- "\u001b[31mUsing TensorFlow backend.\u001b[0m\n",
- "\u001b[31mx train (25000, 400) y train (25000,)\u001b[0m\n",
- "\u001b[31m[[ 0 0 0 ... 19 178 32]\n",
- " [ 0 0 0 ... 16 145 95]\n",
- " [ 0 0 0 ... 7 129 113]\n",
- " ...\n",
- " [595 13 258 ... 72 33 32]\n",
- " [ 0 0 0 ... 28 126 110]\n",
- " [ 0 0 0 ... 7 43 50]]\u001b[0m\n",
- "\u001b[31m[1 0 0 1 0 0 1 0 1 0]\u001b[0m\n",
- "\u001b[31mx test (25000, 400) y test (25000,)\u001b[0m\n",
- "\u001b[31mWARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\u001b[0m\n",
- "\u001b[31mInstructions for updating:\u001b[0m\n",
- "\u001b[31mColocations handled automatically by placer.\u001b[0m\n",
- "\u001b[31mWARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\u001b[0m\n",
- "\u001b[31mInstructions for updating:\u001b[0m\n",
- "\u001b[31mColocations handled automatically by placer.\u001b[0m\n",
- "\u001b[31mWARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\u001b[0m\n",
- "\u001b[31mInstructions for updating:\u001b[0m\n",
- "\u001b[31mPlease use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\u001b[0m\n",
- "\u001b[31mWARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\u001b[0m\n",
- "\u001b[31mInstructions for updating:\u001b[0m\n",
- "\u001b[31mPlease use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\u001b[0m\n",
- "\u001b[31mWARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\u001b[0m\n",
- "\u001b[31mInstructions for updating:\u001b[0m\n",
- "\u001b[31mUse tf.cast instead.\u001b[0m\n",
- "\u001b[31mWARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\u001b[0m\n",
- "\u001b[31mInstructions for updating:\u001b[0m\n",
- "\u001b[31mUse tf.cast instead.\u001b[0m\n",
- "\u001b[31mWARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\u001b[0m\n",
- "\u001b[31mInstructions for updating:\u001b[0m\n",
- "\u001b[31mDeprecated in favor of operator or tf.math.divide.\u001b[0m\n",
- "\u001b[31mWARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\u001b[0m\n",
- "\u001b[31mInstructions for updating:\u001b[0m\n",
- "\u001b[31mDeprecated in favor of operator or tf.math.divide.\u001b[0m\n",
- "\u001b[31mTrain on 25000 samples, validate on 25000 samples\u001b[0m\n",
- "\u001b[31mEpoch 1/1\u001b[0m\n",
- "\u001b[31m 128/25000 [..............................] - ETA: 5:29 - loss: 0.6979 - acc: 0.4531\u001b[0m\n",
- "\u001b[31m 256/25000 [..............................] - ETA: 5:21 - loss: 0.6944 - acc: 0.5000\n",
- " 384/25000 [..............................] - ETA: 4:22 - loss: 0.7009 - acc: 0.4922\u001b[0m\n",
- "\u001b[31m 512/25000 [..............................] - ETA: 3:52 - loss: 0.7005 - acc: 0.4922\u001b[0m\n",
- "\u001b[31m 640/25000 [..............................] - ETA: 3:34 - loss: 0.6990 - acc: 0.4875\u001b[0m\n",
- "\u001b[31m 768/25000 [..............................] - ETA: 3:22 - loss: 0.6979 - acc: 0.4935\u001b[0m\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\u001b[31m 896/25000 [>.............................] - ETA: 3:13 - loss: 0.6988 - acc: 0.4922\n",
- " 1024/25000 [>.............................] - ETA: 3:06 - loss: 0.6980 - acc: 0.4961\u001b[0m\n",
- "\u001b[31m 1152/25000 [>.............................] - ETA: 3:00 - loss: 0.6998 - acc: 0.4931\u001b[0m\n",
- "\u001b[31m 1280/25000 [>.............................] - ETA: 2:56 - loss: 0.7008 - acc: 0.4883\n",
- " 1408/25000 [>.............................] - ETA: 3:06 - loss: 0.7003 - acc: 0.4879\u001b[0m\n",
- "\u001b[31m 1536/25000 [>.............................] - ETA: 3:01 - loss: 0.6993 - acc: 0.4961\u001b[0m\n",
- "\u001b[31m 1664/25000 [>.............................] - ETA: 2:57 - loss: 0.6984 - acc: 0.5072\u001b[0m\n",
- "\u001b[31m 1792/25000 [=>............................] - ETA: 2:53 - loss: 0.6981 - acc: 0.5056\u001b[0m\n",
- "\u001b[31m 1920/25000 [=>............................] - ETA: 2:50 - loss: 0.6969 - acc: 0.5120\n",
- " 2048/25000 [=>............................] - ETA: 2:48 - loss: 0.6969 - acc: 0.5112\u001b[0m\n",
- "\u001b[31m 2176/25000 [=>............................] - ETA: 2:45 - loss: 0.6969 - acc: 0.5078\u001b[0m\n",
- "\u001b[31m 2304/25000 [=>............................] - ETA: 2:43 - loss: 0.6966 - acc: 0.5074\u001b[0m\n",
- "\u001b[31m 2432/25000 [=>............................] - ETA: 2:40 - loss: 0.6965 - acc: 0.5053\u001b[0m\n",
- "\u001b[31m 2560/25000 [==>...........................] - ETA: 2:45 - loss: 0.6960 - acc: 0.5059\u001b[0m\n",
- "\u001b[31m 2688/25000 [==>...........................] - ETA: 2:43 - loss: 0.6956 - acc: 0.5089\u001b[0m\n",
- "\u001b[31m 2816/25000 [==>...........................] - ETA: 2:40 - loss: 0.6950 - acc: 0.5142\n",
- " 2944/25000 [==>...........................] - ETA: 2:38 - loss: 0.6947 - acc: 0.5166\u001b[0m\n",
- "\u001b[31m 3072/25000 [==>...........................] - ETA: 2:36 - loss: 0.6940 - acc: 0.5208\u001b[0m\n",
- "\u001b[31m 3200/25000 [==>...........................] - ETA: 2:34 - loss: 0.6936 - acc: 0.5209\u001b[0m\n",
- "\u001b[31m 3328/25000 [==>...........................] - ETA: 2:33 - loss: 0.6932 - acc: 0.5216\n",
- " 3456/25000 [===>..........................] - ETA: 2:31 - loss: 0.6930 - acc: 0.5214\u001b[0m\n",
- "\u001b[31m 3584/25000 [===>..........................] - ETA: 2:29 - loss: 0.6928 - acc: 0.5209\u001b[0m\n",
- "\u001b[31m 3712/25000 [===>..........................] - ETA: 2:33 - loss: 0.6923 - acc: 0.5229\u001b[0m\n",
- "\u001b[31m 3840/25000 [===>..........................] - ETA: 2:31 - loss: 0.6915 - acc: 0.5289\n",
- " 3968/25000 [===>..........................] - ETA: 2:30 - loss: 0.6906 - acc: 0.5358\u001b[0m\n",
- "\u001b[31m 4096/25000 [===>..........................] - ETA: 2:28 - loss: 0.6901 - acc: 0.5386\u001b[0m\n",
- "\u001b[31m 4224/25000 [====>.........................] - ETA: 2:27 - loss: 0.6895 - acc: 0.5400\u001b[0m\n",
- "\u001b[31m 4352/25000 [====>.........................] - ETA: 2:25 - loss: 0.6892 - acc: 0.5402\u001b[0m\n",
- "\u001b[31m 4480/25000 [====>.........................] - ETA: 2:24 - loss: 0.6883 - acc: 0.5442\n",
- " 4608/25000 [====>.........................] - ETA: 2:22 - loss: 0.6875 - acc: 0.5493\u001b[0m\n",
- "\u001b[31m 4736/25000 [====>.........................] - ETA: 2:21 - loss: 0.6866 - acc: 0.5536\u001b[0m\n",
- "\u001b[31m 4864/25000 [====>.........................] - ETA: 2:22 - loss: 0.6855 - acc: 0.5588\n",
- " 4992/25000 [====>.........................] - ETA: 2:21 - loss: 0.6845 - acc: 0.5623\u001b[0m\n",
- "\u001b[31m 5120/25000 [=====>........................] - ETA: 2:19 - loss: 0.6836 - acc: 0.5652\u001b[0m\n",
- "\u001b[31m 5248/25000 [=====>........................] - ETA: 2:18 - loss: 0.6827 - acc: 0.5665\u001b[0m\n",
- "\u001b[31m 5376/25000 [=====>........................] - ETA: 2:17 - loss: 0.6825 - acc: 0.5664\n",
- " 5504/25000 [=====>........................] - ETA: 2:15 - loss: 0.6816 - acc: 0.5672\u001b[0m\n",
- "\u001b[31m 5632/25000 [=====>........................] - ETA: 2:14 - loss: 0.6806 - acc: 0.5701\u001b[0m\n",
- "\u001b[31m 5760/25000 [=====>........................] - ETA: 2:13 - loss: 0.6795 - acc: 0.5724\u001b[0m\n",
- "\u001b[31m 5888/25000 [======>.......................] - ETA: 2:11 - loss: 0.6785 - acc: 0.5727\u001b[0m\n",
- "\u001b[31m 6016/25000 [======>.......................] - ETA: 2:12 - loss: 0.6762 - acc: 0.5751\u001b[0m\n",
- "\u001b[31m 6144/25000 [======>.......................] - ETA: 2:11 - loss: 0.6753 - acc: 0.5778\u001b[0m\n",
- "\u001b[31m 6272/25000 [======>.......................] - ETA: 2:10 - loss: 0.6736 - acc: 0.5818\u001b[0m\n",
- "\u001b[31m 6400/25000 [======>.......................] - ETA: 2:09 - loss: 0.6719 - acc: 0.5850\n",
- " 6528/25000 [======>.......................] - ETA: 2:07 - loss: 0.6705 - acc: 0.5879\u001b[0m\n",
- "\u001b[31m 6656/25000 [======>.......................] - ETA: 2:06 - loss: 0.6688 - acc: 0.5898\u001b[0m\n",
- "\u001b[31m 6784/25000 [=======>......................] - ETA: 2:05 - loss: 0.6678 - acc: 0.5909\u001b[0m\n",
- "\u001b[31m 6912/25000 [=======>......................] - ETA: 2:04 - loss: 0.6661 - acc: 0.5926\u001b[0m\n",
- "\u001b[31m 7040/25000 [=======>......................] - ETA: 2:03 - loss: 0.6641 - acc: 0.5938\u001b[0m\n",
- "\u001b[31m 7168/25000 [=======>......................] - ETA: 2:03 - loss: 0.6620 - acc: 0.5958\u001b[0m\n",
- "\u001b[31m 7296/25000 [=======>......................] - ETA: 2:02 - loss: 0.6612 - acc: 0.5973\n",
- " 7424/25000 [=======>......................] - ETA: 2:01 - loss: 0.6589 - acc: 0.6001\u001b[0m\n",
- "\u001b[31m 7552/25000 [========>.....................] - ETA: 2:00 - loss: 0.6573 - acc: 0.6029\u001b[0m\n",
- "\u001b[31m 7680/25000 [========>.....................] - ETA: 1:59 - loss: 0.6552 - acc: 0.6065\u001b[0m\n",
- "\u001b[31m 7808/25000 [========>.....................] - ETA: 1:58 - loss: 0.6532 - acc: 0.6092\n",
- " 7936/25000 [========>.....................] - ETA: 1:57 - loss: 0.6502 - acc: 0.6129\u001b[0m\n",
- "\u001b[31m 8064/25000 [========>.....................] - ETA: 1:56 - loss: 0.6481 - acc: 0.6152\u001b[0m\n",
- "\u001b[31m 8192/25000 [========>.....................] - ETA: 1:55 - loss: 0.6467 - acc: 0.6161\u001b[0m\n",
- "\u001b[31m 8320/25000 [========>.....................] - ETA: 1:55 - loss: 0.6445 - acc: 0.6178\n",
- " 8448/25000 [=========>....................] - ETA: 1:54 - loss: 0.6423 - acc: 0.6197\u001b[0m\n",
- "\u001b[31m 8576/25000 [=========>....................] - ETA: 1:53 - loss: 0.6404 - acc: 0.6212\u001b[0m\n",
- "\u001b[31m 8704/25000 [=========>....................] - ETA: 1:52 - loss: 0.6386 - acc: 0.6229\u001b[0m\n",
- "\u001b[31m 8832/25000 [=========>....................] - ETA: 1:51 - loss: 0.6363 - acc: 0.6248\n",
- " 8960/25000 [=========>....................] - ETA: 1:50 - loss: 0.6334 - acc: 0.6282\u001b[0m\n",
- "\u001b[31m 9088/25000 [=========>....................] - ETA: 1:49 - loss: 0.6303 - acc: 0.6316\u001b[0m\n",
- "\u001b[31m 9216/25000 [==========>...................] - ETA: 1:47 - loss: 0.6284 - acc: 0.6331\u001b[0m\n",
- "\u001b[31m 9344/25000 [==========>...................] - ETA: 1:46 - loss: 0.6252 - acc: 0.6360\n",
- " 9472/25000 [==========>...................] - ETA: 1:47 - loss: 0.6222 - acc: 0.6388\u001b[0m\n",
- "\u001b[31m 9600/25000 [==========>...................] - ETA: 1:46 - loss: 0.6206 - acc: 0.6408\u001b[0m\n",
- "\u001b[31m 9728/25000 [==========>...................] - ETA: 1:45 - loss: 0.6181 - acc: 0.6430\u001b[0m\n",
- "\u001b[31m 9856/25000 [==========>...................] - ETA: 1:44 - loss: 0.6151 - acc: 0.6452\n",
- " 9984/25000 [==========>...................] - ETA: 1:42 - loss: 0.6125 - acc: 0.6473\u001b[0m\n",
- "\u001b[31m10112/25000 [===========>..................] - ETA: 1:41 - loss: 0.6102 - acc: 0.6494\u001b[0m\n",
- "\u001b[31m10240/25000 [===========>..................] - ETA: 1:40 - loss: 0.6082 - acc: 0.6511\u001b[0m\n",
- "\u001b[31m10368/25000 [===========>..................] - ETA: 1:39 - loss: 0.6056 - acc: 0.6529\u001b[0m\n",
- "\u001b[31m10496/25000 [===========>..................] - ETA: 1:38 - loss: 0.6038 - acc: 0.6548\u001b[0m\n",
- "\u001b[31m10624/25000 [===========>..................] - ETA: 1:47 - loss: 0.6021 - acc: 0.6558\u001b[0m\n",
- "\u001b[31m10752/25000 [===========>..................] - ETA: 1:46 - loss: 0.6010 - acc: 0.6566\u001b[0m\n",
- "\u001b[31m10880/25000 [============>.................] - ETA: 1:44 - loss: 0.5986 - acc: 0.6590\u001b[0m\n",
- "\u001b[31m11008/25000 [============>.................] - ETA: 1:43 - loss: 0.5961 - acc: 0.6611\u001b[0m\n",
- "\u001b[31m11136/25000 [============>.................] - ETA: 1:42 - loss: 0.5938 - acc: 0.6629\u001b[0m\n",
- "\u001b[31m11264/25000 [============>.................] - ETA: 1:41 - loss: 0.5906 - acc: 0.6652\u001b[0m\n",
- "\u001b[31m11392/25000 [============>.................] - ETA: 1:40 - loss: 0.5890 - acc: 0.6664\u001b[0m\n",
- "\u001b[31m11520/25000 [============>.................] - ETA: 1:39 - loss: 0.5871 - acc: 0.6682\u001b[0m\n",
- "\u001b[31m11648/25000 [============>.................] - ETA: 1:38 - loss: 0.5853 - acc: 0.6696\u001b[0m\n",
- "\u001b[31m11776/25000 [=============>................] - ETA: 1:37 - loss: 0.5827 - acc: 0.6713\u001b[0m\n",
- "\u001b[31m11904/25000 [=============>................] - ETA: 1:36 - loss: 0.5808 - acc: 0.6728\u001b[0m\n",
- "\u001b[31m12032/25000 [=============>................] - ETA: 1:35 - loss: 0.5778 - acc: 0.6750\u001b[0m\n",
- "\u001b[31m12160/25000 [=============>................] - ETA: 1:34 - loss: 0.5753 - acc: 0.6766\u001b[0m\n",
- "\u001b[31m12288/25000 [=============>................] - ETA: 1:33 - loss: 0.5731 - acc: 0.6785\u001b[0m\n",
- "\u001b[31m12416/25000 [=============>................] - ETA: 1:32 - loss: 0.5699 - acc: 0.6807\u001b[0m\n",
- "\u001b[31m12544/25000 [==============>...............] - ETA: 1:31 - loss: 0.5679 - acc: 0.6818\u001b[0m\n",
- "\u001b[31m12672/25000 [==============>...............] - ETA: 1:29 - loss: 0.5656 - acc: 0.6836\u001b[0m\n",
- "\u001b[31m12800/25000 [==============>...............] - ETA: 1:28 - loss: 0.5631 - acc: 0.6855\u001b[0m\n",
- "\u001b[31m12928/25000 [==============>...............] - ETA: 1:28 - loss: 0.5616 - acc: 0.6863\u001b[0m\n",
- "\u001b[31m13056/25000 [==============>...............] - ETA: 1:27 - loss: 0.5592 - acc: 0.6882\u001b[0m\n",
- "\u001b[31m13184/25000 [==============>...............] - ETA: 1:26 - loss: 0.5569 - acc: 0.6899\u001b[0m\n",
- "\u001b[31m13312/25000 [==============>...............] - ETA: 1:25 - loss: 0.5541 - acc: 0.6919\u001b[0m\n",
- "\u001b[31m13440/25000 [===============>..............] - ETA: 1:24 - loss: 0.5518 - acc: 0.6936\u001b[0m\n",
- "\u001b[31m13568/25000 [===============>..............] - ETA: 1:23 - loss: 0.5508 - acc: 0.6944\u001b[0m\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\u001b[31m13696/25000 [===============>..............] - ETA: 1:22 - loss: 0.5482 - acc: 0.6962\u001b[0m\n",
- "\u001b[31m13824/25000 [===============>..............] - ETA: 1:21 - loss: 0.5464 - acc: 0.6970\u001b[0m\n",
- "\u001b[31m13952/25000 [===============>..............] - ETA: 1:20 - loss: 0.5452 - acc: 0.6978\u001b[0m\n",
- "\u001b[31m14080/25000 [===============>..............] - ETA: 1:20 - loss: 0.5433 - acc: 0.6993\u001b[0m\n",
- "\u001b[31m14208/25000 [================>.............] - ETA: 1:18 - loss: 0.5414 - acc: 0.7010\u001b[0m\n",
- "\u001b[31m14336/25000 [================>.............] - ETA: 1:17 - loss: 0.5393 - acc: 0.7026\u001b[0m\n",
- "\u001b[31m14464/25000 [================>.............] - ETA: 1:16 - loss: 0.5380 - acc: 0.7038\u001b[0m\n",
- "\u001b[31m14592/25000 [================>.............] - ETA: 1:15 - loss: 0.5368 - acc: 0.7047\u001b[0m\n",
- "\u001b[31m14720/25000 [================>.............] - ETA: 1:14 - loss: 0.5347 - acc: 0.7064\u001b[0m\n",
- "\u001b[31m14848/25000 [================>.............] - ETA: 1:13 - loss: 0.5331 - acc: 0.7076\u001b[0m\n",
- "\u001b[31m14976/25000 [================>.............] - ETA: 1:12 - loss: 0.5311 - acc: 0.7091\u001b[0m\n",
- "\u001b[31m15104/25000 [=================>............] - ETA: 1:11 - loss: 0.5295 - acc: 0.7103\u001b[0m\n",
- "\u001b[31m15232/25000 [=================>............] - ETA: 1:11 - loss: 0.5276 - acc: 0.7119\u001b[0m\n",
- "\u001b[31m15360/25000 [=================>............] - ETA: 1:10 - loss: 0.5262 - acc: 0.7130\u001b[0m\n",
- "\u001b[31m15488/25000 [=================>............] - ETA: 1:09 - loss: 0.5244 - acc: 0.7143\u001b[0m\n",
- "\u001b[31m15616/25000 [=================>............] - ETA: 1:08 - loss: 0.5229 - acc: 0.7154\u001b[0m\n",
- "\u001b[31m15744/25000 [=================>............] - ETA: 1:07 - loss: 0.5214 - acc: 0.7167\u001b[0m\n",
- "\u001b[31m15872/25000 [==================>...........] - ETA: 1:06 - loss: 0.5203 - acc: 0.7176\u001b[0m\n",
- "\u001b[31m16000/25000 [==================>...........] - ETA: 1:05 - loss: 0.5199 - acc: 0.7180\u001b[0m\n",
- "\u001b[31m16128/25000 [==================>...........] - ETA: 1:04 - loss: 0.5182 - acc: 0.7193\u001b[0m\n",
- "\u001b[31m16256/25000 [==================>...........] - ETA: 1:03 - loss: 0.5165 - acc: 0.7204\u001b[0m\n",
- "\u001b[31m16384/25000 [==================>...........] - ETA: 1:02 - loss: 0.5148 - acc: 0.7215\u001b[0m\n",
- "\u001b[31m16512/25000 [==================>...........] - ETA: 1:01 - loss: 0.5137 - acc: 0.7224\u001b[0m\n",
- "\u001b[31m16640/25000 [==================>...........] - ETA: 1:00 - loss: 0.5131 - acc: 0.7231\u001b[0m\n",
- "\u001b[31m16768/25000 [===================>..........] - ETA: 59s - loss: 0.5115 - acc: 0.7245 \u001b[0m\n",
- "\u001b[31m16896/25000 [===================>..........] - ETA: 58s - loss: 0.5100 - acc: 0.7253\u001b[0m\n",
- "\u001b[31m17024/25000 [===================>..........] - ETA: 57s - loss: 0.5084 - acc: 0.7263\u001b[0m\n",
- "\u001b[31m17152/25000 [===================>..........] - ETA: 56s - loss: 0.5066 - acc: 0.7275\u001b[0m\n",
- "\u001b[31m17280/25000 [===================>..........] - ETA: 55s - loss: 0.5051 - acc: 0.7284\u001b[0m\n",
- "\u001b[31m17408/25000 [===================>..........] - ETA: 54s - loss: 0.5039 - acc: 0.7293\u001b[0m\n",
- "\u001b[31m17536/25000 [====================>.........] - ETA: 54s - loss: 0.5020 - acc: 0.7304\u001b[0m\n",
- "\u001b[31m17664/25000 [====================>.........] - ETA: 53s - loss: 0.5007 - acc: 0.7312\u001b[0m\n",
- "\u001b[31m17792/25000 [====================>.........] - ETA: 52s - loss: 0.4996 - acc: 0.7321\u001b[0m\n",
- "\u001b[31m17920/25000 [====================>.........] - ETA: 51s - loss: 0.4980 - acc: 0.7333\u001b[0m\n",
- "\u001b[31m18048/25000 [====================>.........] - ETA: 50s - loss: 0.4969 - acc: 0.7340\u001b[0m\n",
- "\u001b[31m18176/25000 [====================>.........] - ETA: 49s - loss: 0.4952 - acc: 0.7351\u001b[0m\n",
- "\u001b[31m18304/25000 [====================>.........] - ETA: 48s - loss: 0.4939 - acc: 0.7360\u001b[0m\n",
- "\u001b[31m18432/25000 [=====================>........] - ETA: 47s - loss: 0.4935 - acc: 0.7368\u001b[0m\n",
- "\u001b[31m18560/25000 [=====================>........] - ETA: 46s - loss: 0.4916 - acc: 0.7379\u001b[0m\n",
- "\u001b[31m18688/25000 [=====================>........] - ETA: 45s - loss: 0.4902 - acc: 0.7388\u001b[0m\n",
- "\u001b[31m18816/25000 [=====================>........] - ETA: 44s - loss: 0.4889 - acc: 0.7396\u001b[0m\n",
- "\u001b[31m18944/25000 [=====================>........] - ETA: 43s - loss: 0.4881 - acc: 0.7401\u001b[0m\n",
- "\u001b[31m19072/25000 [=====================>........] - ETA: 42s - loss: 0.4864 - acc: 0.7412\u001b[0m\n",
- "\u001b[31m19200/25000 [======================>.......] - ETA: 41s - loss: 0.4855 - acc: 0.7419\u001b[0m\n",
- "\u001b[31m19328/25000 [======================>.......] - ETA: 40s - loss: 0.4839 - acc: 0.7429\u001b[0m\n",
- "\u001b[31m19456/25000 [======================>.......] - ETA: 39s - loss: 0.4825 - acc: 0.7438\u001b[0m\n",
- "\u001b[31m19584/25000 [======================>.......] - ETA: 38s - loss: 0.4815 - acc: 0.7446\u001b[0m\n",
- "\u001b[31m19712/25000 [======================>.......] - ETA: 37s - loss: 0.4803 - acc: 0.7455\u001b[0m\n",
- "\u001b[31m19840/25000 [======================>.......] - ETA: 36s - loss: 0.4792 - acc: 0.7463\u001b[0m\n",
- "\u001b[31m19968/25000 [======================>.......] - ETA: 36s - loss: 0.4777 - acc: 0.7475\u001b[0m\n",
- "\u001b[31m20096/25000 [=======================>......] - ETA: 35s - loss: 0.4773 - acc: 0.7479\u001b[0m\n",
- "\u001b[31m20224/25000 [=======================>......] - ETA: 34s - loss: 0.4757 - acc: 0.7490\u001b[0m\n",
- "\u001b[31m20352/25000 [=======================>......] - ETA: 33s - loss: 0.4742 - acc: 0.7500\u001b[0m\n",
- "\u001b[31m20480/25000 [=======================>......] - ETA: 32s - loss: 0.4730 - acc: 0.7506\u001b[0m\n",
- "\u001b[31m20608/25000 [=======================>......] - ETA: 31s - loss: 0.4729 - acc: 0.7511\u001b[0m\n",
- "\u001b[31m20736/25000 [=======================>......] - ETA: 30s - loss: 0.4719 - acc: 0.7517\u001b[0m\n",
- "\u001b[31m20864/25000 [========================>.....] - ETA: 29s - loss: 0.4710 - acc: 0.7523\u001b[0m\n",
- "\u001b[31m20992/25000 [========================>.....] - ETA: 29s - loss: 0.4697 - acc: 0.7534\u001b[0m\n",
- "\u001b[31m21120/25000 [========================>.....] - ETA: 28s - loss: 0.4686 - acc: 0.7541\u001b[0m\n",
- "\u001b[31m21248/25000 [========================>.....] - ETA: 27s - loss: 0.4673 - acc: 0.7548\u001b[0m\n",
- "\u001b[31m21376/25000 [========================>.....] - ETA: 26s - loss: 0.4664 - acc: 0.7554\u001b[0m\n",
- "\u001b[31m21504/25000 [========================>.....] - ETA: 25s - loss: 0.4654 - acc: 0.7560\u001b[0m\n",
- "\u001b[31m21632/25000 [========================>.....] - ETA: 24s - loss: 0.4644 - acc: 0.7571\u001b[0m\n",
- "\u001b[31m21760/25000 [=========================>....] - ETA: 23s - loss: 0.4632 - acc: 0.7579\u001b[0m\n",
- "\u001b[31m21888/25000 [=========================>....] - ETA: 22s - loss: 0.4619 - acc: 0.7589\u001b[0m\n",
- "\u001b[31m22016/25000 [=========================>....] - ETA: 21s - loss: 0.4608 - acc: 0.7597\u001b[0m\n",
- "\u001b[31m22144/25000 [=========================>....] - ETA: 21s - loss: 0.4598 - acc: 0.7603\u001b[0m\n",
- "\u001b[31m22272/25000 [=========================>....] - ETA: 20s - loss: 0.4589 - acc: 0.7608\u001b[0m\n",
- "\u001b[31m22400/25000 [=========================>....] - ETA: 19s - loss: 0.4581 - acc: 0.7615\u001b[0m\n",
- "\u001b[31m22528/25000 [==========================>...] - ETA: 18s - loss: 0.4567 - acc: 0.7625\u001b[0m\n",
- "\u001b[31m22656/25000 [==========================>...] - ETA: 17s - loss: 0.4562 - acc: 0.7629\u001b[0m\n",
- "\u001b[31m22784/25000 [==========================>...] - ETA: 16s - loss: 0.4553 - acc: 0.7636\u001b[0m\n",
- "\u001b[31m22912/25000 [==========================>...] - ETA: 15s - loss: 0.4540 - acc: 0.7645\u001b[0m\n",
- "\u001b[31m23040/25000 [==========================>...] - ETA: 14s - loss: 0.4530 - acc: 0.7653\u001b[0m\n",
- "\u001b[31m23168/25000 [==========================>...] - ETA: 13s - loss: 0.4524 - acc: 0.7659\u001b[0m\n",
- "\u001b[31m23296/25000 [==========================>...] - ETA: 12s - loss: 0.4512 - acc: 0.7667\u001b[0m\n",
- "\u001b[31m23424/25000 [===========================>..] - ETA: 11s - loss: 0.4500 - acc: 0.7672\u001b[0m\n",
- "\u001b[31m23552/25000 [===========================>..] - ETA: 10s - loss: 0.4489 - acc: 0.7678\u001b[0m\n",
- "\u001b[31m23680/25000 [===========================>..] - ETA: 9s - loss: 0.4477 - acc: 0.7686 \u001b[0m\n",
- "\u001b[31m23808/25000 [===========================>..] - ETA: 8s - loss: 0.4464 - acc: 0.7695\u001b[0m\n",
- "\u001b[31m23936/25000 [===========================>..] - ETA: 7s - loss: 0.4454 - acc: 0.7703\u001b[0m\n",
- "\u001b[31m24064/25000 [===========================>..] - ETA: 6s - loss: 0.4448 - acc: 0.7707\u001b[0m\n",
- "\u001b[31m24192/25000 [============================>.] - ETA: 5s - loss: 0.4441 - acc: 0.7710\u001b[0m\n",
- "\u001b[31m24320/25000 [============================>.] - ETA: 5s - loss: 0.4437 - acc: 0.7714\u001b[0m\n",
- "\u001b[31m24448/25000 [============================>.] - ETA: 4s - loss: 0.4430 - acc: 0.7719\u001b[0m\n",
- "\u001b[31m24576/25000 [============================>.] - ETA: 3s - loss: 0.4420 - acc: 0.7726\u001b[0m\n",
- "\u001b[31m24704/25000 [============================>.] - ETA: 2s - loss: 0.4411 - acc: 0.7733\u001b[0m\n",
- "\u001b[31m24832/25000 [============================>.] - ETA: 1s - loss: 0.4403 - acc: 0.7739\u001b[0m\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\u001b[31m24960/25000 [============================>.] - ETA: 0s - loss: 0.4393 - acc: 0.7745\u001b[0m\n",
- "\u001b[31m25000/25000 [==============================] - 233s 9ms/step - loss: 0.4387 - acc: 0.7748 - val_loss: 0.2636 - val_acc: 0.8898\u001b[0m\n",
- "\u001b[31m2019-07-16 00:44:17,528 sagemaker_tensorflow_container.training WARNING Your model will NOT be servable with SageMaker TensorFlow Serving container.The model artifact was not saved in the TensorFlow SavedModel directory structure:\u001b[0m\n",
- "\u001b[31mhttps://www.tensorflow.org/guide/saved_model#structure_of_a_savedmodel_directory\u001b[0m\n",
- "\u001b[31m2019-07-16 00:44:17,528 sagemaker-containers INFO Reporting training SUCCESS\u001b[0m\n",
- "\n",
- "2019-07-16 00:45:32 Uploading - Uploading generated training model\n",
- "2019-07-16 00:45:32 Completed - Training job completed\n",
- "Billable seconds: 437\n"
- ]
- }
- ],
- "source": [
- "estimator.fit(inputs)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Batch Prediction\n",
- "\n",
- "\n",
- "If our use case requires individual predictions in near real-time, SageMaker hosted endpoints can be created. Hosted endpoints also can be used for pseudo-batch prediction, but the process is more involved than simply using SageMaker's Batch Transform feature, which is designed for large-scale, asynchronous batch inference.\n",
- "\n",
- "To use Batch Transform, first we must upload to Amazon S3 some test data in CSV format to be transformed."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "csvtestdata_s3_prefix = '{}/data/csv-test'.format(s3_prefix)\n",
- "csvtest_s3 = sagemaker.Session().upload_data(path='./data/csv-test/', key_prefix=csvtestdata_s3_prefix)\n",
- "print(csvtest_s3)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "A Transformer object must be set up to describe the Batch Transform job, including the amount and type of inference hardware to be used. Then the actual transform job itself is started with a call to the `transform` method of the Transformer."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "transformer = estimator.transformer(instance_count=1, instance_type='ml.m5.xlarge')\n",
- "transformer.transform(csvtest_s3, content_type='text/csv')\n",
- "print('Waiting for transform job: ' + transformer.latest_transform_job.job_name)\n",
- "transformer.wait()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can now download the batch predictions from S3 to the local filesystem on the notebook instance; the predictions are contained in a file with a .out extension, and are embedded in JSON. Next we'll load the JSON and examine the predictions, which are confidence scores from 0.0 to 1.0 where numbers close to 1.0 indicate positive sentiment, while numbers close to 0.0 indicate negative sentiment."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import json\n",
- "\n",
- "batch_output = transformer.output_path\n",
- "!mkdir -p batch_data/output\n",
- "!aws s3 cp --recursive $batch_output/ batch_data/output/\n",
- "\n",
- "with open('batch_data/output/csv-test.csv.out', 'r') as f:\n",
- " jstr = json.load(f)\n",
- " results = [float('%.3f'%(item)) for sublist in jstr['predictions'] for item in sublist]\n",
- " print(results)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now let's look at the text of some actual reviews to see the predictions in action. First, we have to convert the integers representing the words back to the words themselves by using a reversed dictionary. Next we can decode the reviews, taking into account that the first 3 indices were reserved for \"padding\", \"start of sequence\", and \"unknown\", and removing a string of unknown tokens from the start of the review."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import re\n",
- "\n",
- "regex = re.compile(r'^[\\?\\s]+')\n",
- "\n",
- "word_index = imdb.get_word_index()\n",
- "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
- "first_decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in x_test[3]])\n",
- "regex.sub('', first_decoded_review)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Overall, this review looks fairly negative. Let's compare the actual label with the prediction:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def get_sentiment(score):\n",
- " return 'positive' if score > 0.5 else 'negative' \n",
- "\n",
- "print('Labeled sentiment for this review is {}, predicted sentiment is {}'.format(get_sentiment(y_test[3]), \n",
- " get_sentiment(results[3])))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Our negative sentiment prediction agrees with the label for this review. Let's now examine another review:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "second_decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in x_test[10]])\n",
- "regex.sub('', second_decoded_review)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "print('Labeled sentiment for this review is {}, predicted sentiment is {}'.format(get_sentiment(y_test[10]), \n",
- " get_sentiment(results[10])))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Again, the prediction agreed with the label for the test data. Note that there is no need to clean up any Batch Transform resources: after the transform job is complete, the cluster used to make inferences is torn down.\n",
- "\n",
- "Now that we've reviewed some sample predictions as a sanity check, we're finished. Of course, in a typical production situation, the data science project lifecycle is iterative, with repeated cycles of refining the model using a tool such as Amazon SageMaker's Automatic Model Tuning feature, and gathering more data. "
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/examples/tensorflow/notebooks/keras-sentiment/sentiment_keras.py b/examples/tensorflow/notebooks/keras-sentiment/sentiment_keras.py
deleted file mode 100644
index 61d9ce0a6..000000000
--- a/examples/tensorflow/notebooks/keras-sentiment/sentiment_keras.py
+++ /dev/null
@@ -1,106 +0,0 @@
-# Standard Library
-import argparse
-import os
-
-# Third Party
-import keras
-import numpy as np
-
-# First Party
-from smdebug import SaveConfig
-from smdebug.tensorflow.keras import SessionHook
-
-max_features = 20000
-maxlen = 400
-embedding_dims = 300
-filters = 250
-kernel_size = 3
-hidden_dims = 250
-
-
-def parse_args():
-
- parser = argparse.ArgumentParser()
-
- # hyperparameters sent by the client are passed as command-line arguments to the script
- parser.add_argument("--epochs", type=int, default=5)
- parser.add_argument("--batch_size", type=int, default=64)
-
- parser.add_argument("--tornasole_outdir", type=str, required=True)
- parser.add_argument("--tornasole_save_interval", type=int, default=10)
-
- # data directories
- parser.add_argument("--train", type=str, default=os.environ.get("SM_CHANNEL_TRAIN"))
- parser.add_argument("--test", type=str, default=os.environ.get("SM_CHANNEL_TEST"))
-
- # model directory: we will use the default set by SageMaker, /opt/ml/model
- parser.add_argument("--model_dir", type=str, default=os.environ.get("SM_MODEL_DIR"))
-
- return parser.parse_known_args()
-
-
-def get_train_data(train_dir):
-
- x_train = np.load(os.path.join(train_dir, "x_train.npy"))
- y_train = np.load(os.path.join(train_dir, "y_train.npy"))
- print("x train", x_train.shape, "y train", y_train.shape)
- print(x_train[:10])
- print(y_train[:10])
-
- return x_train, y_train
-
-
-def get_test_data(test_dir):
-
- x_test = np.load(os.path.join(test_dir, "x_test.npy"))
- y_test = np.load(os.path.join(test_dir, "y_test.npy"))
- print("x test", x_test.shape, "y test", y_test.shape)
-
- return x_test, y_test
-
-
-def get_model():
-
- embedding_layer = keras.layers.Embedding(max_features, embedding_dims, input_length=maxlen)
-
- sequence_input = keras.Input(shape=(maxlen,), dtype="int32")
- embedded_sequences = embedding_layer(sequence_input)
- x = keras.layers.Dropout(0.2)(embedded_sequences)
- x = keras.layers.Conv1D(filters, kernel_size, padding="valid", activation="relu", strides=1)(x)
- x = keras.layers.MaxPooling1D()(x)
- x = keras.layers.GlobalMaxPooling1D()(x)
- x = keras.layers.Dense(hidden_dims, activation="relu")(x)
- x = keras.layers.Dropout(0.2)(x)
- preds = keras.layers.Dense(1, activation="sigmoid")(x)
-
- return keras.Model(sequence_input, preds)
-
-
-if __name__ == "__main__":
-
- args, _ = parse_args()
-
- hook = SessionHook(
- out_dir=args.tornasole_outdir,
- save_config=SaveConfig(save_interval=args.tornasole_save_interval),
- )
-
- x_train, y_train = get_train_data(args.train)
- x_test, y_test = get_test_data(args.test)
-
- model = get_model()
-
- model.compile(
- loss="binary_crossentropy", optimizer="adam", metrics=["accuracy", "mean_squared_error"]
- )
-
- model.fit(
- x_train,
- y_train,
- batch_size=args.batch_size,
- epochs=args.epochs,
- validation_data=(x_test, y_test),
- callbacks=[hook],
- )
-
- model.save(os.path.join(args.model_dir, "sentiment_keras.h5"))
diff --git a/examples/tensorflow/sagemaker-notebooks/tensorflow.ipynb b/examples/tensorflow/sagemaker-notebooks/tensorflow.ipynb
deleted file mode 100644
index bda7f67d5..000000000
--- a/examples/tensorflow/sagemaker-notebooks/tensorflow.ipynb
+++ /dev/null
@@ -1,881 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Debugging SageMaker TensorFlow Training Jobs with Tornasole"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Overview\n",
- "Tornasole is a new capability of Amazon SageMaker that allows debugging machine learning training. \n",
- "It lets you go beyond just looking at scalars like losses and accuracies during training and gives \n",
- "you full visibility into all tensors 'flowing through the graph' during training. Tornasole helps you to monitor your training in near real time using rules and would provide you alerts, once it has detected inconsistency in training flow. \n",
- "\n",
- "Using Tornasole is a two step process: Saving tensors and Analysis. Let's look at each one of them closely.\n",
- "\n",
- "### Saving tensors\n",
- "Tensors define the state of the training job at any particular instant in its lifecycle. Tornasole exposes a library which allows you to capture these tensors and save them for analysis. Tornasole is highly customizable to save the tesnsors you want at different frequencies. Refer [DeveloperGuide_TensorFlow](../../DeveloperGuide_TF.md) for details on how to save the tensors you want to save.\n",
- "\n",
- "### Analysis\n",
- "\n",
- "Analysis of the tensors emitted is captured by the Tornasole concept called ***Rules***. On a very broad level, \n",
- "a rule is a python code used to detect certain conditions during training. Some of the conditions that a data scientist training a deep learning model may care about are monitoring for gradients getting too large or too small, detecting overfitting, and so on.\n",
- "Tornasole will come pre-packaged with certain rules. Users can write their own rules using the Tornasole APIs.\n",
- "You can also analyze raw tensor data outside of the Rules construct in say, a Sagemaker notebook, using Tornasole's full set of APIs. \n",
- "Please refer [DeveloperGuide_Rules](../../../../rules/DeveloperGuide_Rules.md) for more details about analysis.\n",
- "\n",
- "This example guides you through installation of the required components for emitting tensors in a \n",
- "SageMaker training job and applying a rule over the tensors to monitor the live status of the job. \n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup\n",
- "\n",
- "As a first step, we'll do the installation of required tools which will allow emission of tensors (saving tensors) and application of rules to analyze them. This is only for the purposes of this private beta. Once we do this, we will be ready to use smdebug."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "download: s3://tornasole-external-preview-use1/sdk/sagemaker-1.35.2.dev0.tar.gz to ../../../../../tornasole-preview-sdk/sagemaker-1.35.2.dev0.tar.gz\n",
- "download: s3://tornasole-external-preview-use1/sdk/sagemaker-tornasole-latest.tar.gz to ../../../../../tornasole-preview-sdk/sagemaker-tornasole-latest.tar.gz\n",
- "Installing requirements...\n",
- "\u001b[33mYou are using pip version 10.0.1, however version 19.2.3 is available.\n",
- "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
- "Installation completed!\n"
- ]
- }
- ],
- "source": [
- "! aws s3 sync s3://tornasole-external-preview-use1/sdk/ ~/SageMaker/tornasole-preview-sdk/\n",
- "! chmod +x ~/SageMaker/tornasole-preview-sdk/installer.sh && ~/SageMaker/tornasole-preview-sdk/installer.sh"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Training TensorFlow models in SageMaker with Tornasole\n",
- "\n",
- "We'll train a few TensorFlow models in this notebook with Tornasole enabled and monitor the training jobs with Tornasole Rules. This will be done using SageMaker TensorFlow 1.13.1 Container in Script Mode. \n",
- "\n",
- "Let us first train a simple example training script [simple.py](../scripts/simple.py) with Tornasole enabled in SageMaker using the SageMaker Estimator API, along with a ExplodingTensor Rule to monitor the training job in realtime. A Tornasole Rule is essentially python code which analyses tensors saved by tornasole and validates some condition. ExplodingTensor rule is a first party (1P) rule provided by smdebug. During training, Tornasole will capture tensors as specified in its configuration and ExplodingTensor Rule job will monitor whether any tensor exploded (i.e. value became not a number). The rule will emit a cloudwatch event if it finds an exploding tensor during training.\n",
- "\n",
- "### Enabling Tornasole in the script\n",
- "You can see in the script that we have made a couple of simple changes to enable smdebug. We created a SessionHook which we pass when creating a Monitored session, and use this monitored session for training. We passed save_all=True telling the hook to save all tensors in the graph. Note that Tornasole is highly configurable, you can choose exactly what to save. The changes are described in a bit more detail below after we train this example as well as in even more detail in our [Developer Guide for Tensorflow](../../DeveloperGuide_TF.md). \n",
- "\n",
- "We also wrapped the optimizer with Tornasole's optimizer and used this to minimize the loss. This helps Tornasole identify the gradients during training, and changes nothing in the loss minimization process.\n",
- "\n",
- "```python\n",
- "import smdebug.tensorflow as smd\n",
- "# Ask TORNASOLE to save all tensors. Note: SessionHook is highly configurable\n",
- "hook = smd.SessionHook(save_all=True) \n",
- "# Wrap the optimizer with Tornasole optimizer to identify gradients\n",
- "optimizer = hook.wrap_optimizer(optimizer) \n",
- "# pass the hook to hooks parameter of monitored session\n",
- "sess = tf.train.MonitoredSession(hooks=[hook])\n",
- "```\n",
- "\n",
- "### Docker Images with Tornasole\n",
- "We have built SageMaker TensorFlow containers with smdebug. You can use them from ECR from SageMaker. Here are the links to the images. Please use the image from the appropriate region in which you want your jobs to run."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "import sagemaker\n",
- "import boto3\n",
- "from sagemaker.tensorflow import TensorFlow\n",
- "\n",
- "# Below changes the region to be one where this notebook is running\n",
- "REGION = boto3.Session().region_name\n",
- "\n",
- "TAG='latest'\n",
- "\n",
- "gpu_docker_image_name = '072677473360.dkr.ecr.{}.amazonaws.com/tornasole-preprod-tf-1.13.1-gpu:{}'.format(REGION, TAG)\n",
- "cpu_docker_image_name = '072677473360.dkr.ecr.{}.amazonaws.com/tornasole-preprod-tf-1.13.1-cpu:{}'.format(REGION, TAG)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Start training a simple Session based example\n",
- "For the purposes of this demonstration let us bad hyperparameters so that NAN is produced during training for the rule to catch."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {
- "pycharm": {
- "name": "#%%\n"
- }
- },
- "outputs": [],
- "source": [
- "simple_entry_point_script = '../scripts/simple.py'\n",
- "simple_hyperparameters = { 'steps': 1000000, 'save_frequency': 50 }\n",
- "\n",
- "# copy dict\n",
- "bad_simple_hyperparameters = dict(simple_hyperparameters)\n",
- "\n",
- "# These parameters are consumed by simple.py to produce a exploding tensor problem\n",
- "bad_simple_hyperparameters.update({ 'lr': 100, 'scale': 100000000000})"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "sagemaker_simple_estimator = TensorFlow(\n",
- " role=sagemaker.get_execution_role(),\n",
- " base_job_name='tornasole-simple-demo',\n",
- " train_instance_count=1,\n",
- " train_instance_type='ml.m4.xlarge',\n",
- " image_name=cpu_docker_image_name,\n",
- " entry_point=simple_entry_point_script,\n",
- " framework_version='1.13.1',\n",
- " py_version='py3',\n",
- " script_mode=True,\n",
- " hyperparameters=bad_simple_hyperparameters,\n",
- " train_max_run=1800,\n",
- " \n",
- " # These are Tornasole specific parameters, \n",
- " # debug= True means rule specified in rules_specification \n",
- " # will run as rule job. \n",
- " # Below, we specify to run the first party rule ExplodingTensor\n",
- " # on a ml.c5.4xlarge instance\n",
- " debug=True,\n",
- " rules_specification=[\n",
- " {\n",
- " \"RuleName\": \"ExplodingTensor\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " }\n",
- " ])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "sagemaker_simple_estimator.fit(wait=False)\n",
- "# This is a fire and forget event. By setting wait=False, we just submit the job to run in the background.\n",
- "# In the background SageMaker will spin off 1 training job and 1 rule job for you.\n",
- "# Please follow this notebook to see status of the training job and the rule job"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Result\n",
- "As a result of the above command, SageMaker will spin off 1 training job and 1 rule job for you - the first one being the job which produces the tensors to be analyzed and the second one, which analyzes the tensors to check if there are any exploding tensor during training.\n",
- "\n",
- "### Describing the training job\n",
- "We can check the status of the training job by running the following command:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Below command will give the status of training job\n",
- "# Note: In the output of below command you will see DebugConfig parameter \n",
- "\n",
- "job_name = sagemaker_simple_estimator.latest_training_job.name\n",
- "\n",
- "client = sagemaker_simple_estimator.sagemaker_session.sagemaker_client\n",
- "\n",
- "description = client.describe_training_job(TrainingJobName=job_name)\n",
- "\n",
- "# uncomment next line to see full details of training job \n",
- "# description"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "'InProgress'"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "# The status of the training job can be seen below\n",
- "description['TrainingJobStatus']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Once your training job is started SageMaker will spin up a rule execution job to run the ExplodingTensor rule.\n",
- "\n",
- "### Tornasole specific parameters in the description\n",
- "**DebugConfig** parameter has details about Tornasole related configuration. The key parameters to look for below are\n",
- "\n",
- "*S3OutputPath* : This is the path where output tensors from tornasole is getting saved. \n",
- "*RuleConfig*' : This parameter tells about the rule config parameter that was passed when creating the trainning job. In this you should be able to see details of the rule that ran for training. \n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "{'DebugHookConfig': {'LocalPath': '/opt/ml/output/tensors',\n",
- " 'S3OutputPath': 's3://sagemaker-ca-central-1-072677473360/tensors-tornasole-simple-demo-2019-08-30-04-11-54-514',\n",
- " 'DebugHookSpecificationList': []},\n",
- " 'RuleConfig': {'RuleSpecificationList': [{'RuleName': 'ExplodingTensor',\n",
- " 'RuleEvaluatorImage': '453379255795.dkr.ecr.ca-central-1.amazonaws.com/script-rule-executor:latest',\n",
- " 'InstanceType': 'ml.c5.4xlarge',\n",
- " 'VolumeSizeInGB': 100}]}}"
- ]
- },
- "execution_count": 8,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "description['DebugConfig']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Check the status of the Rule Execution Job\n",
- "To get the rule execution job that SageMaker started for you, run the command below and it shows you the `RuleName`, `RuleStatus`, `FailureReason` if any, and `RuleExecutionJobArn`. If the tensors meets a rule evaluation condition, the rule execution job throws a client error with `FailureReason: RuleEvaluationConditionMet`. These details are also available as part of the response `description` above under: `description['RuleMonitoringStatuses']`\n",
- "\n",
- "\n",
- "The logs of the training job are available in the Cloudwatch Logstream `/aws/sagemaker/TrainingJobs` with `RuleExecutionJobArn`. \n",
- "\n",
- "You will see that once the rule execution job starts, that it identifies the exploding tensor situation in the training job, raises the `RuleEvaluationConditionMet` exception and ends the job. \n",
- "\n",
- "**Note that the next cell blocks till the rule execution job ends. Once it says RuleStatus is Started, and shows the `RuleExecutionJobArn`, you can look at the status of the rule being monitored. At that point, we can also look at the logs as shown in the next cell**"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Wait to get status for Rule Execution Jobs...\n",
- "=============================================\n",
- "RuleName: ExplodingTensor\n",
- "RuleStatus: NotStarted\n",
- "=============================================\n",
- "Wait to get status for Rule Execution Jobs...\n",
- "=============================================\n",
- "RuleName: ExplodingTensor\n",
- "RuleStatus: NotStarted\n",
- "=============================================\n",
- "Wait to get status for Rule Execution Jobs...\n",
- "=============================================\n",
- "RuleName: ExplodingTensor\n",
- "RuleStatus: NotStarted\n",
- "=============================================\n",
- "Wait to get status for Rule Execution Jobs...\n",
- "=============================================\n",
- "RuleName: ExplodingTensor\n",
- "RuleStatus: NotStarted\n",
- "=============================================\n",
- "Wait to get status for Rule Execution Jobs...\n",
- "=============================================\n",
- "RuleName: ExplodingTensor\n",
- "RuleStatus: InProgress\n",
- "RuleExecutionJobName: ExplodingTensor-8a2f1f7334ff9ef67c8de51cf7b2a7dd\n",
- "RuleExecutionJobArn: arn:aws:sagemaker:ca-central-1:072677473360:training-job/explodingtensor-8a2f1f7334ff9ef67c8de51cf7b2a7dd\n",
- "=============================================\n",
- "Wait to get status for Rule Execution Jobs...\n",
- "=============================================\n",
- "RuleName: ExplodingTensor\n",
- "RuleStatus: InProgress\n",
- "RuleExecutionJobName: ExplodingTensor-8a2f1f7334ff9ef67c8de51cf7b2a7dd\n",
- "RuleExecutionJobArn: arn:aws:sagemaker:ca-central-1:072677473360:training-job/explodingtensor-8a2f1f7334ff9ef67c8de51cf7b2a7dd\n",
- "=============================================\n",
- "Wait to get status for Rule Execution Jobs...\n",
- "=============================================\n",
- "RuleName: ExplodingTensor\n",
- "RuleStatus: RuleExecutionError\n",
- "FailureReason: ClientError: RuleEvaluationConditionMet: Evaluation of the rule ExplodingTensor at step 0 resulted in the condition being met\n",
- "Traceback (most recent call last):\n",
- " File \"train.py\", line 214, in execute\n",
- " exec(_SYMBOLIC_INVOKE_RULE.format(self.start_step, self.end_step), globals(), exec_local)\n",
- " File \"\", line 2, in \n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 84, in invoke_rule\n",
- " raise e\n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 79, in invoke_rule\n",
- " rule_obj.invoke(step)\n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule.py\", line 56, in invoke\n",
- " raise RuleEvaluationConditionMet(self.rule_name, step)\n",
- "smdebug.exceptions.RuleEvaluationConditionMet: Evaluation of the rule ExplodingTensor at step 0 resulted in the condition being met\n",
- "\n",
- "\n",
- "RuleExecutionJobName: ExplodingTensor-8a2f1f7334ff9ef67c8de51cf7b2a7dd\n",
- "RuleExecutionJobArn: arn:aws:sagemaker:ca-central-1:072677473360:training-job/explodingtensor-8a2f1f7334ff9ef67c8de51cf7b2a7dd\n",
- "=============================================\n"
- ]
- }
- ],
- "source": [
- "statuses = sagemaker_simple_estimator.describe_rule_execution_jobs()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Check logs of the rule execution jobs\n",
- "\n",
- "If you want to access the logs of a particular rule job name, you can do the following:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [],
- "source": [
- "rule_job_name = statuses[0].get('RuleExecutionJobName')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we can attach to this job to see its logs"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "2019-08-30 04:17:01 Starting - Preparing the instances for training\n",
- "2019-08-30 04:17:01 Downloading - Downloading input data\n",
- "2019-08-30 04:17:01 Training - Training image download completed. Training in progress.\n",
- "2019-08-30 04:17:01 Uploading - Uploading generated training model\n",
- "2019-08-30 04:17:01 Failed - Training job failed\u001b[31m[2019-08-30 04:16:45.369 ip-10-0-100-119.ca-central-1.compute.internal:1 INFO s3_trial.py:27] Loading trial base-trial at path s3://sagemaker-ca-central-1-072677473360/tensors-tornasole-simple-demo-2019-08-30-04-11-54-514\u001b[0m\n",
- "\u001b[31m[2019-08-30 04:16:56.457 ip-10-0-100-119.ca-central-1.compute.internal:1 INFO exploding_tensor.py:24] ExplodingTensor rule created. Monitoring tensors for nans or infinities.\u001b[0m\n",
- "\u001b[31m[2019-08-30 04:16:56.457 ip-10-0-100-119.ca-central-1.compute.internal:1 INFO rule_invoker.py:76] Started execution of rule ExplodingTensor at step 0\u001b[0m\n",
- "\u001b[31m[2019-08-30 04:16:57.653 ip-10-0-100-119.ca-central-1.compute.internal:1 INFO exploding_tensor.py:45] Step 0 had 1 tensors with non finite values\u001b[0m\n",
- "\u001b[31mException during rule execution: Customer Error: RuleEvaluationConditionMet: Evaluation of the rule ExplodingTensor at step 0 resulted in the condition being met\u001b[0m\n",
- "\u001b[31mTraceback (most recent call last):\n",
- " File \"train.py\", line 214, in execute\n",
- " exec(_SYMBOLIC_INVOKE_RULE.format(self.start_step, self.end_step), globals(), exec_local)\n",
- " File \"\", line 2, in \n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 84, in invoke_rule\n",
- " raise e\n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 79, in invoke_rule\n",
- " rule_obj.invoke(step)\n",
- " File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule.py\", line 56, in invoke\n",
- " raise RuleEvaluationConditionMet(self.rule_name, step)\u001b[0m\n",
- "\u001b[31msmdebug.exceptions.RuleEvaluationConditionMet: Evaluation of the rule ExplodingTensor at step 0 resulted in the condition being met\n",
- "\n",
- "\u001b[0m\n"
- ]
- },
- {
- "ename": "UnexpectedStatusException",
- "evalue": "Error for Training job ExplodingTensor-8a2f1f7334ff9ef67c8de51cf7b2a7dd: Failed. Reason: ClientError: RuleEvaluationConditionMet: Evaluation of the rule ExplodingTensor at step 0 resulted in the condition being met\nTraceback (most recent call last):\n File \"train.py\", line 214, in execute\n exec(_SYMBOLIC_INVOKE_RULE.format(self.start_step, self.end_step), globals(), exec_local)\n File \"\", line 2, in \n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 84, in invoke_rule\n raise e\n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 79, in invoke_rule\n rule_obj.invoke(step)\n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule.py\", line 56, in invoke\n raise RuleEvaluationConditionMet(self.rule_name, step)\nsmdebug.exceptions.RuleEvaluationConditionMet: Evaluation of the rule ExplodingTensor at step 0 resulted in the condition being met\n\n",
- "output_type": "error",
- "traceback": [
- "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[0;31mUnexpectedStatusException\u001b[0m Traceback (most recent call last)",
- "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msagemaker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mestimator\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mEstimator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mexploding_tensor\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mEstimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mattach\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrule_job_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
- "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/estimator.py\u001b[0m in \u001b[0;36mattach\u001b[0;34m(cls, training_job_name, sagemaker_session, model_channel_name)\u001b[0m\n\u001b[1;32m 460\u001b[0m )\n\u001b[1;32m 461\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_current_job_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlatest_training_job\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 462\u001b[0;31m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlatest_training_job\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 463\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 464\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/estimator.py\u001b[0m in \u001b[0;36mwait\u001b[0;34m(self, logs)\u001b[0m\n\u001b[1;32m 1012\u001b[0m \"\"\"\n\u001b[1;32m 1013\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mlogs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1014\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msagemaker_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlogs_for_job\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwait\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1015\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1016\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msagemaker_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait_for_job\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mlogs_for_job\u001b[0;34m(self, job_name, wait, poll)\u001b[0m\n\u001b[1;32m 1479\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1480\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mwait\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1481\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_job_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdescription\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"TrainingJobStatus\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1482\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mdot\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1483\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36m_check_job_status\u001b[0;34m(self, job, desc, status_key_name)\u001b[0m\n\u001b[1;32m 1092\u001b[0m ),\n\u001b[1;32m 1093\u001b[0m \u001b[0mallowed_statuses\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"Completed\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"Stopped\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1094\u001b[0;31m \u001b[0mactual_status\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mstatus\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1095\u001b[0m )\n\u001b[1;32m 1096\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;31mUnexpectedStatusException\u001b[0m: Error for Training job ExplodingTensor-8a2f1f7334ff9ef67c8de51cf7b2a7dd: Failed. Reason: ClientError: RuleEvaluationConditionMet: Evaluation of the rule ExplodingTensor at step 0 resulted in the condition being met\nTraceback (most recent call last):\n File \"train.py\", line 214, in execute\n exec(_SYMBOLIC_INVOKE_RULE.format(self.start_step, self.end_step), globals(), exec_local)\n File \"\", line 2, in \n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 84, in invoke_rule\n raise e\n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule_invoker.py\", line 79, in invoke_rule\n rule_obj.invoke(step)\n File \"/usr/local/lib/python3.7/site-packages/smdebug/rules/rule.py\", line 56, in invoke\n raise RuleEvaluationConditionMet(self.rule_name, step)\nsmdebug.exceptions.RuleEvaluationConditionMet: Evaluation of the rule ExplodingTensor at step 0 resulted in the condition being met\n\n"
- ]
- }
- ],
- "source": [
- "from sagemaker.estimator import Estimator\n",
- "exploding_tensor = Estimator.attach(rule_job_name)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Receive a CloudWatch Event for Rules\n",
- "When the status of training job or rule execution job change (i.e. starting, failed), TrainingJobStatus [CloudWatch events](https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html) are emitted. More details on this, see [below](#CloudWatch-Event-Integration-for-Rules). "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Making this a good run\n",
- "\n",
- "In above example, we saw how a ExplodingTensor Rule was run which analyzed the tensors when training was running and produced an alert in form of cloudwatch event.\n",
- "\n",
- "You can go back and change the hyperparameters passed to the estimator to `simple_hyperparameters` and start a new training job. You will see that the ExplodingTensor rule is not fired in that case as no tensors go to `nan` with the default good hyperparameters.\n",
- "\n",
- "We have 2 more real life examples at the end section of this notebook for you to try, a GPU example which trains ResNet50, and another CPU example which trains MNIST. Before moving further, let's take some detailed look at Tornasole, some of which were touched upon above."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Enabling Tornasole in the training script\n",
- "\n",
- "The first step to using Tornasole is to save tensors from the training job. The containers we provide in SageMaker come with Tornasole library installed, which needs to be used to enable Tornasole in your training script. We currently support two interfaces for training in TensorFlow: `tf.Session` and `tf.Estimator`. \n",
- "\n",
- "Please note: **Keras** support is Work in Progress. Please stay tuned! We will also support **Eager** mode in the future. Tornasole also currently only works for single process training. We will support distributed training very soon. \n",
- "\n",
- "### TF Session based training\n",
- "When training using this interface you need to create a [MonitoredSession](https://www.tensorflow.org/api_docs/python/tf/train/MonitoredSession) to use for the job which is configured with SessionHook, a construct Tornasole exposes to save tensors from the job. Here's how you will need to modify your training script.\n",
- "\n",
- "First, you need to import `smdebug.tensorflow`. \n",
- "```\n",
- "import smdebug.tensorflow as smd \n",
- "```\n",
- "Then create the SessionHook by specifying what you want to save and when you want to save them.\n",
- "```\n",
- "hook = smd.SessionHook(include_collections=['weights','gradients'],\n",
- " save_config=smd.SaveConfig(save_interval=50))\n",
- "```\n",
- "Tornasole has the concept of modes (TRAIN, EVAL, PREDICT) to separate out different modes of the jobs.\n",
- "Set the mode you are running in your job. Every time the mode changes in your job, please set the current mode. This helps you group steps by mode, for easier analysis. Setting the mode is optional but recommended. If you do not specify this, Tornasole saves all steps under a `GLOBAL` mode. \n",
- "```\n",
- "hook.set_mode(smd.modes.TRAIN)\n",
- "```\n",
- "Wrap your optimizer with wrap_optimizer so that Tornasole can identify your gradients and automatically provide these tensors as part of the `gradients` collection. Use this new optimizer to minimize your loss during training.\n",
- "```\n",
- "optimizer = hook.wrap_optimizer(optimizer)\n",
- "```\n",
- "Create a monitored session with the above hook, and use this for executing your TensorFlow job.\n",
- "```\n",
- "sess = tf.train.MonitoredSession(hooks=[hook])\n",
- "```\n",
- "\n",
- "We have an example script which shows the above [scripts/simple.py](../scripts/simple.py). You will be running this script below.\n",
- "\n",
- "Refer [DeveloperGuide_TensorFlow.md](../../DeveloperGuide_TF.md) for more details on the APIs Tornasole provides to help you save tensors.\n",
- "\n",
- "### TF Estimator based training\n",
- "When training using this interface you need to pass SessionHook, a construct Tornasole exposes to save tensors, to the train, predict or evaluate functions of your [TensorFlow Estimator](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/estimator/Estimator?hl=en). Here's how you will need to modify your training script.\n",
- "\n",
- "First, you need to import `smdebug.tensorflow`. \n",
- "```\n",
- "import smdebug.tensorflow as smd \n",
- "```\n",
- "Then create the SessionHook by specifying what you want to save and when you want to save them.\n",
- "```\n",
- "hook = smd.SessionHook(include_collections=['weights','gradients'],\n",
- " save_config=smd.SaveConfig(save_interval=50))\n",
- "```\n",
- "Tornasole has the concept of modes (TRAIN, EVAL, PREDICT) to separate out different modes of the jobs.\n",
- "Set the mode you are running in your job. Every time the mode changes in your job, please set the current mode. This helps you group steps by mode, for easier analysis. Setting the mode is optional but recommended. If you do not specify this, Tornasole saves all steps under a `GLOBAL` mode. \n",
- "```\n",
- "hook.set_mode(smd.modes.TRAIN)\n",
- "```\n",
- "Wrap your optimizer with wrap_optimizer so that Tornasole can identify your gradients and automatically provide these tensors as part of the `gradients` collection. Use this new optimizer to minimize your loss during training.\n",
- "```\n",
- "opt = hook.wrap_optimizer(opt)\n",
- "```\n",
- "Now pass this hook to the estimator object's train, predict or evaluate methods, whichever ones you want to monitor.\n",
- "```\n",
- "classifier = tf.estimator.Estimator(...)\n",
- "\n",
- "classifier.train(input_fn, hooks=[hook])\n",
- "classifier.predict(input_fn, hooks=[hook])\n",
- "classifier.evaluate(input_fn, hooks=[hook])\n",
- "```\n",
- "\n",
- "Refer our example script for [MNIST](../scripts/mnist.py) or [ResNet50 for ImageNet](../scripts/train_imagenet_resnet_hvd.py) for examples of using Tornasole with the Estimator interface. We will show you to how to run these examples in SageMaker below.\n",
- "\n",
- "Refer [DeveloperGuide_TensorFlow.md](../../DeveloperGuide_TF.md) for more details on the APIs Tornasole provides to help you save tensors."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Enabling Tornasole with SageMaker\n",
- "#### Storage\n",
- "The tensors saved by Tornasole are, by default, stored in the S3 output path of the training job, under the folder **`/tensors-`**. This is done to ensure that we don't end up accidentally overwriting the tensors from a training job with the others. Rules evaluation require separation of the tensors paths to be evaluated correctly.\n",
- "\n",
- "If you don't provide an S3 output path to the estimator, SageMaker creates one for you as: **`s3://sagemaker--/`**\n",
- "\n",
- "This path is used to create a Tornasole Trial taken by Rules (see below).\n",
- "\n",
- "#### New Parameters \n",
- "The new parameters in Sagemaker Estimator to look out for are\n",
- "\n",
- "- `debug` :(bool)\n",
- "This indicates that debugging should be enabled for the training job. \n",
- "Setting this as `True` would make Tornasole available for use with the job\n",
- "\n",
- "- `rules_specification`: (list[*dict*])\n",
- "You can specify any number of rules to monitor your SageMaker training job. This parameter takes a list of python dictionaries, one for each rule you want to enable. Each `dict` is of the following form:\n",
- "```\n",
- "{\n",
- " \"RuleName\": \n",
- " # The name of the class implementing the Tornasole Rule interface. (required)\n",
- "\n",
- " \"SourceS3Uri\": \n",
- " # S3 URI of the rule script containing the class in 'RuleName'. \n",
- " # This is not required if you want to use one of the\n",
- " # First Party rules provided to you by Amazon. \n",
- " # In such a case you can leave it empty or not pass it. If you want to run a custom rule \n",
- " # defined by you, you will need to define the custom rule class in a python \n",
- " # file and provide it to SageMaker as a S3 URI. \n",
- " # SageMaker will fetch this file and try to look for the rule class \n",
- " # identified by RuleName in this file.\n",
- " \n",
- " \"InstanceType\": \n",
- " # The ML instance type which should be used to run the rule evaluation job\n",
- " \n",
- " \"VolumeSizeInGB\": \n",
- " # The volume size to store the runtime artifacts from the rule evaluation \n",
- " \n",
- " \"RuntimeConfigurations\": {\n",
- " # Map defining the parameters required to instantiate the Rule class and\n",
- " # parameters regarding invokation of the rule (start-step and end-step)\n",
- " # This can be any parameter taken by the rule. \n",
- " # Every value here needs to be a string. \n",
- " # So when you write custom rules, ensure that you can parse each argument from a string.\n",
- " #\n",
- " # PARAMS CAN BE\n",
- " #\n",
- " # STANDARD PARAMS FOR RULE EXECUTION\n",
- " # \"start-step\": \n",
- " # \"end-step\": \n",
- " # \"other-trials-paths\": (';' separated list of s3 paths as a string)\n",
- " # \"logging-level\": (can be one of \"CRITICAL\", \"FATAL\", \"ERROR\", \n",
- " # \"WARNING\", \"WARN\", \"DEBUG\", \"NOTSET\")\n",
- " #\n",
- " # ANY OTHER PARAMETER TAKEN BY THE RULE\n",
- " # \"parameter\" : \n",
- " # : \n",
- " }\n",
- "}\n",
- "```\n",
- "\n",
- "### Inputs\n",
- "Just a quick reminder if you are not familiar with script mode in SageMaker. You can pass command line arguments taken by your training script with a hyperparameter dictionary which gets passed to the SageMaker Estimator class. You can see this in the examples below."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Rules\n",
- "Rules are the medium by which Tornasole executes a certain piece of code regularly on different steps of the job.\n",
- "They can be used to assert certain conditions during training, and raise Cloudwatch Events based on them that you can\n",
- "use to process in any way you like. \n",
- "\n",
- "Tornasole comes with a set of **First Party rules** (1P rules).\n",
- "You can also write your own rules looking at these 1P rules for inspiration. \n",
- "Refer [DeveloperGuide_Rules.md](../../../../rules/DeveloperGuide_Rules.md) for more on the APIs you can use to write your own rules as well as descriptions for the 1P rules that we provide. \n",
- " \n",
- "Here we will talk about how to use Sagemaker to evalute these rules on the training jobs.\n",
- "\n",
- "\n",
- "##### 1P Rule \n",
- "If you want to use a 1P rule. Specify the RuleName field with the 1P RuleName, and the rule will be automatically applied. You can pass any parameters accepted by the rule as part of the RuntimeConfigurations dictionary. Rules constructor take trial as parameter. \n",
- "A Trial in Tornasole's context refers to a training job. It is identified by the path where the saved tensors for the job are stored. \n",
- "A rule takes a `base_trial` which refers to the job whose run invokes the rule execution. \n",
- "\n",
- "**Note:** A rule can be written to compare & analyze tensors across training jobs. A rule which needs to compare tensors across trials can be run by passing the argument `other_trials`. The argument `base_trial` will automatically be set by SageMaker when executing the rule. The parameter `other_trials` (if taken by the rule) can be passed by passing `other-trials-paths` in the RuntimeConfigurations dictionary. The value for this argument should be `;` separated list of S3 output paths where the tensors for those trials are stored.\n",
- "\n",
- "Here's a example of a complex configuration for the SimilarAcrossRuns (which accepts one other trial and a regex pattern) where we ask for the rule to be invoked for the steps between 10 and 100.\n",
- "\n",
- "``` \n",
- "rules_specification = [ \n",
- " {\n",
- " \"RuleName\": \"SimilarAcrossRuns\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"other_trials\": \"s3://sagemaker--/past-job\",\n",
- " \"include_regex\": \".*\",\n",
- " \"start-step\": \"10\",\n",
- " \"end-step\": \"100\"\n",
- " }\n",
- " }\n",
- "]\n",
- "```\n",
- "List of 1P rules and details about the rules can be found in *First party rules* section in [DeveloperGuide_Rules.md](../../../../rules/DeveloperGuide_Rules.md) \n",
- "\n",
- "\n",
- "##### Custom rule\n",
- "In this case you need to define a custom rule class which inherits from `smdebug.rules.Rule` class.\n",
- "You need to provide Sagemaker the S3 location of the file which defines your custom rule classes as the value for the field `SourceS3Uri`. Again, you can pass any arguments taken by this rule through the RuntimeConfigurations dictionary. Note that the custom rules can only have arguments which expect a string as the value except the two arguments specifying trials to the Rule. Refer section *Writing a rule* in [DeveloperGuide_Rules.md](../../../../rules/DeveloperGuide_Rules.md) for more details.\n",
- "\n",
- "Here's an example:\n",
- "```\n",
- "rules_specification = [\n",
- " {\n",
- " \"RuleName\": \"CustomRule\",\n",
- " \"SourceS3Uri\": \"s3://weiyou-tornasole-test/rule-script/custom_rule.py\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"threshold\" : \"0.5\"\n",
- " }\n",
- " }\n",
- "]\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### CloudWatch Event Integration for Rules\n",
- "When the status of training job or rule execution job change (i.e. starting, failed), TrainingJobStatus [CloudWatch events](https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html) are emitted.\n",
- "\n",
- "After GA, you can configure a CloudWatch event rule to receive and process these events by setting up a target (Lambda function, SNS) as follows:\n",
- "\n",
- "- The [SageMaker TrainingJobStatus CW event] (https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#sagemaker_event_types) will include rule job statuses associated with the training job\n",
- "- A CW event will be emitted when a RuleStatus changes\n",
- "- Customer can create a CloudWatch event rule that monitors the Training Job customer started\n",
- "- Customer can set a Target (Lambda funtion, SQS) for the CloudWatch event rule that processes the event, and triggers an alarm for the customer based on the RuleStatus. \n",
- "\n",
- "Refer [this page](https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html) for more details. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Train Resnet50 Estimator Interface, with Tornasole\n",
- "Now let us run a more complicated example, let us train ResNet50 on a GPU instance. The script which uses the TensorFlow Estimator interface is available [here](../scripts/train_imagenet_resnet_hvd.py). It supports various modes of using smdebug. Please refer to [this document](../../sm_resnet50.md) which summarizes the changes made to this script to save weights, gradients, activations of certain layers etc. You can also save large layers as reductions instead of saving the full tensor. Full details of Tornasole's APIs to save tensors are available in this document [DeveloperGuide_TensorFlow](../../DeveloperGuide_TF.md).\n",
- "\n",
- "The below hyperparameters initialize the weights of the model badly (to a small constant). This results in training proceeding badly with many gradients vanishing. We can monitor the situation using the VanishingGradient rule."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "resnet_script = '../scripts/train_imagenet_resnet_hvd.py'\n",
- "bad_resnet_hyperparameters = {\n",
- " 'enable_smdebug': True,\n",
- " 'save_weights': True,\n",
- " 'save_gradients': True,\n",
- " 'step_interval' : 100,\n",
- " 'num_epochs': 1,\n",
- " 'constant_initializer': 0.01\n",
- "}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "sagemaker_resnet_estimator = TensorFlow(role=sagemaker.get_execution_role(),\n",
- " base_job_name='tornasole-demo-resnet',\n",
- " train_instance_count=1,\n",
- " train_instance_type='ml.p3.2xlarge',\n",
- " image_name=gpu_docker_image_name,\n",
- " entry_point=resnet_script,\n",
- " framework_version='1.13.1',\n",
- " py_version='py3',\n",
- " script_mode=True,\n",
- " hyperparameters=bad_resnet_hyperparameters,\n",
- " debug=True,\n",
- " train_max_run=1800,\n",
- " rules_specification=[\n",
- " {\n",
- " \"RuleName\": \"VanishingGradient\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " }\n",
- " ])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "sagemaker_resnet_estimator.fit(wait=False)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Note:wait=False above, made fit call fire and forget call. \n",
- "# Sagemaker will run the training job and rule job in the background. \n",
- "# To see status of training job:\n",
- "sagemaker_simple_estimator.sagemaker_session.sagemaker_client.describe_training_job(\n",
- " TrainingJobName=sagemaker_simple_estimator.latest_training_job.name\n",
- ")\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# To check status of rule execution job\n",
- "sagemaker_resnet_estimator.describe_rule_execution_jobs()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Train MNIST Estimator interface, with Tornasole\n",
- "If you do not want to use GPUs at this point, but want to run a slightly more complicated script than the simple example you saw above, you can train a model on CPU on the MNIST dataset as below. Let us monitor for VanishingGradient in this job. **We do not expect this rule to be fired for the below hyperparameters.**"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "mnist_script = '../scripts/mnist.py'\n",
- "mnist_hyperparameters = {'num_epochs': 5}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "sagemaker_mnist_estimator = TensorFlow(role=sagemaker.get_execution_role(),\n",
- " base_job_name='tornasole-demo-mnist',\n",
- " train_instance_count=1,\n",
- " train_instance_type='ml.m4.xlarge',\n",
- " image_name=cpu_docker_image_name,\n",
- " entry_point=mnist_script,\n",
- " framework_version='1.13.1',\n",
- " py_version='py3',\n",
- " script_mode=True,\n",
- " hyperparameters=mnist_hyperparameters,\n",
- " debug=True,\n",
- " train_max_run=1800,\n",
- " rules_specification=[\n",
- " {\n",
- " \"RuleName\": \"VanishingGradient\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " }\n",
- " ])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "sagemaker_mnist_estimator.fit(wait=False)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "sagemaker_mnist_estimator.sagemaker_session.sagemaker_client.describe_training_job(\n",
- " TrainingJobName=sagemaker_mnist_estimator.latest_training_job.name\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# To check status of rule execution job\n",
- "sagemaker_mnist_estimator.describe_rule_execution_jobs()"
- ]
- }
- ],
- "metadata": {
- "celltoolbar": "Raw Cell Format",
- "kernelspec": {
- "display_name": "conda_tensorflow_p36",
- "language": "python",
- "name": "conda_tensorflow_p36"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.5"
- },
- "pycharm": {
- "stem_cell": {
- "cell_type": "raw",
- "metadata": {
- "collapsed": false
- },
- "source": []
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/examples/tensorflow/sagemaker_byoc/mnist.py b/examples/tensorflow/sagemaker_byoc/mnist.py
new file mode 100644
index 000000000..bd354d24c
--- /dev/null
+++ b/examples/tensorflow/sagemaker_byoc/mnist.py
@@ -0,0 +1,139 @@
+"""
+This script is a simple MNIST training script which uses Tensorflow's Estimator interface.
+It has been orchestrated with SageMaker Debugger hooks to allow saving tensors during training.
+These hooks have been instrumented to read from json configuration that SageMaker will put in the training container.
+Configuration provided to the SageMaker python SDK when creating a job will be passed on to the hook.
+This allows you to use the same script with differing configurations across different runs.
+If you use an official SageMaker Framework container (i.e. AWS Deep Learning Container), then
+you do not have to orchestrate your script as below. Hooks will automatically be added in those environments.
+For more information, please refer to https://github.com/awslabs/sagemaker-debugger/blob/master/docs/sagemaker.md
+"""
+
+# Standard Library
+import argparse
+import logging
+import random
+
+# Third Party
+import numpy as np
+import tensorflow as tf
+
+# First Party
+import smdebug.tensorflow as smd
+
+logging.getLogger().setLevel(logging.INFO)
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--lr", type=float, default=0.001)
+parser.add_argument("--random_seed", type=bool, default=False)
+parser.add_argument("--num_epochs", type=int, default=5, help="Number of epochs to train for")
+parser.add_argument(
+ "--num_steps",
+ type=int,
+ help="Number of steps to train for. If this" "is passed, it overrides num_epochs",
+)
+parser.add_argument(
+ "--num_eval_steps",
+ type=int,
+ help="Number of steps to evaluate for. If this"
+ "is passed, it doesnt evaluate over the full eval set",
+)
+parser.add_argument("--model_dir", type=str, default="/tmp/mnist_model")
+args = parser.parse_args()
+
+if args.random_seed:
+ tf.set_random_seed(2)
+ np.random.seed(2)
+ random.seed(12)
+
+# This allows you to create the hook from the configuration you pass to the SageMaker pySDK
+hook = smd.SessionHook.create_from_json_file()
+
+
+def cnn_model_fn(features, labels, mode):
+ """Model function for CNN."""
+ # Input Layer
+ input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])
+
+ # Convolutional Layer #1
+ conv1 = tf.layers.conv2d(
+ inputs=input_layer, filters=32, kernel_size=[5, 5], padding="same", activation=tf.nn.relu
+ )
+
+ # Pooling Layer #1
+ pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)
+
+ # Convolutional Layer #2 and Pooling Layer #2
+ conv2 = tf.layers.conv2d(
+ inputs=pool1, filters=64, kernel_size=[5, 5], padding="same", activation=tf.nn.relu
+ )
+ pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)
+
+ # Dense Layer
+ pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])
+ dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
+ dropout = tf.layers.dropout(
+ inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN
+ )
+
+ # Logits Layer
+ logits = tf.layers.dense(inputs=dropout, units=10)
+
+ predictions = {
+ # Generate predictions (for PREDICT and EVAL mode)
+ "classes": tf.argmax(input=logits, axis=1),
+ # Add `softmax_tensor` to the graph. It is used for PREDICT and by the
+ # `logging_hook`.
+ "probabilities": tf.nn.softmax(logits, name="softmax_tensor"),
+ }
+
+ if mode == tf.estimator.ModeKeys.PREDICT:
+ return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)
+
+ # Calculate Loss (for both TRAIN and EVAL modes)
+ loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)
+
+ # Configure the Training Op (for TRAIN mode)
+ if mode == tf.estimator.ModeKeys.TRAIN:
+ optimizer = tf.train.GradientDescentOptimizer(learning_rate=args.lr)
+
+ # SMD: Wrap your optimizer as follows to help SageMaker Debugger identify gradients
+ # This does not change your optimization logic, it returns back the same optimizer
+ optimizer = hook.wrap_optimizer(optimizer)
+
+ train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
+ return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)
+
+ # Add evaluation metrics (for EVAL mode)
+ eval_metric_ops = {
+ "accuracy": tf.metrics.accuracy(labels=labels, predictions=predictions["classes"])
+ }
+ return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)
+
+
+# Load training and eval data
+((train_data, train_labels), (eval_data, eval_labels)) = tf.keras.datasets.mnist.load_data()
+
+train_data = train_data / np.float32(255)
+train_labels = train_labels.astype(np.int32) # not required
+
+eval_data = eval_data / np.float32(255)
+eval_labels = eval_labels.astype(np.int32) # not required
+
+mnist_classifier = tf.estimator.Estimator(model_fn=cnn_model_fn, model_dir=args.model_dir)
+
+train_input_fn = tf.estimator.inputs.numpy_input_fn(
+ x={"x": train_data}, y=train_labels, batch_size=128, num_epochs=args.num_epochs, shuffle=True
+)
+
+eval_input_fn = tf.estimator.inputs.numpy_input_fn(
+ x={"x": eval_data}, y=eval_labels, num_epochs=1, shuffle=False
+)
+
+# Set training mode so SMDebug can classify the steps into training mode
+hook.set_mode(smd.modes.TRAIN)
+mnist_classifier.train(input_fn=train_input_fn, steps=args.num_steps, hooks=[hook])
+
+# Set eval mode so SMDebug can classify the steps into eval mode
+hook.set_mode(smd.modes.EVAL)
+mnist_classifier.evaluate(input_fn=eval_input_fn, steps=args.num_eval_steps, hooks=[hook])
diff --git a/examples/tensorflow/scripts/simple.py b/examples/tensorflow/sagemaker_byoc/simple.py
similarity index 57%
rename from examples/tensorflow/scripts/simple.py
rename to examples/tensorflow/sagemaker_byoc/simple.py
index b8268d0af..08433dd68 100644
--- a/examples/tensorflow/scripts/simple.py
+++ b/examples/tensorflow/sagemaker_byoc/simple.py
@@ -1,3 +1,14 @@
+"""
+This script is a simple training script which uses Tensorflow's MonitoredSession interface.
+It has been orchestrated with SageMaker Debugger hooks to allow saving tensors during training.
+These hooks have been instrumented to read from json configuration that SageMaker will put in the training container.
+Configuration provided to the SageMaker python SDK when creating a job will be passed on to the hook.
+This allows you to use the same script with differing configurations across different runs.
+If you use an official SageMaker Framework container (i.e. AWS Deep Learning Container), then
+you do not have to orchestrate your script as below. Hooks will automatically be added in those environments.
+For more information, please refer to https://github.com/awslabs/sagemaker-debugger/blob/master/docs/
+"""
+
# Standard Library
import argparse
import random
@@ -13,7 +24,6 @@
def str2bool(v):
if isinstance(v, bool):
return v
-
if v.lower() in ("yes", "true", "t", "y", "1"):
return True
elif v.lower() in ("no", "false", "f", "n", "0"):
@@ -23,25 +33,11 @@ def str2bool(v):
parser = argparse.ArgumentParser()
-parser.add_argument("--script-mode", type=str2bool, default=False)
parser.add_argument("--model_dir", type=str, help="S3 path for the model")
parser.add_argument("--lr", type=float, help="Learning Rate", default=0.001)
parser.add_argument("--steps", type=int, help="Number of steps to run", default=100)
parser.add_argument("--scale", type=float, help="Scaling factor for inputs", default=1.0)
-parser.add_argument("--save_all", type=str2bool, default=True)
-parser.add_argument("--smdebug_path", type=str, default="/opt/ml/output/tensors")
-parser.add_argument("--save_frequency", type=int, help="How often to save TS data", default=10)
parser.add_argument("--random_seed", type=bool, default=False)
-feature_parser = parser.add_mutually_exclusive_group(required=False)
-feature_parser.add_argument(
- "--reductions",
- dest="reductions",
- action="store_true",
- help="save reductions of tensors instead of saving full tensors",
-)
-feature_parser.add_argument(
- "--no_reductions", dest="reductions", action="store_false", help="save full tensors"
-)
args = parser.parse_args()
# these random seeds are only intended for test purpose.
@@ -52,27 +48,7 @@ def str2bool(v):
np.random.seed(2)
random.seed(12)
-
-if args.script_mode:
- # save tensors as reductions if necessary
- rdnc = (
- smd.ReductionConfig(reductions=["mean"], abs_reductions=["max"], norms=["l1"])
- if args.reductions
- else None
- )
-
- # create the hook
- # Note that we are saving all tensors here by passing save_all=True
- hook = smd.SessionHook(
- out_dir=args.smdebug_path,
- save_all=args.save_all,
- include_collections=["weights", "gradients", "losses"],
- save_config=smd.SaveConfig(save_interval=args.save_frequency),
- reduction_config=rdnc,
- )
- hooks = [hook]
-else:
- hooks = []
+hook = smd.SessionHook.create_from_json_file()
# Network definition
# Note the use of name scopes
@@ -84,34 +60,30 @@ def str2bool(v):
y = tf.matmul(x, w0)
loss = tf.reduce_mean((tf.matmul(x, w) - y) ** 2, name="loss")
-smd.get_hook("session", create_if_not_exists=True).add_to_collection("losses", loss)
+hook.add_to_collection("losses", loss)
global_step = tf.Variable(17, name="global_step", trainable=False)
increment_global_step_op = tf.assign(global_step, global_step + 1)
optimizer = tf.train.AdamOptimizer(args.lr)
-if args.script_mode:
- # Wrap the optimizer with wrap_optimizer so Tornasole can find gradients and optimizer_variables to save
- optimizer = hook.wrap_optimizer(optimizer)
+# Wrap the optimizer with wrap_optimizer so smdebug can find gradients to save
+optimizer = hook.wrap_optimizer(optimizer)
# use this wrapped optimizer to minimize loss
optimizer_op = optimizer.minimize(loss, global_step=increment_global_step_op)
-if args.script_mode:
- hook.set_mode(smd.modes.TRAIN)
-
# pass the hook to hooks parameter of monitored session
-sess = tf.train.MonitoredSession(hooks=hooks)
+sess = tf.train.MonitoredSession(hooks=[hook])
# use this session for running the tensorflow model
+hook.set_mode(smd.modes.TRAIN)
for i in range(args.steps):
x_ = np.random.random((10, 2)) * args.scale
_loss, opt, gstep = sess.run([loss, optimizer_op, increment_global_step_op], {x: x_})
print(f"Step={i}, Loss={_loss}")
-if args.script_mode:
- hook.set_mode(smd.modes.EVAL)
+hook.set_mode(smd.modes.EVAL)
for i in range(args.steps):
x_ = np.random.random((10, 2)) * args.scale
sess.run([loss, increment_global_step_op], {x: x_})
diff --git a/examples/tensorflow/sagemaker_byoc/tf_keras_resnet.py b/examples/tensorflow/sagemaker_byoc/tf_keras_resnet.py
new file mode 100644
index 000000000..9ce9d2e74
--- /dev/null
+++ b/examples/tensorflow/sagemaker_byoc/tf_keras_resnet.py
@@ -0,0 +1,73 @@
+"""
+This script is a ResNet training script which uses Tensorflow's Keras interface.
+It has been orchestrated with SageMaker Debugger hooks to allow saving tensors during training.
+These hooks have been instrumented to read from json configuration that SageMaker will put in the training container.
+Configuration provided to the SageMaker python SDK when creating a job will be passed on to the hook.
+This allows you to use the same script with differing configurations across different runs.
+If you use an official SageMaker Framework container (i.e. AWS Deep Learning Container), then
+you do not have to orchestrate your script as below. Hooks will automatically be added in those environments.
+For more information, please refer to https://github.com/awslabs/sagemaker-debugger/blob/master/docs/sagemaker.md
+"""
+
+# Standard Library
+import argparse
+
+# Third Party
+import numpy as np
+import tensorflow as tf
+from tensorflow.keras.applications.resnet50 import ResNet50
+from tensorflow.keras.datasets import cifar10
+from tensorflow.keras.utils import to_categorical
+
+# First Party
+import smdebug.tensorflow as smd
+
+
+def train(batch_size, epoch, model, hook):
+ (X_train, y_train), (X_valid, y_valid) = cifar10.load_data()
+
+ Y_train = to_categorical(y_train, 10)
+ Y_valid = to_categorical(y_valid, 10)
+
+ X_train = X_train.astype("float32")
+ X_valid = X_valid.astype("float32")
+
+ mean_image = np.mean(X_train, axis=0)
+ X_train -= mean_image
+ X_valid -= mean_image
+ X_train /= 128.0
+ X_valid /= 128.0
+
+ model.fit(
+ X_train,
+ Y_train,
+ batch_size=batch_size,
+ epochs=epoch,
+ validation_data=(X_valid, Y_valid),
+ shuffle=True,
+ callbacks=[hook],
+ )
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Train resnet50 cifar10")
+ parser.add_argument("--batch_size", type=int, default=32)
+ parser.add_argument("--epoch", type=int, default=3)
+ parser.add_argument("--model_dir", type=str, default="./model_keras_resnet")
+ opt = parser.parse_args()
+
+ model = ResNet50(weights=None, input_shape=(32, 32, 3), classes=10)
+
+ # Create hook from the configuration provided through sagemaker python sdk
+ hook = smd.KerasHook.create_from_json_file()
+ optimizer = tf.keras.optimizers.Adam()
+ # wrap the optimizer so the hook can identify the gradients
+ optimizer = hook.wrap_optimizer(optimizer)
+ model.compile(loss="categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
+
+ # start the training.
+ train(opt.batch_size, opt.epoch, model, hook)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/tensorflow/scripts/mnist.py b/examples/tensorflow/sagemaker_official_container/mnist.py
similarity index 79%
rename from examples/tensorflow/scripts/mnist.py
rename to examples/tensorflow/sagemaker_official_container/mnist.py
index 078b06a3b..2143c5bc9 100644
--- a/examples/tensorflow/scripts/mnist.py
+++ b/examples/tensorflow/sagemaker_official_container/mnist.py
@@ -1,19 +1,24 @@
+"""
+This script is a simple MNIST training script which uses Tensorflow's Estimator interface.
+It is designed to be used with SageMaker Debugger in an official SageMaker Framework container (i.e. AWS Deep Learning Container).
+You will notice that this script looks exactly like a normal TensorFlow training script.
+The hook needed by SageMaker Debugger to save tensors during training will be automatically added in those environments.
+The hook will load configuration from json configuration that SageMaker will put in the training container from the configuration provided using the SageMaker python SDK when creating a job.
+For more information, please refer to https://github.com/awslabs/sagemaker-debugger/blob/master/docs/sagemaker.md
+"""
+
# Standard Library
import argparse
+import logging
import random
# Third Party
import numpy as np
import tensorflow as tf
-# First Party
-import smdebug.tensorflow as smd
+logging.getLogger().setLevel(logging.INFO)
parser = argparse.ArgumentParser()
-parser.add_argument("--script-mode", type=bool, default=False)
-parser.add_argument("--smdebug_path", type=str)
-parser.add_argument("--train_frequency", type=int, help="How often to save TS data", default=50)
-parser.add_argument("--eval_frequency", type=int, help="How often to save TS data", default=10)
parser.add_argument("--lr", type=float, default=0.001)
parser.add_argument("--random_seed", type=bool, default=False)
parser.add_argument("--num_epochs", type=int, default=5, help="Number of epochs to train for")
@@ -82,13 +87,10 @@ def cnn_model_fn(features, labels, mode):
# Calculate Loss (for both TRAIN and EVAL modes)
loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)
- tf.summary.scalar("loss", loss)
# Configure the Training Op (for TRAIN mode)
if mode == tf.estimator.ModeKeys.TRAIN:
optimizer = tf.train.GradientDescentOptimizer(learning_rate=args.lr)
- if args.script_mode:
- optimizer = smd.get_hook().wrap_optimizer(optimizer)
train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)
@@ -118,25 +120,6 @@ def cnn_model_fn(features, labels, mode):
x={"x": eval_data}, y=eval_labels, num_epochs=1, shuffle=False
)
-if args.script_mode:
- hook = smd.SessionHook(
- out_dir=args.smdebug_path,
- save_config=smd.SaveConfig(
- {
- smd.modes.TRAIN: smd.SaveConfigMode(args.train_frequency),
- smd.modes.EVAL: smd.SaveConfigMode(args.eval_frequency),
- }
- ),
- )
- hooks = [hook]
-else:
- hooks = []
-
-if args.script_mode:
- hook.set_mode(smd.modes.TRAIN)
-# train one step and display the probabilties
-mnist_classifier.train(input_fn=train_input_fn, steps=args.num_steps, hooks=hooks)
-
-if args.script_mode:
- hook.set_mode(smd.modes.EVAL)
-mnist_classifier.evaluate(input_fn=eval_input_fn, steps=args.num_eval_steps, hooks=hooks)
+mnist_classifier.train(input_fn=train_input_fn, steps=args.num_steps)
+
+mnist_classifier.evaluate(input_fn=eval_input_fn, steps=args.num_eval_steps)
diff --git a/examples/tensorflow/sagemaker_official_container/simple.py b/examples/tensorflow/sagemaker_official_container/simple.py
new file mode 100644
index 000000000..a5fb1a3f5
--- /dev/null
+++ b/examples/tensorflow/sagemaker_official_container/simple.py
@@ -0,0 +1,92 @@
+"""
+This script is a simple training script which uses Tensorflow's MonitoredSession interface.
+It is designed to be used with SageMaker Debugger in an official SageMaker Framework container (i.e. AWS Deep Learning Container).
+Here we create the hook object which loads configuration from the json file that SageMaker will
+put in the training container based on the configuration provided using the SageMaker python SDK when creating a job.
+We use this hook object here to add our custom loss to the losses collection and set the mode.
+For more information, please refer to https://github.com/awslabs/sagemaker-debugger/blob/master/docs/
+"""
+
+# Standard Library
+import argparse
+import random
+
+# Third Party
+import numpy as np
+import tensorflow as tf
+
+# First Party
+import smdebug.tensorflow as smd
+
+
+def str2bool(v):
+ if isinstance(v, bool):
+ return v
+ if v.lower() in ("yes", "true", "t", "y", "1"):
+ return True
+ elif v.lower() in ("no", "false", "f", "n", "0"):
+ return False
+ else:
+ raise argparse.ArgumentTypeError("Boolean value expected.")
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--model_dir", type=str, help="S3 path for the model")
+parser.add_argument("--lr", type=float, help="Learning Rate", default=0.001)
+parser.add_argument("--steps", type=int, help="Number of steps to run", default=100)
+parser.add_argument("--scale", type=float, help="Scaling factor for inputs", default=1.0)
+parser.add_argument("--random_seed", type=bool, default=False)
+args = parser.parse_args()
+
+# these random seeds are only intended for test purpose.
+# for now, 2,2,12 could promise no assert failure when running tests
+# if you wish to change the number, notice that certain steps' tensor value may be capable of variation
+if args.random_seed:
+ tf.set_random_seed(2)
+ np.random.seed(2)
+ random.seed(12)
+
+# Network definition
+# Note the use of name scopes
+with tf.name_scope("foobar"):
+ x = tf.placeholder(shape=(None, 2), dtype=tf.float32)
+ w = tf.Variable(initial_value=[[10.0], [10.0]], name="weight1")
+with tf.name_scope("foobaz"):
+ w0 = [[1], [1.0]]
+ y = tf.matmul(x, w0)
+loss = tf.reduce_mean((tf.matmul(x, w) - y) ** 2, name="loss")
+
+hook = smd.SessionHook.create_from_json_file()
+hook.add_to_collection("losses", loss)
+
+global_step = tf.Variable(17, name="global_step", trainable=False)
+increment_global_step_op = tf.assign(global_step, global_step + 1)
+
+optimizer = tf.train.AdamOptimizer(args.lr)
+
+# Do not need to wrap the optimizer if in a zero script change environment
+# i.e. SageMaker/AWS Deep Learning Containers
+# as the framework will automatically do that there if the hook exists
+optimizer = hook.wrap_optimizer(optimizer)
+
+# use this wrapped optimizer to minimize loss
+optimizer_op = optimizer.minimize(loss, global_step=increment_global_step_op)
+
+# Do not need to pass the hook to the session if in a zero script change environment
+# i.e. SageMaker/AWS Deep Learning Containers
+# as the framework will automatically do that there if the hook exists
+sess = tf.train.MonitoredSession()
+
+# use this session for running the tensorflow model
+hook.set_mode(smd.modes.TRAIN)
+for i in range(args.steps):
+ x_ = np.random.random((10, 2)) * args.scale
+ _loss, opt, gstep = sess.run([loss, optimizer_op, increment_global_step_op], {x: x_})
+ print(f"Step={i}, Loss={_loss}")
+
+# set the mode for monitored session based runs
+# so smdebug can separate out steps by mode
+hook.set_mode(smd.modes.EVAL)
+for i in range(args.steps):
+ x_ = np.random.random((10, 2)) * args.scale
+ sess.run([loss, increment_global_step_op], {x: x_})
diff --git a/examples/tensorflow/sagemaker_official_container/tf_keras_resnet.py b/examples/tensorflow/sagemaker_official_container/tf_keras_resnet.py
new file mode 100644
index 000000000..f3e1930a2
--- /dev/null
+++ b/examples/tensorflow/sagemaker_official_container/tf_keras_resnet.py
@@ -0,0 +1,60 @@
+"""
+This script is a ResNet training script which uses Tensorflow's Keras interface.
+It is designed to be used with SageMaker Debugger in an official SageMaker Framework container (i.e. AWS Deep Learning Container).
+You will notice that this script looks exactly like a normal TensorFlow training script.
+The hook needed by SageMaker Debugger to save tensors during training will be automatically added in those environments.
+The hook will load configuration from json configuration that SageMaker will put in the training container from the configuration provided using the SageMaker python SDK when creating a job.
+For more information, please refer to https://github.com/awslabs/sagemaker-debugger/blob/master/docs/sagemaker.md
+"""
+
+# Standard Library
+import argparse
+
+# Third Party
+import numpy as np
+from tensorflow.keras.applications.resnet50 import ResNet50
+from tensorflow.keras.datasets import cifar10
+from tensorflow.keras.utils import to_categorical
+
+
+def train(batch_size, epoch, model):
+ (X_train, y_train), (X_valid, y_valid) = cifar10.load_data()
+
+ Y_train = to_categorical(y_train, 10)
+ Y_valid = to_categorical(y_valid, 10)
+
+ X_train = X_train.astype("float32")
+ X_valid = X_valid.astype("float32")
+
+ mean_image = np.mean(X_train, axis=0)
+ X_train -= mean_image
+ X_valid -= mean_image
+ X_train /= 128.0
+ X_valid /= 128.0
+
+ model.fit(
+ X_train,
+ Y_train,
+ batch_size=batch_size,
+ epochs=epoch,
+ validation_data=(X_valid, Y_valid),
+ shuffle=True,
+ )
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Train resnet50 cifar10")
+ parser.add_argument("--batch_size", type=int, default=128)
+ parser.add_argument("--epoch", type=int, default=3)
+ parser.add_argument("--model_dir", type=str, default="./model_keras_resnet")
+ opt = parser.parse_args()
+
+ model = ResNet50(weights=None, input_shape=(32, 32, 3), classes=10)
+ model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
+
+ # start the training.
+ train(opt.batch_size, opt.epoch, model)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/tensorflow/scripts/distributed_training/horovod_mnist_estimator.py b/examples/tensorflow/scripts/distributed_training/horovod_mnist_estimator.py
deleted file mode 100644
index a9ac4488f..000000000
--- a/examples/tensorflow/scripts/distributed_training/horovod_mnist_estimator.py
+++ /dev/null
@@ -1,268 +0,0 @@
-# Copyright 2018 Uber Technologies, Inc. All Rights Reserved.
-# Copyright 2016 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Convolutional Neural Network Estimator for MNIST, built with tf.layers."""
-
-# Future
-from __future__ import absolute_import, division, print_function
-
-# Standard Library
-import argparse
-import errno
-import os
-
-# Third Party
-import horovod.tensorflow as hvd
-import numpy as np
-import tensorflow as tf
-from tensorflow import keras
-
-# First Party
-import smdebug.tensorflow as smd
-
-tf.logging.set_verbosity(tf.logging.INFO)
-
-
-def cnn_model_fn(features, labels, mode):
- """Model function for CNN."""
- # Input Layer
- # Reshape X to 4-D tensor: [batch_size, width, height, channels]
- # MNIST images are 28x28 pixels, and have one color channel
- input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])
-
- # Convolutional Layer #1
- # Computes 32 features using a 5x5 filter with ReLU activation.
- # Padding is added to preserve width and height.
- # Input Tensor Shape: [batch_size, 28, 28, 1]
- # Output Tensor Shape: [batch_size, 28, 28, 32]
- conv1 = tf.layers.conv2d(
- inputs=input_layer, filters=32, kernel_size=[5, 5], padding="same", activation=tf.nn.relu
- )
-
- # Pooling Layer #1
- # First max pooling layer with a 2x2 filter and stride of 2
- # Input Tensor Shape: [batch_size, 28, 28, 32]
- # Output Tensor Shape: [batch_size, 14, 14, 32]
- pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)
-
- # Convolutional Layer #2
- # Computes 64 features using a 5x5 filter.
- # Padding is added to preserve width and height.
- # Input Tensor Shape: [batch_size, 14, 14, 32]
- # Output Tensor Shape: [batch_size, 14, 14, 64]
- conv2 = tf.layers.conv2d(
- inputs=pool1, filters=64, kernel_size=[5, 5], padding="same", activation=tf.nn.relu
- )
-
- # Pooling Layer #2
- # Second max pooling layer with a 2x2 filter and stride of 2
- # Input Tensor Shape: [batch_size, 14, 14, 64]
- # Output Tensor Shape: [batch_size, 7, 7, 64]
- pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)
-
- # Flatten tensor into a batch of vectors
- # Input Tensor Shape: [batch_size, 7, 7, 64]
- # Output Tensor Shape: [batch_size, 7 * 7 * 64]
- pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])
-
- # Dense Layer
- # Densely connected layer with 1024 neurons
- # Input Tensor Shape: [batch_size, 7 * 7 * 64]
- # Output Tensor Shape: [batch_size, 1024]
- dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
-
- # Add dropout operation; 0.6 probability that element will be kept
- dropout = tf.layers.dropout(
- inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN
- )
-
- # Logits layer
- # Input Tensor Shape: [batch_size, 1024]
- # Output Tensor Shape: [batch_size, 10]
- logits = tf.layers.dense(inputs=dropout, units=10)
-
- predictions = {
- # Generate predictions (for PREDICT and EVAL mode)
- "classes": tf.argmax(input=logits, axis=1),
- # Add `softmax_tensor` to the graph. It is used for PREDICT and by the
- # `logging_hook`.
- "probabilities": tf.nn.softmax(logits, name="softmax_tensor"),
- }
- if mode == tf.estimator.ModeKeys.PREDICT:
- return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)
-
- # Calculate Loss (for both TRAIN and EVAL modes)
- onehot_labels = tf.one_hot(indices=tf.cast(labels, tf.int32), depth=10)
- loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels, logits=logits)
-
- # Configure the Training Op (for TRAIN mode)
- if mode == tf.estimator.ModeKeys.TRAIN:
- # Horovod: scale learning rate by the number of workers.
- optimizer = tf.train.MomentumOptimizer(learning_rate=0.001 * hvd.size(), momentum=0.9)
-
- # Horovod: add Horovod Distributed Optimizer.
- optimizer = hvd.DistributedOptimizer(optimizer)
-
- # Add smdebug Optimizer
- optimizer = smd.get_hook().wrap_optimizer(optimizer)
-
- train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
- return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)
-
- # Add evaluation metrics (for EVAL mode)
- eval_metric_ops = {
- "accuracy": tf.metrics.accuracy(labels=labels, predictions=predictions["classes"])
- }
- return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)
-
-
-def str2bool(v):
- if isinstance(v, bool):
- return v
-
- if v.lower() in ("yes", "true", "t", "y", "1"):
- return True
- elif v.lower() in ("no", "false", "f", "n", "0"):
- return False
- else:
- raise argparse.ArgumentTypeError("Boolean value expected.")
-
-
-def add_cli_args():
- cmdline = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-
- cmdline.add_argument(
- "--steps", type=int, default=20000, help="""Number of training steps to run."""
- )
-
- cmdline.add_argument("--save_all", type=str2bool, default=True)
- cmdline.add_argument("--smdebug_path", type=str, default="/opt/ml/output/tensors")
- cmdline.add_argument("--save_frequency", type=int, help="How often to save TS data", default=10)
- cmdline.add_argument(
- "--reductions",
- type=str2bool,
- dest="reductions",
- default=False,
- help="save reductions of tensors instead of saving full tensors",
- )
-
- return cmdline
-
-
-def main(unused_argv):
- # Get commandline args
-
- cmdline = add_cli_args()
- FLAGS, unknown_args = cmdline.parse_known_args()
-
- # Horovod: initialize Horovod.
- hvd.init()
-
- # Keras automatically creates a cache directory in ~/.keras/datasets for
- # storing the downloaded MNIST data. This creates a race
- # condition among the workers that share the same filesystem. If the
- # directory already exists by the time this worker gets around to creating
- # it, ignore the resulting exception and continue.
- cache_dir = os.path.join(os.path.expanduser("~"), ".keras", "datasets")
- if not os.path.exists(cache_dir):
- try:
- os.mkdir(cache_dir)
- except OSError as e:
- if e.errno == errno.EEXIST and os.path.isdir(cache_dir):
- pass
- else:
- raise
-
- # Download and load MNIST dataset.
- (train_data, train_labels), (eval_data, eval_labels) = keras.datasets.mnist.load_data(
- "/tmp/MNIST-data-%d" % hvd.rank()
- )
-
- # The shape of downloaded data is (-1, 28, 28), hence we need to reshape it
- # into (-1, 784) to feed into our network. Also, need to normalize the
- # features between 0 and 1.
- train_data = np.reshape(train_data, (-1, 784)) / 255.0
- eval_data = np.reshape(eval_data, (-1, 784)) / 255.0
-
- # Horovod: pin GPU to be used to process local rank (one GPU per process)
- config = tf.ConfigProto()
- config.gpu_options.allow_growth = True
- config.gpu_options.visible_device_list = str(hvd.local_rank())
-
- # Horovod: save checkpoints only on worker 0 to prevent other workers from
- # corrupting them.
- model_dir = "./mnist_convnet_model" if hvd.rank() == 0 else None
-
- # Create the Estimator
- mnist_classifier = tf.estimator.Estimator(
- model_fn=cnn_model_fn,
- model_dir=model_dir,
- config=tf.estimator.RunConfig(session_config=config),
- )
-
- # Set up logging for predictions
- # Log the values in the "Softmax" tensor with label "probabilities"
- tensors_to_log = {"probabilities": "softmax_tensor"}
- logging_hook = tf.train.LoggingTensorHook(tensors=tensors_to_log, every_n_iter=500)
-
- # Horovod: BroadcastGlobalVariablesHook broadcasts initial variable states from
- # rank 0 to all other processes. This is necessary to ensure consistent
- # initialization of all workers when training is started with random weights or
- # restored from a checkpoint.
- bcast_hook = hvd.BroadcastGlobalVariablesHook(0)
-
- # Train the model
- train_input_fn = tf.estimator.inputs.numpy_input_fn(
- x={"x": train_data}, y=train_labels, batch_size=100, num_epochs=None, shuffle=True
- )
-
- # Setup the Tornasole Hook
-
- # save tensors as reductions if necessary
- rdnc = (
- smd.ReductionConfig(reductions=["mean"], abs_reductions=["max"], norms=["l1"])
- if FLAGS.reductions
- else None
- )
-
- ts_hook = smd.SessionHook(
- out_dir=FLAGS.smdebug_path,
- save_all=FLAGS.save_all,
- include_collections=["weights", "gradients", "losses", "biases"],
- save_config=smd.SaveConfig(save_interval=FLAGS.save_frequency),
- reduction_config=rdnc,
- )
-
- ts_hook.set_mode(smd.modes.TRAIN)
-
- # Horovod: adjust number of steps based on number of GPUs.
- mnist_classifier.train(
- input_fn=train_input_fn,
- steps=FLAGS.steps // hvd.size(),
- hooks=[logging_hook, bcast_hook, ts_hook],
- )
-
- # Evaluate the model and print results
- eval_input_fn = tf.estimator.inputs.numpy_input_fn(
- x={"x": eval_data}, y=eval_labels, num_epochs=1, shuffle=False
- )
-
- ts_hook.set_mode(smd.modes.EVAL)
-
- eval_results = mnist_classifier.evaluate(input_fn=eval_input_fn, hooks=[ts_hook])
- print(eval_results)
-
-
-if __name__ == "__main__":
- tf.app.run()
diff --git a/examples/tensorflow/scripts/distributed_training/mirrored_strategy_mnist.py b/examples/tensorflow/scripts/distributed_training/mirrored_strategy_mnist.py
deleted file mode 100644
index e3c48318b..000000000
--- a/examples/tensorflow/scripts/distributed_training/mirrored_strategy_mnist.py
+++ /dev/null
@@ -1,268 +0,0 @@
-# Copyright 2016 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Convolutional Neural Network Estimator for MNIST, built with tf.layers."""
-
-# Future
-from __future__ import absolute_import, division, print_function
-
-# Standard Library
-import argparse
-
-# Third Party
-import numpy as np
-import tensorflow as tf
-from tensorflow.python.client import device_lib
-
-# First Party
-import smdebug.tensorflow as smd
-
-tf.logging.set_verbosity(tf.logging.INFO)
-
-
-def cnn_model_fn(features, labels, mode):
- """Model function for CNN."""
- # Input Layer
- # Reshape X to 4-D tensor: [batch_size, width, height, channels]
- # MNIST images are 28x28 pixels, and have one color channel
- input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])
-
- # Convolutional Layer #1
- # Computes 32 features using a 5x5 filter with ReLU activation.
- # Padding is added to preserve width and height.
- # Input Tensor Shape: [batch_size, 28, 28, 1]
- # Output Tensor Shape: [batch_size, 28, 28, 32]
- conv1 = tf.layers.conv2d(
- inputs=input_layer, filters=32, kernel_size=[5, 5], padding="same", activation=tf.nn.relu
- )
-
- # Pooling Layer #1
- # First max pooling layer with a 2x2 filter and stride of 2
- # Input Tensor Shape: [batch_size, 28, 28, 32]
- # Output Tensor Shape: [batch_size, 14, 14, 32]
- pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)
-
- # Convolutional Layer #2
- # Computes 64 features using a 5x5 filter.
- # Padding is added to preserve width and height.
- # Input Tensor Shape: [batch_size, 14, 14, 32]
- # Output Tensor Shape: [batch_size, 14, 14, 64]
- conv2 = tf.layers.conv2d(
- inputs=pool1, filters=64, kernel_size=[5, 5], padding="same", activation=tf.nn.relu
- )
-
- # Pooling Layer #2
- # Second max pooling layer with a 2x2 filter and stride of 2
- # Input Tensor Shape: [batch_size, 14, 14, 64]
- # Output Tensor Shape: [batch_size, 7, 7, 64]
- pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)
-
- # Flatten tensor into a batch of vectors
- # Input Tensor Shape: [batch_size, 7, 7, 64]
- # Output Tensor Shape: [batch_size, 7 * 7 * 64]
- pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])
-
- # Dense Layer
- # Densely connected layer with 1024 neurons
- # Input Tensor Shape: [batch_size, 7 * 7 * 64]
- # Output Tensor Shape: [batch_size, 1024]
- dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
-
- # Add dropout operation; 0.6 probability that element will be kept
- dropout = tf.layers.dropout(
- inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN
- )
-
- # Logits layer
- # Input Tensor Shape: [batch_size, 1024]
- # Output Tensor Shape: [batch_size, 10]
- logits = tf.layers.dense(inputs=dropout, units=10)
-
- predictions = {
- # Generate predictions (for PREDICT and EVAL mode)
- "classes": tf.argmax(input=logits, axis=1),
- # Add `softmax_tensor` to the graph. It is used for PREDICT and by the
- # `logging_hook`.
- "probabilities": tf.nn.softmax(logits, name="softmax_tensor"),
- }
- if mode == tf.estimator.ModeKeys.PREDICT:
- return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)
-
- # Calculate Loss (for both TRAIN and EVAL modes)
- loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)
-
- # Configure the Training Op (for TRAIN mode)
- if mode == tf.estimator.ModeKeys.TRAIN:
- optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
- optimizer = smd.get_hook().wrap_optimizer(optimizer)
- train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
- return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)
-
- # Add evaluation metrics (for EVAL mode)
- eval_metric_ops = {
- "accuracy": tf.metrics.accuracy(labels=labels, predictions=predictions["classes"])
- }
- return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)
-
-
-def per_device_batch_size(batch_size, num_gpus):
- """For multi-gpu, batch-size must be a multiple of the number of GPUs.
- Note that this should eventually be handled by DistributionStrategies
- directly. Multi-GPU support is currently experimental, however,
- so doing the work here until that feature is in place.
- Args:
- batch_size: Global batch size to be divided among devices. This should be
- equal to num_gpus times the single-GPU batch_size for multi-gpu training.
- num_gpus: How many GPUs are used with DistributionStrategies.
- Returns:
- Batch size per device.
- Raises:
- ValueError: if batch_size is not divisible by number of devices
- """
- if num_gpus <= 1:
- return batch_size
-
- remainder = batch_size % num_gpus
- if remainder:
- err = (
- "When running with multiple GPUs, batch size "
- "must be a multiple of the number of available GPUs. Found {} "
- "GPUs with a batch size of {}; try --batch_size={} instead."
- ).format(num_gpus, batch_size, batch_size - remainder)
- raise ValueError(err)
- return int(batch_size / num_gpus)
-
-
-class InputFnProvider:
- def __init__(self, train_batch_size):
- self.train_batch_size = train_batch_size
- self.__load_data()
-
- def __load_data(self):
- # Load training and eval data
- mnist = tf.contrib.learn.datasets.load_dataset("mnist")
- self.train_data = mnist.train.images # Returns np.array
- self.train_labels = np.asarray(mnist.train.labels, dtype=np.int32)
- self.eval_data = mnist.test.images # Returns np.array
- self.eval_labels = np.asarray(mnist.test.labels, dtype=np.int32)
-
- def train_input_fn(self):
- """An input function for training"""
- # Shuffle, repeat, and batch the examples.
- dataset = tf.data.Dataset.from_tensor_slices(({"x": self.train_data}, self.train_labels))
- dataset = dataset.shuffle(1000).repeat().batch(self.train_batch_size)
- return dataset
-
- def eval_input_fn(self):
- """An input function for evaluation or prediction"""
- dataset = tf.data.Dataset.from_tensor_slices(({"x": self.eval_data}, self.eval_labels))
- dataset = dataset.batch(1)
- return dataset
-
-
-def str2bool(v):
- if isinstance(v, bool):
- return v
-
- if v.lower() in ("yes", "true", "t", "y", "1"):
- return True
- elif v.lower() in ("no", "false", "f", "n", "0"):
- return False
- else:
- raise argparse.ArgumentTypeError("Boolean value expected.")
-
-
-def add_cli_args():
- cmdline = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-
- cmdline.add_argument(
- "--steps", type=int, default=20000, help="""Number of training steps to run."""
- )
-
- cmdline.add_argument("--save_all", type=str2bool, default=True)
- cmdline.add_argument("--smdebug_path", type=str, default="/opt/ml/output/tensors")
- cmdline.add_argument("--save_frequency", type=int, help="How often to save TS data", default=10)
- cmdline.add_argument(
- "--reductions",
- type=str2bool,
- dest="reductions",
- default=False,
- help="save reductions of tensors instead of saving full tensors",
- )
-
- return cmdline
-
-
-def get_available_gpus():
- local_device_protos = device_lib.list_local_devices()
- return len([x.name for x in local_device_protos if x.device_type == "GPU"])
-
-
-def main(unused_argv):
- num_gpus = get_available_gpus()
- batch_size = 10 * num_gpus
-
- cmdline = add_cli_args()
- FLAGS, unknown_args = cmdline.parse_known_args()
-
- # input_fn which serves Dataset
- input_fn_provider = InputFnProvider(per_device_batch_size(batch_size, num_gpus))
-
- # Use multiple GPUs by MirroredStragtegy.
- # All avaiable GPUs will be used if `num_gpus` is omitted.
- if num_gpus > 1:
- distribution = tf.contrib.distribute.MirroredStrategy(num_gpus=num_gpus)
- print("### Doing Multi GPU Training")
- else:
- distribution = None
- # Pass to RunConfig
- config = tf.estimator.RunConfig(
- train_distribute=distribution, model_dir="/tmp/mnist_convnet_model"
- )
-
- # save tensors as reductions if necessary
- rdnc = (
- smd.ReductionConfig(reductions=["mean"], abs_reductions=["max"], norms=["l1"])
- if FLAGS.reductions
- else None
- )
-
- ts_hook = smd.SessionHook(
- out_dir=FLAGS.smdebug_path,
- save_all=FLAGS.save_all,
- include_collections=["weights", "gradients", "losses", "biases"],
- save_config=smd.SaveConfig(save_interval=FLAGS.save_frequency),
- reduction_config=rdnc,
- )
-
- ts_hook.set_mode(smd.modes.TRAIN)
-
- # Create the Estimator
- # pass RunConfig
- mnist_classifier = tf.estimator.Estimator(model_fn=cnn_model_fn, config=config)
-
- # Train the model
- mnist_classifier.train(
- input_fn=input_fn_provider.train_input_fn, steps=FLAGS.steps, hooks=[ts_hook]
- )
-
- ts_hook.set_mode(smd.modes.EVAL)
- # Evaluate the model and print results
- eval_results = mnist_classifier.evaluate(
- input_fn=input_fn_provider.eval_input_fn, hooks=[ts_hook]
- )
- print(eval_results)
-
-
-if __name__ == "__main__":
- tf.app.run()
diff --git a/examples/tensorflow/scripts/distributed_training/parameter_server_training/hostfile.txt b/examples/tensorflow/scripts/distributed_training/parameter_server_training/hostfile.txt
deleted file mode 100644
index abe577993..000000000
--- a/examples/tensorflow/scripts/distributed_training/parameter_server_training/hostfile.txt
+++ /dev/null
@@ -1,3 +0,0 @@
-172.31.26.105:6665
-172.31.26.105:6666
-172.31.26.105:6667
diff --git a/examples/tensorflow/scripts/distributed_training/parameter_server_training/parameter_server_mnist.py b/examples/tensorflow/scripts/distributed_training/parameter_server_training/parameter_server_mnist.py
deleted file mode 100644
index a57dfd19a..000000000
--- a/examples/tensorflow/scripts/distributed_training/parameter_server_training/parameter_server_mnist.py
+++ /dev/null
@@ -1,276 +0,0 @@
-# Copyright 2016 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Convolutional Neural Network Estimator for MNIST, built with tf.layers."""
-
-# Future
-from __future__ import absolute_import, division, print_function
-
-# Standard Library
-import argparse
-import json
-import os
-
-# Third Party
-import numpy as np
-import tensorflow as tf
-from tensorflow.python.client import device_lib
-
-# First Party
-import smdebug.tensorflow as smd
-
-tf.logging.set_verbosity(tf.logging.INFO)
-
-
-def cnn_model_fn(features, labels, mode):
- """Model function for CNN."""
- input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])
-
- conv1 = tf.layers.conv2d(
- inputs=input_layer, filters=32, kernel_size=[5, 5], padding="same", activation=tf.nn.relu
- )
-
- pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)
-
- conv2 = tf.layers.conv2d(
- inputs=pool1, filters=64, kernel_size=[5, 5], padding="same", activation=tf.nn.relu
- )
-
- pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)
-
- pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])
-
- dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
-
- dropout = tf.layers.dropout(
- inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN
- )
-
- logits = tf.layers.dense(inputs=dropout, units=10)
-
- predictions = {
- # Generate predictions (for PREDICT and EVAL mode)
- "classes": tf.argmax(input=logits, axis=1),
- # Add `softmax_tensor` to the graph. It is used for PREDICT and by the
- # `logging_hook`.
- "probabilities": tf.nn.softmax(logits, name="softmax_tensor"),
- }
- if mode == tf.estimator.ModeKeys.PREDICT:
- return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)
-
- # Calculate Loss (for both TRAIN and EVAL modes)
- loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)
-
- # Configure the Training Op (for TRAIN mode)
- if mode == tf.estimator.ModeKeys.TRAIN:
- optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
- optimizer = smd.get_hook().wrap_optimizer(optimizer)
- train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
- return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)
-
- # Add evaluation metrics (for EVAL mode)
- eval_metric_ops = {
- "accuracy": tf.metrics.accuracy(labels=labels, predictions=predictions["classes"])
- }
- return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)
-
-
-def per_device_batch_size(batch_size, num_gpus):
- """For multi-gpu, batch-size must be a multiple of the number of GPUs.
- Note that this should eventually be handled by DistributionStrategies
- directly. Multi-GPU support is currently experimental, however,
- so doing the work here until that feature is in place.
- Args:
- batch_size: Global batch size to be divided among devices. This should be
- equal to num_gpus times the single-GPU batch_size for multi-gpu training.
- num_gpus: How many GPUs are used with DistributionStrategies.
- Returns:
- Batch size per device.
- Raises:
- ValueError: if batch_size is not divisible by number of devices
- """
- if num_gpus <= 1:
- return batch_size
-
- remainder = batch_size % num_gpus
- if remainder:
- err = (
- "When running with multiple GPUs, batch size "
- "must be a multiple of the number of available GPUs. Found {} "
- "GPUs with a batch size of {}; try --batch_size={} instead."
- ).format(num_gpus, batch_size, batch_size - remainder)
- raise ValueError(err)
- return int(batch_size / num_gpus)
-
-
-class InputFnProvider:
- def __init__(self, train_batch_size):
- self.train_batch_size = train_batch_size
- self.__load_data()
-
- def __load_data(self):
- # Load training and eval data
- mnist = tf.contrib.learn.datasets.load_dataset("mnist")
- self.train_data = mnist.train.images # Returns np.array
- self.train_labels = np.asarray(mnist.train.labels, dtype=np.int32)
- self.eval_data = mnist.test.images # Returns np.array
- self.eval_labels = np.asarray(mnist.test.labels, dtype=np.int32)
-
- def train_input_fn(self):
- """An input function for training"""
- # Shuffle, repeat, and batch the examples.
- dataset = tf.data.Dataset.from_tensor_slices(({"x": self.train_data}, self.train_labels))
- dataset = dataset.shuffle(1000).repeat().batch(self.train_batch_size)
- return dataset
-
- def eval_input_fn(self):
- """An input function for evaluation or prediction"""
- dataset = tf.data.Dataset.from_tensor_slices(({"x": self.eval_data}, self.eval_labels))
- dataset = dataset.batch(1)
- return dataset
-
-
-def str2bool(v):
- if isinstance(v, bool):
- return v
-
- if v.lower() in ("yes", "true", "t", "y", "1"):
- return True
- elif v.lower() in ("no", "false", "f", "n", "0"):
- return False
- else:
- raise argparse.ArgumentTypeError("Boolean value expected.")
-
-
-def add_cli_args():
- cmdline = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-
- cmdline.add_argument(
- "--steps", type=int, default=20000, help="""Number of training steps to run."""
- )
-
- cmdline.add_argument("--save_all", type=str2bool, default=True)
- cmdline.add_argument("--smdebug_path", type=str, default="/opt/ml/output/tensors")
- cmdline.add_argument("--save_frequency", type=int, help="How often to save TS data", default=10)
- cmdline.add_argument(
- "--reductions",
- type=str2bool,
- dest="reductions",
- default=False,
- help="save reductions of tensors instead of saving full tensors",
- )
-
- cmdline.add_argument(
- "--node_type", type=str, required=True, dest="node_type", help="node type: worker or ps"
- )
-
- cmdline.add_argument(
- "--task_index", type=int, required=True, dest="task_index", help="task index"
- )
-
- cmdline.add_argument(
- "--hostfile",
- default=None,
- type=str,
- required=False,
- dest="hostfile",
- help="Path to hostfile",
- )
-
- return cmdline
-
-
-def get_available_gpus():
- local_device_protos = device_lib.list_local_devices()
- return len([x.name for x in local_device_protos if x.device_type == "GPU"])
-
-
-def main(unused_argv):
- num_gpus = get_available_gpus()
- batch_size = 10 * num_gpus
-
- cmdline = add_cli_args()
- FLAGS, unknown_args = cmdline.parse_known_args()
-
- # input_fn which serves Dataset
- input_fn_provider = InputFnProvider(per_device_batch_size(batch_size, num_gpus))
-
- # Use multiple GPUs by ParameterServerStrategy.
- # All avaiable GPUs will be used if `num_gpus` is omitted.
-
- if num_gpus > 1:
- strategy = tf.distribute.experimental.ParameterServerStrategy()
- if not os.getenv("TF_CONFIG"):
- if FLAGS.hostfile is None:
- raise Exception("--hostfile not provided and TF_CONFIG not set. Please do either.")
- nodes = list()
- try:
- f = open(FLAGS.hostfile)
- for line in f.readlines():
- nodes.append(line.strip())
- except OSError as e:
- print(e.errno)
-
- os.environ["TF_CONFIG"] = json.dumps(
- {
- "cluster": {"worker": [nodes[0], nodes[1]], "ps": [nodes[2]]},
- "task": {"type": FLAGS.node_type, "index": FLAGS.task_index},
- }
- )
-
- print("### Doing Multi GPU Training")
- else:
- strategy = None
- # Pass to RunConfig
- config = tf.estimator.RunConfig(train_distribute=strategy)
-
- # save tensors as reductions if necessary
- rdnc = (
- smd.ReductionConfig(reductions=["mean"], abs_reductions=["max"], norms=["l1"])
- if FLAGS.reductions
- else None
- )
-
- ts_hook = smd.SessionHook(
- out_dir=FLAGS.smdebug_path,
- save_all=FLAGS.save_all,
- include_collections=["weights", "gradients", "losses", "biases"],
- save_config=smd.SaveConfig(save_interval=FLAGS.save_frequency),
- reduction_config=rdnc,
- )
-
- ts_hook.set_mode(smd.modes.TRAIN)
-
- # Create the Estimator
- # pass RunConfig
- mnist_classifier = tf.estimator.Estimator(model_fn=cnn_model_fn, config=config)
-
- hooks = list()
- hooks.append(ts_hook)
-
- train_spec = tf.estimator.TrainSpec(
- input_fn=input_fn_provider.train_input_fn, max_steps=FLAGS.steps, hooks=hooks
- )
- eval_spec = tf.estimator.EvalSpec(
- input_fn=input_fn_provider.eval_input_fn, steps=FLAGS.steps, hooks=hooks
- )
-
- tf.estimator.train_and_evaluate(mnist_classifier, train_spec, eval_spec)
-
- # Evaluate the model and print results
- eval_results = mnist_classifier.evaluate(input_fn=input_fn_provider.eval_input_fn)
- print(eval_results)
-
-
-if __name__ == "__main__":
- tf.app.run()
diff --git a/examples/tensorflow/scripts/keras.py b/examples/tensorflow/scripts/keras.py
deleted file mode 100644
index 0fe13d919..000000000
--- a/examples/tensorflow/scripts/keras.py
+++ /dev/null
@@ -1,84 +0,0 @@
-# Future
-from __future__ import absolute_import, division, print_function, unicode_literals
-
-# Third Party
-import tensorflow as tf
-import tensorflow_datasets as tfds
-
-# First Party
-from smdebug.core.collection import CollectionKeys
-from smdebug.tensorflow import KerasHook
-
-tfds.disable_progress_bar()
-
-
-def train_model():
- print(tf.__version__)
-
- datasets, info = tfds.load(name="mnist", with_info=True, as_supervised=True)
-
- mnist_train, mnist_test = datasets["train"], datasets["test"]
-
- strategy = tf.distribute.MirroredStrategy()
-
- # You can also do info.splits.total_num_examples to get the total
- # number of examples in the dataset.
-
- num_train_examples = info.splits["train"].num_examples
- num_test_examples = info.splits["test"].num_examples
-
- BUFFER_SIZE = 10000
-
- BATCH_SIZE_PER_REPLICA = 64
- BATCH_SIZE = BATCH_SIZE_PER_REPLICA * strategy.num_replicas_in_sync
-
- def scale(image, label):
- image = tf.cast(image, tf.float32)
- image /= 255
-
- return image, label
-
- train_dataset = mnist_train.map(scale).cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
- eval_dataset = mnist_test.map(scale).batch(BATCH_SIZE)
-
- hook = KerasHook(
- out_dir="/tmp/ts_outputs/",
- include_collections=[
- # CollectionKeys.WEIGHTS,
- # CollectionKeys.GRADIENTS,
- # CollectionKeys.OPTIMIZER_VARIABLES,
- CollectionKeys.DEFAULT,
- # CollectionKeys.METRICS,
- # CollectionKeys.LOSSES,
- # CollectionKeys.OUTPUTS,
- # CollectionKeys.SCALARS,
- ],
- save_all=True,
- )
-
- with strategy.scope():
- model = tf.keras.Sequential(
- [
- tf.keras.layers.Conv2D(32, 3, activation="relu", input_shape=(28, 28, 1)),
- tf.keras.layers.MaxPooling2D(),
- tf.keras.layers.Flatten(),
- tf.keras.layers.Dense(64, activation="relu"),
- tf.keras.layers.Dense(10, activation="softmax"),
- ]
- )
- model.compile(
- loss="sparse_categorical_crossentropy",
- optimizer=hook.wrap_optimizer(tf.keras.optimizers.Adam()),
- metrics=["accuracy"],
- )
-
- # get_collection('default').include('Relu')
-
- callbacks = [
- hook
- # tf.keras.callbacks.TensorBoard(log_dir='/tmp/logs'),
- ]
-
- model.fit(train_dataset, epochs=1, callbacks=callbacks)
- model.predict(eval_dataset, callbacks=callbacks)
- model.fit(train_dataset, epochs=1, callbacks=callbacks)
diff --git a/examples/tensorflow/scripts/train_imagenet_resnet_hvd.py b/examples/tensorflow/scripts/train_imagenet_resnet_hvd.py
deleted file mode 100644
index f1a1ae098..000000000
--- a/examples/tensorflow/scripts/train_imagenet_resnet_hvd.py
+++ /dev/null
@@ -1,1502 +0,0 @@
-#!/usr/bin/env python
-# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions
-# are met:
-# * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-# * Redistributions in binary form must reproduce the above copyright
-# notice, this list of conditions and the following disclaimer in the
-# documentation and/or other materials provided with the distribution.
-# * Neither the name of NVIDIA CORPORATION nor the names of its
-# contributors may be used to endorse or promote products derived
-# from this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
-# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
-# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
-# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
-# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
-# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-# Future
-from __future__ import print_function
-
-# Standard Library
-import argparse
-import logging
-import math
-import os
-import random
-import re
-import shutil
-import sys
-import time
-from operator import itemgetter
-
-# Third Party
-import horovod.tensorflow as hvd
-import numpy as np
-import tensorflow as tf
-from tensorflow.contrib.image.python.ops import distort_image_ops
-from tensorflow.python.ops import data_flow_ops
-from tensorflow.python.util import nest
-
-# First Party
-import smdebug.tensorflow as smd
-
-try:
- from builtins import range
-except ImportError:
- pass
-
-
-def rank0log(logger, *args, **kwargs):
- if hvd.rank() == 0:
- if logger:
- logger.info("".join([str(x) for x in list(args)]))
- else:
- print(*args, **kwargs)
-
-
-class LayerBuilder(object):
- def __init__(
- self,
- activation=None,
- data_format="channels_last",
- training=False,
- use_batch_norm=False,
- batch_norm_config=None,
- conv_initializer=None,
- adv_bn_init=False,
- ):
- self.activation = activation
- self.data_format = data_format
- self.training = training
- self.use_batch_norm = use_batch_norm
- self.batch_norm_config = batch_norm_config
- self.conv_initializer = conv_initializer
- self.adv_bn_init = adv_bn_init
- if self.batch_norm_config is None:
- self.batch_norm_config = {
- "decay": 0.9,
- "epsilon": 1e-4,
- "scale": True,
- "zero_debias_moving_mean": False,
- }
-
- def _conv2d(self, inputs, activation, *args, **kwargs):
- x = tf.layers.conv2d(
- inputs,
- data_format=self.data_format,
- use_bias=not self.use_batch_norm,
- kernel_initializer=self.conv_initializer,
- activation=None if self.use_batch_norm else activation,
- *args,
- **kwargs
- )
- if self.use_batch_norm:
- x = self.batch_norm(x)
- x = activation(x) if activation is not None else x
- return x
-
- def conv2d_linear_last_bn(self, inputs, *args, **kwargs):
- x = tf.layers.conv2d(
- inputs,
- data_format=self.data_format,
- use_bias=False,
- kernel_initializer=self.conv_initializer,
- activation=None,
- *args,
- **kwargs
- )
- param_initializers = {
- "moving_mean": tf.zeros_initializer(),
- "moving_variance": tf.ones_initializer(),
- "beta": tf.zeros_initializer(),
- }
- if self.adv_bn_init:
- param_initializers["gamma"] = tf.zeros_initializer()
- else:
- param_initializers["gamma"] = tf.ones_initializer()
- x = self.batch_norm(x, param_initializers=param_initializers)
- return x
-
- def conv2d_linear(self, inputs, *args, **kwargs):
- return self._conv2d(inputs, None, *args, **kwargs)
-
- def conv2d(self, inputs, *args, **kwargs):
- return self._conv2d(inputs, self.activation, *args, **kwargs)
-
- def pad2d(self, inputs, begin, end=None):
- if end is None:
- end = begin
- try:
- _ = begin[1]
- except TypeError:
- begin = [begin, begin]
- try:
- _ = end[1]
- except TypeError:
- end = [end, end]
- if self.data_format == "channels_last":
- padding = [[0, 0], [begin[0], end[0]], [begin[1], end[1]], [0, 0]]
- else:
- padding = [[0, 0], [0, 0], [begin[0], end[0]], [begin[1], end[1]]]
- return tf.pad(inputs, padding)
-
- def max_pooling2d(self, inputs, *args, **kwargs):
- return tf.layers.max_pooling2d(inputs, data_format=self.data_format, *args, **kwargs)
-
- def average_pooling2d(self, inputs, *args, **kwargs):
- return tf.layers.average_pooling2d(inputs, data_format=self.data_format, *args, **kwargs)
-
- def dense_linear(self, inputs, units, **kwargs):
- return tf.layers.dense(inputs, units, activation=None)
-
- def dense(self, inputs, units, **kwargs):
- return tf.layers.dense(inputs, units, activation=self.activation)
-
- def activate(self, inputs, activation=None):
- activation = activation or self.activation
- return activation(inputs) if activation is not None else inputs
-
- def batch_norm(self, inputs, **kwargs):
- all_kwargs = dict(self.batch_norm_config)
- all_kwargs.update(kwargs)
- data_format = "NHWC" if self.data_format == "channels_last" else "NCHW"
- return tf.contrib.layers.batch_norm(
- inputs, is_training=self.training, data_format=data_format, fused=True, **all_kwargs
- )
-
- def spatial_average2d(self, inputs):
- shape = inputs.get_shape().as_list()
- if self.data_format == "channels_last":
- n, h, w, c = shape
- else:
- n, c, h, w = shape
- n = -1 if n is None else n
- x = tf.layers.average_pooling2d(inputs, (h, w), (1, 1), data_format=self.data_format)
- return tf.reshape(x, [n, c])
-
- def flatten2d(self, inputs):
- x = inputs
- if self.data_format != "channel_last":
- # Note: This ensures the output order matches that of NHWC networks
- x = tf.transpose(x, [0, 2, 3, 1])
- input_shape = x.get_shape().as_list()
- num_inputs = 1
- for dim in input_shape[1:]:
- num_inputs *= dim
- return tf.reshape(x, [-1, num_inputs], name="flatten")
-
- def residual2d(self, inputs, network, units=None, scale=1.0, activate=False):
- outputs = network(inputs)
- c_axis = -1 if self.data_format == "channels_last" else 1
- h_axis = 1 if self.data_format == "channels_last" else 2
- w_axis = h_axis + 1
- ishape, oshape = [y.get_shape().as_list() for y in [inputs, outputs]]
- ichans, ochans = ishape[c_axis], oshape[c_axis]
- strides = (
- (ishape[h_axis] - 1) // oshape[h_axis] + 1,
- (ishape[w_axis] - 1) // oshape[w_axis] + 1,
- )
- with tf.name_scope("residual"):
- if ochans != ichans or strides[0] != 1 or strides[1] != 1:
- inputs = self.conv2d_linear(inputs, units, 1, strides, "SAME")
- x = inputs + scale * outputs
- if activate:
- x = self.activate(x)
- return x
-
-
-def resnet_bottleneck_v1(builder, inputs, depth, depth_bottleneck, stride, basic=False):
- num_inputs = inputs.get_shape().as_list()[1]
- x = inputs
- with tf.name_scope("resnet_v1"):
- if depth == num_inputs:
- if stride == 1:
- shortcut = x
- else:
- shortcut = builder.max_pooling2d(x, 1, stride)
- else:
- shortcut = builder.conv2d_linear(x, depth, 1, stride, "SAME")
- if basic:
- x = builder.pad2d(x, 1)
- x = builder.conv2d(x, depth_bottleneck, 3, stride, "VALID")
- x = builder.conv2d_linear(x, depth, 3, 1, "SAME")
- else:
- x = builder.conv2d(x, depth_bottleneck, 1, 1, "SAME")
- x = builder.conv2d(x, depth_bottleneck, 3, stride, "SAME")
- # x = builder.conv2d_linear(x, depth, 1, 1, 'SAME')
- x = builder.conv2d_linear_last_bn(x, depth, 1, 1, "SAME")
- x = tf.nn.relu(x + shortcut)
- smd.get_hook().add_to_collection("relu_activations", x)
- return x
-
-
-def inference_resnet_v1_impl(builder, inputs, layer_counts, basic=False):
- x = inputs
- x = builder.pad2d(x, 3)
- x = builder.conv2d(x, 64, 7, 2, "VALID")
- x = builder.max_pooling2d(x, 3, 2, "SAME")
- for i in range(layer_counts[0]):
- x = resnet_bottleneck_v1(builder, x, 256, 64, 1, basic)
- for i in range(layer_counts[1]):
- x = resnet_bottleneck_v1(builder, x, 512, 128, 2 if i == 0 else 1, basic)
- for i in range(layer_counts[2]):
- x = resnet_bottleneck_v1(builder, x, 1024, 256, 2 if i == 0 else 1, basic)
- for i in range(layer_counts[3]):
- x = resnet_bottleneck_v1(builder, x, 2048, 512, 2 if i == 0 else 1, basic)
- return builder.spatial_average2d(x)
-
-
-def inference_resnet_v1(
- inputs,
- nlayer,
- data_format="channels_last",
- training=False,
- conv_initializer=None,
- adv_bn_init=False,
-):
- """Deep Residual Networks family of models
- https://arxiv.org/abs/1512.03385
- """
- builder = LayerBuilder(
- tf.nn.relu,
- data_format,
- training,
- use_batch_norm=True,
- conv_initializer=conv_initializer,
- adv_bn_init=adv_bn_init,
- )
- if nlayer == 18:
- return inference_resnet_v1_impl(builder, inputs, [2, 2, 2, 2], basic=True)
- elif nlayer == 34:
- return inference_resnet_v1_impl(builder, inputs, [3, 4, 6, 3], basic=True)
- elif nlayer == 50:
- return inference_resnet_v1_impl(builder, inputs, [3, 4, 6, 3])
- elif nlayer == 101:
- return inference_resnet_v1_impl(builder, inputs, [3, 4, 23, 3])
- elif nlayer == 152:
- return inference_resnet_v1_impl(builder, inputs, [3, 8, 36, 3])
- else:
- raise ValueError("Invalid nlayer (%i); must be one of: 18,34,50,101,152" % nlayer)
-
-
-def get_model_func(model_name):
- if model_name.startswith("resnet"):
- nlayer = int(model_name[len("resnet") :])
- return lambda images, *args, **kwargs: inference_resnet_v1(images, nlayer, *args, **kwargs)
- else:
- raise ValueError("Invalid model type: %s" % model_name)
-
-
-def deserialize_image_record(record):
- feature_map = {
- "image/encoded": tf.FixedLenFeature([], tf.string, ""),
- "image/class/label": tf.FixedLenFeature([1], tf.int64, -1),
- "image/class/text": tf.FixedLenFeature([], tf.string, ""),
- "image/object/bbox/xmin": tf.VarLenFeature(dtype=tf.float32),
- "image/object/bbox/ymin": tf.VarLenFeature(dtype=tf.float32),
- "image/object/bbox/xmax": tf.VarLenFeature(dtype=tf.float32),
- "image/object/bbox/ymax": tf.VarLenFeature(dtype=tf.float32),
- }
- with tf.name_scope("deserialize_image_record"):
- obj = tf.parse_single_example(record, feature_map)
- imgdata = obj["image/encoded"]
- label = tf.cast(obj["image/class/label"], tf.int32)
- bbox = tf.stack(
- [obj["image/object/bbox/%s" % x].values for x in ["ymin", "xmin", "ymax", "xmax"]]
- )
- bbox = tf.transpose(tf.expand_dims(bbox, 0), [0, 2, 1])
- text = obj["image/class/text"]
- return imgdata, label, bbox, text
-
-
-def decode_jpeg(imgdata, channels=3):
- return tf.image.decode_jpeg(
- imgdata, channels=channels, fancy_upscaling=False, dct_method="INTEGER_FAST"
- )
-
-
-def crop_and_resize_image(image, original_bbox, height, width, distort=False, nsummary=10):
- with tf.name_scope("crop_and_resize"):
- # Evaluation is done on a center-crop of this ratio
- eval_crop_ratio = 0.8
- if distort:
- initial_shape = [
- int(round(height / eval_crop_ratio)),
- int(round(width / eval_crop_ratio)),
- 3,
- ]
- bbox_begin, bbox_size, bbox = tf.image.sample_distorted_bounding_box(
- initial_shape,
- bounding_boxes=tf.constant([0.0, 0.0, 1.0, 1.0], dtype=tf.float32, shape=[1, 1, 4]),
- min_object_covered=0.1,
- aspect_ratio_range=[3.0 / 4.0, 4.0 / 3.0],
- area_range=[0.08, 1.0],
- max_attempts=100,
- seed=11 * hvd.rank(),
- # Need to set for deterministic results
- use_image_if_no_bounding_boxes=True,
- )
- bbox = bbox[0, 0] # Remove batch, box_idx dims
- else:
- # Central crop
- ratio_y = ratio_x = eval_crop_ratio
- bbox = tf.constant(
- [0.5 * (1 - ratio_y), 0.5 * (1 - ratio_x), 0.5 * (1 + ratio_y), 0.5 * (1 + ratio_x)]
- )
- image = tf.image.crop_and_resize(image[None, :, :, :], bbox[None, :], [0], [height, width])[
- 0
- ]
- return image
-
-
-def parse_and_preprocess_image_record(
- record,
- counter,
- height,
- width,
- brightness,
- contrast,
- saturation,
- hue,
- distort=False,
- nsummary=10,
- increased_aug=False,
-):
- imgdata, label, bbox, text = deserialize_image_record(record)
- label -= 1 # Change to 0-based (don't use background class)
- with tf.name_scope("preprocess_train"):
- try:
- image = decode_jpeg(imgdata, channels=3)
- except:
- image = tf.image.decode_png(imgdata, channels=3)
- image = crop_and_resize_image(image, bbox, height, width, distort)
- if distort:
- image = tf.image.random_flip_left_right(image)
- if increased_aug:
- image = tf.image.random_brightness(image, max_delta=brightness)
- image = distort_image_ops.random_hsv_in_yiq(
- image,
- lower_saturation=saturation,
- upper_saturation=2.0 - saturation,
- max_delta_hue=hue * math.pi,
- )
- image = tf.image.random_contrast(image, lower=contrast, upper=2.0 - contrast)
- tf.summary.image("distorted_color_image", tf.expand_dims(image, 0))
- image = tf.clip_by_value(image, 0.0, 255.0)
- image = tf.cast(image, tf.uint8)
- return image, label
-
-
-def make_dataset(
- filenames,
- take_count,
- batch_size,
- height,
- width,
- brightness,
- contrast,
- saturation,
- hue,
- training=False,
- num_threads=10,
- nsummary=10,
- shard=False,
- synthetic=False,
- increased_aug=False,
-):
- if synthetic and training:
- input_shape = [height, width, 3]
- input_element = nest.map_structure(
- lambda s: tf.constant(0.5, tf.float32, s), tf.TensorShape(input_shape)
- )
- label_element = nest.map_structure(
- lambda s: tf.constant(1, tf.int32, s), tf.TensorShape([1])
- )
- element = (input_element, label_element)
- ds = tf.data.Dataset.from_tensors(element).repeat()
- else:
- shuffle_buffer_size = 10000
- num_readers = 1
- if hvd.size() > len(filenames):
- assert (hvd.size() % len(filenames)) == 0
- filenames = filenames * (hvd.size() / len(filenames))
-
- ds = tf.data.Dataset.from_tensor_slices(filenames)
- if shard:
- # split the dataset into parts for each GPU
- ds = ds.shard(hvd.size(), hvd.rank())
-
- if not training:
- # make sure all ranks have the same amount
- ds = ds.take(take_count)
-
- if training:
- ds = ds.shuffle(1000, seed=7 * (1 + hvd.rank()))
-
- ds = ds.interleave(tf.data.TFRecordDataset, cycle_length=num_readers, block_length=1)
- counter = tf.data.Dataset.range(sys.maxsize)
- ds = tf.data.Dataset.zip((ds, counter))
- preproc_func = lambda record, counter_: parse_and_preprocess_image_record(
- record,
- counter_,
- height,
- width,
- brightness,
- contrast,
- saturation,
- hue,
- distort=training,
- nsummary=nsummary if training else 0,
- increased_aug=increased_aug,
- )
- ds = ds.map(preproc_func, num_parallel_calls=num_threads)
- if training:
- ds = ds.apply(
- tf.data.experimental.shuffle_and_repeat(
- shuffle_buffer_size, seed=5 * (1 + hvd.rank())
- )
- )
- ds = ds.batch(batch_size)
- return ds
-
-
-def stage(tensors):
- """Stages the given tensors in a StagingArea for asynchronous put/get.
- """
- stage_area = data_flow_ops.StagingArea(
- dtypes=[tensor.dtype for tensor in tensors],
- shapes=[tensor.get_shape() for tensor in tensors],
- )
- put_op = stage_area.put(tensors)
- get_tensors = stage_area.get()
- tf.add_to_collection("STAGING_AREA_PUTS", put_op)
- return put_op, get_tensors
-
-
-class PrefillStagingAreasHook(tf.train.SessionRunHook):
- def after_create_session(self, session, coord):
- enqueue_ops = tf.get_collection("STAGING_AREA_PUTS")
- for i in range(len(enqueue_ops)):
- session.run(enqueue_ops[: i + 1])
-
-
-class LogSessionRunHook(tf.train.SessionRunHook):
- def __init__(self, global_batch_size, num_records, display_every=10, logger=None):
- self.global_batch_size = global_batch_size
- self.num_records = num_records
- self.display_every = display_every
- self.logger = logger
-
- def after_create_session(self, session, coord):
- rank0log(self.logger, " Step Epoch Speed Loss FinLoss LR")
- self.elapsed_secs = 0.0
- self.count = 0
-
- def before_run(self, run_context):
- self.t0 = time.time()
- return tf.train.SessionRunArgs(
- fetches=[tf.train.get_global_step(), "loss:0", "total_loss:0", "learning_rate:0"]
- )
-
- def after_run(self, run_context, run_values):
- self.elapsed_secs += time.time() - self.t0
- self.count += 1
- global_step, loss, total_loss, lr = run_values.results
- if global_step == 1 or global_step % self.display_every == 0:
- dt = self.elapsed_secs / self.count
- img_per_sec = self.global_batch_size / dt
- epoch = global_step * self.global_batch_size / self.num_records
- self.logger.info(
- "%6i %5.1f %7.1f %6.3f %6.3f %7.5f"
- % (global_step, epoch, img_per_sec, loss, total_loss, lr)
- )
- self.elapsed_secs = 0.0
- self.count = 0
-
-
-def _fp32_trainvar_getter(
- getter, name, shape=None, dtype=None, trainable=True, regularizer=None, *args, **kwargs
-):
- storage_dtype = tf.float32 if trainable else dtype
-
- bn_in_name = False
- for x in ["BatchNorm", "batchnorm", "batch_norm", "Batch_Norm"]:
- if x in name:
- bn_in_name = True
-
- if trainable and not bn_in_name:
- use_regularizer = regularizer
- else:
- use_regularizer = None
-
- variable = getter(
- name,
- shape,
- dtype=storage_dtype,
- trainable=trainable,
- regularizer=use_regularizer,
- *args,
- **kwargs
- )
- if trainable and dtype != tf.float32:
- cast_name = name + "/fp16_cast"
- try:
- cast_variable = tf.get_default_graph().get_tensor_by_name(cast_name + ":0")
- except KeyError:
- cast_variable = tf.cast(variable, dtype, name=cast_name)
- cast_variable._ref = variable._ref
- variable = cast_variable
- return variable
-
-
-def fp32_trainable_vars(name="fp32_vars", *args, **kwargs):
- """A varible scope with custom variable getter to convert fp16 trainable
- variables with fp32 storage followed by fp16 cast.
- """
- return tf.variable_scope(name, custom_getter=_fp32_trainvar_getter, *args, **kwargs)
-
-
-class MixedPrecisionOptimizer(tf.train.Optimizer):
- """An optimizer that updates trainable variables in fp32."""
-
- def __init__(self, optimizer, scale=None, name="MixedPrecisionOptimizer", use_locking=False):
- super(MixedPrecisionOptimizer, self).__init__(name=name, use_locking=use_locking)
- self._optimizer = optimizer
- self._scale = float(scale) if scale is not None else 1.0
-
- def compute_gradients(self, loss, var_list=None, *args, **kwargs):
- if var_list is None:
- var_list = tf.trainable_variables() + tf.get_collection(
- tf.GraphKeys.TRAINABLE_RESOURCE_VARIABLES
- )
-
- replaced_list = var_list
-
- if self._scale != 1.0:
- loss = tf.scalar_mul(self._scale, loss)
- gradvar = self._optimizer.compute_gradients(loss, replaced_list, *args, **kwargs)
-
- final_gradvar = []
- for orig_var, (grad, var) in zip(var_list, gradvar):
- if var is not orig_var:
- grad = tf.cast(grad, orig_var.dtype)
- if self._scale != 1.0:
- grad = tf.scalar_mul(1.0 / self._scale, grad)
- final_gradvar.append((grad, orig_var))
- return final_gradvar
-
- def apply_gradients(self, *args, **kwargs):
- return self._optimizer.apply_gradients(*args, **kwargs)
-
-
-class LarcOptimizer(tf.train.Optimizer):
- """ LARC implementation
- -------------------
- Parameters:
- - optimizer: initial optimizer that you wanna apply
- example: tf.train.MomentumOptimizer
- - learning_rate: initial learning_rate from initial optimizer
- - clip: if True apply LARC otherwise LARS
- - epsilon: default value is weights or grads are 0.
- - name
- - use_locking
- """
-
- def __init__(
- self,
- optimizer,
- learning_rate,
- eta,
- clip=True,
- epsilon=1.0,
- name="LarcOptimizer",
- use_locking=False,
- ):
- super(LarcOptimizer, self).__init__(name=name, use_locking=use_locking)
- self._optimizer = optimizer
- self._learning_rate = learning_rate
- self._eta = float(eta)
- self._clip = clip
- self._epsilon = float(epsilon)
-
- def compute_gradients(self, *args, **kwargs):
- return self._optimizer.compute_gradients(*args, **kwargs)
-
- def apply_gradients(self, gradvars, *args, **kwargs):
- v_list = [tf.norm(tensor=v, ord=2) for _, v in gradvars]
- g_list = [tf.norm(tensor=g, ord=2) if g is not None else 0.0 for g, _ in gradvars]
- v_norms = tf.stack(v_list)
- g_norms = tf.stack(g_list)
- zeds = tf.zeros_like(v_norms)
- # assign epsilon if weights or grads = 0, to avoid division by zero
- # also prevent biases to get stuck at initialization (0.)
- cond = tf.logical_and(tf.not_equal(v_norms, zeds), tf.not_equal(g_norms, zeds))
- true_vals = tf.scalar_mul(self._eta, tf.div(v_norms, g_norms))
- false_vals = tf.fill(tf.shape(v_norms), self._epsilon)
- larc_local_lr = tf.where(cond, true_vals, false_vals)
- if self._clip:
- ones = tf.ones_like(v_norms)
- lr = tf.fill(tf.shape(v_norms), self._learning_rate)
- # We need gradients to compute local learning rate,
- # so compute_gradients from initial optimizer have to called
- # for which learning rate is already fixed
- # We then have to scale the gradients instead of the learning rate.
- larc_local_lr = tf.minimum(tf.div(larc_local_lr, lr), ones)
- gradvars = [
- (tf.multiply(larc_local_lr[i], g), v) if g is not None else (None, v)
- for i, (g, v) in enumerate(gradvars)
- ]
- return self._optimizer.apply_gradients(gradvars, *args, **kwargs)
-
-
-def get_with_default(obj, key, default_value):
- return obj[key] if key in obj and obj[key] is not None else default_value
-
-
-def get_lr(
- lr,
- steps,
- lr_steps,
- warmup_it,
- decay_steps,
- global_step,
- lr_decay_mode,
- cdr_first_decay_ratio,
- cdr_t_mul,
- cdr_m_mul,
- cdr_alpha,
- lc_periods,
- lc_alpha,
- lc_beta,
-):
- if lr_decay_mode == "steps":
- learning_rate = tf.train.piecewise_constant(global_step, steps, lr_steps)
- elif lr_decay_mode == "poly" or lr_decay_mode == "poly_cycle":
- cycle = lr_decay_mode == "poly_cycle"
- learning_rate = tf.train.polynomial_decay(
- lr,
- global_step - warmup_it,
- decay_steps=decay_steps - warmup_it,
- end_learning_rate=0.00001,
- power=2,
- cycle=cycle,
- )
- elif lr_decay_mode == "cosine_decay_restarts":
- learning_rate = tf.train.cosine_decay_restarts(
- lr,
- global_step - warmup_it,
- (decay_steps - warmup_it) * cdr_first_decay_ratio,
- t_mul=cdr_t_mul,
- m_mul=cdr_m_mul,
- alpha=cdr_alpha,
- )
- elif lr_decay_mode == "cosine":
- learning_rate = tf.train.cosine_decay(
- lr, global_step - warmup_it, decay_steps=decay_steps - warmup_it, alpha=0.0
- )
- elif lr_decay_mode == "linear_cosine":
- learning_rate = tf.train.linear_cosine_decay(
- lr,
- global_step - warmup_it,
- decay_steps=decay_steps - warmup_it,
- num_periods=lc_periods,
- # 0.47,
- alpha=lc_alpha, # 0.0,
- beta=lc_beta,
- ) # 0.00001)
- else:
- raise ValueError("Invalid type of lr_decay_mode")
- return learning_rate
-
-
-def warmup_decay(warmup_lr, global_step, warmup_steps, warmup_end_lr):
- from tensorflow.python.ops import math_ops
-
- p = tf.cast(global_step, tf.float32) / tf.cast(warmup_steps, tf.float32)
- diff = math_ops.subtract(warmup_end_lr, warmup_lr)
- res = math_ops.add(warmup_lr, math_ops.multiply(diff, p))
- return res
-
-
-def cnn_model_function(features, labels, mode, params):
- labels = tf.reshape(labels, (-1,)) # Squash unnecessary unary dim
- lr = params["lr"]
- lr_steps = params["lr_steps"]
- steps = params["steps"]
- use_larc = params["use_larc"]
- leta = params["leta"]
- lr_decay_mode = params["lr_decay_mode"]
- decay_steps = params["decay_steps"]
- cdr_first_decay_ratio = params["cdr_first_decay_ratio"]
- cdr_t_mul = params["cdr_t_mul"]
- cdr_m_mul = params["cdr_m_mul"]
- cdr_alpha = params["cdr_alpha"]
- lc_periods = params["lc_periods"]
- lc_alpha = params["lc_alpha"]
- lc_beta = params["lc_beta"]
-
- model_name = params["model"]
- num_classes = params["n_classes"]
- model_dtype = get_with_default(params, "dtype", tf.float32)
- model_format = get_with_default(params, "format", "channels_first")
- device = get_with_default(params, "device", "/gpu:0")
- model_func = get_model_func(model_name)
- inputs = features # TODO: Should be using feature columns?
- is_training = mode == tf.estimator.ModeKeys.TRAIN
- momentum = params["mom"]
- weight_decay = params["wdecay"]
- warmup_lr = params["warmup_lr"]
- warmup_it = params["warmup_it"]
- loss_scale = params["loss_scale"]
-
- adv_bn_init = params["adv_bn_init"]
- conv_init = params["conv_init"]
-
- if mode == tf.estimator.ModeKeys.TRAIN:
- with tf.device("/cpu:0"):
- preload_op, (inputs, labels) = stage([inputs, labels])
- smd.get_hook().add_to_collection("inputs", inputs)
-
- with tf.device(device):
- if mode == tf.estimator.ModeKeys.TRAIN:
- gpucopy_op, (inputs, labels) = stage([inputs, labels])
- inputs = tf.cast(inputs, model_dtype)
- imagenet_mean = np.array([121, 115, 100], dtype=np.float32)
- imagenet_std = np.array([70, 68, 71], dtype=np.float32)
- inputs = tf.subtract(inputs, imagenet_mean)
- inputs = tf.multiply(inputs, 1.0 / imagenet_std)
- if model_format == "channels_first":
- inputs = tf.transpose(inputs, [0, 3, 1, 2])
- with fp32_trainable_vars(regularizer=tf.contrib.layers.l2_regularizer(weight_decay)):
- top_layer = model_func(
- inputs,
- data_format=model_format,
- training=is_training,
- conv_initializer=conv_init,
- adv_bn_init=adv_bn_init,
- )
- logits = tf.layers.dense(
- top_layer, num_classes, kernel_initializer=tf.random_normal_initializer(stddev=0.01)
- )
- predicted_classes = tf.argmax(logits, axis=1, output_type=tf.int32)
- logits = tf.cast(logits, tf.float32)
- if mode == tf.estimator.ModeKeys.PREDICT:
- probabilities = tf.softmax(logits)
- predictions = {
- "class_ids": predicted_classes[:, None],
- "probabilities": probabilities,
- "logits": logits,
- }
- return tf.estimator.EstimatorSpec(mode, predictions=predictions)
- loss = tf.losses.sparse_softmax_cross_entropy(logits=logits, labels=labels)
- loss = tf.identity(loss, name="loss") # For access by logger
-
- if mode == tf.estimator.ModeKeys.EVAL:
- # Allow fallback to CPU if no GPU support for these ops
- with tf.device(None):
- accuracy = tf.metrics.accuracy(labels=labels, predictions=predicted_classes)
- top5acc = tf.metrics.mean(tf.cast(tf.nn.in_top_k(logits, labels, 5), tf.float32))
- newaccuracy = (hvd.allreduce(accuracy[0]), accuracy[1])
- newtop5acc = (hvd.allreduce(top5acc[0]), top5acc[1])
- metrics = {"val-top1acc": newaccuracy, "val-top5acc": newtop5acc}
- return tf.estimator.EstimatorSpec(mode, loss=loss, eval_metric_ops=metrics)
-
- assert mode == tf.estimator.ModeKeys.TRAIN
- reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
- total_loss = tf.add_n([loss] + reg_losses, name="total_loss")
-
- batch_size = tf.shape(inputs)[0]
-
- global_step = tf.train.get_global_step()
-
- # Allow fallback to CPU if no GPU support for these ops
- with tf.device("/cpu:0"):
- learning_rate = tf.cond(
- global_step < warmup_it,
- lambda: warmup_decay(warmup_lr, global_step, warmup_it, lr),
- lambda: get_lr(
- lr,
- steps,
- lr_steps,
- warmup_it,
- decay_steps,
- global_step,
- lr_decay_mode,
- cdr_first_decay_ratio,
- cdr_t_mul,
- cdr_m_mul,
- cdr_alpha,
- lc_periods,
- lc_alpha,
- lc_beta,
- ),
- )
- learning_rate = tf.identity(learning_rate, "learning_rate")
- tf.summary.scalar("learning_rate", learning_rate)
-
- opt = tf.train.MomentumOptimizer(learning_rate, momentum, use_nesterov=True)
- opt = hvd.DistributedOptimizer(opt)
- if use_larc:
- opt = LarcOptimizer(opt, learning_rate, leta, clip=True)
-
- opt = MixedPrecisionOptimizer(opt, scale=loss_scale)
- opt = smd.get_hook().wrap_optimizer(opt)
-
- update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS) or []
- with tf.control_dependencies(update_ops):
- gate_gradients = tf.train.Optimizer.GATE_NONE
- train_op = opt.minimize(
- total_loss, global_step=tf.train.get_global_step(), gate_gradients=gate_gradients
- )
- train_op = tf.group(preload_op, gpucopy_op, train_op)
-
- return tf.estimator.EstimatorSpec(mode, loss=total_loss, train_op=train_op)
-
-
-def get_num_records(filenames):
- def count_records(tf_record_filename):
- count = 0
- for _ in tf.python_io.tf_record_iterator(tf_record_filename):
- count += 1
- return count
-
- nfile = len(filenames)
- return count_records(filenames[0]) * (nfile - 1) + count_records(filenames[-1])
-
-
-def str2bool(v):
- if isinstance(v, bool):
- return v
-
- if v.lower() in ("yes", "true", "t", "y", "1"):
- return True
- elif v.lower() in ("no", "false", "f", "n", "0"):
- return False
- else:
- raise argparse.ArgumentTypeError("Boolean value expected.")
-
-
-def sort_and_load_ckpts(log_dir):
- ckpts = []
- for f in os.listdir(log_dir):
- m = re.match(r"model.ckpt-([0-9]+).index", f)
- if m is None:
- continue
- fullpath = os.path.join(log_dir, f)
- ckpts.append(
- {
- "step": int(m.group(1)),
- "path": os.path.splitext(fullpath)[0],
- "mtime": os.stat(fullpath).st_mtime,
- }
- )
- ckpts.sort(key=itemgetter("step"))
- return ckpts
-
-
-def add_cli_args():
- cmdline = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
- # Basic options
- cmdline.add_argument(
- "-m",
- "--model",
- default="resnet50",
- help="""Name of model to run:
- resnet[18,34,50,101,152]""",
- )
- cmdline.add_argument(
- "--data_dir",
- help="""Path to dataset in TFRecord format
- (aka Example protobufs). Files should be
- named 'train-*' and 'validation-*'.""",
- )
- cmdline.add_argument(
- "--synthetic",
- type=str2bool,
- default=False,
- help="""Whether to use synthetic data for training.
- If data_dir is not given, uses synthetic data
- by default""",
- )
- cmdline.add_argument(
- "-b", "--batch_size", default=128, type=int, help="""Size of each minibatch per GPU"""
- )
- cmdline.add_argument(
- "--num_batches",
- type=int,
- default=1000,
- help="""Number of batches to run.
- Ignored during eval or if num epochs given""",
- )
- cmdline.add_argument(
- "--num_epochs",
- type=int,
- help="""Number of epochs to run.
- Overrides --num_batches. Ignored during eval.""",
- )
- cmdline.add_argument(
- "--log_dir",
- default="tf_logs",
- help="""Directory in which to write training
- summaries and checkpoints. If the log directory
- already contains some checkpoints, it tries
- to resume training from the last saved checkpoint.
- Pass --clear_log if you want to clear all
- checkpoints and start a fresh run""",
- )
- cmdline.add_argument("--model_dir", default=None, type=str)
- cmdline.add_argument("--random_seed", type=str2bool, default=False)
- cmdline.add_argument(
- "--clear_log",
- type=str2bool,
- default=False,
- help="""Clear the log folder passed
- so a fresh run can be started""",
- )
- cmdline.add_argument("--log_name", type=str, default="hvd_train.log")
- cmdline.add_argument(
- "--local_ckpt",
- type=str2bool,
- default=False,
- help="""Performs local checkpoints
- (i.e. one per node)""",
- )
- cmdline.add_argument(
- "--display_every",
- default=20,
- type=int,
- help="""How often (in iterations) to print out
- running information.""",
- )
- cmdline.add_argument(
- "--eval",
- type=str2bool,
- default=False,
- help="""Evaluate the top-1 and top-5 accuracy of
- the latest checkpointed model. If you want to
- evaluate using multiple GPUs ensure that all
- processes have access to all checkpoints.
- Either if checkpoints were saved using
- --local_ckpt or they were saved to a
- shared directory which all processes can access.""",
- )
- cmdline.add_argument(
- "--eval_interval",
- type=int,
- help="""Evaluate accuracy per eval_interval
- number of epochs""",
- )
- cmdline.add_argument(
- "--fp16",
- type=str2bool,
- default=True,
- help="""Train using float16 (half) precision instead
- of float32.""",
- )
- cmdline.add_argument(
- "--num_gpus",
- default=1,
- type=int,
- help="""Specify total number of GPUS used to
- train a checkpointed model during eval.
- Used only to calculate epoch number to
- print during evaluation""",
- )
- cmdline.add_argument("--save_checkpoints_steps", type=int, default=1000)
- cmdline.add_argument("--save_summary_steps", type=int, default=0)
- cmdline.add_argument(
- "--adv_bn_init",
- type=str2bool,
- default=True,
- help="""init gamme of the last BN of
- each ResMod at 0.""",
- )
- cmdline.add_argument(
- "--adv_conv_init", type=str2bool, default=True, help="""init conv with MSRA initializer"""
- )
- cmdline.add_argument("--lr", type=float, help="""Start learning rate""")
- cmdline.add_argument("--mom", default=0.90, type=float, help="""Momentum""")
- cmdline.add_argument("--wdecay", default=0.0001, type=float, help="""Weight decay""")
- cmdline.add_argument("--loss_scale", default=1024.0, type=float, help="""loss scale""")
- cmdline.add_argument(
- "--warmup_lr", default=0.001, type=float, help="""Warmup starting from this learning rate"""
- )
- cmdline.add_argument(
- "--warmup_epochs",
- default=0,
- type=int,
- help="""Number of epochs in which to warmup
- to given lr""",
- )
- cmdline.add_argument(
- "--lr_decay_steps",
- default="30,60,80",
- type=str,
- help="""epoch numbers at which lr is decayed
- by lr_decay_lrs. Used when lr_decay_mode is steps""",
- )
- cmdline.add_argument(
- "--lr_decay_lrs", default="", type=str, help="""learning rates at specific epochs"""
- )
- cmdline.add_argument(
- "--lr_decay_mode",
- default="poly",
- help="""Takes either `steps`
- (decay by a factor at specified steps)
- or `poly`(polynomial_decay with degree 2)""",
- )
-
- cmdline.add_argument(
- "--use_larc",
- type=str2bool,
- default=False,
- help="""Use Layer wise Adaptive Rate Control
- which helps convergence at really
- large batch sizes""",
- )
- cmdline.add_argument(
- "--leta",
- default=0.013,
- type=float,
- help="""The trust coefficient for LARC optimization,
- LARC Eta""",
- )
- cmdline.add_argument(
- "--cdr_first_decay_ratio",
- default=0.33,
- type=float,
- help="""Cosine Decay Restart First
- Decay Steps ratio""",
- )
- cmdline.add_argument(
- "--cdr_t_mul", default=2.0, type=float, help="""Cosine Decay Restart t_mul"""
- )
- cmdline.add_argument(
- "--cdr_m_mul", default=0.1, type=float, help="""Cosine Decay Restart m_mul"""
- )
- cmdline.add_argument(
- "--cdr_alpha", default=0.0, type=float, help="""Cosine Decay Restart alpha"""
- )
- cmdline.add_argument(
- "--lc_periods", default=0.47, type=float, help="""Linear Cosine num of periods"""
- )
- cmdline.add_argument("--lc_alpha", default=0.0, type=float, help="""linear Cosine alpha""")
- cmdline.add_argument("--lc_beta", default=0.00001, type=float, help="""Liner Cosine Beta""")
-
- cmdline.add_argument(
- "--increased_aug",
- type=str2bool,
- default=False,
- help="""Increase augmentations helpful when training
- with large number of GPUs such as 128 or 256""",
- )
- cmdline.add_argument("--contrast", default=0.6, type=float, help="""contrast factor""")
- cmdline.add_argument("--saturation", default=0.6, type=float, help="""saturation factor""")
- cmdline.add_argument(
- "--hue",
- default=0.13,
- type=float,
- help="""hue max delta factor,
- hue delta = hue * math.pi""",
- )
- cmdline.add_argument("--brightness", default=0.3, type=float, help="""Brightness factor""")
-
- # tornasole arguments
- cmdline.add_argument(
- "--enable_smdebug", type=str2bool, default=False, help="""enable Tornasole"""
- )
- cmdline.add_argument(
- "--smdebug_path",
- default="tornasole_outputs/default_run",
- help="""Directory in which to write tornasole data.
- This can be a local path or
- S3 path in the form s3://bucket_name/prefix_name""",
- )
- cmdline.add_argument(
- "--tornasole_save_all", type=str2bool, default=False, help="""save all tensors"""
- )
- cmdline.add_argument(
- "--tornasole_dryrun",
- type=str2bool,
- default=False,
- help="""If enabled, do not write data to disk""",
- )
- cmdline.add_argument(
- "--tornasole_exclude",
- nargs="+",
- default=[],
- type=str,
- action="append",
- help="""List of REs for tensors to exclude from
- Tornasole's default collection""",
- )
- cmdline.add_argument(
- "--tornasole_include",
- nargs="+",
- default=[],
- type=str,
- action="append",
- help="""List of REs for tensors to include from
- Tornasole's default collection""",
- )
- cmdline.add_argument(
- "--step_interval", default=10, type=int, help="""Save tornasole data every N runs"""
- )
- cmdline.add_argument("--save_weights", type=str2bool, default=False)
- cmdline.add_argument("--save_gradients", type=str2bool, default=False)
- cmdline.add_argument("--tornasole_save_inputs", type=str2bool, default=False)
- cmdline.add_argument("--tornasole_save_relu_activations", type=str2bool, default=False)
- cmdline.add_argument(
- "--tornasole_relu_reductions",
- type=str,
- help="""A comma separated list of reductions can be
- passed. If passed, saves relu activations
- in the form of these reductions.""",
- )
- cmdline.add_argument(
- "--tornasole_relu_reductions_abs",
- type=str,
- help="""A comma separated list of absolute reductions
- can be passed. If passed, saves relu activations
- in the form of these reductions on absolute values
- of the tensor.""",
- )
- cmdline.add_argument(
- "--constant_initializer",
- type=float,
- help="""if passed sets that constant as initial
- weight, if not uses default initialization
- strategies""",
- )
- return cmdline
-
-
-def create_hook(FLAGS):
- abs_reductions = []
- reductions = []
- if FLAGS.tornasole_relu_reductions:
- for r in FLAGS.tornasole_relu_reductions:
- reductions.append(r)
- if FLAGS.tornasole_relu_reductions_abs:
- for r in FLAGS.tornasole_relu_reductions_abs:
- abs_reductions.append(r)
- if reductions or abs_reductions:
- rnc = smd.ReductionConfig(reductions=reductions, abs_reductions=abs_reductions)
- else:
- rnc = None
-
- include_collections = ["losses"]
-
- hook = smd.SessionHook(
- out_dir=FLAGS.smdebug_path,
- save_config=smd.SaveConfig(save_interval=FLAGS.step_interval),
- reduction_config=rnc,
- include_collections=include_collections,
- save_all=FLAGS.tornasole_save_all,
- )
- if FLAGS.save_weights is True:
- include_collections.append("weights")
- if FLAGS.save_gradients is True:
- include_collections.append("gradients")
- if FLAGS.tornasole_save_relu_activations is True:
- include_collections.append("relu_activations")
- if FLAGS.tornasole_save_inputs is True:
- include_collections.append("inputs")
- if FLAGS.tornasole_include:
- hook.get_collection("default").include(FLAGS.tornasole_include)
- include_collections.append("default")
- return hook
-
-
-def main():
- gpu_thread_count = 2
- os.environ["TF_GPU_THREAD_MODE"] = "gpu_private"
- os.environ["TF_GPU_THREAD_COUNT"] = str(gpu_thread_count)
- os.environ["TF_USE_CUDNN_BATCHNORM_SPATIAL_PERSISTENT"] = "1"
- os.environ["TF_ENABLE_WINOGRAD_NONFUSED"] = "1"
- hvd.init()
-
- config = tf.ConfigProto()
- config.gpu_options.visible_device_list = str(hvd.local_rank())
- config.gpu_options.force_gpu_compatible = True # Force pinned memory
- config.intra_op_parallelism_threads = 1 # Avoid pool of Eigen threads
- config.inter_op_parallelism_threads = 5
-
- cmdline = add_cli_args()
- FLAGS, unknown_args = cmdline.parse_known_args()
-
- # these random seeds are only intended for test purpose.
- # if you wish to change the seed settings, notice that certain
- # steps' tensor value may be capable of variation
- if FLAGS.random_seed:
- random.seed(5 * (1 + hvd.rank()))
- np.random.seed(7 * (1 + hvd.rank()))
- tf.set_random_seed(31 * (1 + hvd.rank()))
-
- if len(unknown_args) > 0:
- for bad_arg in unknown_args:
- print("ERROR: Unknown command line arg: %s" % bad_arg)
- raise ValueError("Invalid command line arg(s)")
-
- FLAGS.data_dir = None if FLAGS.data_dir == "" else FLAGS.data_dir
- FLAGS.log_dir = None if FLAGS.log_dir == "" else FLAGS.log_dir
- FLAGS.model_dir = None if FLAGS.model_dir == "" else FLAGS.model_dir
-
- if FLAGS.eval:
- FLAGS.log_name = "eval_" + FLAGS.log_name
- if FLAGS.local_ckpt:
- do_checkpoint = hvd.local_rank() == 0
- else:
- do_checkpoint = hvd.rank() == 0
- if hvd.local_rank() == 0 and FLAGS.clear_log and os.path.isdir(FLAGS.log_dir):
- shutil.rmtree(FLAGS.log_dir)
- barrier = hvd.allreduce(tf.constant(0, dtype=tf.float32))
- tf.Session(config=config).run(barrier)
-
- if hvd.local_rank() == 0 and not os.path.isdir(FLAGS.log_dir):
- os.makedirs(FLAGS.log_dir)
- barrier = hvd.allreduce(tf.constant(0, dtype=tf.float32))
- tf.Session(config=config).run(barrier)
-
- logger = logging.getLogger(FLAGS.log_name)
- logger.setLevel(logging.INFO) # INFO, ERROR
- if not hvd.rank():
- fh = logging.FileHandler(os.path.join(FLAGS.log_dir, FLAGS.log_name))
- fh.setLevel(logging.DEBUG)
- formatter = logging.Formatter("%(message)s")
- fh.setFormatter(formatter)
- # add handlers to logger
- logger.addHandler(fh)
-
- height, width = 224, 224
- global_batch_size = FLAGS.batch_size * hvd.size()
- rank0log(logger, "PY" + str(sys.version) + "TF" + str(tf.__version__))
- rank0log(logger, "Horovod size: ", hvd.size())
-
- if FLAGS.data_dir:
- filename_pattern = os.path.join(FLAGS.data_dir, "%s-*")
- train_filenames = sorted(tf.gfile.Glob(filename_pattern % "train"))
- eval_filenames = sorted(tf.gfile.Glob(filename_pattern % "validation"))
- num_training_samples = get_num_records(train_filenames)
- rank0log(logger, "Using data from: ", FLAGS.data_dir)
- if not FLAGS.eval:
- rank0log(logger, "Found ", num_training_samples, " training samples")
- else:
- if not FLAGS.synthetic:
- FLAGS.synthetic = True
- rank0log(
- logger,
- "data_dir missing. Using synthetic data. "
- "If you want to run on real data"
- "pass --data_dir PATH_TO_DATA",
- )
- train_filenames = eval_filenames = []
- num_training_samples = 1281167
- training_samples_per_rank = num_training_samples // hvd.size()
-
- if FLAGS.num_epochs:
- nstep = num_training_samples * FLAGS.num_epochs // global_batch_size
- elif FLAGS.num_batches:
- nstep = FLAGS.num_batches
- FLAGS.num_epochs = max(nstep * global_batch_size // num_training_samples, 1)
- else:
- raise ValueError("Either num_epochs or num_batches has to be passed")
- nstep_per_epoch = num_training_samples // global_batch_size
- decay_steps = nstep
-
- if FLAGS.lr_decay_mode == "steps":
- steps = [int(x) * nstep_per_epoch for x in FLAGS.lr_decay_steps.split(",")]
- lr_steps = [float(x) for x in FLAGS.lr_decay_lrs.split(",")]
- else:
- steps = []
- lr_steps = []
-
- if not FLAGS.lr:
- if FLAGS.use_larc:
- FLAGS.lr = 3.7
- else:
- FLAGS.lr = (hvd.size() * FLAGS.batch_size * 0.1) / 256
- if not FLAGS.save_checkpoints_steps:
- # default to save one checkpoint per epoch
- FLAGS.save_checkpoints_steps = nstep_per_epoch
- if not FLAGS.save_summary_steps:
- # default to save one checkpoint per epoch
- FLAGS.save_summary_steps = nstep_per_epoch
-
- if not FLAGS.eval:
- rank0log(logger, "Using a learning rate of ", FLAGS.lr)
- rank0log(logger, "Checkpointing every " + str(FLAGS.save_checkpoints_steps) + " steps")
- rank0log(logger, "Saving summary every " + str(FLAGS.save_summary_steps) + " steps")
-
- warmup_it = nstep_per_epoch * FLAGS.warmup_epochs
- if FLAGS.constant_initializer:
- initializer_conv = tf.constant_initializer(FLAGS.constant_initializer)
- elif FLAGS.adv_conv_init:
- initializer_conv = tf.variance_scaling_initializer()
- else:
- initializer_conv = None
-
- if FLAGS.model_dir is None:
- FLAGS.model_dir = FLAGS.log_dir
-
- if FLAGS.enable_smdebug is True and hvd.rank() == 0:
- hook = create_hook(FLAGS)
-
- classifier = tf.estimator.Estimator(
- model_fn=cnn_model_function,
- model_dir=FLAGS.model_dir,
- params={
- "model": FLAGS.model,
- "decay_steps": decay_steps,
- "n_classes": 1000,
- "dtype": tf.float16 if FLAGS.fp16 is True else tf.float32,
- "format": "channels_first",
- "device": "/gpu:0",
- "lr": FLAGS.lr,
- "mom": FLAGS.mom,
- "wdecay": FLAGS.wdecay,
- "use_larc": FLAGS.use_larc,
- "leta": FLAGS.leta,
- "steps": steps,
- "lr_steps": lr_steps,
- "lr_decay_mode": FLAGS.lr_decay_mode,
- "warmup_it": warmup_it,
- "warmup_lr": FLAGS.warmup_lr,
- "cdr_first_decay_ratio": FLAGS.cdr_first_decay_ratio,
- "cdr_t_mul": FLAGS.cdr_t_mul,
- "cdr_m_mul": FLAGS.cdr_m_mul,
- "cdr_alpha": FLAGS.cdr_alpha,
- "lc_periods": FLAGS.lc_periods,
- "lc_alpha": FLAGS.lc_alpha,
- "lc_beta": FLAGS.lc_beta,
- "loss_scale": FLAGS.loss_scale,
- "adv_bn_init": FLAGS.adv_bn_init,
- "conv_init": initializer_conv,
- },
- config=tf.estimator.RunConfig(
- tf_random_seed=31 * (1 + hvd.rank()),
- session_config=config,
- save_summary_steps=FLAGS.save_summary_steps if do_checkpoint else None,
- save_checkpoints_steps=FLAGS.save_checkpoints_steps if do_checkpoint else None,
- keep_checkpoint_max=None,
- ),
- )
-
- if not FLAGS.eval:
- num_preproc_threads = 5
- rank0log(logger, "Using preprocessing threads per GPU: ", num_preproc_threads)
- training_hooks = [hvd.BroadcastGlobalVariablesHook(0), PrefillStagingAreasHook()]
- if hvd.rank() == 0:
- training_hooks.append(
- LogSessionRunHook(
- global_batch_size, num_training_samples, FLAGS.display_every, logger
- )
- )
- if FLAGS.enable_smdebug is True:
- training_hooks.append(hook)
- try:
- if FLAGS.enable_smdebug is True:
- hook.set_mode(smd.modes.TRAIN)
- start_time = time.time()
- classifier.train(
- input_fn=lambda: make_dataset(
- train_filenames,
- training_samples_per_rank,
- FLAGS.batch_size,
- height,
- width,
- FLAGS.brightness,
- FLAGS.contrast,
- FLAGS.saturation,
- FLAGS.hue,
- training=True,
- num_threads=num_preproc_threads,
- shard=True,
- synthetic=FLAGS.synthetic,
- increased_aug=FLAGS.increased_aug,
- ),
- max_steps=nstep,
- hooks=training_hooks,
- )
- rank0log(logger, "Finished in ", time.time() - start_time)
- except KeyboardInterrupt:
- print("Keyboard interrupt")
- elif FLAGS.eval and not FLAGS.synthetic:
- rank0log(logger, "Evaluating")
- rank0log(logger, "Validation dataset size: {}".format(get_num_records(eval_filenames)))
- barrier = hvd.allreduce(tf.constant(0, dtype=tf.float32))
- tf.Session(config=config).run(barrier)
- time.sleep(5) # a little extra margin...
- if FLAGS.num_gpus == 1:
- rank0log(
- logger,
- """If you are evaluating checkpoints of a
- multi-GPU run on a single GPU, ensure you set --num_gpus to
- the number of GPUs it was trained on.
- This will ensure that the epoch number is
- accurately displayed in the below logs.""",
- )
- try:
- ckpts = sort_and_load_ckpts(FLAGS.log_dir)
- for i, c in enumerate(ckpts):
- if i < len(ckpts) - 1:
- if (not FLAGS.eval_interval) or (i % FLAGS.eval_interval != 0):
- continue
- eval_result = classifier.evaluate(
- input_fn=lambda: make_dataset(
- eval_filenames,
- get_num_records(eval_filenames),
- FLAGS.batch_size,
- height,
- width,
- FLAGS.brightness,
- FLAGS.contrast,
- FLAGS.saturation,
- FLAGS.hue,
- training=False,
- shard=True,
- increased_aug=False,
- ),
- checkpoint_path=c["path"],
- )
- c["epoch"] = c["step"] / (
- num_training_samples // (FLAGS.batch_size * FLAGS.num_gpus)
- )
- c["top1"] = eval_result["val-top1acc"]
- c["top5"] = eval_result["val-top5acc"]
- c["loss"] = eval_result["loss"]
- rank0log(logger, " step epoch top1 top5 loss checkpoint_time(UTC)")
- barrier = hvd.allreduce(tf.constant(0, dtype=tf.float32))
- for i, c in enumerate(ckpts):
- tf.Session(config=config).run(barrier)
- if "top1" not in c:
- continue
- rank0log(
- logger,
- "{:5d} {:5.1f} {:5.3f} {:6.2f} {:6.2f} {time}".format(
- c["step"],
- c["epoch"],
- c["top1"] * 100,
- c["top5"] * 100,
- c["loss"],
- time=time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(c["mtime"])),
- ),
- )
- rank0log(logger, "Finished evaluation")
- except KeyboardInterrupt:
- logger.error("Keyboard interrupt")
-
-
-if __name__ == "__main__":
- main()
diff --git a/examples/xgboost/README.md b/examples/xgboost/README.md
new file mode 100644
index 000000000..9818d371a
--- /dev/null
+++ b/examples/xgboost/README.md
@@ -0,0 +1,2 @@
+## Example Notebooks
+Please refer to the example notebooks in [Amazon SageMaker Examples repository](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger)
diff --git a/examples/xgboost/notebooks/xgboost_abalone.ipynb b/examples/xgboost/notebooks/xgboost_abalone.ipynb
deleted file mode 100644
index 312cc5f42..000000000
--- a/examples/xgboost/notebooks/xgboost_abalone.ipynb
+++ /dev/null
@@ -1,813 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# XGBoost for Regression in Tornasole\n",
- "This notebook will demonstrate the simplest kind of interactive analysis that can be run in smdebug. It will focus on the predicting the age of abalone ([Abalone dataset](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html)) using [XGBoost](https://github.com/dmlc/xgboost) for regression.\n",
- "\n",
- "## Setup\n",
- "\n",
- "Some basic setup that's always helpful"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Make sure that you can run `xgboost`. You can install `xgboost` by doing\n",
- "```shell\n",
- "$ pip3 install xgboost\n",
- "```\n",
- "You'll probably have to restart this notebook after doing this.\n",
- "\n",
- "Let's import some basic libraries for ML"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "import numpy as np\n",
- "import xgboost as xgb\n",
- "import matplotlib.pyplot as plt\n",
- "import seaborn as sns"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's copy the Tornasole libraries to this instance, this step has to be executed only once. \n",
- "Please make sure that the AWS account you are using can access the `tornasole-external-preview-use1` bucket.\n",
- "\n",
- "To do so you'll need the appropriate AWS credentials. There are several ways of doing this:\n",
- "- inject temporary credentials \n",
- "- if running on EC2, use [EC2 roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html) that can access all S3 buckets\n",
- "- (preferred) run this notebook on a [SageMaker notebook instance](https://docs.aws.amazon.com/sagemaker/latest/dg/nbi.html)\n",
- "\n",
- "The code below downloads the necessary `.whl` files and installs them in the current environment. Only run the first time!\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "#WARNING - uncomment this code only if you haven't done this before\n",
- "#!aws s3 sync s3://tornasole-external-preview-use1/sdk/ts-binaries/tornasole_xgboost/py3/latest/ tornasole_xgboost/\n",
- "#!pip install tornasole_xgboost/tornasole-*\n",
- "\n",
- "# If you run into a version conflict with boto, run the following\n",
- "#!pip uninstall -y botocore boto3 aioboto3 aiobotocore && pip install botocore==1.12.91 boto3==1.9.91 aiobotocore==0.10.2 aioboto3==6.4.1"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Fetching the dataset\n",
- "\n",
- "We use the [Abalone data](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html) originally from the [UCI data repository](https://archive.ics.uci.edu/ml/datasets/abalone). More details about the original dataset can be found [here](https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.names). In the libsvm converted [version](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html), the nominal feature (Male/Female/Infant) has been converted into a real valued feature. Age of abalone is to be predicted from eight physical measurements."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "CPU times: user 28.5 ms, sys: 9.39 ms, total: 37.9 ms\n",
- "Wall time: 6.01 s\n"
- ]
- }
- ],
- "source": [
- "%%time\n",
- "import random\n",
- "import tempfile\n",
- "import urllib.request\n",
- "\n",
- "\n",
- "def load_abalone(train_split=0.8, seed=42):\n",
- "\n",
- " if not (0 < train_split <= 1):\n",
- " raise ValueError(\"'train_split' must be between 0 and 1.\")\n",
- "\n",
- " url = \"https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression/abalone\"\n",
- "\n",
- " response = urllib.request.urlopen(url).read().decode(\"utf-8\")\n",
- " lines = response.strip().split('\\n')\n",
- " n = sum(1 for line in lines)\n",
- " indices = list(range(n))\n",
- " random.seed(seed)\n",
- " random.shuffle(indices)\n",
- " train_indices = set(indices[:int(n * 0.8)])\n",
- "\n",
- " with tempfile.NamedTemporaryFile(mode='w', delete=False) as train_file:\n",
- " with tempfile.NamedTemporaryFile(mode='w', delete=False) as valid_file:\n",
- " for idx, line in enumerate(lines):\n",
- " if idx in train_indices:\n",
- " train_file.write(line + '\\n')\n",
- " else:\n",
- " valid_file.write(line + '\\n')\n",
- "\n",
- " return train_file.name, valid_file.name\n",
- "\n",
- "train_file, validation_file = load_abalone()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "## Model Training\n",
- "\n",
- "At this point we have all the ingredients installed on our machine. We can now start training."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "from smdebug import SaveConfig\n",
- "from smdebug.xgboost import SessionHook\n",
- "from smdebug.trials import LocalTrial"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can change the logging level if appropriate "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "#import logging\n",
- "#logging.getLogger(\"tornasole\").setLevel(logging.WARNING)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Clean up from previous runs, we remove old data (warning - we assume that we have set `ts_output` as the directory into which we send data)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [],
- "source": [
- "!rm -rf ./ts_output/"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We load the datasets into [DMatrix](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.DMatrix) objects and define some hyperparameters - it doesn't really matter what it is."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[20:28:55] 3341x9 matrix with 26728 entries loaded from /var/folders/8f/_jkz_jm54y56k334j6vkhlr51s_6bx/T/tmps8ervlbn\n",
- "[20:28:55] 836x9 matrix with 6688 entries loaded from /var/folders/8f/_jkz_jm54y56k334j6vkhlr51s_6bx/T/tmprvihe8rx\n"
- ]
- }
- ],
- "source": [
- "dtrain = xgb.DMatrix(train_file)\n",
- "dval = xgb.DMatrix(validation_file)\n",
- "\n",
- "watchlist = [(dtrain, 'train'), (dval, 'validation')]\n",
- "\n",
- "params = {\n",
- " \"max_depth\": 5,\n",
- " \"eta\": 0.2,\n",
- " \"gamma\": 4,\n",
- " \"min_child_weight\": 6,\n",
- " \"subsample\": 0.7,\n",
- " \"silent\": 0,\n",
- " \"objective\": \"reg:squarederror\",\n",
- " \"eval_metric\": [\"rmse\", \"mae\"]\n",
- "}\n",
- "\n",
- "num_round = 100"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Importantly - we **add the Tornasole Hook**. This hook will be run at every iteration and will save selected performance metrics, feature importances, or [SHAP](https://github.com/slundberg/shap) values (in this case, all of them) to the desired directory (in this case, `'{base_loc}/{run_id}'`.\n",
- "\n",
- "`{base_loc}` can be either a path on a local file system (for instance, `./ts_output/`) or an S3 bucket/object (`s3://mybucket/myprefix/`).\n",
- "\n",
- "See the documentation for more details."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[2019-09-04 20:28:55.303 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:86] Saving to ./ts_output\n"
- ]
- }
- ],
- "source": [
- "save_config = SaveConfig(save_interval=5)\n",
- "\n",
- "hook = SessionHook(\n",
- " out_dir=\"./ts_output\",\n",
- " save_config=save_config,\n",
- " shap_data=dtrain\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "At this point we are ready to train. We will train this simple model.\n",
- "\n",
- "Behind the scenes, the `SessionHook` is saving the data requested."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[2019-09-04 20:28:55.359 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 0.\n",
- "[0]\ttrain-rmse:8.12809\ttrain-mae:7.57979\tvalidation-rmse:7.9501\tvalidation-mae:7.42376\n",
- "[1]\ttrain-rmse:6.66075\ttrain-mae:6.0853\tvalidation-rmse:6.50774\tvalidation-mae:5.95214\n",
- "[2]\ttrain-rmse:5.49178\ttrain-mae:4.87746\tvalidation-rmse:5.35986\tvalidation-mae:4.76687\n",
- "[3]\ttrain-rmse:4.58214\ttrain-mae:3.91531\tvalidation-rmse:4.47214\tvalidation-mae:3.81942\n",
- "[4]\ttrain-rmse:3.87782\ttrain-mae:3.14893\tvalidation-rmse:3.79344\tvalidation-mae:3.07788\n",
- "[2019-09-04 20:28:55.438 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 5.\n",
- "[5]\ttrain-rmse:3.35051\ttrain-mae:2.57049\tvalidation-rmse:3.28752\tvalidation-mae:2.51613\n",
- "[6]\ttrain-rmse:2.95214\ttrain-mae:2.14503\tvalidation-rmse:2.91198\tvalidation-mae:2.11696\n",
- "[7]\ttrain-rmse:2.66585\ttrain-mae:1.85231\tvalidation-rmse:2.63266\tvalidation-mae:1.83957\n",
- "[8]\ttrain-rmse:2.45031\ttrain-mae:1.65847\tvalidation-rmse:2.45043\tvalidation-mae:1.6836\n",
- "[9]\ttrain-rmse:2.29575\ttrain-mae:1.53516\tvalidation-rmse:2.30491\tvalidation-mae:1.57687\n",
- "[2019-09-04 20:28:55.698 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 10.\n",
- "[10]\ttrain-rmse:2.18057\ttrain-mae:1.4608\tvalidation-rmse:2.21775\tvalidation-mae:1.51841\n",
- "[11]\ttrain-rmse:2.09701\ttrain-mae:1.41371\tvalidation-rmse:2.16059\tvalidation-mae:1.4858\n",
- "[12]\ttrain-rmse:2.04006\ttrain-mae:1.38316\tvalidation-rmse:2.12174\tvalidation-mae:1.46568\n",
- "[13]\ttrain-rmse:1.99211\ttrain-mae:1.35925\tvalidation-rmse:2.08493\tvalidation-mae:1.45008\n",
- "[14]\ttrain-rmse:1.96792\ttrain-mae:1.35184\tvalidation-rmse:2.06902\tvalidation-mae:1.44609\n",
- "[2019-09-04 20:28:55.906 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 15.\n",
- "[15]\ttrain-rmse:1.94583\ttrain-mae:1.34568\tvalidation-rmse:2.05138\tvalidation-mae:1.44183\n",
- "[16]\ttrain-rmse:1.92788\ttrain-mae:1.34136\tvalidation-rmse:2.04385\tvalidation-mae:1.43984\n",
- "[17]\ttrain-rmse:1.91056\ttrain-mae:1.33725\tvalidation-rmse:2.03618\tvalidation-mae:1.43834\n",
- "[18]\ttrain-rmse:1.89749\ttrain-mae:1.33499\tvalidation-rmse:2.0322\tvalidation-mae:1.4393\n",
- "[19]\ttrain-rmse:1.88413\ttrain-mae:1.33075\tvalidation-rmse:2.03752\tvalidation-mae:1.44543\n",
- "[2019-09-04 20:28:56.163 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 20.\n",
- "[20]\ttrain-rmse:1.87967\ttrain-mae:1.32948\tvalidation-rmse:2.03493\tvalidation-mae:1.44581\n",
- "[21]\ttrain-rmse:1.86798\ttrain-mae:1.32499\tvalidation-rmse:2.03925\tvalidation-mae:1.44907\n",
- "[22]\ttrain-rmse:1.8582\ttrain-mae:1.32131\tvalidation-rmse:2.03707\tvalidation-mae:1.44972\n",
- "[23]\ttrain-rmse:1.85303\ttrain-mae:1.31908\tvalidation-rmse:2.03875\tvalidation-mae:1.45157\n",
- "[24]\ttrain-rmse:1.84696\ttrain-mae:1.31813\tvalidation-rmse:2.04676\tvalidation-mae:1.45646\n",
- "[2019-09-04 20:28:56.518 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 25.\n",
- "[25]\ttrain-rmse:1.83281\ttrain-mae:1.31091\tvalidation-rmse:2.04628\tvalidation-mae:1.45422\n",
- "[26]\ttrain-rmse:1.821\ttrain-mae:1.30427\tvalidation-rmse:2.04828\tvalidation-mae:1.45734\n",
- "[27]\ttrain-rmse:1.81488\ttrain-mae:1.30071\tvalidation-rmse:2.04599\tvalidation-mae:1.45618\n",
- "[28]\ttrain-rmse:1.8002\ttrain-mae:1.29268\tvalidation-rmse:2.05193\tvalidation-mae:1.4594\n",
- "[29]\ttrain-rmse:1.79308\ttrain-mae:1.29057\tvalidation-rmse:2.05075\tvalidation-mae:1.46388\n",
- "[2019-09-04 20:28:56.806 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 30.\n",
- "[30]\ttrain-rmse:1.78557\ttrain-mae:1.28705\tvalidation-rmse:2.05292\tvalidation-mae:1.46461\n",
- "[31]\ttrain-rmse:1.77816\ttrain-mae:1.28089\tvalidation-rmse:2.05105\tvalidation-mae:1.46399\n",
- "[32]\ttrain-rmse:1.76314\ttrain-mae:1.27199\tvalidation-rmse:2.05361\tvalidation-mae:1.46855\n",
- "[33]\ttrain-rmse:1.75584\ttrain-mae:1.2695\tvalidation-rmse:2.04789\tvalidation-mae:1.4681\n",
- "[34]\ttrain-rmse:1.75165\ttrain-mae:1.26756\tvalidation-rmse:2.04984\tvalidation-mae:1.47014\n",
- "[2019-09-04 20:28:57.182 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 35.\n",
- "[35]\ttrain-rmse:1.74615\ttrain-mae:1.26296\tvalidation-rmse:2.0479\tvalidation-mae:1.4715\n",
- "[36]\ttrain-rmse:1.73885\ttrain-mae:1.25856\tvalidation-rmse:2.04948\tvalidation-mae:1.4712\n",
- "[37]\ttrain-rmse:1.736\ttrain-mae:1.257\tvalidation-rmse:2.04992\tvalidation-mae:1.47051\n",
- "[38]\ttrain-rmse:1.73002\ttrain-mae:1.25165\tvalidation-rmse:2.05359\tvalidation-mae:1.47123\n",
- "[39]\ttrain-rmse:1.72742\ttrain-mae:1.25118\tvalidation-rmse:2.05419\tvalidation-mae:1.47141\n",
- "[2019-09-04 20:28:57.649 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 40.\n",
- "[40]\ttrain-rmse:1.72441\ttrain-mae:1.2498\tvalidation-rmse:2.0541\tvalidation-mae:1.47108\n",
- "[41]\ttrain-rmse:1.71452\ttrain-mae:1.24448\tvalidation-rmse:2.05683\tvalidation-mae:1.47221\n",
- "[42]\ttrain-rmse:1.70619\ttrain-mae:1.23868\tvalidation-rmse:2.0529\tvalidation-mae:1.47012\n",
- "[43]\ttrain-rmse:1.6994\ttrain-mae:1.23462\tvalidation-rmse:2.05173\tvalidation-mae:1.4685\n",
- "[44]\ttrain-rmse:1.69531\ttrain-mae:1.2325\tvalidation-rmse:2.05062\tvalidation-mae:1.46729\n",
- "[2019-09-04 20:28:57.973 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 45.\n",
- "[45]\ttrain-rmse:1.68994\ttrain-mae:1.22921\tvalidation-rmse:2.05038\tvalidation-mae:1.46773\n",
- "[46]\ttrain-rmse:1.68733\ttrain-mae:1.22771\tvalidation-rmse:2.04775\tvalidation-mae:1.46629\n",
- "[47]\ttrain-rmse:1.6745\ttrain-mae:1.22109\tvalidation-rmse:2.05052\tvalidation-mae:1.46931\n",
- "[48]\ttrain-rmse:1.6677\ttrain-mae:1.21652\tvalidation-rmse:2.05003\tvalidation-mae:1.46771\n",
- "[49]\ttrain-rmse:1.66666\ttrain-mae:1.21571\tvalidation-rmse:2.05055\tvalidation-mae:1.46835\n",
- "[2019-09-04 20:28:58.286 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 50.\n",
- "[50]\ttrain-rmse:1.66069\ttrain-mae:1.21231\tvalidation-rmse:2.05735\tvalidation-mae:1.47412\n",
- "[51]\ttrain-rmse:1.65219\ttrain-mae:1.20659\tvalidation-rmse:2.05643\tvalidation-mae:1.47175\n",
- "[52]\ttrain-rmse:1.6502\ttrain-mae:1.20494\tvalidation-rmse:2.05467\tvalidation-mae:1.46944\n",
- "[53]\ttrain-rmse:1.64501\ttrain-mae:1.20097\tvalidation-rmse:2.0534\tvalidation-mae:1.46682\n",
- "[54]\ttrain-rmse:1.63611\ttrain-mae:1.1947\tvalidation-rmse:2.05339\tvalidation-mae:1.46562\n",
- "[2019-09-04 20:28:58.703 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 55.\n",
- "[55]\ttrain-rmse:1.63324\ttrain-mae:1.19463\tvalidation-rmse:2.05613\tvalidation-mae:1.47189\n",
- "[56]\ttrain-rmse:1.62148\ttrain-mae:1.18714\tvalidation-rmse:2.05909\tvalidation-mae:1.47115\n",
- "[57]\ttrain-rmse:1.61668\ttrain-mae:1.18364\tvalidation-rmse:2.05695\tvalidation-mae:1.46885\n",
- "[58]\ttrain-rmse:1.6119\ttrain-mae:1.18027\tvalidation-rmse:2.05439\tvalidation-mae:1.46791\n",
- "[59]\ttrain-rmse:1.61104\ttrain-mae:1.1797\tvalidation-rmse:2.05469\tvalidation-mae:1.46735\n",
- "[2019-09-04 20:28:59.112 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 60.\n",
- "[60]\ttrain-rmse:1.60972\ttrain-mae:1.17818\tvalidation-rmse:2.05268\tvalidation-mae:1.46727\n",
- "[61]\ttrain-rmse:1.60483\ttrain-mae:1.17471\tvalidation-rmse:2.05238\tvalidation-mae:1.46723\n",
- "[62]\ttrain-rmse:1.60038\ttrain-mae:1.17187\tvalidation-rmse:2.05876\tvalidation-mae:1.46983\n",
- "[63]\ttrain-rmse:1.58954\ttrain-mae:1.16512\tvalidation-rmse:2.05812\tvalidation-mae:1.46628\n",
- "[64]\ttrain-rmse:1.58059\ttrain-mae:1.16055\tvalidation-rmse:2.06236\tvalidation-mae:1.4686\n",
- "[2019-09-04 20:28:59.606 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 65.\n",
- "[65]\ttrain-rmse:1.57333\ttrain-mae:1.15614\tvalidation-rmse:2.06741\tvalidation-mae:1.47281\n",
- "[66]\ttrain-rmse:1.57042\ttrain-mae:1.15445\tvalidation-rmse:2.06701\tvalidation-mae:1.47235\n",
- "[67]\ttrain-rmse:1.56843\ttrain-mae:1.15295\tvalidation-rmse:2.0663\tvalidation-mae:1.47219\n",
- "[68]\ttrain-rmse:1.56508\ttrain-mae:1.15063\tvalidation-rmse:2.06515\tvalidation-mae:1.47276\n",
- "[69]\ttrain-rmse:1.56182\ttrain-mae:1.14841\tvalidation-rmse:2.06628\tvalidation-mae:1.47551\n",
- "[2019-09-04 20:29:00.062 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 70.\n",
- "[70]\ttrain-rmse:1.55922\ttrain-mae:1.14672\tvalidation-rmse:2.06947\tvalidation-mae:1.47674\n",
- "[71]\ttrain-rmse:1.55271\ttrain-mae:1.14336\tvalidation-rmse:2.07123\tvalidation-mae:1.4787\n",
- "[72]\ttrain-rmse:1.54876\ttrain-mae:1.14062\tvalidation-rmse:2.07051\tvalidation-mae:1.47639\n",
- "[73]\ttrain-rmse:1.54667\ttrain-mae:1.13933\tvalidation-rmse:2.075\tvalidation-mae:1.47814\n",
- "[74]\ttrain-rmse:1.54487\ttrain-mae:1.13787\tvalidation-rmse:2.07309\tvalidation-mae:1.47687\n",
- "[2019-09-04 20:29:00.610 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 75.\n",
- "[75]\ttrain-rmse:1.53595\ttrain-mae:1.1307\tvalidation-rmse:2.07264\tvalidation-mae:1.47924\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[76]\ttrain-rmse:1.53418\ttrain-mae:1.12846\tvalidation-rmse:2.07262\tvalidation-mae:1.47965\n",
- "[77]\ttrain-rmse:1.52683\ttrain-mae:1.12359\tvalidation-rmse:2.07258\tvalidation-mae:1.47693\n",
- "[78]\ttrain-rmse:1.52175\ttrain-mae:1.12025\tvalidation-rmse:2.0736\tvalidation-mae:1.47748\n",
- "[79]\ttrain-rmse:1.51673\ttrain-mae:1.11775\tvalidation-rmse:2.07555\tvalidation-mae:1.47927\n",
- "[2019-09-04 20:29:01.110 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 80.\n",
- "[80]\ttrain-rmse:1.51552\ttrain-mae:1.11628\tvalidation-rmse:2.07579\tvalidation-mae:1.47937\n",
- "[81]\ttrain-rmse:1.51042\ttrain-mae:1.11335\tvalidation-rmse:2.07407\tvalidation-mae:1.47882\n",
- "[82]\ttrain-rmse:1.49888\ttrain-mae:1.10658\tvalidation-rmse:2.07054\tvalidation-mae:1.47881\n",
- "[83]\ttrain-rmse:1.48993\ttrain-mae:1.10055\tvalidation-rmse:2.07382\tvalidation-mae:1.48155\n",
- "[84]\ttrain-rmse:1.48741\ttrain-mae:1.09893\tvalidation-rmse:2.07477\tvalidation-mae:1.48274\n",
- "[2019-09-04 20:29:01.640 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 85.\n",
- "[85]\ttrain-rmse:1.48121\ttrain-mae:1.09505\tvalidation-rmse:2.07463\tvalidation-mae:1.48405\n",
- "[86]\ttrain-rmse:1.47717\ttrain-mae:1.09075\tvalidation-rmse:2.07385\tvalidation-mae:1.48084\n",
- "[87]\ttrain-rmse:1.47266\ttrain-mae:1.08722\tvalidation-rmse:2.07319\tvalidation-mae:1.4827\n",
- "[88]\ttrain-rmse:1.46499\ttrain-mae:1.08129\tvalidation-rmse:2.07362\tvalidation-mae:1.4862\n",
- "[89]\ttrain-rmse:1.46304\ttrain-mae:1.08039\tvalidation-rmse:2.07298\tvalidation-mae:1.48554\n",
- "[2019-09-04 20:29:02.197 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 90.\n",
- "[90]\ttrain-rmse:1.45968\ttrain-mae:1.07896\tvalidation-rmse:2.07361\tvalidation-mae:1.487\n",
- "[91]\ttrain-rmse:1.45246\ttrain-mae:1.07513\tvalidation-rmse:2.07227\tvalidation-mae:1.48551\n",
- "[92]\ttrain-rmse:1.44731\ttrain-mae:1.07157\tvalidation-rmse:2.07416\tvalidation-mae:1.48621\n",
- "[93]\ttrain-rmse:1.44069\ttrain-mae:1.06819\tvalidation-rmse:2.07838\tvalidation-mae:1.49\n",
- "[94]\ttrain-rmse:1.43273\ttrain-mae:1.06434\tvalidation-rmse:2.08272\tvalidation-mae:1.49347\n",
- "[2019-09-04 20:29:03.068 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 95.\n",
- "[95]\ttrain-rmse:1.4278\ttrain-mae:1.06026\tvalidation-rmse:2.08366\tvalidation-mae:1.49517\n",
- "[96]\ttrain-rmse:1.42101\ttrain-mae:1.05588\tvalidation-rmse:2.0787\tvalidation-mae:1.49234\n",
- "[97]\ttrain-rmse:1.41775\ttrain-mae:1.05389\tvalidation-rmse:2.07722\tvalidation-mae:1.49251\n",
- "[98]\ttrain-rmse:1.41412\ttrain-mae:1.05038\tvalidation-rmse:2.07725\tvalidation-mae:1.49371\n",
- "[2019-09-04 20:29:03.090 38f9d36a2c42.ant.amazon.com:42958 INFO hook.py:182] Saved iteration 99.\n",
- "[99]\ttrain-rmse:1.40749\ttrain-mae:1.04556\tvalidation-rmse:2.07435\tvalidation-mae:1.49251\n"
- ]
- }
- ],
- "source": [
- "bst = xgb.train(\n",
- " params=params,\n",
- " dtrain=dtrain,\n",
- " evals=watchlist,\n",
- " num_boost_round=num_round,\n",
- " callbacks=[hook])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Data Analysis - Manual\n",
- "Now that we have trained the system we can analyze the data. Notice that this notebook focuses on after-the-fact analysis. Tornasole also provides a collection of tools to do automatic analysis as the training run is progressing, which will be covered in a different notebook.\n",
- "\n",
- "We import a basic analysis library, which defines a concept of `Trial`. A `Trial` is a single training run, which is depositing values in a local directory (`LocalTrial`) or S3 (`S3Trial`). In this case we are using a `LocalTrial` - if you wish, you can change the output from `./ts_output` to `s3://mybucket/myprefix` and use `S3Trial` instead of `LocalTrial`."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And we read the data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[2019-09-04 20:29:03.117 38f9d36a2c42.ant.amazon.com:42958 INFO local_trial.py:22] Loading trial myrun at path ./ts_output\n",
- "[2019-09-04 20:29:03.119 38f9d36a2c42.ant.amazon.com:42958 INFO local_trial.py:58] Loaded 3 collections\n"
- ]
- }
- ],
- "source": [
- "trial = LocalTrial(\"myrun\", \"./ts_output\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can list all the tensors we know something about. Each one of these names is the name of a tensor - the name is a combination of the feature name (which, in these cases, is auto-assigned by XGBoost) and whether it's an evaluation metric, feature importance, or SHAP value."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[2019-09-04 20:29:06.817 38f9d36a2c42.ant.amazon.com:42958 INFO trial.py:98] Training has ended, will try to do a final refresh in 5 sec\n",
- "[2019-09-04 20:29:11.839 38f9d36a2c42.ant.amazon.com:42958 INFO trial.py:103] Marked loaded all steps to True\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "['train-rmse',\n",
- " 'train-mae',\n",
- " 'validation-rmse',\n",
- " 'validation-mae',\n",
- " 'f8/feature_importance',\n",
- " 'f2/feature_importance',\n",
- " 'f6/feature_importance',\n",
- " 'f1/feature_importance',\n",
- " 'f4/feature_importance',\n",
- " 'f1/average_shap',\n",
- " 'f2/average_shap',\n",
- " 'f4/average_shap',\n",
- " 'f6/average_shap',\n",
- " 'f3/feature_importance',\n",
- " 'f5/feature_importance',\n",
- " 'f7/feature_importance',\n",
- " 'f3/average_shap',\n",
- " 'f7/average_shap',\n",
- " 'f5/average_shap']"
- ]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "trial.tensors()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For each tensor we can ask for which steps we have data - in this case, every 5 steps"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "[0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95]"
- ]
- },
- "execution_count": 13,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "trial.tensor(\"f8/feature_importance\").steps()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can obtain each tensor at each step as a `numpy` array"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "numpy.ndarray"
- ]
- },
- "execution_count": 14,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "type(trial.tensor(\"f8/feature_importance\").value(30))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Performance metrics\n",
- "\n",
- "We can also create a simple function that visualizes the training and validation errors\n",
- "as the training progresses.\n",
- "We expect each gradient to get smaller over time, as the system converges to a good solution.\n",
- "Now, remember that this is an interactive analysis - we are showing these tensors to give an idea of the data. \n",
- "\n",
- "Later on in this notebook we will run an automated analysis."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Define a function that, for the given tensor name, walks through all \n",
- "# the iterations for which we have data and fetches the value.\n",
- "# Returns the set of steps and the values\n",
- "\n",
- "def get_data(trial, tname):\n",
- " tensor = trial.tensor(tname)\n",
- " steps = tensor.steps()\n",
- " vals = [tensor.value(s) for s in steps]\n",
- " return steps, vals"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEGCAYAAABvtY4XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZwdZZno8d9ztj69L0ln6e5sKIaQBEJoNhk0EIYbUSKgCBqG5V5FlDHojDroOIPeK6MzOgz6EdG4MC6MiBEUV0QEEVSWYAwhCYYlMZ21u9P7evqc5/5R1Z3Tne509VKncs55vp/P+dRe9ZyczlNVb731vqKqGGOMyT2hoAMwxhjjD0vwxhiToyzBG2NMjrIEb4wxOcoSvDHG5KhI0AGkmzlzpi5cuDDoMIwxJmts2rSpSVWrR1t2XCX4hQsX8uyzzwYdhjHGZA0R2T3WMiuiMcaYHGUJ3hhjcpQleGOMyVHHVRm8McYfiUSChoYGent7gw7FTFI8Hqeuro5oNOp5G0vwxuSBhoYGSktLWbhwISISdDhmglSV5uZmGhoaWLRokeftrIjGmDzQ29vLjBkzLLlnKRFhxowZE74DswRvTJ6w5J7dJvP7ZX2CT6WUL/1mJ7/9S2PQoRhjzHEl6xN8KCRsePwVHtl+MOhQjDFjaG1t5ctf/vKEt7v44otpbW31IaL84GuCF5EPicgLIrJVRL4nInE/jlNTUci+VqsdYMzxaqwEPzAwcMztfv7zn1NRUTGpY6oqqVRqUtvmCt8SvIjUAuuBelVdBoSBq/w41tzyOPvbevzYtTFmGtxyyy28/PLLrFixgjPOOIPzzjuPtWvXcvLJJwNw6aWXcvrpp7N06VI2bNgwtN3ChQtpampi165dLFmyhPe85z0sXbqUiy66iJ6eo//P79q1i8WLF3PNNdewbNky9uzZQ0lJCR/5yEdYunQpF154IU8//TSrVq3ihBNO4MEHHwTghRde4Mwzz2TFihWccsop7Ny5E4Dvfve7Q/Pf+973kkwmM/CvNX38riYZAQpFJAEUAfv8OMjcikL+3NDmx66NyTmf+skLbNvXPq37PLmmjFsvWTrm8s9+9rNs3bqVzZs389hjj/HmN7+ZrVu3DlX5++Y3v0lVVRU9PT2cccYZvO1tb2PGjBnD9rFz506+973v8bWvfY13vOMd/PCHP+Tqq68+6lg7d+7kW9/6FmeffTYAXV1dXHDBBXzuc5/jsssu4xOf+AQPP/ww27Zt49prr2Xt2rV85Stf4eabb2bdunX09/eTTCbZvn073//+93nyySeJRqO8//3v55577uGaa66Zxn85f/mW4FV1r4h8Hvgr0AP8SlV/NXI9EbkBuAFg/vz5Ez9QKsVKdvCH7jZ6E0ni0fDUAjfG+O7MM88cVp/7i1/8Ig888AAAe/bsYefOnUcl+EWLFrFixQoATj/9dHbt2jXqvhcsWDCU3AFisRhr1qwBYPny5RQUFBCNRlm+fPnQPs455xxuu+02GhoauPzyyznxxBN55JFH2LRpE2eccQYAPT09zJo1a1q+f6b4luBFpBJ4K7AIaAV+ICJXq+p309dT1Q3ABoD6+vqJ9wAuwmXP30RLeDX7297JopnFUw/emBx2rCvtTCkuPvL/9LHHHuPXv/41f/jDHygqKmLVqlWj1vcuKCgYGg+Hw/T09LBnzx4uueQSAG688UbWrFkzbN8A0Wh0qIphKBQa2k8oFBp6BvCud72Ls846i5/97GdcfPHFfPWrX0VVufbaa/nMZz4zvV8+g/x8yHoh8KqqNqpqArgfeP20H0WE/pIa5koz+1utHN6Y41FpaSkdHR2jLmtra6OyspKioiJ27NjBH//4R8/7nTdvHps3b2bz5s3ceOONk47vlVde4YQTTmD9+vW89a1vZcuWLaxevZqNGzdy6NAhAA4fPszu3WO2zHtc8rMM/q/A2SJShFNEsxrwp7H38jpqW/fzcpvVpDHmeDRjxgzOPfdcli1bRmFhIbNnzx5atmbNGr7yla+wZMkSFi9ePKx4JVPuu+8+vvOd7xCNRpkzZw4f//jHqaqq4tOf/jQXXXQRqVSKaDTKnXfeyYIFCzIe32SJ6sRLRTzvXORTwJXAAPAn4N2q2jfW+vX19TqZDj8GHng/zZt/xn1v+DUfWH3ipOM1Jldt376dJUuWBB2GmaLRfkcR2aSq9aOt72stGlW9FbjVz2MARCrmMVtaOdgyvTUDjDEmm2X9m6wAlNcB0He4IeBAjDHm+JFTCT7VZgneGGMG5UiCnwdArNOX96iMMSYr5UiCrwWgauAQHb2JgIMxxpjjQ24k+GghfbEqaqWJ/VZV0hhjgFxJ8MBAaQ010sw+e9nJmJxQUlICwL59+3j7298+6jqrVq1ivKrVd9xxB93d3UPT+dQEcc4k+FDFPOdtVruCNyan1NTUsHHjxklvPzLBT6UJ4rEcr00T50yCL5gx3ymiaekef2VjTMbdcsst3HnnnUPTn/zkJ/n0pz/N6tWrWblyJcuXL+fHP/7xUdvt2rWLZcuWAU6DX1dddRVLlizhsssuG9Zk8Pve9z7q6+tZunQpt97qvH7zxS9+kX379nH++edz/vnnA0eaIAa4/fbbWbZsGcuWLeOOO+4YOl6uNE3sd3PBGROqmEeJ9HK4pSnoUIw5vv3iFjjw/PTuc85yeNNnj7nKlVdeyQc/+EFuuukmwGke4KGHHmL9+vWUlZXR1NTE2Wefzdq1a8fsf/Suu+6iqKiI7du3s2XLFlauXDm07LbbbqOqqopkMsnq1avZsmUL69ev5/bbb+fRRx9l5syZw/a1adMm7r77bp566ilUlbPOOos3vvGNVFZW5kzTxDmT4Afrwg8c/mvAgRhjRnPaaadx6NAh9u3bR2NjI5WVlcyZM4cPfehDPP7444RCIfbu3cvBgweZM2fOqPt4/PHHWb9+PQCnnHIKp5xyytCy++67jw0bNjAwMMD+/fvZtm3bsOUjPfHEE1x22WVDrU9efvnl/O53v2Pt2rU50zRx7iT4MifBh9r3BhyIMce5ca60/XTFFVewceNGDhw4wJVXXsk999xDY2MjmzZtIhqNsnDhwlGbCh7Pq6++yuc//3meeeYZKisrue666ya1n0G50jRxzpTBD17Bx7r24WcDasaYybvyyiu599572bhxI1dccQVtbW3MmjWLaDTKo48+Om5zvG94wxv4n//5HwC2bt3Kli1bAGhvb6e4uJjy8nIOHjzIL37xi6Ftxmqq+LzzzuNHP/oR3d3ddHV18cADD3DeeeeNeexsbJo4d67gS2aTlAiztImW7gRVxbGgIzLGjLB06VI6Ojqora1l7ty5rFu3jksuuYTly5dTX1/PSSeddMzt3/e+93H99dezZMkSlixZwumnnw7AqaeeymmnncZJJ53EvHnzOPfcc4e2ueGGG1izZg01NTU8+uijQ/NXrlzJddddx5lnngnAu9/9bk477bQxi2OmSyabJva1ueCJmmxzwYO6P7eUX7XP57U33suy2vJpjMyY7GbNBeeGiTYXnDtFNECqtI4aqwtvjDFAjiX4SOVggre3WY0xJqcSfMGMBczhMPtbuoIOxZjjzvFUHGsmbjK/X04leKmoIyIpupqtXXhj0sXjcZqbmy3JZylVpbm5mXg8PqHtcqcWDQy1C59qsQRvTLq6ujoaGhpobGwMOhQzSfF4nLq6ugltk2MJ3vny4Q5L8Maki0ajLFq0KOgwTIb5VkQjIotFZHPap11EPujX8QAoczr+KO7dTyplt6LGmPzm2xW8qr4IrAAQkTCwF3jAr+MBEC+jP1LK7IEmmjr7mFU2sfIqY4zJJZl6yLoaeFlVp/7u7Tj6imuokcPstY4/jDF5LlMJ/irgexk5UnkdNdZ1nzHG+J/gRSQGrAV+MMbyG0TkWRF5djqe8Meq5lvXfcYYQ2au4N8EPKeqB0dbqKobVLVeVeurq6unfLDYjPlUSidNhw9PeV/GGJPNMpHg30mmimcAcatK9lvHH8aYPOdrgheRYuBvgfv9PM4wboJPtVrHH8aY/Obri06q2gXM8PMYR3ETfLTTErwxJr/lVFs0AJTOJUWI0r4DJJKpoKMxxpjA5F6CD0fpjVdTI80cbLeqksaY/HXMBC8iYRH5UKaCmS6JkhpqsLrwxpj8dswEr6pJnFowWSVUMc/qwhtj8p6XIponReRLInKeiKwc/Pge2RQUzFjAXDnMvpbuoEMxxpjAeKlFs8Id/t+0eQpcMP3hTI9Y1XyQBO3N+4ETgw7HGGMCMW6CV9XzMxHItHKrSibsZSdjTB4bt4hGRMpF5PbB9mJE5D9FpDwTwU1audMuvLRbxx/GmPzlpQz+m0AH8A730w7c7WdQU+Z23Rfv2hdwIMYYExwvZfCvUdW3pU1/SkQ2+xXQtCisJBGKU95/iN5Ekng0HHRExhiTcV6u4HtE5G8GJ0TkXOD4rn8oQk9RDTXSbHXhjTF5y8sV/I3At9PK3VuAa/0LaXokS2upad/L/tYeFs0sDjocY4zJuGMmeBEJAYtV9VQRKQNQ1faMRDZFkcp51O7bwm/tCt4Yk6fGe5M1BXzUHW/PluQOEJ85n2pp4+DhtqBDMcaYQHgpg/+1iHxYROaJSNXgx/fIpihaOR+A7iarC2+MyU9eyuCvdIc3pc1T4ITpD2cauS87DbTsCTgQY4wJhpcy+KtV9ckMxTN93AQf6bCXnYwx+clLGfyXMhTL9Cpz3mYt7N4fcCDGGBMML2Xwj4jI20REfI9mOkXjdEermJFspKM3EXQ0xhiTcV4S/HuBHwD9ItIuIh0ikhW1afqKa6gV6/jDGJOfxk3wqlqqqiFVjapqmTtdlongpqy8zmkX3jr+MMbkIS+tSYqIXC0i/+JOzxORM73sXEQqRGSjiOwQke0ics5UA56IaNV8aqSJ/ZbgjTF5yEsRzZeBc4B3udOdwJ0e9/8F4JeqehJwKrB9whFOQeHMBRRLHy1NBzN5WGOMOS54SfBnqepNQC+AqrYAsfE2ctuueQPwDXe7flVtnUKsExaucKpK9jbby07GmPzjJcEnRCSM83ITIlINpDxstwhoBO4WkT+JyNdF5KhWv0TkhsHORBobGycS+/jcduG1zV52MsbkHy8J/ovAA8AsEbkNeAL4Nw/bRYCVwF2qehrQBdwyciVV3aCq9apaX11d7T1yL4ZedrKOP4wx+cdLn6z3iMgmYDUgwKWq6qUsvQFoUNWn3OmNjJLgfVVczYBEKerdj6qSbVX5jTFmKry0RYOq7gB2TGTHqnpARPaIyGJVfRHnBLFtEjFOXihEd3wOszubaOlOUFU87qMDY4zJGZ4S/BR8ALhHRGLAK8D1Ph/vKImSGmq6mtnX2mMJ3hiTV3xN8Kq6Gaj38xjjkfI6ag7t5IW2XpbVlo+/gTHG5AgvD1mzWsHM+cymhQMtHUGHYowxGTXmFbyIdOBWjRxNtjRXUDhzISFROhobgNcGHY4xxmTMmAleVUsBROT/AfuB7+DUolkHzM1IdNMg5L7slDi8O+BIjDEms7wU0axV1S+raofbL+tdwFv9DmzauC870bY32DiMMSbDvCT4LhFZJyJhEQmJyDqcl5ayg9vxR0GXJXhjTH7xkuDfBbwDOOh+ruBIw2PHv4ISesJllPYdIJka85GCMcbkHC9vsu4im4pkRtFTNJfZ/c00dfYxuywedDjGGJMRXtqDf52IPCIiW93pU0TkE/6HNn2SpbXUSrN1/GGMySteimi+BnwMSACo6hbgKj+Dmm7hinlOxx/WdZ8xJo94SfBFqvr0iHkDfgTjl8Lq+ZRLN41N09wcsTHGHMe8JPgmEXkNR9qDfztOvfisEZ+xAICeJuv4wxiTP7y0RXMTsAE4SUT2Aq/ivOyUNaTCqQs/0GIdfxhj8scxE7yIhIB6Vb3Q7Y0ppKrZ16iL2/FHqN3qwhtj8scxi2hUNQV81B3vysrkDlAyhyRh4t3Ws5MxJn94KYP/tYh8WETmiUjV4Mf3yKZTOEJXQTUViYMkkl66kzXGmOznpQz+Snd4U9o8BU6Y/nD801c0l5qeZg6291JXWRR0OMYY4zsvb7IuykQgfkuV1TG3+Sn2tVqCN8bkh3ETvIhcM9p8Vf329Ifjn1jVPCp3/ZzNrV1AdpUwGWPMZHgpojkjbTyO03n2c0BWJfii6oXEJEnrob3AvKDDMcYY33kpovlA+rSIVAD3+haRTwrcl536mncDZwcbjDHGZMBkOt3uAjyVy4vILqADSAIDqhpcB9zlTrvwydaGwEIwxphM8lIG/xOO9M0aAk4G7pvAMc5X1aZJxDa93JedIh32spMxJj94uYL/fNr4ALBbVbPvMjheQV+okOLeA0FHYowxGeGlDP63U9i/Ar8SEQW+qqobprCvqRGhMz6Xqo5D9CaSxKPhwEIxxphM8FJE08GRIpphiwBV1bJjbP43qrpXRGYBD4vIDlV9fMT+bwBuAJg/f773yCchUVxDTede9rf1smhmsa/HMsaYoHlpquAO4BagFqgD/gm4Q1VLx0nuqOped3gIeAA4c5R1NqhqvarWV1dXTzT+iSmvpUaa2W89Oxlj8oCXBL9WVb+sqh2q2q6qd+Ghj1YRKRaR0sFx4CJg69TCnZqCGQuYKe3sb24JMgxjjMkILwm+S0TWiUhYREIisg6nquR4ZgNPiMifgaeBn6nqL6cS7FQVz1oIQGejdfxhjMl9XmrRvAv4gvtR4El33jGp6ivAqVOKbprFqpwy/kTz7oAjMcYY/3mpRbMLD0UyWcGtC09b9tXyNMaYiRq3iEZE/kNEykQkKiKPiEijiFydieCmXVkNKYRYl3X8YYzJfV7K4C9S1XbgLcAu4LXAR/wMyjeRAroiVZTYy07GmDzgJcEPFuO8GfiBqrb5GI/vugvnMjPVREdvIuhQjDHGV14S/E9FZAdwOvCIiFQDvf6G5Z9kaS210sT+tqz9CsYY48m4CV5VbwFeD9SragLoJosfuoYq66iRZva1dAcdijHG+MrLFTyqelhVk+54l6pmbSF2fOZCCqWf5qas/QrGGOOJpwSfS0qrnY4/eg7tCjYQY4zxWd4l+HCl011ff8uegCMxxhh/eerRSURqgQXp649sFTJrlDsJPtRuHX8YY3Kbl+aC/x24EtiG0/UeOE0WZGeCL55JQqLEu+1lJ2NMbvNyBX8psFhV+/wOJiNEaI/NoaznAKqKiAQdkTHG+MJLGfwrQNTvQDKpr2guc2iipdtedjLG5C4vV/DdwGYReQQYuopX1fW+ReWzVFktNYd3sq+1h6riWNDhGGOML7wk+AfdT86IVM1n1q5WtrZ0sKy2POhwjDHGF16aC/5WJgLJpKLqBYREaTu0G6cXQmOMyT1emgs+UUQ2isg2EXll8JOJ4PxSUr0IgF7r2ckYk8O8PGS9G7gLGADOB74NfNfPoPwWcl92SrVagjfG5C4vCb5QVR8BRFV3q+oncZoOzl5ltQBEOqwuvDEmd3l5yNonIiFgp4j8PbAXKPE3LJ/FiugMlxHv2R90JMYY4xsvV/A3A0XAepw24a8GrvUzqEzoLJhLReIgyZQGHYoxxvjCSy2aZwBEJKWq10/0ACISBp4F9qrqWyYeoj8SJTXM7XyJps4+ZpfFgw7HGGOmnZdaNOeIyDZghzt9qoh8eQLHuBnYPsn4/FNeR400sa+1J+hIjDHGF16KaO4A/hfQDKCqfwbe4GXnIlKH80D265MN0C+xqvmUSQ+NTY1Bh2KMMb7w2qPTyMbTk6OueLQ7gI8CqbFWEJEbRORZEXm2sTFzybZkltPxR8fBXRk7pjHGZJKXBL9HRF4PqIhEReTDeChyEZG3AIdUddOx1lPVDapar6r11dXV3qKeBkXVCwHob7a68MaY3OQlwd8I3ATU4lSRXOFOj+dcYK2I7ALuBS4QkePmBSmpcF52os16djLG5CYvtWiagHUT3bGqfgz4GICIrAI+rKpXT3Q/vimZzQBhop32spMxJjd56dFpEfABYCHDu+xb619YGRAK0x6tprjvQNCRGGOML7y8yfoj4BvATzjGw9JjUdXHgMcms62fugvnMqP1EP0DKWKRvOt/3BiT47wk+F5V/aLvkQRgoKSGua1Pc7C9l3lVRUGHY4wx08rLZesXRORW94WnlYMf3yPLgFDFPObIYfa3dAUdijHGTDsvV/DLgb8DLuBIEY2601ktPnMBUUnScmgPvCZzVTSNMSYTvCT4K4ATVLXf72AyrWz2QgC6Du0CcuKmxBhjhngpotkKVPgdSBDiM523WQda7GUnY0zu8XIFXwHsEJFngL7BmVlfTRKg3O2Ptd3qwhtjco+XBH+r71EEJV5OtxQR77IEb4zJPV7eZP1tJgIJSnvBHEp77WUnY0zuyfu3e3qL5lKdaqQ34bWBTGOMyQ55n+BTpbXUSBP723qDDsUYY6aVlx6dbvYyL1uFK+dRJZ0caDocdCjGGDOtvFzBj9bB9nXTHEdgBtuFbz3warCBGGPMNBvzIauIvBN4F7BIRB5MW1QK5MzlbtmcRQD0Nu0OOBJjjJlex6pF83tgPzAT+M+0+R3AFj+DyqSCqvkA7N21E1VFRAKOyBhjpseYRTSqultVH1PVc4AdOFfupUCDqg5kKkDfldWgCKnW3Tz1as7cmBhjjKeHrFcAT+O0SfMO4CkRebvfgWVMOIrWnMblkT/w9cdeDDoaY4yZNl4esn4COENVr1XVa4AzgX/xN6zMCq26hToOUvXy/ew40B50OMYYMy28JPiQqh5Km272uF32OPEiBuau5ObIj+wq3hiTM7wk6l+KyEMicp2IXAf8DPi5v2FlmAiR1f9MrTQS3/o9Glq6g47IGGOmbNwEr6ofAb4KnOJ+NqjqP/kdWMa9ZjX9c+u5KfwAd//WruKNMdnPa1HLk8CjwG/c8XGJSFxEnhaRP4vICyLyqckGmREixC78BHPlMDz3bVq6cq5/E2NMnvFSi+YdOLVo3s7EatH0AReo6qnACmCNiJw9lWB9d8IqeuaexXvkAe55wq7ijTHZzcsV/D8ziVo06uh0J6PuRycdaSaIUHjRvzBHWuj54zfo6bcWJo0x2cvXWjQiEhaRzcAh4GFVfWoSMWbWovNon3MO16Xu54dP/SXoaIwxZtImW4vmF152rqpJVV0B1AFnisiykeuIyA0i8qyIPNvY2DiR2H1T9qZ/pVraaP3tVxhIpoIOxxhjJmWytWg+OpGDqGorzkPaNaMs26Cq9apaX11dPZHd+mfB62mefS5XJe7nl396KehojDFmUjwVtajq/ar6D6r6D8CPRWTdeNuISLWIVLjjhcDf4rRpkxUqL76VmdLOoUfuRPX4fnRgjDGjGTPBi0iZiHxMRL4kIheJ4++BV3Bq04xnLvCoiGwBnsEpg//p9ITtv9CCs9hffR6Xdm/kyW27gg7HGGMm7FjNBX8HaAH+ALwb+DggwKWqunm8HavqFuC06QgyKDMv+STRb66m4Zd3wNIvBB2OMcZMyLES/AmquhxARL6O0zb8fFXNm85Lo/Pr2T3zDaxp/AHPv/yPLH/N/KBDMsYYz45VBp8YHFHVJE478HmT3AdVX/IpKqSLXT/7z/FXNsaY48ixEvypItLufjqAUwbHRSRv2tQtWrCSv1St4o3N97G7YW/Q4RhjjGfH6tEprKpl7qdUVSNp42WZDDJoM99yK2XSzUsP/kfQoRhjjGe51a67T6pOWMnz5edz1sHv03hoX9DhGGOMJ5bgPaq6+F8popeXfvTZoEMxxhhPLMF7VLt4JZtKz+fUfffSeXh/0OEYY8y4LMFPQPmaT1Cg/bz0wL8FHYoxxozLEvwEvG7Z6fy++AJO2nMvfa12FW+MOb5Zgp+g+OqPEdEBdv3YruKNMcc3S/ATVL+ynkcLLmDhq/eSarOreGPM8csS/ASJCLLqo4R1gD0/uS3ocIwxZkyW4Cdh1Vln8IvIBcx96V60rSHocIwxZlSW4CchEg6ROPcfQFMc+vlngg7HGGNGZQl+ki4+72weDK1mxov3QuueoMMxxpijWIKfpHg0TPsZ60kqtD5kV/HGmOOPJfgpuHzVWWxkNaXb74WWXUGHY4wxw1iCn4KKohgHT7mJhIZI3XUu3HctbLkPug8HHZoxxhyzRyfjwVWrz2Tdc//K9bEnWP3yExRu+xFIGBa8HhZfDIvfBFWLgg7TGJOHRFWDjmFIfX29Pvvss0GHMWEPvXCA/3r4L7x4oI1zC//KTXP/Qn3fH4k273BWmHWyk+gXvxlqToOQ3TgZY6aHiGxS1fpRl1mCnx6qylOvHubuJ1/l4W0HERGufl2Sd896kbpDjyK7/wCahJI5sHiNc3W/6I0QjQcdujHGL6kUpBKQGoCkOxwaT0DSnRaBWUsmdYhAEryIzAO+DcwGFNigql841jbZnODT7TnczXf+uJt7n/4r7b0DLK8t54YzKlhT8DzRnb+El34N/Z0QLYLXXOAk++rFECmASNwZhguGT4tMX4CqkEq6f2CJI394mnTmDw1T7h/kyHlp04PbISAh5xMKu+PhtHmhI+OD84fWEyemoX2nRsQycnqU+ZC2f3efkhZTenxDy0PDjz/a9xo6xsDw44789xr8fzT0O8nR0+MtGxbnaPNCY3wn9zcd6INkHwz0jxj2QbL/6GH6eGrA/Q46zpBjLE8d+R2HTY+3PH2ao5cfa93B8WFG+fdNG4y9fArTQ/9X0pJ2KjFKbGMongUf2elt3RGCSvBzgbmq+pyIlAKbgEtVddtY2+RKgh/U3T/AD5/by38/+SovN3Yxs6SAdWfNZ139bGY1PwM7fg4v/gI6PPQSFY4dSfaR+IjpAmedZP+RZD3a+NBVQwL3f4fJJ+GYc+EQjroXEbEjFxOhsJuwZIJD0qZHnISGnazcYsmxlh+1fdoxRt3XyGVuLEP5TDM7LQKhKIQizr9vKDLGeBTC7nQo6s4LQ7QYXneRp59xpOOiiEZEfgx8SVUfHmudXEvwg1SV3+1s4r9/v4vf7DhENCxcckoN15+7iOW1ZXDgeeg4AAO97lVVr/sZHO8bsawvbeiOD/6BhaNpf0gjx2PuH1rsyB/e0HvEMB0AAAw3SURBVHjYubJOH4YiaVfa6cvcq/BQ5Mg8OHLlPexKXI/MG3mFnn7VLjJ830cdd7Q40uY7/9DDr/SGXe2N/GjaekmG7ixCx/g3CEVGX2cw3mMmgXGWpQ9HjX+see64hCDiJvChYXoSj6VdcZpcEniCF5GFwOPAMlVtH7HsBuAGgPnz55++e/du3+MJ0qtNXXzr97v4wbN76OpPcvqCSq4/dyHnL55FcYFVajLGTEygCV5ESoDfArep6v3HWjdXr+BH09Gb4AfPNvCtP+xid3M3ABVFUeoqC6mtKKS2oohad7yu0vmUF0YRuwozxqQJLMGLSBT4KfCQqt4+3vr5lOAHpVLK715q4oV9bext6WFvaw8NLT3sbemhJ5Ectm5xLJyW9I+cAGorC5lTFqeqOEY8Gg7omxhjgnCsBO9bmYA4l5rfALZ7Se75KhQS3vi6at74uuph81WVlu6Em/S7aWhxE3+rk/yf+2srbT2Jo/ZXWhBhRkmMGSUFzCh2hjNLYkPjM0pizHSXVRTFCIfsjsCYXOVnoe+5wN8Bz4vIZnfex1X15z4eM2eICFXFMaqKYyyvKx91nY7exFDCb+zoo7mrn6bOPpo7neHu5m6e+2sLh7v6SY1yoxYSqCqOMaO4gJmlznDwBOCcFNKnCyiM2d2BMdnEtwSvqk+QVvPUTL/SeJST5kQ5aU7ZMddLppTW7v5hJ4DmzsETwuC8Pv7c0kpzZz+dfQOj7qcoFnau/t3kP3PwbqDEORFVFMWoKIxSWRSjojhKaUHEnhkYEyCrtpEHwiFxi2cKeN3s0nHX7+lP0tzlngi6+mjq6Kep68idQXNnPw0t3fy5oZXDXf0kR7s9cI9bURiloihKRVGMyqIo5YXOsLI4RvngyaAoSmk8QmE0TDwapigWpjAWJh4JE7IiJGMmzRK8OUphLExdrIi6yqJx102llJbuflq6E7T19NPSlaClu5+2HmfY2p2gtdsZ39fay7Z97bT2JOjuT467b4CCSMhJ+NEwcXdYGHVOAOnDkoIIZYVRyuLOsLww6k5HKSuMUF4YpTAatjsKk1cswZspCaXdHUxEbyJJe0+CFjf5d/UN0JNI0tOfHD4cHHenexNJuvuTdPYN0NjRNzTd1TdA1zgnjUhIjj4JuCeA0niUkoKI84lHho2XusPiggjFsYg9mDZZwxK8CUTcLY6ZVTZ9ja0lkik6egdo70nQ3pugvWeAtqFxZ9jW48wfnLevtYe2ngE6+xL0Jry1G1IcCx91EiiOOePF7qekIJw2fmRe0bD1whRE7MG18Y8leJMzouHQUM2jyRhIpujqS9LRl6Czb4DO3gFnmDbe0TtAlzuvI21+c2c3nX3Osq6+JP1JbyeLaFiG7gwKY87zB+fjTg89k4gMLStMW2dwXkmBc0dSXhglHg1ZUZQBLMEbMyQSDlFeFKK8KDrlffUPpIZOBF39gyeF5JF5fcPndfUP0NPvFDf19Cc51NE7ND449HrSiIVDbhFUhIqi2FDiH3wuUV4YpWJwXpEzLC6IEAuHnE8kRDQsRMLWb0G2swRvjA9ikRCxSIzKSd5NjCaRTA09k+jqG3AS/+AziV6nOGr4x3nYfaijl52HOmjrTtDRN4DXl9dDgpvsQxREnOQfHRy6J4KYOx2Phpy7jMEH3+7dR2HanUb6HUphdOQdiT3b8IMleGOyRNRNrGXxyd9hJFNKR+/IE0GC7r4kfckUiYEU/ckU/QPOJ5FM0efOS182NH8gRXf/AIe7Uu7JZmDojmNgjOqzY4mFQxREQ0O1o+IRp+ZUPBIaqi0VH/qEhmpUxaNhigrClMXTa09FhmpRxSL5eydiCd6YPBIOifNCWtH03VmMpX8g5RQxJQaGFTd1pxVHdSeS9Lgnhd5Eil63plTvYA0qd97hrn56+pP0DiTp6U/Rl3C2HesdjHTxaCitxtTw5O+cECKUFEQpjIXS3sWIuCeaEHH3RFIUi1AQCWXVuxmW4I0xvhgswiln6s80xpJIOieArr7ksNpSQzWo0qbbe53xxs4+Xm7sGlp/gjcaw+8eBoudohFK44Of6IhhhLJR5hXHIr6fLCzBG2Oy1mCxVWk8ypzyiVe5VVW6+pN09DrVZEd7D6M3bdy50zh6eWffAPvbevnLoQQdvU5tq/HuLkRwXtCLR6mpiPODG18/2X+GMVmCN8bkLREZep9hOqkqPYmkm+wTtLtJv6M3MWLo3FlEQ/48J7AEb4wx00xE3NpDEWZP48t8E5W/j5eNMSbHWYI3xpgcZQneGGNylCV4Y4zJUZbgjTEmR1mCN8aYHGUJ3hhjcpQleGOMyVGiXtsOzQARaQR2T3LzmUDTNIaTbez72/e375+fFqhq9WgLjqsEPxUi8qyq1gcdR1Ds+9v3t++fv99/LFZEY4wxOcoSvDHG5KhcSvAbgg4gYPb985t9f3OUnCmDN8YYM1wuXcEbY4xJYwneGGNyVNYneBFZIyIvishLInJL0PH4TUTmicijIrJNRF4QkZvd+VUi8rCI7HSHlUHH6icRCYvIn0Tkp+70IhF5yv07+L6I+N+rdEBEpEJENorIDhHZLiLn5NPvLyIfcv/2t4rI90Qknk+//0RkdYIXkTBwJ/Am4GTgnSJycrBR+W4A+EdVPRk4G7jJ/c63AI+o6onAI+50LrsZ2J42/e/Af6nqa4EW4P8EElVmfAH4paqeBJyK8++QF7+/iNQC64F6VV0GhIGryK/f37OsTvDAmcBLqvqKqvYD9wJvDTgmX6nqflV9zh3vwPnPXYvzvb/lrvYt4NJgIvSfiNQBbwa+7k4LcAGw0V0lZ7+/iJQDbwC+AaCq/araSh79/jhdjRaKSAQoAvaTJ7//RGV7gq8F9qRNN7jz8oKILAROA54CZqvqfnfRAWB2QGFlwh3AR4GUOz0DaFXVAXc6l/8OFgGNwN1uEdXXRaSYPPn9VXUv8HngrziJvQ3YRP78/hOS7Qk+b4lICfBD4IOq2p6+TJ26rzlZ/1VE3gIcUtVNQccSkAiwErhLVU8DuhhRHJPjv38lzt3KIqAGKAbWBBrUcSzbE/xeYF7adJ07L6eJSBQnud+jqve7sw+KyFx3+VzgUFDx+excYK2I7MIpkrsAp0y6wr1lh9z+O2gAGlT1KXd6I07Cz5ff/0LgVVVtVNUEcD/O30S+/P4Tku0J/hngRPcJegznYcuDAcfkK7e8+RvAdlW9PW3Rg8C17vi1wI8zHVsmqOrHVLVOVRfi/N6/UdV1wKPA293Vcvn7HwD2iMhid9ZqYBt58vvjFM2cLSJF7v+Fwe+fF7//RGX9m6wicjFOmWwY+Kaq3hZwSL4Skb8Bfgc8z5Ey6I/jlMPfB8zHaXL5Hap6OJAgM0REVgEfVtW3iMgJOFf0VcCfgKtVtS/I+PwiIitwHjDHgFeA63Eu1vLi9xeRTwFX4tQo+xPwbpwy97z4/Sci6xO8McaY0WV7EY0xxpgxWII3xpgcZQneGGNylCV4Y4zJUZbgjTEmR1mCNzlJRDrd4UIRedc07/vjI6Z/P537N2a6WII3uW4hMKEEn/ZG5FiGJXhVff0EYzImIyzBm1z3WeA8EdnstiMeFpHPicgzIrJFRN4LzktTIvI7EXkQ581IRORHIrLJbXv8BnfeZ3FaMtwsIve48wbvFsTd91YReV5Erkzb92Npbbjf476FaYyvxrtSMSbb3YL7tiuAm6jbVPUMESkAnhSRX7nrrgSWqeqr7vT/VtXDIlIIPCMiP1TVW0Tk71V1xSjHuhxYgdNG+0x3m8fdZacBS4F9wJM47ac8Mf1f15gj7Are5JuLgGtEZDNO8w4zgBPdZU+nJXeA9SLyZ+CPOI3ancix/Q3wPVVNqupB4LfAGWn7blDVFLAZp+jIGF/ZFbzJNwJ8QFUfGjbTadema8T0hcA5qtotIo8B8SkcN71dlCT2f89kgF3Bm1zXAZSmTT8EvM9tchkReZ3bYcZI5UCLm9xPwukecVBicPsRfgdc6ZbzV+P0vPT0tHwLYybBriJMrtsCJN2ilv/GaTt+IfCc+6CzkdG7d/slcKOIbAdexCmmGbQB2CIiz7lNFQ96ADgH+DNOhxsfVdUD7gnCmIyz1iSNMSZHWRGNMcbkKEvwxhiToyzBG2NMjrIEb4wxOcoSvDHG5ChL8MYYk6MswRtjTI76/99Rh662yY3BAAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "metrics_to_plot = [\"train-rmse\", \"validation-rmse\"]\n",
- "for metric in metrics_to_plot:\n",
- " steps, data = get_data(trial, metric)\n",
- " plt.plot(steps, data, label=metric)\n",
- "plt.xlabel('Iteration')\n",
- "plt.ylabel('Root mean squred error')\n",
- "plt.legend()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Feature importances\n",
- "\n",
- "We can also visualize the feature importances as determined by\n",
- "[xgboost.get_fscore()](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.Booster.get_fscore).\n",
- "Note that feature importances with zero values are not included here\n",
- "(which means that those features were not used in any split condisitons)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {},
- "outputs": [],
- "source": [
- "def plot_collections(trial, endswith, ylabel=''):\n",
- " \n",
- " plt.figure(\n",
- " num=1, figsize=(8, 8), dpi=80,\n",
- " facecolor='w', edgecolor='k')\n",
- "\n",
- " features_to_plot = [\n",
- " tname for tname in trial.tensors()\n",
- " if tname.endswith(endswith)\n",
- " ]\n",
- "\n",
- " for feature in sorted(features_to_plot):\n",
- " steps, data = get_data(trial, feature)\n",
- " label = feature.replace(endswith, '')\n",
- " plt.plot(steps, data, label=label)\n",
- "\n",
- " plt.legend(bbox_to_anchor=(1.04,1), loc='upper left')\n",
- " plt.xlabel('Iteration')\n",
- " plt.ylabel(ylabel)\n",
- " plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnsAAAITCAYAAAB/k9qgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAMTQAADE0B0s6tTgAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXxU5b0/8M+ZJfu+LzOZ7GTfSMSiVhFlqS3i0hav3go/Eao/5frTX7XlVm57u9let0KvBaul8rNSewUEaXEBRKLQEgjZSUIgk0x2sq+T2c7vj5CRCCSTZE4ymXzer9e8ksw858k3vE57Pj7Pc54jiKIogoiIiIickmy2CyAiIiIi6TDsERERETkxhj0iIiIiJ8awR0REROTEGPaIiIiInBjDHhEREZETY9gjIiIicmKK2S5ASq6urggODp7tMoiIiGgSLl26hOHh4dkuw2k4ddgLDg5GQ0PDbJdBREREk6BSqWa7BKfCaVwiIiIiJ8awR0REROTEnHoal4iIiGg8oihaX3ONIAiQySYet2PYIyIionnHYrGgra0N3d3dczLojVIqlYiKioKLi8t12zDsERER0bxTV1cHmUyG6OhoKJXK2S5nSkRRREdHB+rr6xEfH3/ddgx7RERENK9YLBbo9XokJCRAoZjbUSgwMBCdnZ2wWCzXndLlDRpEREQ0r4xO2wqCMMuVTN/o3zDeVDTDHhEREZETY9gjIiIicgD79+9HcnIysrKysGTJEkRHR0MQBBQVFU2r37k9UU1ERETkJLZv344tW7bggQcewPHjxxEbG4ubb7552v0y7BERERHNsk2bNiE/Px+VlZXYtm0bTpw4Ybe+GfaIiIho3lv/VgHqOgYl6VsT6IE3Hs4bt83WrVtRUlKCp556CqtXr7br7+eaPSIiIiInxpE9IiIimvcmGnmbyziyR0REROTEGPaIiIiIHMzGjRuhUqnQ0NCA5cuXj/s4tIlIHvbOnz+PxYsXIzExEXl5eSgvL79uW1EUcfvtt8PPz2/M+wcPHkRSUhISEhJw7733ore3V+qyiYiIiGbUsWPHrDdn7NixAw0NDTCZTGhtbUVNTc2U+5U87G3cuBEbNmxAdXU1nnvuOaxdu/a6bV955RXExcWNea+/vx+PPPII3n//fZw/fx4RERH42c9+JnHVRERERM5B0rDX1taG06dP46GHHgIA3HfffdDpdNdMp+Xl5Xj//ffxwx/+cMz7hw4dQnZ2NpKSkgAAjz/+OHbv3i1l2eMa7OvFh3v+B70d7bNWAxEREZGtJA17Op0O4eHhUChGbvoVBAFRUVGor68f085oNOLRRx/Fjh07IJfLx3xWX18PjUZj/Tk6OhrNzc0wmUxSln5dRw5+gH+UluPI3/82K7+fiIiIaDIc4gaNn/70p7j33nuRnJw8rX5efvllqFQq66u/v99OFX7p5qVLIRiHUXZRi8FBaTZfJCIiIrIXScOeWq0eMwoniiLq6+sRFRU1pt1nn32Gbdu2ITo6GjfffDN6e3sRHR2NS5cuISoqCnV1dda2Wq12zGjhlZ5++mk0NDRYX15eXnb/m/xDwhBgGIRZFHH8+HG7909ERERkT5KGvZCQEOTk5ODtt98GAOzZswcqleqq24fz8/NRV1cHrVaLzz//HD4+PtBqtQgODsaKFStQWFiIyspKAMBrr72GNWvWSFn2hBbEx0M2NIBTp06hs7NzVmshIiIiGo/k07g7duzAjh07kJiYiBdeeAE7d+4EAKxfvx4HDhyY8Hhvb2+88cYbWL16NeLj49HQ0IDnn39e6rLHFZWWAddWHSwWCw4fPjyrtRARERGNR/LHpS1YsAAnT5686v033njjmu2jo6PR3d095r1Vq1Zh1apVktQ3FerUDCiG+uHvokBFRcU1p6aJiIiIJmP//v344Q9/CFEUERkZCZ1OB3d3d4SEhOD3v//9lDdWdogbNOYaTz9/BESqoWiqhUwmw8cffwxRFGe7LCIiIprDtm/fji1btqCoqAjPPPMMqqqqUFxcjLvvvhvr16+fcr+Sj+w5K3VqBoo//hvSb12O4vIKlJeXIy0tbbbLIiIioql4Zw3QVStN3/4xwL/8ZdwmmzZtQn5+PiorK7Ft2zacOHHC+tmNN96IF198ccq/niN7UxSVmg4AUHm5w9XVFYcPH561vf+IiIhobtu6dStyc3PxyiuvjAl6APDb3/4Wd99995T75sjeFKlSRsJea/U53HLLLTh8+DBOnTqFxYsXz3JlRERENGkTjLzNll/+8peoqanBkSNHptwHR/amyMPHF0FR0dCVl2DRokXw9fXF8ePHudEyERER2cWLL76IvXv34tChQ/Dw8JhyPwx706BOTcdgTzd6W1uwdOlS6PV6brRMRERE0/byyy9j9+7d+OSTT+Dn5zetvhj2pkGdmgEA0FWUIi0tDRERETh16hQ6OjpmuTIiIiKaqxoaGvDMM8+gu7sbS5YsQVZWFhYtWjTl/rhmbxrUyemAIEBXXoKsZd/AsmXL8Kc//QlHjhzBd77zndkuj4iIiOaQY8eOWb+355ZuHNmbBjcvL4RoYqGrKIUoioiOjkZSUpJ1o2UiIiKi2cawN03q1HQM9fagQ1cHALjjjju40TIRERE5DIa9aRpdt1dfXgoACAoKQm5uLhoaGlBeXj6bpREREREx7E2XKjkVgiCDrrzE+t6tt97KjZaJiIjIITDsTZOrhydCY+PQcK4MosUCAPD09MQtt9yC7u5unDp1apYrJCIiovmMYc8O1KkZ0Pf34VK91voeN1omIiIiR8CwZwfqy49Ou3IqV6lU4o477uBGy0RERGST/fv3Izk5GVlZWViyZAkyMjKQlZWFW265BWfPnp1yvwx7dhCZlAJBJkP9FWEPAFJTU7nRMhEREdlk+/bt2LJlC4qKirBv3z6UlJSgqKgITz/9NNauXTvlfhn27MDF3QNhcQloPFcOi8VsfV8mk2HZsmWwWCzTeoAxERERObdNmzYhPz8fmzdvxuLFi8c8Iq2npweCIEy5bz5Bw07UqRloPl+FttqLCItLsL7/1Y2Wo6KiZrFKIiIiupYnjzwJXZ9Okr7V3mpsW7pt3DZbt25FSUkJnnrqKaxevRoA8L3vfQ+ffvopAODvf//7lH8/R/bsxPqc3K9M5QJfbrT80UcfcaNlIiIissmuXbug0+nw85//HM8999yU++HInp1ELkiGTK6ArrwEeavuG/PZ6EbLp06dQnl5OdLS0mapSiIiIrqWiUbeZtPDDz+M73//++jo6EBgYOCkj+fInp0oXd0QnpCIhsoKmK+xkTI3WiYiIiJbdHd3o6mpyfrz+++/j8DAQAQEBEypP4Y9O1KnZsCoH0LrxZqrPuNGy0RERGSLnp4erF69Gunp6cjMzMTvfvc7HDx4cMo3aXAa147UKRn4x56/QFdegojEpKs+X7RoEQoKCnD8+HFkZWXBw8NjFqokIiIiR3Ts2DHr9/YcGOLInh1FJCZBrlRCV1F6zc+50TIRERHNNIY9O1K4uCAiIQmNVRUwm4zXbMONlomIiGgmMezZmTo1A6bhYbTUnL/m5zKZDMuXL4fFYsHhw4dnuDoiIiKabxj27EydevVzcr9Ko9EgKSkJ586dQ319/UyVRkRERPMQw56dhcUvgMLFFbqK64c9gBstExER0cxg2LMzhVKJiAXJaKqqhMl47XV7wJcbLTc2NqK8vHwGKyQiIqL5hGFPAlGpGTAZDWg+XzluO260TERERFJj2JOALev2gJGNlr/+9a9zo2UiIiLC/v37kZycjKysLJSWjmzjtnPnTgiCgPfff3/K/TLsSSA0NgFKVzfoyq+9396VbrjhBvj6+uL48eMYHBycgeqIiIjIEW3fvh1btmxBUVER0tPTodVq8Yc//AE33njjtPrlEzQkIFcoEJmUAl15CYyGYShdXK/bdnSj5T179uCzzz7DypUrZ7BSIiIiAgDdY4/DoJNmhwwXdRTUv39t3DabNm1Cfn4+KisrsW3bNnz++edYv349tm3bhmeeeWZav58jexJRp2bAbDKhqerchG3T0tIQGRmJgoICbrRMREQ0D23duhW5ubl45ZVXcOLECbz88su46aabsHDhwmn3zZE9iXy5bq8UmvSscdsKgoBly5Zh586dOHz4ML773e/ORIlERER02UQjbzOprKwMe/bssdujVRn2JBIaEw8Xd/cJb9IY9dWNlqOioiSukIiIiBxRfn4+tFotEhISAAAtLS3YsGEDmpub8dhjj026P07jSkQml0OVnIaWC9Uw6IdsOoYbLRMREdFjjz2G5uZmaLVaaLVa3HjjjXj99denFPQAhj1JqVPSYTGb0VRZYVP7oKAgZGdno7GxEZcuXZK4OiIiIpoPOI0rIXVqBgCgvqIU0Vm2LbBMSEjAmTNnUFdXh5CQECnLIyIiIgdy7NixSb1vK47sSSg4Ogaunp42r9sDYF2rp9VqJaqKiIiI5hOGPQnJZHKoktPRerEGwzZumOzh4YGQkBDU1dVx3R4RERFNG8OexKJS0yFaLGisKrf5mOjoaPT396Ozs1PCyoiIiGg+YNiT2Oi6PVsenTZKo9EAAOrq6iSpiYiIiOYPhj2JBak1cPP2mdS6vdGwx3V7RERENF0MexITZDKoU9LQVnsR+oF+m47x8vJCYGAgR/aIiIho2hj2ZoA6NQOiaEHDucmt2+vp6UF3d7eElREREZGj2L9/P5KTk5GVlYXAwEAsWLAAWVlZyMrKwrvvvjvlfhn2ZkCUdd3e5KdyObpHREQ0P2zfvh1btmxBUVERvL298e6776KoqAhFRUX47ne/O+V+GfZmQECkGh6+fgx7REREdE2bNm1Cfn4+Nm/ejMWLF9u1bz5BYwYIggB1SjqqTuZjqK8X7t4+Ex7j6+sLPz8/3qRBREQ0A/72Wgl6Ltn2LPvJ8g12x12PZ4zbZuvWrSgpKcFTTz2F1atXIzo6Gt/73vcgiiJuuOEGvPDCCwgODp7S7+fI3gwZ3YKloaLM5mM0Gg06OzvR19cnVVlERETkgI4fP46SkhIUFhYiKCgIDz/88JT74sjeDLE+J7e8BAmLbBuejY6ORnFxMerq6pCWliZleURERPPaRCNvM2308alKpRJPPfUUEhMTp9yX5CN758+fx+LFi5GYmIi8vDyUl199R+rJkyetd5ukpqZi48aNGB4eBjDy8F93d3fr51lZWRgakmaYVUr+4RHw9A/guj0iIiIa18DAwJjdOHbv3o3s7Owp9yd52Nu4cSM2bNiA6upqPPfcc1i7du1VbTIzM1FQUICioiKUlpaira0Nr732mvXzBQsWWO9GKSoqgru7u9Rl293our2OhnoM9ti2nYq/vz+8vb25bo+IiGgeaW1txZIlS5CRkYH09HR89tln2LVr15T7k3Qat62tDadPn8bHH38MALjvvvvwxBNPoKamBvHx8dZ2Hh4e1u8NBgOGhoYgCIKUpc0KdWoGKr/4DLqKUiz42i0TthcEARqNBmVlZRgYGICnp+cMVElERESz4dixY9bvz549a7d+JR3Z0+l0CA8Ph0IxkikFQUBUVBTq6+uvaqvVapGZmYmgoCD4+vri8ccft3524cIF5OTkIC8vb8yI31wzlf32oqOjAeCa/2ZEREREE3GYu3FHb0ZoaWnB8PAw9u7dCwDIyclBQ0MDCgsLsW/fPmzfvh1//etfr9nHyy+/DJVKZX3199v2eLKZ4hsaBu/AYNSXl9p8DNftERER0XRIGvbUajWam5thMpkAAKIoor6+3nqHybV4eXlhzZo1+POf/wwA8PHxga+vLwBApVLhgQceQH5+/jWPffrpp9HQ0GB9eXl52fkvmh5BEKBOTUdXUwP6OztsOiYoKAgeHh4Me0RERDQlkoa9kJAQ5OTk4O233wYA7NmzByqVasx6PQCoqamB0WgEMLJmb9++fcjIGJnybG5uhsViAQD09fXh4MGD07ojZbaNbsGiq7BtdG903V5LSwv0er2UpREREZETknwad8eOHdixYwcSExPxwgsvYOfOnQCA9evX48CBAwCAo0ePIjs7G5mZmcjOzkZoaCief/55ACMBMT09HZmZmbjxxhtx5513Yt26dVKXLZmpPid3dFSUiIiIaDIEURTF2S5CKiqVCg0NDbNdxlXeePIRCDIZHvntH2xq39LSgu3bt+Omm27CnXfeKXF1REREs0vq67fZbEZ1dTUSExMhl8sl+z0zwZa/xWFu0JhP1KkZ6G5pRl9Hu03tQ0JC4ObmxnV7RERENGkMe7NAPcmpXJlMhqioKDQ1NcFgMEhZGhEREc2S/fv3Izk5GVlZWSguLsYTTzyBhIQEpKen46GHHppyvwx7s0Cdmg5g5Dm5ttJoNLBYLNDpdFKVRURERLNo+/bt2LJlC4qKivCnP/0JgiCguroapaWlePHFF6fcr6RP0KBr8w4Ign94BHST2G9vdHPluro6xMXFSVQZERHR/LTvN/+JntYWSfr2DQ3DPc9uGbfNpk2bkJ+fj8rKSvzmN7/BhQsX0NDQYH2iWFhY2JR/P0f2Zok6JQO9l1rR09ZqU/uwsDC4uLhw3R4REZET2rp1K3Jzc/HKK6/grbfeQkBAAH75y18iNzcXt9xyC44cOTLlvjmyN0vUqekoOfIhdOUl8A2Z+A5buVwOtVoNrVYLo9EIpVI5A1USERHNDxONvM0kk8mEuro6pKSk4IUXXsDZs2dx5513ory8HKGhoZPujyN7s2SyN2kAI+v2zGYzGhsbpSqLiIiIZllUVBRkMhkefPBBAEB2djZiYmJQWmr78q8rMezNEk8/fwREqlFfUQpbtzq8ct0eEREROaegoCAsXboUH330EQCgtrYWtbW1SE5OnlJ/DHuzSJ2agf6OdnS3NtvUPiIiAgqFgmGPiIjIyW3fvh3/9V//hfT0dKxevRo7duxAZGTklPrimr1ZpE5JR/HHf4OuvAT+YRETtlcoFFCpVNDpdDCbzXN+128iIiL60rFjx6zfx8bG4tNPP7VLvxzZm0Wj++1NZgsWjUYDo9GIpqYmqcoiIiIiJ8KwN4s8fHwRpNZAV15i87o9jUYDgOv2iIiIyDYMe7NMnZqBge4udDbZ9sBnlUoFmUzGsEdEREQ2YdibZZOdynVxcUFkZCTq6+thsVikLI2IiIicAMPeLFOlpAOCMOn99oaHh9HSIs1jXYiIiMh5MOzNMncvbwRrYqCbxH57XLdHREREtmLYcwBRqekY6u1BR0O9be2joiAIAsMeERGRE9m/fz+Sk5ORnp4OuVyOrKwsZGVlITExEQqFAp2dnVPql2HPAUz20Wmurq4IDw9HXV0d1+0RERE5ie3bt2PLli0oLS2F2WxGUVERioqKsGHDBqxcuRIBAQFT6pdhzwGoktMgCLJJ77c3NDSE9vZ2CSsjIiKimbBp0ybk5+dj8+bNWLx48ZjP3nzzTTzyyCNT7ptP0HAArh6eCImJG1m3Z7FAkE2cwTUaDU6ePAmtVouQkJAZqJKIiMh5tb9VDlOHXpK+FYFuCHo4ddw2W7duRUlJCZ566imsXr3a+v6JEyfQ1dWFb37zm1P+/RzZcxDq1HTo+/twqV5rU3vepEFEROT83nzzTXzve9+DQjH18TmO7DmIqNQMnP5gL3TlpQiJjp2wvbu7O0JDQ1FXVwdRFCEIwgxUSURE5JwmGnmbDf39/fjrX/+KgoKCafXDkT0HEZmUAkEmg65icvvt9ff3T/nuHCIiInJc7777LjIzM5GUlDStfhj2HISLuwfC4hLQUFEGi8Vs0zGjU7larVbCyoiIiGg2TPfGjFGcxnUg6tQMNJ+vwiVtLUJj4ydsf+W6vYULF0pdHhEREUno2LFjY34+ceKEXfrlyJ4DGd1vr97G/fa8vLwQFBTEmzSIiIjouhj2HEjkgmTI5IpJPye3p6cH3d3dElZGREREcxXDngNRurohPCERjZXlsJgnt26Po3tERER0LQx7DiYqLROGoSE0VZ+zqT1v0iAiIqLxMOw5mJjsXABA7dnTNrX39fWFv78/R/aIiIjomhj2HExYbALcfXxx0cawB4yM7nV2dqKvr0/CyoiIiGguYthzMIJMhpishWiv16K3vc2mY7huj4iIiK6HYc8BxebkAQBqz56xqT3X7REREc19+/fvR3JyMrKysvDb3/4WOTk5yMrKQlpaGt56660p98tNlR2QJiMbgkyGi2cLkHnnygnb+/v7w8fHhyN7REREc9j27duxZcsWrFmzBoGBgTh27BgyMjKg1WqRlJSEe++9F97e3pPulyN7DsjN0wuRC1JQX1YMk8EwYXtBEKDRaHDp0iUMDAzMQIVERERkT5s2bUJ+fj42b96MxYsXQxAE6x66vb29CAwMhKur65T65sieg4rJzkXDuTI0VJQiOmviR6FpNBqUlpaivr4eycnJM1AhERGR83jnnXfQ1dUlSd/+/v74l3/5l3HbbN26FSUlJXjqqaewevVqHD58GPfeey88PT3R1dWFvXv3wsXFZUq/nyN7Dir28hYstt6Vy3V7REREzsFkMuHnP/859u7di7q6Ohw5cgT/+q//ivb29in1x5E9BxWo1sA7KBgXzxZgydoNEARh3PZBQUHw9PTkuj0iIqIpmGjkbSYVFRWhqakJX//61wEAeXl5UKlUOHv2LO68885J98eRPQclCAJis/PQ09qCruZGm9prNBq0tLRAr9fPQIVEREQkBbVajebmZpw7N/I0rZqaGly4cAELFiyYUn8Mew5ssk/TGJ3Kra+vl6wmIiIiklZoaChef/11fOc730FmZibuuece/O53v0NUVNSU+uM0rgOLSsuAXKnExcICLLxr9YTtr9xcOTExUeryiIiIyI6OHTtm/f6BBx7AAw88YJd+ObLnwJSublCnZqDhXDkMQ4MTtg8JCYGbmxtv0iAiIiIrhj0HF5udC4vZhLrSognbymQyaDQaNDc3Y3h4eAaqIyIiIkfHsOfgYrJHHp12sdD2dXsWiwUNDQ1SlkVERERzBMOeg/MLDUNAhAq1RachiuKE7a9ct0dERERXG93OzJbrqqMb/RvG26KNN2jMATE5eThzcB/atBcRGhM3btuwsDC4uLhw3R4REdF1yGQyuLm5obGxEaGhoVAqlbNd0pSIooiOjg4olUrIZNcfv2PYmwNis0fCXm1hwYRhTy6XIyoqCrW1tTAajXP2BCYiIpKSRqNBW1sbtFrtnB7hUyqVE27JwrA3B0QmJcPF3R0Xi07jxvvWTNheo9GgpqYGjY2NiI6Olr5AIiKiOUYmkyEsLAyhoaEQRXFOBj5BEMYd0RvFsDcHyBVKaDKycf7USQz29sDDx3fc9leu22PYIyIiuj5BECZ8JOlcxxs05oiY7FxAFFFXXDhh24iICCgUCt6kQURERAx7c0Xs6BYsNjw6TaFQQKVSQafTwWw2S10aEREROTDJw9758+exePFiJCYmIi8vD+Xl5Ve1OXnyJLKyspCVlYXU1FRs3LhxzKbAb775JhISEhAXF4dHH30URqNR6rIdjqefP0Jj46EtOgOLDQEuOjoaRqMRTU1NM1AdEREROSrJw97GjRuxYcMGVFdX47nnnsPatWuvapOZmYmCggIUFRWhtLQUbW1teO211wAAtbW1eP7555Gfn4+amhq0trbi9ddfl7pshxSTnQv9QD+az1dN2Jb77REREREgcdhra2vD6dOn8dBDDwEA7rvvPuh0OtTU1Ixp5+HhYd0ixGAwYGhoyLpY8r333sOqVasQFhYGQRDw/e9/H7t375aybIf15VRuwYRtVSoVZDIZwx4REdE8J2nY0+l0CA8Ph0IxctOvIAiIiopCfX39VW21Wi0yMzMRFBQEX19fPP744wCA+vp66ygVMDI9ea3j54PQuHi4e/ug1oZ1e0qlEpGRkaivr4fFYpmB6oiIiMgROcwNGtHR0SguLkZLSwuGh4exd+/eSffx8ssvQ6VSWV/9/f0SVDp7ZDI5YrIW4lJdLfo62idsHx0djeHhYbS0tMxAdUREROSIJA17arUazc3NMJlMAEYe61FfXz/uTs9eXl5Ys2YN/vznPwMAoqKixkxFarXa6x7/9NNPo6Ghwfry8vKy41/jGGJyRqZybRnd47o9IiIikjTshYSEICcnB2+//TYAYM+ePVCpVIiPjx/TrqamxnqHrcFgwL59+5CRkQFgZJ3fgQMH0NLSAlEUsX37dqxZM/FTJJxVdEYOBJnMpi1Y1Go1BEFg2CMiIprHJJ/G3bFjB3bs2IHExES88MIL2LlzJwBg/fr1OHDgAADg6NGjyM7ORmZmJrKzsxEaGornn38eABAbG4uf/vSnuOmmmxAfH4/g4GBs3LhR6rIdlpuXFyISk1FfWgTTBFvQuLq6Ijw8HHV1dVy3R0RENE8J4lx8GJyNVCoVGhoaZrsMuzu1/z3kv/Mn3PfvP0N0Rva4bT/66COcPHkSjz32GEJDQ2eoQiIioqlz1uv3bHGYGzTIdjHZuQCA2sKJt2AZfTYup3KJiIjmJ4a9OShIrYF3YLBN++2N3szCsEdERDQ/MezNQYIgICZ7IbpbmtHV3DhuW3d3d4SGhqKurg5OPGNPRERE18GwN0fFXt6C5WKhbVuw9Pf3o6OjQ+qyiIiIyMEw7M1RUamZkCuVqC2aOOxx3R4REdH8xbA3Rynd3KBOSUdDRSkM+qFx23LdHhER0fzFsDeHxWTnwWwyoa60aNx2Xl5eCAoKYtgjIiKahxj25rDY0S1YbHx0Wk9PD7q6uqQui4iIiBwIw94c5hcWDv8IFWrPnp7wTluu2yMiIpqfGPbmuNjsXPR3duBSXe247TQaDQCGPSIiovmGYW+Oi7FxKtfHxwf+/v4Me0RERPMMw94cp0pOhYu7Oy7a8Og0jUaDzs5O9Pb2zkBlRERE5AgY9uY4uUIJTXo2ms9XYahv/BDHqVwiIqL5h2HPCcRk50IULdAWF47bjjdpEBERzT8Me04gJmshgInX7fn5+cHX1xeVlZUwGAwzURoRERHNMoY9J+AVEIiQ6DjUFhfCYjFft50gCLjpppvQ39+Pzz//fAYrJCIiotnCsOckYnNyoe/rRSlJvmAAACAASURBVPP56nHbLVy4ECEhIThx4gQ3WCYiIpoHGPacREx2HoCJp3LlcjlWrFgBk8mETz75ZCZKIyIiolnEsOckwuIT4O7tg4tnJ96CJTY2FklJSaioqEBt7fibMRMREdHcxrDnJGQyOaKzFuKS9iL6OtsnbL9s2TLI5XIcOnQIZvP11/kRERHR3Maw50S+fJrGmQnbBgQEYPHixWhra0Nh4fhbthAREdHcxbDnRKIzcyAIMtTaMJULADfffDO8vb1x9OhRDA4OSlwdERERzQaGPSfi7uWNiAVJqCsthslonLC9q6sr7rjjDgwNDeHYsWPSF0hEREQzjmHPycRk5cKoH0LjuXKb2mdkZEClUqGgoACtra0SV0dEREQzjWHPycTmXN6Cpci2qVxBELBy5UqIoogPP/wQoihKWR4RERHNMIY9JxMUFQ2vwCBcLBx/v70rRUZGIisrC7W1taisrJSwOiIiIpppDHtORhAExGbloqu5EV0tTTYft3TpUri4uODjjz+G0Yb1fkRERDQ3MOw5oZgc256mcSVvb2/ceuut6Orqwj/+8Q+pSiMiIqIZxrDnhKLSMiBXKHCx0LZ1e6MWLVqEgIAAHD9+HL29vRJVR0RERDOJYc8Jubi5Q5WSjoaKUhj1epuPUygUWL58OYxGIw4fPixhhURERDRTGPacVGxOHswmE+rKiid1XGJiIuLj41FSUgKdTidRdURERDRTGPaclPXRaZOcyhUEAcuXL4dMJsOhQ4dgsVikKI+IiIhmCMOek/IPi4B/eCQuFp2e9N55wcHBuOGGG9DU1ITi4smNDBIREZFjYdhzYjHZuejvaEd7vXbSx956663w8PDA4cOHoZ/Euj8iIiJyLAx7Tiw2e2QLlouT2IJllLu7O5YuXYqBgQHk5+fbuzQiIiKaIQx7TiwyORVKN3fUnp3cur1R2dnZCAsLw8mTJ9HR0WHn6oiIiGgmMOw5MYVSCU16JpqqKjHU3zfp42UyGVauXAmLxYKPPvpIggqJiIhIagx7Ti4mOw+iaEFdceGUjtdoNEhLS0N1dTVqamrsXB0RERFJjWHPycVkLwQwtXV7o+68804oFAp8+OGHMJvN9iqNiIiIZgDDnpPzDghCcHQstEVnYLFMLaj5+vri5ptvRnt7O06dOmXnComIiEhKDHvzQGx2Hob6etFSc37KfSxevBi+vr44duwYBgYG7FgdERERScmmsNfT04MnnngC3/zmNwEAFRUV2L17t6SFkf1Yn6YxxbtyAcDFxQXLli3D8PAwjh49aq/SiIiISGI2hb2NGzciLCwMWq0WABATE4Nf//rXUtZFdhSekAg3b59prdsDgJSUFGg0Gpw5cwbNzc12qo6IiIikZFPYq66uxo9//GMolUoAIxvuTvYRXDR7ZDI5YjJz0FZ7Af1dnVPuRxAErFy5EoIg4NChQzwHiIiI5gCbwp6Li8uYn4eGhnihn2O+nMqd3uheWFgYFi5ciPr6epSXl9ujNCIiIpKQTWFvyZIl+MUvfgG9Xo/Dhw/j/vvvx7333it1bWRH0Zk5EATZtMMeMHI+uLm54eOPP4bBYLBDdURERCQVm8Lez372M8hkMvj4+GDz5s246aab8Pzzz0tdG9mRu7cPwhMWoK70LMwm47T68vT0xG233Ybe3l588cUXdqqQiIiIpGBT2FMoFPjRj36Ef/7znzh16hQ2b94MuVwudW1kZ7E5eTAMDaGxsmLafeXl5SE4OBhffPEFuru77VAdERERScGmsLd+/Xp0dHRYf25vb8fGjRslK4qkMbpu72Lh1LdgGSWXy7FixQqYTCZ88skn0+6PiIiIpGFT2Dtz5gwCAwOtPwcFBaGgYPqBgWZWsCYG3kHBKPr4byg8dACixTKt/uLi4rBgwQKUl5dbt+UhIiIix2JT2DOZTGN+FkWRC/PnIEEQsOrpzfAOCMKnf3ode1/4Cfo7OyY+cBzLly+HXC7HoUOHYJlmeCQiIiL7syns3XjjjXjiiSdQV1cHrVaLJ598EjfeeKNNv+D8+fNYvHgxEhMTkZeXd83tOo4ePYobbrgBKSkpSE1NxbPPPmsNDlqtFnK5HFlZWdbXhQsXJvEn0pXC4hLwr7/+LdKWLIO2uBBvPfskzp86MeX+AgIC8LWvfQ2tra0oLCy0Y6VERERkDzaFvZdeegmDg4PIy8vDokWLMDw8jFdeecWmX7Bx40Zs2LAB1dXVeO6557B27dqr2vj7++Mvf/kLKioqcObMGZw4cQK7du2yfu7t7Y2ioiLrKy4uzra/jq7Jxd0Dy7+/Caue2QwAOPDSL/HR9t/CMDQ4pf5uueUWeHl54ejRoxgaGrJnqURERDRNgijh7shtbW2Ij49HZ2cnFAoFRFFEeHg4Pv/8c8THx1/3uCeeeAJBQUH4yU9+Aq1Wi6ysrCnd8alSqdDQ0DCdP8Hp9Xd14qPfvwptcSF8Q8PwjSeeQURi8qT7KSoqwvvvv49FixZh5cqVElRKRETzBa/f9mXTyB4A/POf/8Q777yDXbt2WV8T0el0CA8Ph0KhADCyZiwqKgr19fXXPaalpQXvvfcevvnNb1rfGxgYQF5eHnJycvCf//mfMJvNtpZNE/DyD8C9P/oplqzdiIHOTvzlP57Dif/5MyyT/DfOyMhAZGQkTp06hba2NomqJSIiosmyKew99thjeOCBB/Dee+/hgw8+wAcffICDBw/avZje3l5861vfwrPPPovc3JFtQsLDw9HY2IiCggIcPnwY+fn5eOmll655/MsvvwyVSmV99ff3271GZyQIAnJWfgsP/uoVBKk1OPnebvxly7PoammyuQ+ZTIaVK1dCFEXs2rWL/0VGRETkIGyaxk1ISEBpaSnc3Nwm1flkpnH7+vqwfPlyfOMb38CPf/zj6/a5e/duvPPOO/jggw8m/P0cBp48k9GIE399GwUf7IXSxRW3Pfwo0m9fBkEQbDq+rKwM77//PgDg7rvvRnp6upTlEhGRE+L1275sGtkLDw+Hq6vrpDsPCQlBTk4O3n77bQDAnj17oFKprgp6/f39WLFiBVasWHFV0Gtra4PROPJ4r+HhYezduxfZ2dmTroVso1Aq8fUH1+HbP/4FXL288Mnr23DgpV9gsLfHpuPT0tKwbt06uLu7Y8+ePTh69Ci3ZCEiIppFNo3s/eAHP8DFixfx3e9+d8zo3qpVqyb8BVVVVVi7di06Ojrg4+ODnTt3Ij09HevXr8eqVauwatUq/OIXv8BPfvITpKamWo/79re/jX//93/H3r17sWXLFsjlcphMJtx+++148cUXbQqf/C+D6dH39+Pwm6+h6sRxePr5Y/ljTyEma6FNx/b29uIvf/kLmpqakJycjHvuuQcuLi4SV0xERM6A12/7sinsLVmy5OoDBQFHjx6VpCh74cliH+fyP8XhN38Pw9Agsld8C7c8uBZKl4nDtsFgwP79+1FeXo6wsDA88MAD8PX1nYGKiYhoLuP1274k3XpltvFksZ/eS2049N8vo+FcGQIi1fjGk/8XoTET73coiiI+++wzHDt2DF5eXlizZg1UKtUMVExERHMVr9/2ZXPYMxqNqK2thV6vt76XkZEhWWH2wJPFviwWM05/sA9fvDuyBvOm7z6E3G/dA5lMPuGx5eXl2LdvH0RRxN133+3w5w4REc0eXr/ty6awd/DgQTz66KPo6uqCp6cnurq6oNFoUFtbOxM1ThlPFmm0XqzB37e9iM6mBqhT0rHif/8f+ASFTHhcU1MTdu/ejb6+Ptxyyy1YsmQJZDKbt3okIqJ5gtdv+7LpSvv888/jH//4B5KTk9HR0YFdu3bh/vvvl7o2clChsfF46IVXkbX8LugqSrHrB0/i3BefTXhcREQEHn30UURERCA/Px9//etfYTAYZqBiIiKi+cumsCeTyaDRaGAymQAADz30kMPfnEHSUrq6Yen/egz3/PA/IFcq8fet/4W/b3sR+oHxN7L28fHBunXrkJqaisrKSvzxj39ET49t27oQERHR5NkU9pRKJYCRYdV9+/bh7Nmz6OrqkrQwmhtis/Pw8H/9DnG5i3Du82PY9eyTaDhXNu4xSqUS999/P5YsWYKWlha8/vrr0Ol0M1QxERHR/GLTmr3du3djxYoVuHjxItasWYPu7m68+uqrePDBB2eixinjnP/MEUURpUc/wqdv/QFmgxFf+/YDWHTPdya8eYM3bhAR0Vfx+m1fNoW99vZ2BAUFTfieo+HJMvM6GnX426u/xqV6LaLSMrDyif8LL/+AcY/hjRtERHQlXr/ty6Yr6rJly2x6jygwUo0HfvESMu/8BurLSrDr2SdRW3Rm3GMiIiKwYcOGMTduDA8Pz1DFREREzm3csGcwGNDb2wuz2Yy+vj709vait7cXOp0OAwMDM1UjzTFKF1fcsf5xfOvpH8FiMmHvr/4Dx/+8E+bLN/hci7e3N9atW4e0tDTrjRvd3d0zWDUREZFzGjfs/epXv4Kfnx9KS0vh6+sLPz8/+Pn5IT09HQ899NBM1UhzVOKim/Cvv96K8PgFKDiwB+/+x3PoaWu5bnulUon77rsPS5YsQWtrK/7whz/wxg0iIqJpsmnN3mOPPYbf//73M1GPXXHO3zGYTSZ88e7/Q8GBPXD18MSyjU8i8cabxz2moqIC+/btg8ViwapVq5CZmTlD1RIR0Wzj9du+JlyzZzab8dlnE2+YS3Q9coUCX39wHe790U8hUyjwwSsv4PAb/w2j4frr8lJSUrBu3Tp4eHhg3759OHz4MCwWywxWTURE5BwmDHtyuRzBwcEYHByciXrIicVkLcT3frMNUWkZKP7kEHb/+zPoaLz+NO3ojRuRkZH4/PPPeeMGERHRFNg0jfvII4+gsLAQ3/72t+Hl5WV9f9OmTZIWN10cBnZMFosZp/b9D078zzuQuyix9H89htRbl0IQhGu2NxqN2L9/P8rKyhAaGoqlS5ciPj6e27MQETkpXr/ty6awt27duqsPFAT88Y9/lKQoe+HJ4tgazpXhb9teRH9HO5JvWYI7HnkMLu4e12wriiLy8/Px6aefQhRF+Pn5ITc3F9nZ2fD09JzhyomISEq8ftuXTWFvruLJ4viG+nrx4e9fxcUzp+AfHoG7/u05hMbEXbd9d3c3zpw5g8LCQgwMDEAulyMlJQW5ubmIioq67uggERHNHbx+25dNYc9kMuGVV17BJ598AgBYvnw5/u3f/g0KhULyAqeDJ8vcIIoizh46gM/e3glBAL7+0CPIXvHNcYObyWRCZWUlCgoKUFdXBwAICQlBXl4eMjIy4OrqOlPlExGRnfH6bV82hb1NmzbhwoULePTRRyEIAt544w3ExMRg69atM1HjlPFkmVtaL9bg4Ku/RndrM+LzbsSy7/8b3L28Jzyura0Np0+fRnFxMYaHh+Hi4oKMjAzk5uYiLCxsBionIiJ74vXbvmwKexkZGSgqKrIuiDeZTMjJyUFJSYnkBU4HT5a5Z3hwEIff+G9UfvEZvIOCcdeTP0BkUopNxxoMBpSWlqKgoAAtLSObN6vVauTl5SElJcXhR6KJiGgEr9/2ZVPYS09Px9mzZ60XS6PRiJycHJSWlkpe4HTwZJmbRFFE2aef4OjOHTCbjLjpOw/hhrvvh2Dj3beiKKKxsREFBQUoLy+HyWSCh4cHsrOzsXDhQgQEBEj8FxAR0XTw+m1fNoW9H/zgBzhz5gzWrl0LANi1axdycnLwm9/8Rur6poUny9zWrqvDwVd/jY6GekSlZ+EbTzwDTz//SfUxODiIoqIinD59Gp2dnQCA+Ph45OXlISEhgdu3EBE5IF6/7cumsGexWLBjxw4cOXIEAHDHHXdgw4YNDn+h5Mky9xmH9Tj21hsoOfIhPHz9sPLx/wNNZs6k77q1WCyora1FQUEBqqqqIIoifH19sXDhQuTk5IzZP5KIiGYXr9/2NamtV0abzpXtLXiyOI/KE8fxyeu/g2FoEEo3d/iHRyAgQgX/8Aj4R6gQEB4J//CI6+7Td6Wenh4UFhbizJkz6O/vh0wmQ3JyMvLy8qDRaObM+U1E5Kx4/bYvm8JefX09Hn30UeszcpcsWYIdO3YgKipK8gKngyeLc+lubcGZv72PzsZ6dDY3ob+j/ao2Xv4B8L8cAgMiVPCPiERAuAo+wSGQyeVj2prNZlRVVaGgoAC1tbUAgMDAQCQnJyMpKQkREREOP3pNROSMeP22L5vC3m233Ya77roLGzZsgCiKeOONN3Dw4EEcO3ZsBkqcOp4szs2gH0JXcxO6mhvR1dSIzqaGyz83wDA0NKatTK6AX1g4AiIi4R8eaQ2B/hGRcPf2QUdHB06fPo2ysjL09/cDALy9vZGUlISkpCRER0dD/tWwaDLCqB+GQT8E47AeRr1+zFeDfghG/TCM+iEYh4dhHNbD088fkUmpCI2Nh0KpnLF/KyKiuYTXb/uyKeylpqaivLx8zHtpaWkoKyuTrDB74MkyP4miiIHurq+EwEZ0NTeiu7UFosUypr2bpxf8L4dAD18/dPb1o61vAJ16A4YtI//zkIkWuBmH4TLYC3lfF0xDQ7CYzVOuUaF0QVh8IiKTUhC5IAURC5Lh6sHHvhERAbx+25tNG4/Fx8ejuroaiYmJAIDq6mokJCRIWhjRVAmCAC//AHj5B0Cdkj7mM7PJiJ62VnQ2NaLrcgjsbBoJgs3nq8a0VQJw9fSB2TcABg8fDLq4Y9DFHYJfCLzlAgLdXBDs5QkPDw+4uLlB4eoGpasbXNzcoBx9ubrBxc0dCldXKF1d0d3SgsaqcjRWVqCpuhIN58pGi0ZwVPRI+EtKRWRSCrwDgmboX4yIiJyZTSN7S5cuxYkTJ7B48WIAwMmTJ7F48WL4+PgAAPbu3SttlVPE/zKgydAP9GOorxcubu5QurpC4eoKmezLqdvu7m5UVVWhsrISWq0WoihCEARERUUhOTkZCxYsgL+/7VvDWMxmXKqrRWNVBRrPlaOxqgID3V3Wz32CQ6G6IvwFRKp58wgRzQu8ftuXTWHvrbfeGvfzhx9+2G4F2RNPFpLK4OAgqqurUVlZiZqaGphMJgBAWFgYkpKSkJycjJCQkEmFM1EU0dPagsaqCjRcDn9dTV+ev27ePohckIzIBSmX1/3FQa7guj8icj68ftvXpLZemWt4stBMMBgMuHDhAiorK1FVVQW9Xg8A8Pf3t97goVarp3Rn72BP98jIX2UFGqsq0FZ7wbpWUKF0QVhCIlRJqYhckILwxGS4eky89QwRkaPj9du+bAp7zc3NePXVV3H+/HnrCAYAHDhwQNLiposnC800s9mMuro6VFZWorKyEr29vQAAT09PLFiwAAsWLEB0dDRcXV2n1L9Rr0dzTZU1/DVVV8KoH7nzWBBkiM7MRvodKxCbnQc5nwVMRHMUr9/2ZVPY+9rXvoaFCxfihhtuGLP9xIMPPihpcdPFk4VmkyiKaG5uxrlz51BZWYlLly4BAGQyGaKiohAfH4/4+HiEhoZOeS2edd1fZTnqSotQW3QGosUCT/8ApN12J9JvvxO+IWH2/LOIiCTH67d92RT2MjIyUFJSMhP12BVPFnIkHR0dOH/+PGpqaqDVaq2j5F5eXoiPj0dcXBzi4uLgMY2p2P7ODpQdO4zSox+h91IbIAjQpGch444ViFu4aMqjfQaDAQMDA/Dz8+NNIkQkOV6/7cumsLdu3Tps3rx5zm23wpOFHJXRaER9fT1qampQU1NjHfUDgMjISOuoX0RExFWbOdtCtFhQV3IWJUc+woUz/4TFbIaHrx9Sb7sD6bcvg39YxLWPE0X09vaipaUFra2t1q8dHR0ARjaaHg2l0w2mowa6u9BwrgzDAwPwj4hEYKQa7j6+DJVE8xiv3/ZlU9grLS3F7bffjoSEBLi5uVnfP3r0qKTFTRdPFporenp6rMHv4sWLGB4eBgC4ubkhNjbWGv5GtzuajIHuLutoX09rCwAgKi0DKUuWwVcTh/aODrS0tFiD3dBXnj4SGBiIsLAweHh4QKvVXhVM4+LiEB8fj8jISJuC6UB3F3QVpWioKIWuvBSdTVf/b9TN2weBkSoERKoRGKm2fvUODILAR9gROT1ev+3LprCXmZmJu+++G7m5uWP+z/yuu+6StLjp4slCc5HZbEZjY6M1/DU1NVk/CwkJsU75ajQaKGyclh0YGEBLczMqi4twofIcuvv7YXFxBYQvg5NSqURYWBhCQ0MRFhaGsLAwhISEwMXFZUxfo8H0woULuHjxovXuY1dXV2swjYuLg5+fH4CRqWXdubKRcFdRNmY7Ge/AYKhT0qBKTYennz86GxvQ2ahDR2MDOhvqoR/oH/O7la5uCLgyBKpGvvqFhl/17GMimrt4/bYvm8NecXHxTNRjVzxZyBkMDAzgwoULuHDhAmpqajAwMABgJJxFR0dbR/0CAgIgiiI6OzvHjNS1tLSgr69vTJ/e3t5wlwH61iYY2tsg1w9CHZ+AzDtWIOGGxVB8JeBdz2gwHa2tsbHR+pm7Qg5Ffw9MbU2QD/ZBEEX4BIdAnZIOVXIa1Knp8Am+/s0poihisKfbGv46GurR2ahDZ6MO/V2dY9rK5Ar4h0cgIFKFQFWUNQz6R0RC6TK1O5+JaPbw+m1fNoW9xx9/HBs2bEBWVtZM1GQ3PFnI2VgsFrS2tlpH/XQ6HSyXn/Xr7e0NvV4Po9FobS+XyxEcHDxmxC40NNS61k4URTRUlKLkyEc4/88vYDaZ4ObljZSv346MpcsRqIqasKa+jnbrtKz2XDk6+wdh8vKF2dMHonIkNMpkAtSRKixITkZ8fDyCg4OntSZveHAAnZcDYMflANjZ2IDuthbgyv9LEwT4BocgIFKNILUGQVHRCFJrEBCphkLJDamJHBWv3/ZlU9hLT09HVVUV4uPjx6zZKywslLS46eLJQs5Or9dDq9WipqYGdXV18Pb2HjMNGxQUZPMNHoO9PTiX/ylKDn9oXUcXsSAFGUuXI/FrN1tHyHrbL12eki1FQ0UZulubrX34hoZZR+5UyWnQi7BO+dbV1cF8eUNoHx8f61q/2NhYuLu72+Xfw2gYRldT45dTwY06dDTUo6u5CRbzl3uEyuRy+IdHIkitQbAmBkFRGgSpo+ETPLmnnhCRNHj9ti+bwt5nn312zfdvvfVWuxdkTzxZiCZPFEU0VlWg9PCHqPrH5zAbjXD19IQ6JQOX6i6ip63V2tYvLByq5HSoU0cCnk9Q8HX7NRgM0Gq11inf0Tt8BUFAZGQkEhMTkZ6ePqnnC9vKYjajq7kJ7Tot2uu1uFRfh/b62jF/CwC4uLsjUK1BsDoaQVEaBEfFICgqGm5eXnaviYiuj9dv++Lj0ojouob6+3Au/1OUHvkI7bo6+IdHQJWSPjJ6l5IG74CgKffd1dVlDX4XL16EwWAAAGg0GmRmZiIlJWXMTIIUDEODaNfVXw6BdSNBUFcHfV/vmHZeAYHWKeDgqGgERUVzKphIQrx+29e4Ye+ZZ57BSy+9hHvuueeaUxt79+6VtLjp4slCZB+iKMKoH4KLuzTP3jWbzbhw4QKKi4tRWVkJs9kMhUKBpKQkZGRkIC4ubkr7DU6FKIoY6O5Ce10tLulGAmB7fR06GuthvmI9pCCTISBCZV0LGBARCf/wSPiFhUPpKm1IJXJ2vH7b17j7Ntx2220AgNWrV89ELUTkoARBkCzoASM3kiQmJiIxMRFDQ0OoqKhAcXExysrKUFZWBk9PT6SnpyMzMxNhYWGSrqsTBAFe/gHw8g9AdNZC6/sWsxldLU0jI4A6LS7VadGu06LqZD6qTuaP6cM7MBj+4eHwDx8NgBHwD4+Eb0gon1lMRDOO07hE5LA6OztRUlKCkpISdHaObLcSEhKCzMxMpKenT2mTaXsz6IesN4F0NTeO+WrUj92gWpDJ4BcaZg1//uGR8A+LgH9EBLwDpNsw2mQ0YnigH/qB/i+/9vdDPziA4f5+mIxGCDIZZDIZhNGXIFz+WX75PWHM5zKZHIIgXPGzDIIgG/vz5ZfCxRUubm5QurmPfHV3h0Lpwpth6Lp4/bYvhj0icniiKKKhocE62je6kXNsbCwyMzORnJx81ebPs210Ori7uQmdzY3oam5Ed0sTupqb0N3SBLPJNKa9QukCv7DR0cAI+IWPBMKA8Ei4+/jCZBj+MqQN9EM/MIDhK8LblZ8NDw5A3z/62QBMhuFZ+le4PkGQQenmZg2BI9+PfB0Jhe5f+eyKsHj5c6WbG1zcPeDp58ep868QLRYMdHfBZDDAYrFAtFggWswj34vi5Z8tlz8zf/m9zZ+NvB+bnQevgEC718/rt30x7BHRnGIymVBdXY3i4mKcP38eFosFSqUSKSkpyMzMRHR0NGQO/kg1i8WMvvZLV48GtjSht60NomgZ016QySBaLNfpbSyZXAE3Ly+4enjCzdMLrl5eI189R7+OvG99z8sLcqXSehEfvaCPXsxFi3jFhd8CUbRc/+erwoAI0WKGyWCAQT8Eo14Pg14Po34IhqEhGIf11veN+qGRz4aGYDIaJv1v6urhCc/L0+9e/gHwDAiEl58/PP0DR94LCICnX4DNG4bPBWaTEb2X2tDd0ozu1mZ0t7aMfG1pRm9b65T+HSfr/h//HJp0++/By+u3fTHsEdGcNTAwgPLychQXF1uf3uHt7Y2MjAxkZmYiJCRkliu0nclkQl9fHzo7OtCiq0dbczO6OjvQ29cHg9EIhVwOV6USLq4ucHd1h7uHOzw8PeHp5Q0vHx/4+PnBxz8AXj4+cHNzc/jAOx6L2fyVIDjyvWFo6MtQeDkgDg8NYqC7CwNdHejv7ER/VycMQ4PX7dvNy3skDPoHwMs/cCQE+gfAy+/y14AAePr5Q65wjDutDfohdLc0o2c0yF0Oc92tLehrv3TVfxjI5Ar4hoTCLzQMvqFhULq5fzmlLoxOx8uvnm4X6Q/H7gAAIABJREFUBAgy+XWm8mXWz3DFdL5MkCE0LgHu3vZfTsHrt33ZHPb27NmDqqoqbN68GY2Njejs7ER6errU9U0LTxai+aO9vR3FxcUoKSlBT08PACA8PByZmZlIS0uD1yzulSeKIgYHB9HT03PNV29v71WPtBulUCjg7u6O4eFh6/Y0tnB1dYW7uzvc3NysX6/8/sr33NzcoFQq4eLiYn0pFIo5GxgN+iEMdI0Ev/6uTuv3I187MNDVib7ODpiGrz+97e7jCy//ALj7+ELp6gaFiwuUrq5QuLhA4eJ6+fXle8or3lN89T3XL9vK5IoxaxVFUcRQX+/lQHd5dK6lyTpKN9jTfVVtSjf3kbWfoeHwDQ2Df1gEfC//7B0UBJls7j8nmtdv+7Ip7G3ZsgUFBQW4cOECqqur0dzcjPvuuw8nTpyYiRqnjCcL0fxjsVhQX1+P4uJilJeXw2AwQBAE+Pr6QqlUQqFQQKlUXvf7qXwul8vR399/3TDX09MD01fW6I3y8vKCr6/vdV8eHh7WcGA2m6HX66HX6zE0NDTm60Tvja5znIzRAPjVIDjez9f6bLR+URStr4l+tqUNMHL3dEBAAAIDAye1PY8oijAMDVnDX39XJ/o7OzDQ3XU5GHagv6sTQ709MA4P2zyNPhFBkI0JhMMDA9cciXT38b0i0IXDP2zkq19oGDx8/Zz+5hZev+3LprCXmZmJwsJC5Obm4uzZswCAjIwMlJSUSF7gdPBkIZrfDAYDqqqqUFZWht7eXhiNRhiNRphMJuv3Uq5kUSqV+P/s3Xl8XHd9//vXWWYfzWgZSZaszbIt2/GSxElISKBJWLIWA+UHJFAaB0ICbS60oZcuwOMHLYW2tLn9cVsovaUJNCGhJFAgFAIkBAzZcWI7jmNbtiXZ1r6NNOuZs9w/zmg0I8m2bEvW4s/z8TiPs87MV7Kk8/Z3O+Xl5USjUSKRyLQgF4lE0M/RVCy2bZPNZmcMhRO1hoZhkMvlZtyeur/YegCpqkpVVRU1NTVUV1cX1pWVlXMyR6NlmpiGgWlk84tBLpvfzmbJ5QzMbLZwTa5ou+TaKce8gUBhhLbb9FpHeW0dvuD8TXW0FMj9e27N6q9MIBCY9ssy21/0gwcPcttttzE4OEg0GuX+++9n48aNJdc8+eST/Pmf/zmJRAJFUbj55pv527/920ITwmOPPcaf/umfYlkWmzdv5v77718UUy4IIRY3r9fL5s2bT9rlxLKsaQHwVNtTj5mmOWMNnd/vXzQ1MKqqEggECAQCZ/1IOsdxsCzrhEFwpmCoKErJApzy2GyusSyLwcFBBgYG6O/vZ+/evSVl1TSNWCxWEgBramqoqKg4rWZqTdfRdP28D2FiaZpV2GtubmbHjh0oikIul+MLX/gCF100u9E3d911F3feeSfbt2/nkUceYfv27bzwwgsl11RUVPDwww/T2tpKJpPhLW95C9/85jfZvn07iUSCD33oQ/zyl79k/fr13H333fz1X/81X/rSl07/qxVCiCk0TTtnT+dYLhRFQdd1dF0nuMjCj2EYDA4O0t/fXwiA/f39vPLKKyXX6bpOLBabVhNYXl6+ZPsqCnEis2rG7evr47bbbuOJJ55AURSuvfZaHnzwQWKxkz8Xs7+/nzVr1jA8PIyu6ziOQ11dHb/+9a9Zs2bNCV939913E4vF+OxnP8t3vvMdvv71r/OTn/wEgFdffZXrrrtuVtW7Ug0shBACIJvNMjAwUBIABwYGGBsrfQ6yx+Ohurq6EACrqqooKysjFAoRDofPWbP7+U7u33PrlD+1tm3T0dHBT37yE1KpFI7jEAqFZvXmR48epa6urvDLoSgKTU1NdHV1nTDs9fb28sgjj/DYY48B0NXVRXNzc+F8S0sLPT09mKYpv3RCCCFmxefz0dDQQENDQ8nxTCZTCIDFQbC7u3vG9/H7/YTD4UL4K16Kj4VCIakxFovGKdOSqqrceeed7Nq1a96r68fGxnjb297GJz/5SS699NLTfv29997LvffeW9hPJBJzWTwhhBDLjN/vp7GxkcbGxpLj6XSa/v5+hoeHSSaTJBKJkqW/v5/Ozs6TvncwGJwWCqfuBwKBwshlCYdivsyqamzt2rW0t7eftOl1Jo2NjSW1cI7j0NXVRVNT07Rrx8fHueGGG3j729/OPffcUzje1NTEz372s8J+R0dHSW1hsXvuuafktVP/ByeEEELMRiAQoLm5uaRlaSrTNEuC4EyhMJlM0t3dTfYkc/pN0HW9ZNoan8834/Zs93VdXzQDhMTCmlXYGx4e5qKLLuLKK68smZj0u9/97klfV1NTw9atW3nggQfYvn07jz76KA0NDdNCYyKR4IYbbuCGG27g05/+dMm5G264gT/6oz/itddeY/369XzlK1/hlltume3XJ4QQQswLXdcLI69PJZfLzRgIp059YxhGYT+VShWOnQlVVampqaGlpYXm5maamppm3Q1LLC+zGqDxjW98Y8bjt9122yk/YP/+/Wzfvp2hoSEikQj33Xcfmzdv5o477mDbtm1s27aNv/mbv+Gzn/1syZQs7373u/nUpz4FwA9+8AM++clPYpommzZt4hvf+Masfrmkg6cQQoilzrbtkqlsThQOp+6n02m6u7tJJpOF96quri7UWDY3Ny/aaczk/j235Nm4QgghxDLlOA6Dg4N0dnYWluIRyBUVFTQ3Nxdq/8rLF8fTOeT+PbdmFfY++MEPznj8P/7jP+a8QHNJfliEEEKISY7jMDo6WhL+hoeHC+cjkUhJzV8sFluQ8Cf377k1qz57l1xySWE7k8nw6KOPsnXr1nkrlBBCCCHmnqIoVFRUUFFRUXg4wtjYWEn427NnD3v27AHcEcUTwa+lpYWamhqZdHoJOqNm3FQqxbZt2/j5z38+H2WaM/I/AyGEEOL0JJNJurq66OzspKOjg97e3sI5v99PU1NTIQDW1dXNy5Qxcv+eW2c0K7Hf75d/BCGEEGIZCoVCbNiwgQ0bNgBui95E+Ovs7KS9vZ0DBw4A8N73vrdwnVi8ZhX2iueusyyLF198kU2bNs1boYQQQgixOPj9ftra2mhrawPc5w8fO3aMzs7Ok85DKBaPWYW94mlOdF3nYx/7GL/3e783b4USQgghxOLk9XppbW2ltbV1oYsiZmlWYe+d73wnW7ZsKTm2e/fuaceEEEIIIcTiMqshNdu3b5/VMSGEEEIIsbictGavv7+f3t5e0uk0e/bsYWLgbjweL5mRWwghhBBCLE4nDXsPPfQQ//RP/0R3dzfbtm0rHI9Go3zyk5+c98IJIYQQQoizM6t59v76r/+az3zmM+eiPHNK5ukRQgghlh65f8+t05pUOZvNks1mC/uL9QHKE+SHRQghhFh65P49t2Y1QOO5555jw4YNBIPBwmNWKioq5rtsQgghhBDiLM1q6pWPfexj3H///XzkIx/hV7/6FV/+8pfx+/3zXTYhhBBCCHGWZlWzl8vluPzyyzFNk7KyMj71qU/x8MMPz3fZhBBCCCHEWZpV2PN4PABUVVWxc+dOBgYGGBgYmNeCCSGEEEKIszerZtxbbrmFoaEh/vIv/5Krr76aXC7H5z//+fkumxBCCCGEOEunNRoX3CbdTCZDWVnZfJVpzshoHiGEEGLpkfv33JpVM65pmvzjP/4jH/3oR/F4PPT39/Pkk0/Od9mEEEIIIcRZmlUz7t13341lWfz6178G3L57733ve3nxxRfntXBCCCGEEOLszCrsPfvss7z88stcfPHFAJSXl5PL5ea1YEIIIYQQ4uzNqhl36px6lmVh2/a8FEgIIYQQQsydWYW9LVu28MADD2DbNu3t7XzkIx/hmmuumeeiCSGEEEKIszWrsHfvvfeyY8cOent7ueqqq1BVlb/7u7+b77IJIYQQQoizdNpTrywlMnRbCCGEWHrk/j23Tlqzd+eddxa2v//97897YYQQQgghxNw6adgrnlrlc5/73LwXRgghhBBCzK1Z9dkDWMatvUIIIYQQy9ZJ59lLp9Ps2bMHx3HIZDKF7QlbtmyZ9wIKIYQQQogzd9IBGi0tLSiKMvMLFYXDhw/PW8HmgnTwFEIIIZYeuX/PrZPW7HV0dJyjYgghhBBCiPkw6z57QgghhBBi6ZGwJ4QQQgixjEnYE0IIIYRYxiTsCSGEEEIsYxL2hBBCCCGWMQl7QgghhBDLmIQ9IYQQQohlTMKeEEIIIcQyJmFPCCGEEGIZk7AnhBBCCLGMSdgTQgghhFjGJOwJIYQQQixj+kIXQAghhBCLl5ExiQ+kifenGRtMu9sDKeIDaW76yBaqm8oWuojiFCTsCSGEEOcxx3HIJHL5EDcZ5sby2+nx3LTXeP0a0ZogZs5egBKL0yVhTwghhFjmHNshMZolPpAuhLiJ2rn4QJpcxpr2mkCZh2h1kKYLAkSqA0QnlpoA/pAHRVEW4CsRZ0LCnhBCLBDbsskkTdLjBplEjnQiRyZhkC5s50iPGziOQ21LhLo15dStjuILeha66AvOcRyyKRMjbaJqKh6fiu7VUDVlSYcQ27Ixcza26WDmbCzTxppY57fNGY5Z5sTr7PzrHEzDIjGccQPeYAbLnFILp0C4wkdNc2QyyFVPBjuvXyLCciH/kkIIMQccxyGXsSZDWiIf4MZzZJIG6fHctHPZlHnK9/X4NXDg+P5ReLwLFKiqD1O/Jkrd2nLq15QTKvedg69wfpk5y/1eFX2PJr9fbgie2J447tjOtPdRVAXdq+LxauheNwDqXq0QBnWPhserovu0/LmJa7XS1/k0PPnwWBywbNPBNC2snHPCIGZPXF8475wklBVdl7Nxpn9JZ0XVFCKxAA3rK0pr56oDRKoCaB4Zp3k+UBxnrn+0Fo+GhgaOHTu20MUQQiwjpmEx0pti6HiCoeMJhruTDPckSY0b2ObJ/5yqqoI/7MEf9hAo8+APed112EMg7CEQ9paeC3vQPCq27TB0PEFP+yjdB+P0tI+SGjMK7xuJ+alb4wa/ujVRymuDC1675TgOqTGDscEM6TFjSngzJkPdeI50MoeZnd6MOJUvpBMIu9+Xie+ZN+hxg5NhkTPctbvY5PJr95y7bS1AHzNVV9B0Fd2joun5ZYZt97wyec4z0/npr9V1Fc2joOlafj15LlDmRVWXXk2n3L/nloQ9IYSYgW07jA2kGepOMHQ8yXB+He9PldS+aB6VihVBQuW+fAgpDSOBMi/+kBvgvAF9TkKY4ziMDaYLwa+7fZR4f7pwPlDmKQl/sYYwqjb3NThGxmRsMMPYYDq/FG0PZU4arHSPir+sKOAWvmfeou+dG3r9YQ/+kD4nX4NtO4UwWBoKpwdD07CwTBvdo80cwk4WxCaCm6aiLMGwtdDk/j23JOwJIc5rEzVQw8eTbrDrTjKcr7ErGWmoQLQ6QNXKMFX1IXe9MkykOrAoak6S8Sw97ZPhb+hYohBKPT6NFaujbtPv6nJqV0XQvdop39OybBLDWcaG3E79Y4OZye2hDJnE9FGaqq4QqQoQqfITiQUoi/kJRX2TYa7MDW+eWXy+OH/J/XtuzXvYO3jwILfddhuDg4NEo1Huv/9+Nm7cWHJNR0cH27dv56WXXmLVqlW8/PLLhXNPPfUUN954I+vWrSsce+aZZwgEAqf8bPlhEUIUMzImw93JQvPrRK3d1NASjHqpqg9RuTJMVX2YqpUhKupCSyqgGGmT3sNxuttH6WmP03dkrNBBX9UUaprLCrV//jIP44MZ4lNq6BIj2Rn7xYWiXiKxQCHMRWMBIjE33IWiPqnJEmdN7t9za94HaNx1113ceeedbN++nUceeYTt27fzwgsvlFwTiUT4/Oc/Tzwe51Of+tS091i3bl1JABRCiNmwbYej+4bZ/0wPfR1jjA1mSs57/Job6i6qpmpliKr6MJUrQwTC3gUq8dzxBnSaNlbRtLEKACtn0985Vgh/PYfi9B4e46Wfdk17rcevEYkFaGlway4jVW6Yi1YHKKv0z6pWUAixeMxr2Ovv7+fFF1/kpz/9KQDvete7uPvuu2lvb2fNmjWF6yorK3nDG97AU089NZ/FEUKcJ+IDaV57pofXnukhMZIFoLI+xNrLaqmcaIKtD1FW5V/wgQzniuZR3alb1pQDbhAe7k7S0z6KkTELNXXRWABfaG76FgohFod5DXtHjx6lrq4OXXc/RlEUmpqa6OrqKgl7p3Lo0CG2bt2Kpmncfvvt/OEf/uF8FVkIsUSZhsWhlwbY93S3O00J7hxil97cwobX1xGJnbrrx/lEVRViDWFiDeGFLooQYp4t+nn2tm7dyrFjx4hGoxw7doybbrqJWCzGe97znmnX3nvvvdx7772F/UQicS6LKoQ4xxzHYaBrnFd/08PBF/rcCXZ1hTWX1LDhqjoa1lcuisETQgixkOY17DU2NtLT04Npmui6juM4dHV10dTUNOv3iEQihe2GhgZuvfVWduzYMWPYu+eee7jnnntKrhdCLD+ZRI79z/Wy7+keho67/6mrWhlmw1V1rHvdCvxhecKEEEJMmNewV1NTw9atW3nggQfYvn07jz76KA0NDafVhNvT00NtbS2qqjI+Ps5jjz3Ghz70oXkstRBiMbJth2P7htn3dA+Hdw1gmw7egM6m31nJhqvqqG4qk35mQggxg3lvxv3a177G9u3b+cIXvkAkEuG+++4D4I477mDbtm1s27aNVCpFW1sb2WyWeDxOQ0MDH/jAB/jiF7/Io48+yle/+lV0Xcc0Td797ndz++23z3exhRCLxNhgmn1Plw62WLmunA1X1rP64moZGSqEEKcgkyoLIRYd07A4/PIAr/6mh+P7RwB3sMX619ex/vV1RKtlsIUQy5ncv+fWoh+gIYQ4f7iDLbo5+EIf2ZSJqims3lrNhqvqadwggy2EEOJMSNgTQpxzjuOQHs8R708x2p8m3p+ic+8Qg0fdwRaV9SEuu3kVbZfXLosJjoUQYiFJ2BNCzJtMMsdof4p4f9pd902GOyNjlVzr9WtsfGM9G66qp6ZZBlsIIcRckbAnhDgrRtosCXTF29mkOe36UNRLdVMZ0Zog0ZoA5RPr6iCaR12Ar0AIIZY3CXtCiFNybIeh7iSjfSniAyl3nQ906fHctOsDES+VdSGiNUHKawJEq4OU17prj09GzwohxLkkYU8IcUI5w2L/s73seuIoo32pknP+kIdoTYCmC6pKa+hqgngD8qdFCCEWC/mLLISYJjmaZfdTx9i74zjZpIkvqHPhmxqpbi4rBDp/SJ5SIYQQS4GEPSFEwUDXOC8/0UX7i/3YlkO0OsDlb2tl3RUr8Prlz4UQS4XjODiGhZO1cXIWTs7OLxZ2zsYxbBzTdq8pOje5nd8/xXWx2zfha40u9JcrTkH+egtxnrNth47dg+x64ijdB0cBWNlWzoVvbqRlcwxF5rYTYsE5ORsrlcNO5pdUDjtpYpXs54/lt7Hm4JkJmoLiUVE8Wn6togZ8KLqK6lVRpA/ukiBhT4jzlJExee2ZXnY9eZSxgTSqprDu8hVc+OZGqpvKFrp4Qix71lgWczCDncq5oS0f4Oxk8b57zDGsU76f4tfRQjp6uQ+1Powa8qD6NPCoqPmgVhzaCtteFUVXUbxafl10Tv6ztyxI2BPiPDM+nGHPL47x6m+6yaZMfCGdS25oZvM1DYTKfQtdPCGWLWssS/Zw3F0OjWIOZU54reJR3bAW8qDHAqghD1rQUzimhnTUoAdtYj+oo2gydZGYmYQ9Ic4TfUfG2PVEF+07B3Bsh4oVQa54x2rWXbECj1eaYoSYa9aYQfbIKNlDbsAzB9OFc1qln+CltXhWhNDCHtRgaYhT5XdSzCEJe0IsY7btcOTlAXY9cZSeQ3EAGtZXcOGbG2neWCVNNELMIWvcyNfcjbrhbqAo3FX4CF5Si681iq81il7hX8CSivONhD0hliEjbbLv6R52PXmU8aEMqq6w/so6LnxTI7GG8EIXT4hlwUoYk82yh0cx+4vCXdRHcGsNvtZyN9xVSrgTC0fCnhDLyNhgmt35/ni5jEWgzMNlN7ew6eoGghHvQhdPiCXNShhkj4xN1twVTTSuRb0EL65xa+5Wl6NV+OT5zmLRkLAnxDIwNpjmme8d4tDOfhwHKutDXPjmRtpeV4vukb4/QpwJK2FgdIyRPRwnc2i0JNypES/Bi6rxrXZr7rRKv4Q7sWhJ2BNiCbMtm11PHOP5xw5jGjZNF1Ry4VsaadxQKTceIU6DYzuY/SmynWMYnWMYXeMlAyrUMi+BC6vxrY7iay1Hr5JwJ5YOCXtCLFH9nWP84oHXGDyaIFod4Or3r6NxfeVCF0uIJcHOmhhHx92au65xjK4xnMzkXHZ6lZ/g1hq8zRG3z10sIOFOLFkS9oRYYoyMyfM/OMLuXxxFURQuuaGZS29qQZepGoSYkeM4WCNZjM6xQs1drjcJEw+Y0BW8K8vcYNdchrcpglYmfVzF8iFhT4glpGP3IL98aD+JkSy1qyJc+/vrqVopo2uFKOaYNkZ3wm2O7Rgj2zWGPZ4rnFfDHgIXVOFtjuBtieCtD6PoMiGxWL4k7AmxBCTjWXZ8+yCHdvbj9Wv8zi1tbPqdlTJPnhDkB1J0jpHtHHcD3vFxMPPVdgp4VoQIbIzha47gbY7ISFlx3pGwJ8Qi5tgOe3/dzTPfO4SRNmm9uJo3vqeNcIU81kycX+ysiTmcxRpOYw5nMIczWMMZcgNprOHJx44pPs2d267JbZb1Npah+uVWJ85v8hsgxCI11J3gqQf203s4TrjCx1u2b2DVhdULXSwh5oVjO1jxbCHEmUWLNZzGTprTX6Qr6BVFAymaI+g1QanxFmIKCXtCLDJmzuK3P+5k5+Od2LbDlmsbuPztrXildkIscXbazAe49PRAN5IF25n2GrXMix4Lorf50Sr96EWLWuaVYCfELMjdQ4hF5Nj+EZ568DXi/WmqGsJc+/711K6KLHSxxDngmDZ2MoeVyGEnDHed38e00av86DVBPLVBN+Qs0j5njmljDqXJ9aUw+1Pk+lOYQ26gc9LTa+cUj4pW6ce/rqIQ4iZCnVbhR5VR5kKcNQl7QiwCmUSO33y3ndee7kH3qLz+nau58C2NaJqMEFyqHMfBSZtYyRx2IoeVMPJrN8QVAl3+mJOZoZnyBBSfhqcm6Ia/mgB6tRsCtQr/OavpcnI2ucE0Zn/SDXZ9E8EuDXZxYUGL+PCsCE3WylW5QU6v9KOGPYs2uAqxXEjYE2IBOY7Dgef7+PV3DpJJ5Gi8oJKrb11HtDqw0EUTM5ix9q04yCVLa+WwpjdLFiigBj2oYQ/elSHUsBct5O5rYS9qOL8d8oCmYg7mA9VAulBjZhwdL31PXcETC6LXBIrCYNCdEPgMpxaxDQtzIO0GuYlANxHqir88BfSqAP71VXhqg4XP16sDUjsnxAKTsCfEAokPpPjlt/ZzdN8IgTIPb7n9AtpeVyu1HOeQ4zg4hoU9ng9q40bpekqN3EzNkMUUr4oa9qJHfagrw5OhLeRBC08JckHPadXC6eU+WFNRcsxO5fLhKx/GBlLk+lKkdw+SLr5QBb0ygF4dwFMbdGsCa9xQqPrc24CdtQohsjjYWSOZ0lCnuqEusLGq0Kys14TwxAIoHqmJFmIxkrAnxDlmWTa7fn6U5x87gpWz2XBlHVf+3hr8Yc9CF21ZcRwHe9zIh6A0VtwoCm6TAQ7TPun7qEEdNezFWxeaXuuW39ZCHtQy7zmvwVKDHnwtUXwt0ZLjE7VxJeGtP0Vm/wiZfcMl12pRLygK1mi29M01BT0WILA5NllLWBtErzrzWkIhxMKQsCfEOdR7JM5TD+xn6HiC8tog17xvHSvXVZz6heKEHMvBHMlMDgbIh7tcfwona01/gaYUwpmnNoia39aK1xNBLuRB0ZZeTavq1fCuDOOd8nQVx7QxhzOTgycG3DU2eC+M4CnU1AXRq/wo0mdUiGVBwp4Qef2dYxzdN4xjO9i2WzOE487/5Tj5Jr/ibQewHez8PlPOlb4OTMOic+8Qqqpw6U0tXHJjM7pH+jLNVqG2aiBVqK3L9acwB9PT+sapYQ+e+vDk4IWaIFqFDy3kQQno521TuaKrbqCrCS50UYQQ55CEPXHeM9Imz37/MHt+eay0b9I8qF9bztW3rqOyPjS/H7SEWcncZKAr6otmjWanDQjQKvz415RPDkSoCeKpDqAGpUlcCCEmSNgT5y3HcTj88gA7Hj5AMm6wojXKFW9vxRfK1/wooCgKqjq5rSigqIq7rU49xuTrVAVVUaD4miVem+SY9uRiOZCzcSwbx3QKx7Gckuswnfw1k9dh2Tg59z3ca9xta9zAHEhNf1KCruCJBfBujuVr6fK1ddUBFKkZFUKIU5KwJ85L48MZfvXwATp2D+IL6lzz/nVccFW9zMYP2Blzsk9XX4pcXxKzP4UVN+b1cxW/7ga59ZMjRd3m13M3d5wQQixHEvbEecW2bHb/4hjP/fAIZtZi7aU1XPXutYSivoUu2jk361Cnq3hqAnibIygeDcWjomiKOyJTV1F0BUVT8/uT20r+HBPbJ3iNoqugKRLohBBinkjYE+eN/s4xnnpwPwNd40Rifn7nzk00b6xa6GLNu9MNdb7WcvT8pLjn+qkMQggh5p6EPbGk2BkTxaedVv83I2Py/A+OsPsXR0FR2Hp9E5fevArPMpvV/4xCXT7QSagTQojlS8KeWBLsrMnId9tJ7xpArw4Q2FJNcEsMT+3JR7Ue2TXArx4+QGIkS+2qCNe8fz2xhvBJX7NUWGNZskfGyHbEMY6MketLlo5WnQh1q6LotaHCI6y0Sgl1QghxPpGwJxY9oyfJ8IP7MAfTeJvKMIczjD/RxfgTXeg1QYJbYgS2VJfMHZYYybLj2wc4/PIAXr/G1be2sfGNK5dsyHEcB3Mog3EkTrbDDXjWUKZwXi3zENgUy89nNHxcAAAgAElEQVQtl6+pk1AnhBACCXtiEXMch9SLfYx8/xDYDtGbVxF+w0pwIHs4TnrPAOlXBhn7eRdjP+/CsyKIf1OMoxmLp39+lFzGYvXWGt74nrWEypfWAAzHdsj1JkvCnT2eK5zXqvwEL6nFtyqCryWKVuVf8lO7CCGEmB+K4zjzPI3swmloaODYsWMLXQxxBmzDYvR77aRe6kcr91H5vvX4miLTrnMsh+zhUdJ7BknuGoD847HGgeCF1ax8azN6LHCOS3/6HNPGODZOtmPMDXidYziZ/KO+FPCsCOFtieBb5T4HVYt4F7bAQggxj+T+PbekZk8sOrm+JEMP7sPsT+NfX0nle9pO+EQERVPQmiK8unuI3QNZYprC5uYwkWQOZ9cAvbsG8KwME9gcI7g5hl61OIKfnTUxOsfJdsTJHoljHE2AabsnNQVvQxm+lgjeVVF8zRHUgPyqCiGEODNyBxGLSvK3fYz+dzuOZRO9cRXhU/Sz69gzyK8eOsD4cIaa5jKu+f31VDeW4Zg2mUOjpHcPkt47yNhPOhj7SQeehjDBLdUENsfQK/zz+rU4joOdMrETBta4gTWeI3c8QfZInFxPAvLZTvGqheZY36oI3sYyeTKEEEKIOSPNuGJRsA2L0e8fIvXbPrSI1222bYme8PpkPMuObx/k0M5+PH6NK97eyqarG9xHm03hmDaZgyNu8Ht1CCff1OttLCOwJUZgczX6afTpsw0Le9zASuSwxoxCmLPHc26oSxiF81jTf73UkI63JVoId566MIom/e2EEGKC3L/nloQ9seBy/Sm32bYvha+tgsr3tKGFZ+6T5tgOe3cc55nvHcLIWLReVM0b37uW8Cxr6ZzcRPAbIP3qMI6RD35NZQS2VONrjmAlc/mwVhTgxg3shLs9ERZnpCloYS9qmQetzItW5kUt86KVedDCXvSaIHp1QAZTCCHEScj9e25JM65YUKmX+hn53kGcnE3k+hbKrm44YbNtfCDFz/7jVfqOjBGu8PHm7RfQelH1aX2e4lEJXFBF4IIqnJxFZv8IqT2DZPYNYXSNn/B1asiDVubB21SWD3P5AFfmRQ1PbisBXYKcEEKIRUXCnlgQTs5i9IeHST7fi1rmJbZ9Hb7W8hNe331wlB//6x4yqRwXvqmR121bhdd/dj++ikcjsClGYFMM27DI7B/GHEwXhTk3xKkhD4qmntVnCSGEEAtFwp4453IDKYa/9Rq5niS+NeVU3rLuhM22APuf6+XJ/9yHqqnc/NEttGyJzXmZVK9GcPPp1RIKIYQQS4GEPXFOpXYNMPLoQZycReQtTZS9qemEzbaO7fD8Y0d48X86CFf4uPmPthBrKDvHJRZCCCGWNgl74pxwcjajPzpM8tke1LCHqj+4AP+aEzfbmobFE9/cR/uL/dQ0l3HTH24hFF1aT8EQQgghFgMJe2LemUNphr71GrnjCXytUSpvXY9WduJm29SYwf98dTd9R8Zovbiat9x+AR6vzDsnhBBCnIl573V+8OBBrrzyStra2rjsssvYu3fvtGs6Ojq45ppriEajXHTRRdPOf/3rX2ft2rWsXr2aD3/4w+RyuWnXiMUptWeQvi+/RK47Qdmbm4jdsfmkQW/oeIJH/vZF+o6MsfX6Jm748CYJekIIIcRZmPewd9ddd3HnnXdy4MAB/uzP/ozt27dPuyYSifD5z3+eb33rW9POHTlyhM985jPs2LGD9vZ2+vr6+Ld/+7f5LrY4S45pM/qDQww/uA9FV4l9cBPRtzaf9GkYnXuHePRLvyU5muVNf7Ce179zzUmvF0IIIcSpzWvY6+/v58UXX+T3f//3AXjXu97F0aNHaW9vL7musrKSN7zhDYRCoWnv8cgjj7Bt2zZWrFiBoih85CMf4aGHHprPYi9r5mCa9N5BMgdHyHaOketNYg5n3AmEDYu5mGPbHM7Q/6+7SDzdjXdVhNqPX4x/bcVJX7PnqWP86J93oaoK2z5+ERuurD/rcgghhBBinvvsHT16lLq6OnTd/RhFUWhqaqKrq4s1a9bM6j26urpobm4u7Le0tNDV1TUv5V3uHMdh4N/3YI1mT3yR4s4/p/hUFK+G6tVQvBqKV3W3fRP7GqrXvUbxTVynku3qJfncGE7GouyaRiJvbT7po8Bs2+E3jxxk95PHiFYH+N27L6S8NjgPX/0ikE2AbYI/CjLxshBCiHNkWQ3QuPfee7n33nsL+4lEYgFLs/jkupNYo1n8F1ThX1eBk7VwDAvbsHEMq2jfwskfsw0LJ2HgZG2cnAWzqPizjQRknsNJbMIai6JXzFyrZ2RMfvr1vXTuGaJ+bTk33rUZf9gzx1/1ArAtGD4Cfa9A39788gqMdrrnFQ2ClRCscpdAxeR2sKr0XLASApUSEIUQQpyxeQ17jY2N9PT0YJomuq7jOA5dXV00NTXN+j2ampo4dOhQYb+jo+OEr7/nnnu45557CvsNDQ1nXvhlKPPaMADhq+rxrz7xtCcn4jgOTq4oGOZs7HxAzHYcZeCf/hnFG8BTDcmnnyT55Lfhf3+W4OsuI3L99ZS95S3oMXdC5PHhDD/6l90MHU+w/ooVXPP+9WieJfiUitRwaaDr2wv9+8BMT16jaBBrg03vAk/QfU1qyF0GXnP3T5WiVd0NfYUwmN8OTAmGvjL3M7xh8IbAGwRPCLRl9f86IYQQp2Fe7wA1NTVs3bqVBx54gO3bt/Poo4/S0NAw6yZccPv5veENb+Czn/0stbW1/Ou//iu33HLLPJZ6+UrvG0Lxa/haImf0ekVRULwaeDUITx43R0Y49tFPkOvtpfn++wheeilWPM74L37B+OM/JfnrX5N65ll6P/dXBC+5BOPK32XH4TrSCZMr3tHK1uubF//zZE0Dhg4WhbpX3e3x7tLrQjXQdAXUboTaTe66eh3oJ5kj0LYgEy8NgakhSBfvj0xu9++D9AizqmadoPuLQmAwHwRDbhD0zrAUjhcFx0AFVK2V4CiEEEvMvP/V/trXvsb27dv5whe+QCQS4b777gPgjjvuYNu2bWzbto1UKkVbWxvZbJZ4PE5DQwMf+MAH+OIXv0hrayuf+9znuOqqqwC45ppruOuuu+a72MuONW6QO5YgsCU2p895dXI5jn/8j8kdO8aKv/ocwUsvBUCLRil/xzsof8c7sBIJEk/9kvHHH+fw3lH2vhQFUmzN/IqWnjbM7rfiWblyzsp0VhwHEn1TmmD3wsB+sIum/NF8ULMeVl+bD3YboWYjhM/gkWuqNllbxyz/I1QIiEOlIdFIgpFw17nU5LaRmjyXS8F47+Q525x9Wb1hWHmJG2gbXwcNl7lNzEIIIRYtxZmL4ZeLVENDA8eOHVvoYiwKyRd6GXn0IJXvXUfw4po5e9/ev/orRr71EBXvfz8rPvPpE17nOA47H+/k2f8+jN/ncDm/Qd/xfZxUCgD/pk2UXX8dkeuuw1s0IGe+WIkERmcnua4ujM4ujIN7MV59AXt0GM2TQ/PZaD4b3WejRSNoKxrQ6lejN12Atnor2qqLUIPTR48vSaYxGQKNZOmSK9oe74VjL8Lx3xY1Uytu0G18HTTmA2BFi/QvFEKcFbl/zy0Je+eJwW++SmbfEPWfuQI1ODeDIEYefpjez36O4BVX0PT//RuKZ+b3tUybp761n9ee7qFqZYib/nALkaoAdiZD8je/Yezxx0k8+Qvs/IAa3/r1RK6/jrLrrsO3evUZl8+KxzEmwlxXJ7nOLozOToyuLqzh4WnXa14LNejBNjSs1Kkn7laCQfSKCrT8oldWoJXn9yvzxyoq0Cor3WORCIq2DCaINg3o3QNHn4Ojz0LXc5DonTwfroXGy92l6QpYsQX0E0+kLYQQU8n9e25J2DsPODmb7r96Bs/KMDUfuXBO3jP5/PN0ffBDeOrqaPmvb59wxG0mmePH/7qH7oOjNG2s4vo7NuINTO89YBsGqWeeYezxn5J44gmseBwA75rVRK67nrLrr8fXtrakb5/jOFijo+Q6O90Q19nlhruuLnKdnYX3KKZVVeFtasJbV4XXOoJnfCfesIF3/cVoN3waVv0OKAqOaWLF41gjI5jDw1gjo1gjI1gjw5gjI+7+8LB7fnQEa3gEJ5M5+TdNVdGrqvDU1+NZWY+nvh693l27y0q08BKsLXQcGO2Co89Phr/+veDY7nndD/Vboenyydq/YOXCllkIsajJ/XtuSdg7D6T3DzN0316iN66i7OqzH6FsHDtGx/96N45h0PLth/GtXTvjdaN9KR77l13E+9NsvraBN/yvNaiz6C/o5HIkn3+e8cd/yvjPf16ohfM2NxO66iqs0ZFCsLPHx6e9Xq+uxtvcjKe5CW9TM97mJrxNTXiamtDMYfjVP8DLD7p91RqvgGv/AlZdfdZNj3Y6jTU8jFkUDK2RETccDrv7ub5+ct3dWIODM76HGo0WgqBnShD0rKxHKy9f/INZADJjcPxFNwB2Pes2/xpF/1axtsmm36YroGqNNP3OMcdxcFIprEQSO5nATriLlUhgJ5LuftLdV31+vKta8La04G1uRouc2SAuscQ5DuTSkB2D7Lj7e5yN59fj7vHMWOn62k+5/ZfnmNy/55aEvfPAyH+3k3y2h9p7LsFTc3YTFtvJJB23vo/swYM0/Mu/UPama2e87viBEX78tT0YKZM3vKeNLdeeWch0LIvUi79l/PHHGf/ZzzAHBgDQ6+rcGrqmJrzNbpDzNjfjbWxEDc7wNY52wY5/hJcedAdazGHIOxN2JkOup4dcd/fkcvx4Ydvs6wfLmvY6JRCYIQjmawpXrkSvqVmcYdC2oP9VN/hN1ACOFk2OHqiEuguhohmijVDeDOWNUN4E4RWgLsFpec6AY1nY6TR2KoWTShW27VRqelBLJLCSpftueJsIckmw7TMqh1ZV5Qa/lub8ugVfSwuepiZU30lGlotzw3Hc/6yaGbdbhZnJL1mwsu56Yn/GkFZ8LF4U7MZOb8AWCnzgu7D6TXP+Jcr9e25J2FvmHMeh929fAF1hxZ9eelZBwLFtjn/844z/7OdU33MPsTs/PON1+57u4akHX0PzqFx/xyaaN1Wd8WdO/fxcdzd6LIbq98/uRaNH8yHvgXzIuxyu+QtovWZR1yQ5ponZ11caBru7yR3Ph8KeHhzDmPY6LRYjePHFBLZuJbj1YvwbNqB4F2l/ubGefL+/59wQOHV+wgmaFyIr3eBXng+C0XwQLG+CsrqTTgczEayd7JQnx8z0p2+GY9P+RM70F9OxsdNpnEI4y6/TaexU8gTHUzjpFHayaH9qGWdJDQZRw+HCooVDqKFw0bEQWjhcdCy/Hw6jhkLYqRTGkQ6MjtLFGh0t/SBFwVNfXwiAhWVVC566utI+qYVAkgXN4/47LuLfuXlj25AZLZ1WKTlYNII+URTOjCmhLXPic86ZBfkCVQdfBPwRd35OXzS/XXwsMnmseHvivLds3v4jJvfvuSVhb5kzuhP0f/klwlfVU/62Mx/sADDw5S8z+JWvErn5Zur/4UvTgqOVs3n+scPsfLyLcKWP3/2jC6laGT7Bu82z0aPw63th53+6Ia/hdW5NXuu1y+KG49g21tBQSRA0uo6S3r2b7P79hRodxecjsHkzga1bCWy9mOBFF6GVn/6E2ueE47g3v9FO999vtAvi+fXEvjE+7SVWTien1JGzY5i5MnJpD7lxh9xImtxgHGtk9AQfuEA8HjecBQIlayU4sR+cPB5yt5VAAK2sLB/WioJaOOyeP5OBP44D8WPutELJ/hkDhRkfI9c7QrZ3FKN/HGMggTGYxhjK4piltw5FBU/EwRux8IVNvKEM3jITb5mJ5rdRVC0/h2NwcrJvbzC/nz/uCZz6mqnnPQE3TKq6W4j5/P12HHdkeiG4DUNqcIYgN1w6X+Zsg5miuX1cdV9+7Z2y73OnfSrsz3CNVrzvc6dL8kdKw5yvzP2+LeK/hXL/nlsS9pa5sSe7GPtpJ7E7NuFfM/Mgilm9z49/zPE/uQf/xo00P/hASc2a4zi0/7afZ//7EGODGWpaItz00c2EogvQ3BM/5tbkFULeZW5N3uo3Leo/bHPJSiRI79pFeudLpF/aSfrlXdj5KW7AHfQSvHhrofbP09S0OJt+Karh7OlxazS7DpPrPETu+FFyfQPkBsdwjOnN3eCgB2w8QQtPyMJTEUD150eLOw6F6rnCnz8nf8iZXIrPzXh98XlQw+WoFTWolfUoVQ2oNS2otatRq5tQQyE3vAUC576m1bZgpAMGD7hPbBnYn18fcKfWOV2aF0fzY2a9GAkfxriGMaZhxB2MURtj1Jxe+6mCFtDRgyqaX0HzO2heE91ronkMND3jbvtsNL+N7rVRznTguqK5c1eq+mQAnNhW3XMOKrapYxkaVk7BNGDUgHjWIZl1SGUtclkHK2Oh5BwCikNIsSizDcLk0FQHVXVQNFBUB0Vz3Jw5cSwQQgmUoQYjKKEISqgcJVyBEq5EicRQymIokRhqpNYNYLrPDWlq/ovO/2yV3J4d56TrwrWFn+P8Oad025k4P+O5ot+DWZ7zNDbNy8AyuX/PLQl7y1z/v7xMrj9F/WeuQNHPrLo98+qrdLzv/ahlYVZ95zt4VqwonOs+OMpvHm2nv2MMb0Dnkhua2fKmBnTPOZ5iJH4MdtwLO7953oa8E3FMk+yBA6R2vkR6505SL72E2dNTOK9VVRHcejGBi/NNvxdcMO+BxMnl3NHOo6OFtTkw6NZSFvVlNPv6Zux3pvh8eOrq8iOa6yb7LlZF8IQdPJ4USrI7XyuYryHMZYpqfhR3XdjOH1fUonNq6Tk48XW26X7OSMf0Pk+eEFS2QtXq/LLGXSpXu6OS5+rn08rB8OHJIDcR7AYPuM1+xcIr3Ce7VK9315H60togbUrNkla0PkWznZPLYRw7lm8K7sTo6MAcHHQHLQ27o9ntGUbKT6WGAmiRMFpZEL0sgFbmQwt60EM6ml9FC4Duc1B1EztjYKWy2CkDK53DSuWw0zmstImRMcmkTXIZCztjQ9ZGz4IyyzufqYJ+li2my1nT/fcRuuKKOX9fuX/PLQl7y5g1btDzhecIbI5R9b4NZ/Qe5uAgR979HqzBQZr/85sELroIgJHeJM987xBHdg2iqgqbrl7JpTe3EAif41qL+DH49f/jhjzLgJWXus21q9983oe8k8n19JDauZP0zpdIvbST7GulTb/+zZvytX8XE7z44hM2/Tq2jT0+PhncJsLbyGjp/mjp/sSciieiVVS4YW5lPXo+1HnqJgak1KFVVs6qNtJxHBK5BPFsHL/up9xXjq7O44ODLNNthh46BMOHYKjd3R465AbOqVVe/vLJAFi5ejIQVq52a3xmksu471tcSzd4wD02NWhGG0tDXfV6dyR0YPLfM56NkzbTVPmr8GhzMwfnqTim6f4sjIxgDheNXJ+Y5mh4GGs0fy4/xZGTO/XclyeS9EHSP7EohW07HESJlOGJVuAvryJUVUtZVR2V1Q1U1TRTW70KzevjyOhh2gf30zFwkM6hdo4OH2ZorBfdAt0CjwkBR6fBV0ujfwUrvTWs8Mao9VRSroRQTBPHMLANA8cwcIxcfm0UNfFO/Idi6prC/uTP/ImunXxNybXF5xWl9PzU//xM+7ypry8tS+Rt2/A2zP0TkM73+/dck7C3jCVf7GXkkYNUvKeN0Nba0369bRh03bad9EsvUffFL1L+zneQHjd44bEjvLKjG8d2WL21mivesZrysxzle9rix/N98iZC3iVwzV/CGgl5Z8JKJMns3lWo/Uvv2uWO5szzrl6Nf9067FSqNMDF47Ma8akEg2jlUbTycvTyctSou128X5iDsK5uxhHVhmUwmh0lno2XrKduj2XHSrZNZzIAKShU+Cuo9FdSFaiiyl91wnVloBKPOofhJ5eBkSNFAbDdrYkbancf0TdVqCZfC9jqPpd46JAb7EY6pvQBU6ByFcTWlQa7WBv4JvvMZswMh+OHOThykIMjB2kfbefgyEH60/2Fa6K+KDF/zP0e5L8PsUCMWMA9NrE9X6HZcRzGc+MMpgcZSg8xmB50l9QA8ZFeUoN9GEMDmCMjqPFxwkmHUNYh7Z0McAk/mEEf/ooYocpaymP1VIdrqQ3WUhOsKaxjgdhZfQ2pXIoj8SMcHD3IodFDtI+2c2j0ED3JnpLr/JqfVdFVrClfw+ry1aytWMvq8tXUhepQlfNjlPmZON/v33NNwt4yNvifr5J5dYi6T1+BFjq9m5bjOPR8+tPEH/0ulbffTuWffIJdTxxl5+Od5DIWK1ojXPmutdStPsfPRY0fz9fkfaMo5P0FrHmLhLw55FhWvul3svbP7O4BjwetPHrCwDaxrxVvl5ejTmkWtmyLkewIw5lhhtJDDGWGGMmMFILbTIEuPdNI3Sk0RSPqixL1RSn3lU+uvVGyVpahzBBD6SGGM8MMpgdJ5E5ewxj1RU8aCIvXXu0sarWz40W1gYdKA2EmP8BE1d0av+opoa5qjdvZvuh72zXeVQhzE8Gua7wLuygk+jQfrdFW1lasJewJF743g+lBhjJDjBvT57CcMBGaqwJVxPylYbA4JFYFqij3lWNYBkOZyfBWEuSm7Bv29FHmE/yavxA4C58VqJoW5CLeyIL1Q00YCQ7FD5UEwPaR9pJQDRDQA6yOrqa1vJXWqLusiq6ioaxhfmufl4jz/f491yTsLVOOmX9qRl2Ymo+e/lMzhr/5n/R94QsE3/hGUtv/N88/1kFiJEukOsDr37Ga1Vurz90fU9uCw0+5EyHv+6Eb8uq3uiFv7Vsl5J0jdiaD4vOd8N/dsIyS8Fa8Hs4Ml2yPZkdLgsdMwp7wjMGtJMRNWYc94dP6ucxaWbecM5R56nrMGDtleSv9lYWaw+Jl6rFyf/nsaw1Tw+5S3lTy2DnHcehP9XNw9CDtI+0cHHWD3eH4YbJFffRURaWprIm1FWtZW76WtRVrWVO+hsayRjT1xH1rs1aW4fTwZCDLh7Xi78nEuZSZOuH7qIp60n9rXdGpDFROBriioFgc7GKBGEE9uGgHE51KPBvncPxwSQBsH21nKDNUcp2u6jSXNdNa3kpLpIXWcjcEroqsIug5xy0oC+h8vn/PBwl7y1TmwAiD//EKkRtaiFzTeFqvTfzmNxz98J2MrXsjRy6+jaHuNL6QzmU3r2LT76xEO8OBHqdt8CC8/C3Y9TCMd7vHmq+Cq/74vA95juNgORambRbWpm1iOza2Y+Pgnrcdu3Ct4zjYju1u4xSuLXmNPXmu+DW2Y2PYBiOZkcLNvjjYDaeHGc+duCYI3DBUFahym1CLm0vzTaoV/opCcIv6onPbhDoHclbODTkTIXBKIBzJuDWVw5lhRjIjWM5Mo4QnRbyRaaGwMlBJha+CyoD7PZrY1hStUFN0YORAobZuagCtCdZMC3Wt0Vb8+iznpTxDqVxq2vdlIggOZ4bx6/5pNYATS9QXPa+bM+PZOEfiRzgSP8Lh+OHC+nji+LSQvCK0olADOLFeFV1Flb9qyYbgEzmf79/zQcLeMjXy/XaSz/RQ+ydb8dRODovvHOvk6PhRVgRXsCK0grC3dB48o6ODPX/wMQ6uvJGh6Do0XWXLmxq45IZmfMFzcPPNxOGV77oh79jz7rFoI1x4K7kt7yZVVrugTTSnK2flODB6gL2De9k7tJeRzMi0gGY6JpZtlR7LX2PZFqZTemxivdBK+r/l+7jNFOImzvm08+fJC7ZjM26Ml4TAkcxIIRhPNGEXn3NmnK15ZmWeskKYW1sxGeyivnPcrULMm6yVpXOssyQEHokfoSPeQcYqfQ53xBspCYAT65XhlSetvV3Mzuf793yQsLcMOY5D79+9AKrCiv978qkZI5kRtv33Nkazk5PMhj1hVoRWUBuqpSm9gjXfCjIQ2QqKysqLw1z1zjaqa+Z3El7HMokf/DH9ux+iv2sHA1j0ef30x1oZKKumD4uB9ABD6SEcHCp8FbRVtNFW2ca6inWsq1xHa7T17PpMzQHLtugY6+CVwVd4ZfAV9g7tZf/w/pI+SLqio6s6mqq5a0Wb+ZiqoyuTx4q3Zzo/cUxVVDRFQ1EUVFRUVXXXyuQycU5TNRSUknOFBfc6TdEKr/GoHrefVj7QzfvI1vOIZVvEjXghCJaEwvQwWStLa3lrocauNli7ZP7DI+aW7dj0JHs4PHp4WhAcyY6UXKurOgE9MPl7jFL4+1C8nvr3QVO0wrUz/X0ofv0fb/1j1lfKs3EXO/lLvQyZfSms0SzhK+tLbgj/Z+f/YTQ7ym0X3IZf99OT7KEv2cfA2BDWngqiXZczEPWR4QA/2vwDBvxH4cdQ4atgRWhF6RKc3K4OVp+wyS1jZhhIDdCX6mMgPUB/qt/dTg3QP9ZJf7yLATNBdqKc1UXBMteHPjJEdbCa+nA9F9dcTFAPcmTsCLsHd/Nc73OFS3VFpyXawrrKdayrWEdbRRvrKtcRC8Tm5XvsOA7HE8d5ZegV9g7u5ZXBV3h16NWSvkvlvnIuq7uMTVWb2BRzl/kqj1jaNFUrNOUKcTKqorIyvJKV4ZW8seGNJedGMiMlTcKdY51kzAw29rRuGyULNrad77phO2ScTGkXj4muIDN0DznZQB6xeEjYW4bS+4YB8G+YvHG83P8yjx58lMtXXM4nLv0EiqJgWzb7nu7huSeOkB4zCKZ7WVN5iNyfXEVd+vfpTfXSm+ylL9lHb7KXgyMHS6axmKAqKjF/jBXhFdQGa0nmkvSn+ulP9Z+0U3uFZVFtWlymeKitaKW6/jJqYuuoCdRQE3SXCn/FjP15bMfm2Pgx9o/sZ//wfg6MHODAyAF+dPhH/IgfFa6r9FeWhL+2ijZao62nPZ/YYHqwUGP3ytArvDr4asn/ooN6kAuqLmBTbBMbYxvZVLWJleGVUvsihDhnKvwVVPgr2Fq7daGLIhYZCXvLUGbfEGKHl3kAACAASURBVIpPw7fK7b9j2iaff/bz6KrOX17xlwB07B7k6e+2M9Kbwu+1Wbf/27TWJmn50v3TpsmYYNkWQ5khepO99CR76E3mw2DKDYM9iR72DOzBp/moCdbQVtHmhrZANTXpMaq7X6G263lqjBTVePBu2AYXvQ9WXX3aD9NWFZWmSBNNkSbe2vzWwvFxY5wDIwdKAuBL/S/xTM8zhWt0Vac12lpoAm6raKOtoo2qQBUAY8ZYoY/dRMDrS03Og+ZRPayvXM/1LdcXauxaIi1Ltm+MEEKI5U3C3jJjJQyMo+MENsUKj0d7+LWH2T+ynw9v/jCR8Wq+/x8vcXz/KLpH5aJLA0T/+eP4ykM0/b//dcKgB25T00SN25bqLTNeY9pmoS8HQ4dg10Ow4z4Yy/e9aLwcLno/bHwH+Oe+M3mZt4xLai/hktpLCscm5h0rDoH7R/bzw8M/5IeHf1i4bmJqh67xrsIxVVFZXb6aK+uvLNTatZW3nbMnDQghhBBnS8LeMpPZPwIO+Ne7TbgDqQH++eV/pj5Uz+8G3sOjf/9bzJzNhivr2Pr6CAMfvBXLMWj45/vQq6vP+vP1XBr2fs8dTduVr02LrIQ3fgIufB/E1pz1Z5wuTdUKUxRc33J94Xg8Gy/U/k0EwWQuyY2rbiz0s1tfuf68mttKCCHE8iNhb5nJ7BsCBfzrKgD40otfIplL8pmmL/Kzr76Goir83ie2Utvgp/MDf4A5MED9P/4DgU0bz/7D+16F+2+C9Ij70PTN7y5qpl18TZxRX5TLVlzGZSsuW+iiCCGEEPNGwt4y4pg2mQOjeBvL0MJenu15lh8f+TE3+t7Fsf9SUFSFbR+7iNpVEbo/+Wdk9uyh6q67iN5889l/eHIQHnqv+9inG78EW95T8rB1IYQQQiwMCXvLSPZIHMew8G+owrAM/ubZv6EpsZ7W315dCHorWqMM/fu/M/bDHxJ+05uo/vjHzv6DTQP+6w9gtAve9mW45Lazf08hhBBCzInz9xk1y1AmP+VKYEMl39j7DTLHVG7adxeqovK2/8ut0Rv7yU/o/8d78a1dQ/3f/z3KaY6CncZx4H8+AZ2/gcs/IkFPCCGEWGSkZm+ZcByH9GvDaOU++oLDfG/H4/zuax/Fo3u48ZZ69Mcf5PD3f4DR2YkWjdLwla+ghUOnfuNTee5rsPOb0HotXPc3Z/9+QgghhJhTEvaWCbM/hTWcIfT6Or7443/nur0fxIvGpcPfJ3H7j0kAWlUVlbf9ARW33oq3sfHsP7T9CXj8L6BqDbz7PtDkx0kIIYRYbOTuvExMPDXj1cPP07zvcjQbLtz9TwSzxym76Sai73g7oSuvRNHn6J988CB853bwlcGt34ZAxdy8rxBCCCHmlIS9Jc5xHDJ79jD20y5sK8DeveVoisIlxk9Z+2cfouz669DC4bn90PQIPHQLGOPw/kcWZO48IYQQQsyOhL0lyjh2nLEf/oD/v707D6+qPPQ9/l17TnbmmZCEmSBDjUBQUBkUHPAUsFZB0YrVQ9XSo1X7VG+r0mPL0XssrZW2XL1enCpahVLHKs56ihoKgSoQIRBCICGQhMw72cO6f6xkkxQUQhIC29/nkWftNb97sd37x7vW+761f30Z/76DeC99mP3BEC12g6QZFeR/9zedtg+FTPYeambXwUZSYtyMzIw7sRMHA/DS96FqB1zyEAy9sAfejYiIiPQWhb3TSLC+nrq//Y26v75M0/r1ANiTkrDN+g+MgI2yYCsbz3mdH024nzUb97LzQAPFBxopPtBASVUjPn8ofKzR/eOYm5/D7LxM4jxdGPrrrZ9D8bsw9no4+wc9/RZFRESkhxmmaZp9XYjekpWVRVlZWV8Xo1tMv5+Gjz+m9uWXaXjnXczWVgy3G/t5Uzgw6QK2uIcwel0V/R0GP+63nA215xFqHhDe3zAgMz6KwalehqTGMDjVy9byel4u3EtjaxCP08ZlYzK5ekI24wYkWmPafpV/PAmv3AYDzoXr1oDjq8fRFREROVGR8Pt9KlHN3inINE18n3/OwdVrqH/jDYxDNQDsHXAGHw0cz8sJI6gx3PTbaDC3YT/p8U52usrZ641jVsZEBreFusEpMQxK8RLlOnKosp9fdgavbS7n+YJSVm0oY9WGMoamxTAvP5vLz+pPcoy78w4l/wOv3QkJOXDVMwp6IiIipwnV7J0EvqZm6qsOUV9VS1ON9cd3qI7Wunr8dXUE6+sJNTRiNjRgNDeSsK+E1OpyAEpj0ng3exzvZZ/FAW8S/ROiGJIaw3CHi4SCQ6TaYFKUnRcy3uLGm+8kwdP1IcqKKup5oWAPqzeWcajJj9NucNGoDK7Oz2HSkGRstbvh8Qsg0AI3roX0kT19iURERMJOld/vSKGw10V7t22n5vlfs881Cn9DM8HGBsyGBmhsxNbUiL25EaevGVdLE56WZjx+H65QoEvnOOSJ5Yvh+VSecwGx3xrD4LRYhqR5GZjsxeO0U7GrllceKSQUMhk6uJrcilT+eXk1l549u1vvzecP8uYXFbxQsIe/F1cBMDwRVhr3ktS0E2PeczBiZrfOISIiciwKez1Lt3G76NCqR7E//wHZfHDEuiAGza4ofE4PLZ5ommITCXqiCEV7MaO9GN4YbLGxOGJjcMbF4oqLw5MYR3RiPN6EeGKTE4hNjsflcTPxK87fMeiNuTYe5yvVHHLXc1H+v3X7vXmcdmbn9Wd2Xn9KDjbyYkEJZ3/2HySbxTwUmMf2T1OZG9rPtNxUHHaNtCciInI6UNjrosTv3oZ93wdEmQfZM/0RXFnDiUlOICYpAW98DLbujjX7NfbvqrOCXtBk5qIxLP3nfdzpn4f/rCjstiOfy+uOgSlefuJ8Ecz17MuZRZFxI+9vq+TtrZWkxbq5cnwWc8fnkJMc3aPnFRERkZ6lsNdFmblD4K5l8NS/Mcr3Ikx44aScd39JHS//zgp6ly06k0+Nd4kvtRpJZOQN7vkTbv4zfPwb6D+ezOse5/85Pew71MxL/yjjhYI9/P69Yn7/XjHnDk1mXn4OF41Kx+3o2cApIiIi3adn9k7Uqpvgny/CvJW9/hzb/pI6Xn6kkFAgxGWLzsQ7AL695tss3vEDRrQOpv+9EzGcPVijWLYeVsyE6GRY+B7EZnRaHQyZ/M+OgzxfUMraLfvxB00So518Z2wW8/KzGZYe23NlERGRbxw9s9ezVLN3oi76JRT9Dd74KQyeCq7euZ1ZubtD0Pvht8jKTeT+v99PqDFAbuMAPGck9mzQq90Lz18Dhg2ufu6IoAdgtxlMHp7K5OGpHGxoYfWGMp7/bA9PfLyLJz7eRWa8hzP6xTEyM46RbdPsxGhstq/pw09ERER6hcLeiYrNgGn/C968Bz5eChf8vMdP0R70gu1Bb0QShZWFrN6+mpuc8zBMg6gzknruhK1NVtBr2A9XPgmZZx1zl5QYNwsnD+Hfzx9MQUkNqzeUsamslg++PMA72yrD23ldds7oF9cpBOZmxOJx6taviIhIb1LY644JC2Hjs/A/j8CZV0PykB47dHvQC/itoJc9IolAKMADnzyAw+bg8tDFQDOeET0U9kwT/norlBfClLth1OVd2t0wDCYMSmLCIKs8LYEgOyob2Fpez5Z9dWwpr2VreT3rd9eE97EZMDg1hpH/EgJTY91fdRoRERHpIoW97rA74LKHYcWl8PpP4NpV1vhk3XSgtP5w0LvVCnoAK7et5MuaL1k46t+xv9qKIysGe2wPjWTx4X/DF3+BM2bBlJ92+3Buh51RmfGMyoyHcdYy0zTZV+tj6746tpTXsWVfHVsr6nh50z5e3rQvvG9KjDsc/M7oF8uozDgGpcRg121gERGRLlPY664Bk6xavU0rYesrMHJWtw53oLSev/524+Gg13abtrKpkt8X/p7+Mf25Pm4udb4viTojuSfeAWz5K7z3K8gYA5cvh17qPsYwDPonRNE/IYrpI9PDy+t9fooq6g8HwPI6Pt1ZxYdfHghv43bYGJER2+k5wBEZcXjd+giLiIh8Hf1S9oQZ/wnbXoe/3QNDLwSX94QPtXFtqRX0bjkc9AAeLniYRn8jD53/EKFNjQB4euJ5vfLN8JebwZtmtSzuRtlPVKzHyfiBSYwfePj9BIIhdh1stAJgeV3b7eBaNpXVhrcxDBiY7A2Hv5GZcYxquw1s9EANq4iISCRQ1ys95dPH4I2fwHk/humLT/gwAX+QqrJG0gfFhZet27eOhWsXMi17Gr+74HdUPLwe0x8k4+4J3Qs1DZXw2DRorIQFr0H2hBM/1klSWe9rewawLjzddbCRjp/ilBhXp2cAdRtYROT0oq5XepZq9npK/o2w8Rn4+zI48xpIHX5Ch3E47Z2CXmuwlSWfLsFj93D3hLvxH2gicLAZ79kZ3Qt6gRZ4fj7UlcGc5adF0ANIi/WQlutham5aeFlTa4BtFVZDkC/aAuBnu6r5aPvB8DYep43cDCv4Hb4NHEu0S/8LiIhIZNMvXU+x2eGyX8MTM+D1u+B7f+2RxhpPfvEkJXUl3Db2NjJjMqnfYP1Lx9Od5/VME165Hco+g0n/AXlXd7ucfSna5WBsTiJjcxLDywLBECVVjeHwt2Wf9WfTnkPhbQwDBqV0uA3cz3oOMD1Ot4FFRCRyKOz1pOwJcNZ1Vg3fF6th9BXdOlxZfRmPbX6MQfGDuH7k9QA0b63GcNrwDIk/8QOvWwabnoNhF3frlvOpzGG3MTQtlqFpsczO6w9YrYEr61uOuA386uZyXt1cHt43ymlnQHJ02x8vA5KjGdg27RcfpdvBIiJyWlHY62nTF1utct/8GQy7CNwnNnSYaZr812f/RUuwhZ+d/TOcdiehJj+tu2vx5CZhnGhnxF++BW/dC6kj4Ir/a9VIfkMYhkF6nIf0OA/TRhy+DdzQEqCowgp/Rfvr2V3VxO6qJt7eWkkw1PmRVpfdRlZSVDj8DUz2ktM2zUqMwmnvnZbMIiIiJ0phr6d5U2D6/fDqj+H9B+HiX53QYd7b8x4fln3IzEEzObvf2QD4vqyBUDda4e7+O7x4PUQlwtXPgyfu2Pt8A8S4HYwbkMS4AZ2vqz8YYm9NMyVVjeEAuLuqkZKqRj7efpB3t4U6bW+3WV3LtNcKWoHQy8DkaLKTojVaiIiI9AmFvd4w9nrY8DR88kfImw/pI7u0e5O/iYc+e4gYZwx3jb8rvLx5azUAUScyasaez+BPV4Jhh2v+DEmDun6Mbxin3cbAFC8DU47sjiYYMqmo87H7YCMlHULg7qom1pfUdGoc0q5fvIchqTEMT48lNyOG3Iw4hqXFqK9AERHpVb3+K7N9+3auv/56Dh48SHx8PE8++SSjRo06YrsnnniCBx98kFAoxAUXXMAf/vAHnE4n77//Ppdeeim5ubnhbdetW0dUVFRvF/3EtTfWePxCq7HGgte61Fjj8X8+zr7Gfdw94W5So1MBMIMmvqIanP1jsMd1cTixvRvg2SushhnXvgTZ+V3bX47QXovXPyGKSUM7rzNNkwP1LeEQuLuqiZK2MPiP3TV8vKNzEMxOiiI3PS4cAHPTYxmU4sXl0C1hERHpvl4Pez/4wQ9YuHAhCxYs4KWXXmLBggUUFBR02mbXrl3ce++9bNiwgfT0dGbPns1jjz3GD3/4QwByc3MpLCzs7aL2rP7jYNwC+McK+OeL8K2rjmu3nYd28uQXT3JG0hnMzZ0bXt66uxbTFyCqq7dwyzfDM3Mg6If5L8KAiV3bX7rMMAzS4jykxXnCYwW3C4VMymqaKdpfT1FFHUX7G/iyop73iyp5e+v+8HYOm2HVAmbEkpt+OARmJUZhUwMRERHpgl4Ne5WVlaxfv5633noLgCuuuIJFixaxY8cOhg49XB3y0ksvMWvWLDIyMgC4+eabWbJkSTjsnbYuvM8aiuzNn8Hwi8Fz7Ba0j258lGAoyM/P+TkO2+G/nvZbuF3qcmX/Fnh6Nvh9cM3zMOj8Lr8F6Vk2m0FOcjQ5ydHM6DBkXGvAGjFkW0UdX+6vp6iigaL9dbyyaR+vdNg/2mVnWLoVAIenxzIiI47hGTGkxqi7GBERObpeDXt79uyhX79+OBzWaQzDICcnh9LS0k5hr7S0lAEDBoTnBw4cSGlpaXi+uLiYsWPHYrfbueGGG7j11luPer6lS5eydOnS8HxDQ0NPv6WuiU6CGb+Al38E7y2BSx865i73T7yfC3Iu4Fup3+q03LetGlucC2fmcQ5ndqAInp4FrQ0w7zkYcsGJvAM5SVwOG7kZseRmdG693dgSYHtlg1UL2BYAiyoaOvUXCJAY7WR42+3f7CSrQUh2YhTZSdEke10KgiIi32Cn/JPhY8eOpaysjPj4eMrKypg5cyYpKSlcddWRt0XvuOMO7rjjjvB8VlbWySzq0eVdazXW+OwxOOtayBjztZsneBL49pBvd1rmP9hM4EAz3gnHOWrGwR3w1LehuQauegaGzejOO5A+5HU7yMtOIC87odPyqoYWivbX82VFPUX7G8Jdx3y6q/qIY0S77GQnRpOdFNUWAqPJaQ+ESVEaRUREJML16rd8dnY25eXlBAIBHA4HpmlSWlpKTk5Op+1ycnIoLi4Oz5eUlIS3iYs73D1IVlYWV199NR999NFRw94pyWazGms8NhVeuxNu+Ju1rAt8W6sA8BxPK9zqnVbQazwIV66AETNPoNByqkuOcTMpxs2kISnhZaZpUtPkZ091E3tqmiitbmJPdXN4/v2iAwRCRw6FnRLjIiscAKM6hcF+8R4c6jtQROS01qthLy0tjbFjx/Lss8+yYMECVq1aRVZWVqdbuGA9y3feeeexePFi0tPTWb58OfPmzQOgvLyc9PR0bDYb9fX1vPrqq9x44429Weye1+9MyL/Jqt3btBLOmt+l3X1bq8Fhwz004es3PFQKT82Chgr4zuMwcnY3Ci2nG8MwSPK6SPK6ODP7yM9KMGRSXttsBcCaJisEVjexp6aZ0uomCv/l1jBYrY4zEzxkJ0aTlRhFotdFUrSLxGgXCdFOkrwuEqJdJEY7SYh2aXQREZFTkGGa5pH/1O9BRUVFLFiwgKqqKuLi4lixYgVjxozhpptuYtasWcyaNQuAxx9/nAcffBCAqVOnsnz5cpxOJ8uWLeOPf/wjDoeDQCDAlVdeyf33339ctzOzsrIoKyvrzbd3/JoPwbLxVvcnP1pvdWx8HELNAfY98Ame4YmkLDiyy5qw2r2w4lIr8M3542k/3q2cfD5/kLIONYKlHcLgnuomGloCX7u/YUCcpz0AOklsC4WJ0U4SvYdfJ0RbgbT9tbqYEZF/dUr9fkeAXg97femU+7AUroQ1N1u1fJf9+rh2adp0gOqV20i4fCgxZ/c7+kb1FbBiJlQXw6xHYez3erDQItYt4jpfgENNrVQ3tnKoyU91Yys1TW2vm1o7ratpaqWm0U9rMHTMY8e4HSREO0n2ukiOcZPsdZES2zaNcZMS4yY5xkVyjFWreLreVjZNE58/RENLgMaWAI2tARpbgjS2BMLLrGmQptaOy4L4/EFCpolp0nmKNQ2Z1vHblx85b23bad48PG8zDNwOGy6HDbfDhtthD7/uPP3X5Udu96/7ux120uPcpMaqxbgcv1Pu9/s0pyezT6Yz58GGp6DgCauxRuZZx9yl/Xm9rxw1o6HSekavuhguW6qgJ73CMAzio5zERzkZkHx8LcJN06SpNdg5ADa1UtPYSnWT/4hwuL+uha3l9V8bEA0DEqNd4SCYHNMeCDsHxRSvte54RycJhUxaAiF8/uBxTVsCQXz+DlN/MBzewmGuLbw1tR5edpRHJo/JYTOIctmxGQY2A2yGgWFYfyc2AwzapoaBzXZ43mYY0Da12wwMrP3a928/HoZBKGTSGgjREghR12yF9BZ/iJZgiNbAsQP78Yhy2g8PI5gS3Wl86Yw4j/qPFOlFCnsnk2HAzIfh/0y2Gmvc+PbXNtYwgybNRTU4M73Y448yakZjldWP3sEv4ZKHIP80e5ZRIpphGHjdDrxuB9nH2Rd4ew1iVUMLVY2tVDW0cKDBmlY1tHKwfdrYwpbyOmqb/V97vCinva1W0I3doHNAC1ghrSUQOq4ayONltxl4XXZi2t57epyHIW4H0R2Wed0OYtz28Guvy4HXfXj94e3suOy2Pq0RM03TCn+BUDgQtrYF3tajLGsJdN7W5w9SUesLDyf41paKI0Kvy2FjQFJ0eCzpASleBiRZQTAzQY2ERLpLYe9kyxgNZ/8APvkDbHzaGmXjK7SW1mE2B/BMyjxyZVM1PDMbKrfAjAfgnJt7r8wiJ0nHGsTBqcfevjUQorrRCoHtQbCqsYWDHYJhVWML+2t9mJi4HXY8ThtJXhcepy0873bYcf/LfFen3rbw5nb0bTjraYZhWNfHYe+R47UGQuw91GyFvw5jS++uauL9osojWow7bAbZSdGHawU7TLMSo/XMp8hx0DN7fcFXZzXWCLbCjzZYnS8fxaHXd9HwYRlpi/JwZXXobLf5kFWjV14IF9wLk+86SQUXEek9gWCI8rZawJKqps5hsLrpiFvKNgP6xUeREuu2GgJ1aBSUEG45frhRUEK0E4+zZ0Kr9K5T9vf7NKWavb7giYOLfgWrb4K3F8Os3x11M9+2KmyxTpyZMYcXttTDn75rBb0pdyvoiUjEcNht4RFgzh/WeV0oZFJRd/h2sFUzaLUer6zzsa28jpbjeL4wymnv3GK8Q8vwpLaW49Zra5v4aCcehx2n3YioGlv5ZlHY6ytjvms11tjwtNWoImt8p9WBqmYClc148zMw2h9cbm2EP10JZQVw3h0w9e4+KLiIyMlnsxlkJkSRmRDFpCFH36a5NdihIZC/rbV4K9UdX7c1DqppamV31bG7FAqf3wCP0279cdhwO+24Hba2ZW1TR4fXbevd7esd9s7bti3zuh0kRruIj3YS63aooYr0CoW9vtLeWGP5ufDaHfDv74Ht8O2F5q3WsFfhUTNam+C5uVC6DiYuggvvs44hIiIARLnsRLmsQHi8WgOhtvDnD7cW7/i6zufH5+/cItvX1rjH5w9yqMmPL2C99vm719DHZkB8lFXLGB/lDNc4WsucJLStS2hbntC2PNbjVIfm8rUU9vpS2gg451b4++/gHyus/vfa+LZVg8PAPSwB/D54/hoo+QgmLISLfqmgJyLSA1wOG2lxHtLiPN0+VnvL5fbueHz+UKcg6GsLiB1DY4MvwKHmVmqb/OF+K2ub/VTU+thWUXdcAdJoD4lRTuI7hMD4KCsEtj+ZH+5fkfbp4f4Y25eHOrxu+6/Dfh22bzvGbdOHMSIj7mtKJ6cChb2+NuWn8M+X4J3/hDNmQ0wqIV+Alp21eIYlYLMF4IXrYOd7MO4GuPR/K+iJiJyCOrVcjnL2yDHbaw8PNVtB0AqDba+brVvS7csPNfs5UOdj+/56mlqDPXL+Y7n2nAEn5TzSPQp7fc0dA5csgRcXWI015vwe35c1EDLxDI+3lm9/C/KutTpNVtATEfnG8DjtZMTbyYjvWs1jSyBIbbOfUMj62bD6zzY6v+Zw59tHW97+c2O0dd4d7pS7fZl+j04bCnungpFzYPBUKHwWxl6Hb6s1bq5n56+g+HX41lyrxe7XdMAsIiLSzu2wkxarbmbEovRwKmhvrGFzYr76E3xF1TijqnAUr4RRl8PsP3RqvCEiIiJyvBT2ThUpw2DSj2it8BNqCuBpXQsj/g2+8zjYVQErIiIiJ0Yp4lQy+S58f/dDK0QNssF3V4C9Zx7yFRERkW8m1eydSlxeHJO+jSe1Cuf3/hscrr4ukYiIiJzmVLN3ivFOPxvv9L4uhYiIiEQK1eyJiIiIRDCFPREREZEIprAnIiIiEsEU9kREREQimMKeiIiISART2BMRERGJYAp7IiIiIhFMYU9EREQkginsiYiIiEQwhT0RERGRCKawJyIiIhLBFPZEREREIpjCnoiIiEgEU9gTERERiWAKeyIiIiIRTGFPREREJIIp7ImIiIhEMIU9ERERkQimsCciIiISwRT2RERERCKYYZqm2deF6C1ut5vU1NReOXZDQwMxMTG9cmz5arrufUfXvu/o2vcdXfu+ceDAAVpaWvq6GBEjosNeb8rKyqKsrKyvi/GNo+ved3Tt+46ufd/RtZdIoNu4IiIiIhFMYU9EREQkgtkXL168uK8LcbqaOHFiXxfhG0nXve/o2vcdXfu+o2svpzs9syciIiISwXQbV0RERCSCKeyJiIiIRDCFvS7avn07kyZNYvjw4eTn5/PFF1/0dZEiks/nY86cOQwfPpwzzzyTGTNmsGPHDgAqKyu55JJLGDZsGKNHj+bDDz/s49JGrhUrVmAYBmvWrAF07U+GlpYWFi1axLBhwxgzZgzXXnstoO+ek+H1119n7Nix5OXlMXr0aJ566ilAn3uJAKZ0ybRp08wVK1aYpmmaL774ojl+/Pi+LVCEam5uNl977TUzFAqZpmmajz76qDllyhTTNE3zhhtuMO+//37TNE3zs88+M/v372+2trb2UUkj165du8yJEyea55xzjvmXv/zFNE1d+5Ph9ttvNxctWhT+7JeXl5umqe+e3hYKhczExERz06ZNpmlan3+3223W1dXpcy+nPYW9Lti/f78ZGxtr+v1+0zStL4f09HRz+/btfVyyyFdQUGAOGDDANE3T9Hq94R9A0zTN/Px8c+3atX1UssgUDAbNCy+80Fy/fr05ZcqUcNjTte9dDQ0NZmxsrFlbW9tpub57el8oFDKTkpLMDz74wDRN09y0aZOZmZlptrS06HMvpz3dxu2CPXv20K9fPxwOBwCGYZCTk0NpaWkflyzyPfLII8yePZuqqir8fj8ZGRnhdQMHDtTfQQ9bunQp5557LuPGjQsv07XvfcXFxSQlJbFkyRLGjx/P+eefzzvvvKPvnpPAMAxeeOEFFT5h6QAABY5JREFUvvOd7zBgwADOO+88nnrqKerr6/W5l9Oeo68LIHIsS5YsYceOHbzzzjs0Nzf3dXEi3ueff86qVav0XFIfCAQC7N69m5EjR/Lggw+yceNGZsyYwWuvvdbXRYt4gUCAX/7yl6xevZrJkydTUFDArFmzKCws7OuiiXSbava6IDs7m/LycgKBAACmaVJaWkpOTk4flyxyPfzww6xevZo33niD6OhokpOTcTgcVFRUhLcpKSnR30EP+uijjygpKWHYsGEMHDiQTz75hIULF/LnP/9Z176X5eTkYLPZmD9/PgBnnXUWgwYNYvfu3fru6WWFhYXs27ePyZMnA5Cfn09WVhabN2/W515Oewp7XZCWlsbYsWN59tlnAVi1ahVZWVkMHTq0j0sWmZYuXcrKlStZu3YtCQkJ4eVXXnkly5cvB6CgoIC9e/cyZcqUvipmxLnlllsoLy+npKSEkpISzjnnHB577DFuueUWXftelpKSwoUXXsibb74JwK5du9i1axfnnnuuvnt6Wfs/5rdu3QrAjh07KC4uJjc3V597Oe1pBI0uKioqYsGCBVRVVREXF8eKFSsYM2ZMXxcr4pSVlZGdnc3gwYOJjY0FwO128+mnn7J//36uu+46du3ahcvlYtmyZUybNq2PSxy5pk6dyu23386cOXN07U+CnTt3cuONN3Lw4EFsNhv33XcfV1xxhb57ToKVK1eyZMkSbDYboVCIe+65h2uuuUafezntKeyJiIiIRDDdxhURERGJYAp7IiIiIhFMYU9EREQkginsiYiIiEQwhT0RERGRCKawJyK9ZuDAgRQWFvLkk0+ybdu2XjnH4sWL8fl84fn77ruPP/3pT71yLhGR05G6XhGRXjNw4EDWrFnD7bffHu6rrytCoRAANttX/7vUMAxqamo6dbwtIiKHqWZPRHrV22+/zfr16/nxj39MXl4er7/+OmANhTdhwgTGjh3LJZdcwu7duwGrpu6KK67g4osvZvTo0ZSXl3PXXXeRn59PXl4ekydPpqioCICbb74ZgPPPP5+8vDwqKytZsGABv/3tbwFoaGjg+9//PqNHj2b06NH84he/CJdr6tSp3HXXXZx//vkMGTIkfCwRkUijsCcivWr69OmMHz+e3/zmNxQWFjJz5kyee+45ioqKWLduHRs2bGD+/Pnceuut4X3WrVvH008/zZYtW+jfvz8//elPKSgooLCwkFtvvZXbbrsNIDyE1UcffURhYSFpaWmdzv3AAw/Q0tLC5s2b+fTTT1mzZg0vvPBCeH1xcTHvvfcen3/+OW+++Sbr1q07CVdEROTkcvR1AUTkm2fNmjUUFBQwbtw4AILBYKf1M2fOJD09PTy/du1aHn30Uerr6wmFQlRXVx/Xed5++21+/etfY7PZ8Hq9fO9732Pt2rXMnTsXgLlz5+JwOHA4HOTl5VFcXMzEiRN76F2KiJwaFPZE5KQzTZN77rmHhQsXHnV9TExM+HVpaSmLFi2ioKCAIUOGsHnzZiZPnnxC5zUMo9O8x+MJv7bb7QQCgRM6rojIqUy3cUWk18XFxVFbWxuenzNnDsuXLw/X0Pn9fjZu3HjUfWtra3E6nfTr1w/TNFm2bFmn9bGxsZ2O3dH06dN54oknME2TxsZGnnnmGS666KIeelciIqcHhT0R6XULFy5kyZIl4QYa8+fPZ8GCBUybNo0zzzyTvLw83n333aPuO2bMGObNm8eoUaPIz88nJyen0/o777yTGTNmhBtodHTvvffidDoZM2YMZ599NrNmzeKqq67qtfcpInIqUtcrIiIiIhFMNXsiIiIiEUxhT0RERCSCKeyJiIiIRDCFPREREZEIprAnIiIiEsEU9kREREQimMKeiIiISART2BMRERGJYAp7IiIiIhHs/wOlwH5U97Jw+QAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "plot_collections(trial, \"/feature_importance\", \"Feature importance\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### SHAP\n",
- "\n",
- "[SHAP](https://github.com/slundberg/shap) (SHapley Additive exPlanations) is\n",
- "another approach to explain the output of machine learning models.\n",
- "SHAP values represent a feature's contribution to a change in the model output."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnsAAAITCAYAAAB/k9qgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAMTQAADE0B0s6tTgAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeVzUdf4H8Ndc3DeCIMOAyCGicgjmVV4daIqmWWpuYZlubeua9dN2Lbdjt6xtrXC3tK21WstqPfLONCUxzBsElFNxGDlFTjln5vv7A5zVVByO4Qszr+fjwcM5vvP5vr9lzKvP5/v5fCSCIAggIiIiIrMkFbsAIiIiIjIdhj0iIiIiM8awR0RERGTGGPaIiIiIzBjDHhEREZEZY9gjIiIiMmMMe0RERERmTC52AaZkbW0NDw8PscsgIiKidigrK0NjY6PYZZgNsw57Hh4e0Gg0YpdBRERE7aBUKsUuwaxwGJeIiIjIjDHsEREREZkxsx7GJSIiImqLIAiGn95GIpFAKr1zvx3DHhEREVkcvV6P0tJSVFZW9sqgd41CoYBKpYKVldVtj2HYIyIiIotz8eJFSKVS+Pv7Q6FQiF1OhwiCgPLycqjVagQGBt72OIY9IiIisih6vR4NDQ0ICgqCXN67o5C7uzuuXLkCvV5/2yFdTtAgIiIii3Jt2FYikYhcSeddu4a2hqIZ9oiIiIjMGMMeERERUQ+wbds2hIaGIiIiAuPHj4e/vz8kEglSUlI61W7vHqgmIiIiMhNr167FypUrMWfOHBw6dAgBAQEYM2ZMp9tl2CMiIiIS2eLFi5GUlITMzEysWbMGycnJXdY2wx4RERFZvAWfH8fF8jqTtO3nbodPnohp85iEhAScOXMGS5YswfTp07v0/Lxnj4iIiMiMsWePiIiILN6det56M/bsEREREZkxhj0iIiKiHmbRokVQKpXQaDR44IEH2twO7U44jEtERETUAyQmJhoer1u3rsvaZc8eERERkRlj2CMiIqJ20WmbkXbghzb3Y6Weg8O4REREZLSaK5ex471VKMrOhNzKCqFjxoldEt0Bwx4REREZpeBsGna+/zbqqioRPXUGQkbeLXZJZASGPSIiImqTIAg4ues7HPpyPeRW1pj6/EsIHtH5PVupezDsERER0W01NdRj79oEZB9Jgls/JeJeWAF3pa/YZVE7MOwRERHRLV0p1GDbu3/FlUsFCLprFGKfWQIrWzuxy6J2YtgjIiKim+QcTcb3H72H5oZG3DPvSURPeQgSiUTssszatm3b8NJLL0EQBPj4+KCgoAC2trbw9PTERx991OGFlbn0ChERERnodToc+uozbF/9JmQKKzz88l8QM3UGg143WLt2LVauXImUlBS88MILyMrKQmpqKqZNm4YFCxZ0uF327BEREf2KIAhI+uoznNn/PSABpFIZpDIZJDIZZLLWx62vSaUySOUywzFSmRQSqQwyubz1GGnrMfLWY6SGz3v49Ufo3eOgsLIW+5IBAHXVVdj1wTtQp6fCOzAEU5f+EY7ufcQuq3t8NRuouGCatl37A3O/bvOQxYsXIykpCZmZmVizZg2Sk5MN740YMQLvvvtuh0/PsEdERHQdQRBwYP06pOzdCdd+Sji6uUOv10Gv00PQ6Voea7XQ6/XQNTejWd8Avbb1dZ3uumP+91pbfv7mP4h8YArC758MW0enbrrKmxXlZmHH6lWoKS9D+H2TMe6JpyFXKESrx9IkJCTgzJkzWLJkCaZPn37Dex988AGmTZvW4bYZ9oiIiFq1BL21SNm7C75hQ/HQspVQ2Nh0uk1B0N8Q/vQ6HXTNzcg5+jNO7PoOP3+7AUe3/RdDxt+PYQ9Og7OnVxddkXH1pf24FwfWr4VEIkXss88jbOzEbjt/j3GHnjexvPnmm8jNzcWPP/7Y4TYY9oiIiNASen7891qk/tAa9JavhMK6c0EPACQSCSQSGaRWspvei5o8DeH3P4jsXw7j+I4tOP39DqTs3YXgEaMREzcTfQM6dkO+sZqbGnHg32uRfnAfnDz6Iu6FP6Fv/wEmPScZ791338WWLVuwf/9+2Nl1fBY0wx4REVk8Qa9vCXr7dkM1eCimL+uaoGcMmVyO0DHjMHD0WKjTUnF8x2ZkHUlC1pEk+IYNRczUGfCPGNblEySqSkuwffWbKL2QB/+IYZj8+xdh6+DYpeegjlu9ejU2btyI/fv3w8XFpVNtMewREZFFawl6HyF13x6oBodj+rJXui3oXU8ikcBvaAT8hkagNP88TuzciqzkQyjIOIM+vn6InjoDA0ffA5m88/fR5aecxK4176KhtgYjZs7ByIdnQyq9ueeRxKHRaPDCCy8gICAA48ePBwBYW1vj6NGjHWpPIgiC0JUF9iRKpRIajUbsMoiIqIcS9Hrs//RDnNn/PVRDIjD9/14WJejdTvXlMpzavQ1nftyL5oZ6OLi5I2pSHIbeGwtrO/t2tyfo9Ti69Vv8/N8vYW1nh0m/ewEDhg03QeWdY+rvb51Oh+zsbAQHB0Mm690h15hrYdgjIrJgZep8VBRq4DMwDPYurmKX060EvR77P/kQZ35sDXrLXukxS6D8WsPVWpzZ/z1O7dmOqxVXYGVrh6H3xiJqUpzRS6M0XK3Fnn+uxvmTx+Ch8kfcCyvg4uVt4so7hmHPeMZcC4dxiYgslCAI+O6dN1BdVgIAcPX2gXLQYChDW36c+niIXKHpCHo99n3yT6T9uBd+QyMx7f9e7rFBDwBs7B0wfNrDiJo8DZmHE3Fi51ac2LEFp3Zvw8DRYxE9dQY8VP63/XyZOh/b//5XVBYXIfTu8bjv6d/1qB5MMi2GPSIiC1WUk4nqshL0j4yGnZMzCs6mI+3HvUj7cS8AwNmzL5ShQ6AcNBi+gwbDyaOvWeyiIOj12PevfyDtwA+9IuhdT65QYPD4+xA2diIupJzE8R2bcfbQAZw9dAD9I4YheupM+IYNueHf07nDifhh3RrodVpMePK3iLj/QbP490jGM3nYy8nJwRNPPIHLly/D2dkZn332GcLCwm445siRI3jmmWcAAM3NzRgzZgwSEhJgbW2NxMRETJo0CSEhITccb2tra+rSiYjMWmbyIQDA6EfmGZb4qL5cCs3ZdBScTYfmXBoyftqPjJ/2AwAc3T2gDA1r7f0bAlfvfr0uNFwf9PzDoxD34opeE/SuJ5FKERAVg4CoGBTlZuHE9i3IOXYEF1JOom9AIKKnzsCA6LuQ9NVnOL1nBxxc3TDl+T/CJyRU7NJJBCa/Z2/ChAl4/PHHER8fj02bNuHtt9/G8ePHbzimrq4OCoUCCoUCer0eM2fOxD333IPnn38eiYmJWLJkCVJSUtp9bt6zR0R0a3q9Dh8/Ox8Ka2s8+f7Htw1ttVfKUXAuHZqzadCcy8CVSwWG9+xd3aAcGAbloCHwHTQYbj6+PTr8CXo9fvj4H0g/2BL0pr34MuRWVmKX1WUqi4twYtd3yEjcD21TI+RW1tA2NUIZOhhTlizvVfdk8p4944l+z15paSlOnDiBH374AQAwc+ZMPPfcc8jNzUVg4P8Wirx+ocCmpibU19f36F8YRES93aVzGbhacQUjZjza5u9bBzd3hI4ei9DRYwEAdVWV0Jy71vOXblgPDgBsnZxbev5a7/nzUPlDIpV2y/XcSUvQW4P0g/vgHzEM015YYVZBDwBcvLxx71PPYNSsuUjZuwvpifsQMvJujJn9OGRy3rVlyUz6b7+goADe3t6Qt/4lk0gkUKlUUKvVN4Q9AMjPz8e0adOQl5eHBx98EM8++6zhvby8PERFRUEmk2H+/Pk3vHe91atXY/Xq1YbntbW1JrgqIqLe79oQbsioe9r1OTtnFwSPGIPgEWMAAPU11dBkZkBzNh2as+nIOXYEOUdbNnC3sXeAT2gYQkaMQfDIMV2yPlxHCHo99q5LQEbifvSPGIY4Mwx617NzcsaoWXMxatZcsUuhdtq2bRteeuklWFtbw9XVFeXl5ZBKpXB0dERCQgIiIyM71G6Pifr+/v5ITU1FbW0t5s2bhy1btmD27NmIioqCRqOBs7MzNBoNJk+ejD59+uCRRx65qY2lS5di6dKlhudKpbI7L4GIqFfQabXIOZoMd6UKfXz9OtWWraMTgmJGIihmJICW5T0Ks86h4GwaNOfScf7UceSdOIqfvlyP8PsmIfzeSbBz7txuAO1xQ9CLjEbc0j+ZddCj3m3t2rVYuXIl5syZg8rKSsPOGVu3bkV8fDxSU1M71K5Jw56vry+Kioqg1Wohl8shCALUajVUKtVtP+Pg4IDZs2fjyy+/xOzZs+Hk5GR4T6lUYs6cOUhKSrpl2CMiojsrSE9FfU01oibFdXnbNvYOhokDQEvPX9qBH5CydxeSv/0SR7d8g4GjxyFy0lST78Gq1+vww9o1yPipNei9sAJyhTi9i0R3snjxYiQlJSEzMxNr1qxBcnKy4b2qqqpO3d5m0rDn6emJqKgobNiwAfHx8di8eTOUSuVNQ7i5ubnw8/ODQqFAU1MTtm7diqFDhwIAioqK0LdvX0ilUtTU1GDnzp146qmnTFk2EZFZy0xuuccuZNTdJj+XraMThk97GNFTHkLOsSM4tWe7YYavz8AwRE2OQ2D0CEi7+Cb5lqCXgIyffkRAVAymLv0Tgx616fc//h4FNQV3PrADfB19sWbimjaPSUhIwJkzZ7BkyRJMnz4dAPD444/j4MGDAIDdu3d3+PwmH8Zdt24d4uPj8eabb8LJyQnr168HACxYsABxcXGIi4vDgQMHkJCQAJlMBq1Wi4kTJ+KVV14BAGzevBkfffQR5HI5tFotZs2ahfnz55u6bCIis6Rtbkbu8SPw7D8Art4+3XZeqUyGkJFjEDJyDErO5+LU7m3ITE7CjtVvwbGPByIfmIIhEx6AjYNDp8+l1+uw96MPcPbQAQY96tW++OILAMDnn3+O5cuXdzjwcbs0IiILknviKLb97Q3c89h8xMTNFLWWq5UVSN23B6n7dqOuqhJya2uE3TMBkbFxcFf6dqhNvV6HvR++j7NJBxn0ejFLXXpl3LhxN/TsXc/W1hYajQbu7u43vC760itERNSzZF2bhTvS9EO4d2Lv4opRs+Zi+PRZyD6ShFN7treGvz3wGxqJqMlx6B8+zOjlW24IesOGY+rzf2TQo16psrISdXV16NevHwDgu+++g7u7O9zc3DrUHsMeEZGFaG5sQN6Jo/AOHggnD0+xyzGQKxQYdM8EhN49HoVZ53Bqz3bkHEvGxTOn4ertg8jYKQgbOxFWtna3bUOv1+H7D9/HOQY9MgNVVVWYNWsW6uvrIZVK4eHhgZ07d3Z4kgbDHhGRhTh/6gSaGxswsJ1r63UXiUQCn4GD4DNwEKovlyLlh91I2/89Dqxfh8Nf/weDx9+HyNipcOnrdcPn9Hodvv/nezh3OBEDou/C1OdfEm1NP6LOSExMNDw+duxYl7XLsEdEZCGykg8BEgmC7xotdil35NTHE/fMjcfImbNxLikRp/Zsx6nd23Bqz3YMGDYcUZPi4Bs2FIJejz3/XI3Mn3/CgOgRmPr8cgY9ol9h2CMisgCNdXW4cPoEfEMHw8HN/c4f6CEU1jYYem8shkx8AOq0VJz6fjvyTh5D3omj6KPyh4ObO/JTTiIwZgSmLGHQI7oVhj0iIguQd/IotM1N7d4eraeQSCTwGxoBv6ERqCguRMr3O5GeuA+X1fkIjBmJKUuWMegR3QbDHhGRBchKPgSJVIqgu0aJXUqnuXr1w/j4hRj1yDwUZp+DanA4ZHJ+nRHdDv/rICIyc/W1NchPPQ2/IRGwc3IWu5wuY21nh/4Rw8Qug6jHM27xIiIi6rVyjx2BXqfttUO4RNQ5DHtERGYu60gSZHI5AmNGiF0KEYmAYY+IyIzVVVVCnZYK/4hhsLHv/L6zRGQ627ZtQ2hoKCIiIpCWlgYAWL9+PSQSCb777rsOt8t79oiIzFj2Lz9DEPQcwiXqBdauXYuVK1dizpw5AID8/Hz861//wogRneuVZ9gjIjJjWUeSILeyxoBhw8UuhahHK3jmWTQVqE3StpWvCr4ffdjmMYsXL0ZSUhIyMzOxZs0aHD58GAsWLMCaNWvwwgsvdOr8HMYlIjJTNeWXocnMQEBUDKxsbMUuh4jakJCQgOjoaLz33ntITk7G6tWrMXr0aAwb1vkZ5+zZIyIyU9m/HAYEASGj7ha7FKIe7049b90pPT0dmzdvxqFDh7qkPYY9IiIzlZWcBIWNLfpHRotdChG1Q1JSEvLz8xEUFAQAKC4uxsKFC1FUVIRnnnmm3e1xGJeIyAxVlRajKDcLgTEjoLCyFrscImqHZ555BkVFRcjPz0d+fj5GjBiBjz/+uENBD2DYIyIyS5nJSQCAkJEcwiWydBzGJSIyQ1lHkmBtbw//8EixSyEiIyUmJrbrdWOxZ4+IyMxcKdSgLP88goaPhkyuELscIhIZwx4RkZnJujaEy1m4RASGPSIisyIIAjJ//gm2Ts5QhQ0Vuxwi6gEY9oiIzMhldT6uFGoQPGIMpDKZ2OUQUQ/AsEdEZEayjrQM4Q7kLFwiasWwR0RkJgRBQGbyITi4usFn4CCxyyGiHoJhj4jITJTk5aCqpBjBI++GRMpf70S9zbZt2xAaGoqIiAi4u7sjJCQEERERiIiIwDfffNPhdrnOHhGRmci8NoQ76h6RKyGijli7di1WrlyJOXPmwN/fH9988w0iIiI63S7DHhGRGRD0emQdSYKTR194BQaLXQ4RtdPixYuRlJSEzMxMrFmzpkvbZtgjIjIDl7LPobb8MmKmPQyJRCJ2OUS9zq4Pz6CqrN4kbTt72OLBZ9teCikhIQFnzpzBkiVLMH36dPj7++Pxxx+HIAgYPnw4Vq1aBQ8Pjw6dnzd1EBGZgWsLKXMIl8g8HDp0CGfOnMGpU6fQp08fPPHEEx1uiz17RES9nF6nQ/Yvh+HaTwkPv/5il0PUK92p5627qVQqAIBCocCSJUsQHNzx2zPYs0dE1MsVnE1DXVUlQkbezSFcIjNw9epVVFZWGp5v3LgRkZGRHW6PPXtERL2cYSFl7oVLZBZKSkowc+ZM6HQ6CIKAgIAAfPHFFx1uj2GPiKgX02m1yDmaDA+VP9yVKrHLIaJOSExMNDw+ffp0l7XLYVwiol5MnZaChtoahHBiBhHdBsMeEVEvlpl8CAAQwr1wieg2GPaIiHopbVMTco8fQd+AILh4eYtdDhH1UAx7RES91IXUk2iqr+fEDCJqE8MeEVEvlfVzyxBuMIdwiagNDHtERL1Qc0MD8k4dQ7+QQXDq07EtlIjIMjDsERH1QnmnjkHb2MghXCK6I4Y9IqJOqCotxo///ghFOVndet6s5EOQSKQIHjGmW89LRKazbds2hIaGIiIiAqmpqXjuuecQFBSEIUOGYN68eR1ul4sqExF10PnTx7Fnzd/RcLUWqfv2YMSMR3HXQ49CJjftr9bGuqu4cPoEfMMGw97F1aTnIqLus3btWqxcuRJz5szB888/D4lEguzsbEgkEhQXF3e4XYY9IqJ20ut1OLJpI37Z/DVsHBxx39PPIeWHXTiyaSMupJzEpN+9ALd+PiY7f+7xX6DTarmQMlEX2vrO66gq6XigaotzXy88tGxlm8csXrwYSUlJyMzMxDvvvIO8vDxoNBrDftdeXl4dPj+HcYmI2qGuugpb3noVv2z+Gn0DAjHvrfcx9N5YzP3rasTEzURxXg7+89JipO7bDUEQTFJDVvIhSGUyBA0fZZL2iaj7JSQkIDo6Gu+99x4+//xzuLm54c0330R0dDTuvvtu/Pjjjx1umz17RERGKszOxI73V6G2/DKG3huL8U8shNzKCgAgVyhwz2PzERAZgz0frsb+Tz5E3sljeOC3f+jSodb6mmpcTEuB35AI2Do6dVm7RJbuTj1v3Umr1eLixYsYNGgQVq1ahdOnT+O+++5DRkYG+vbt2+722LNHRHQHgiDg9N6d+ObVl9BQXY3YZ5/HfU8/Zwh611MOGozH31mDsLETceH0CXz+4u+Qc/xIl9WScywZep2OQ7hEZkylUkEqleKxxx4DAERGRqJ///5IS0vrUHsMe0REbWhuaMDuNe/iwL/XwqmPB+b85V2EjZ3Y5mes7ewR++zzmLr0jxAEAdvf/Su+/+h9NNXXdbqerORDkMnlCIwZ0em2iKhn6tOnDyZOnIi9e/cCAC5cuIALFy4gNDS0Q+1xGJeI6DauFGqw/e9volyjRmDMCDzwzBLY2DsY/fngu0ajX3Ao9q79ABmJ+6E5m4ZJv3sBPgMHdaieq5UVKMhIx4Do4bC2s+9QG0TUO6xduxZPPfUUli9fDqlUinXr1sHHp2MTvxj2iIhuIfuXw/j+ow+gbWzEPY/NR/TUGYZZce3h4OqGGS+9ipQfduHQhvX45tWXMHz6wxj58BzI5Ip21yQIeg7hEpmpxMREw+OAgAAcPHiwS9pl2CMiuo5Oq0XSV5/h5K7vYOfsgoeWvQLfsKGdalMikSDygSlQDQ7Hnn/8HUe3fosLKScx+bkX4a70NbqdzOQkyK2tMSBqeKfqISLLYvJ79nJycjBq1CgEBwcjJiYGGRkZNx1z5MgRREREICIiAmFhYVi0aBEaGxsN73/66acICgrCgAED8PTTT6O5udnUZRORBaq9Uo7/vvEnnNz1HXwGDsJvVn3Q6aB3PXcfX8x5412MmPEoyvIvYMNLf8CpPdsh6PV3/Gz15TIUZp3FgKjhUNjYdFlNRGT+TB72Fi1ahIULFyI7OxvLly9HfHz8TceEh4fj+PHjSElJQVpaGkpLS/Hhhx8CaLkp8ZVXXkFSUhJyc3NRUlKCjz/+2NRlE5GFKTibhv+89AdcyjyLYQ9Ox6xX3oSDm3uXn0cml2P0o7/B7NffhoObOw5+9jE2v/Vn1Fy53Obnso8kAQBCRnMIl4jax6Rhr7S0FCdOnDDs5zZz5kwUFBQgNzf3huPs7OygULTcu9LU1IT6+nrDvTGbNm1CXFwcvLy8IJFI8Nvf/hYbN240ZdlEZEEEQcDx7Zvx3zdWoLmxEVOffwnjHl9g8i3P+gWH4jfvJGDIhPtx8cxpfPHic8hqDXS3kpmcBCtbO/QPH2bSuojI/Jg07BUUFMDb2xvy1l+aEokEKpUKarX6pmPz8/MRHh6OPn36wNnZGc8++ywAQK1Ww8/Pz3Ccv7//LT8PAKtXr4ZSqTT81NbWmuCqiMhcNNZdxfa//xWHvlwPt35KzHvrPQSPGNNt57eyscX9ixZj2v+9AolMhp3vv43d/2jZa/d6lcVFKDmfg8CYEbdc24+IqC09Zp09f39/pKamori4GI2NjdiyZUu721i6dCk0Go3hx8HB+CUSiMiylF28gA1/XILc479g4OixmPvXv8Otn1KUWgKj78ITf/sHAqJicC7pIL74v9+jIOOM4f1rPX4DOQuXiDrApGHP19cXRUVF0Gq1AFqGS9RqNVQq1W0/4+DggNmzZ+PLL78E0LKK9MWLFw3v5+fnt/l5IqI7OXvoAL56+UVUl5VhwpO/xeTfvwgrG1tRa7J3ccX0ZStx38Ln0FBbg2/fWIHE/3wKbXMzMpMPwcbBEaohEaLWSESmtW3bNoSGhmLIkCGQyWSGyavBwcGQy+W4cuVKh9o1adjz9PREVFQUNmzYAADYvHkzlEolAgMDbzguNzfXMMO2qakJW7duxdChLTPgZs6cie3bt6O4uBiCIGDt2rWYPXu2Kcsmom52pVCDvJPHUJSTharSYjQ3NJjkPNqmJuz71z+w55+rYePoiEdfXYXIB6Z0aP08U5BIJBg6MRa/eScB3oHBOLlzK774v9/hsjofQXeNMvl9hEQkrrVr12LlypVIS0uDTqdDSkoKUlJSsHDhQkyaNAlubm4datfkvznWrVuH+Ph4vPnmm3BycsL69esBAAsWLEBcXBzi4uJw4MABJCQkQCaTQavVYuLEiXjllVcAtCwq+Nprr2H06NEAgHHjxmHRokWmLpuITKy5oQFZvxxG+sEfcCnz7E3vy62sYefsDDsnZ9g5u8DW0fnG5043PpYr2l6guKq0BDveW4WS8zlQDYnAg4v/D3ZOzqa6vE5x9eqH2a+9g6PffYsjm1ompIWMvFvkqojIlBYvXoykpCRkZmZizZo1SE5ONrz36aef4q233upw2xJBEISuKLInUiqV0Gg0YpdBRK0EQUDJ+VykHdiLzJ9/QlN9PeQKKwSNGA1l6GA01NagrroK9VWVqKupRl1VpeG5rvV2kNuxsrVrDYMusHN2bg2DLY8lUimSv/0SDbU1GDHjUYycNRdSqaybrrpzSs7nojgvB0Pvje0xPZBEpmbq72+dTofs7GwEBwdDJmv5XXD58wxoy00zqiB3t0GfJ8LueNy4ceOwZMkSTJ8+3fBacnIyZsyYAY1GY5jwer1bXctN5+946URExmmorcW5wweRduAHlF28AADw8OuPIRMfQOjocbC5w2QqQRDQVF+PuupK1FVVob666lePqwzBsKq0GEW5WTctVGxtb4+Hlv8ZAVExJrtOU+gbEIi+AYF3PpCIzNKnn36Kxx9//JZBz1gMe0RkEoJej4Kz6Ug7sBc5x5Kha26Gla0dwu+bhCETHoBn/wFG91RJJBJY29nB2s4Orl79jDp3w9Xa1l7BKtTXVMM7KMQkiyQTkXkwpuetu9XW1uLbb7/F8ePHO9UOwx4RdanaK+XI+OlHpB/ch8qSIgCAz8AwDJlwP4JHjIbC2vRbfUmkUtg6OsHW0QnwMX7vWSKinuSbb75BeHg4Bg4c2Kl2GPaIqNP0Oh0upJxA2oEfcP7UcQh6PeycXRA9dQaGTLhftPXriIh6s08//RRPP/10p9th2COiDqsoLkT6wX3I+OlHXK24AolECv+IKAyZcD8CooZzqRAionZITEy84fn1M3I7g7+JiahdtE1NyDmWjLQDPxh2eXDy8MSoRx5D2Nh74dTHQ+QKiYjoegx7RGSU0vzzSD+4D+eSDqLhai1kcjlCRt6NwRPuh9/gcEikPWb3RSIiug7DHhG1qaq0BIlf/Au5x38BALgrVRgxcw5C7x7XYxclJiKi/2HYI6Jb0ivkwHMAACAASURBVDY348SOLTi69VtomxoRfNdoDJvyELyDQri4LxH1atd+h5nDvhLXrqGt38sMe0R0kwspJ3Fg/VpUFhfBrZ8SE578LfyGRIhdFhFRl5BKpbCxscGlS5fQt29fKO6w3WJPJQgCysvLoVAoIG3jVhqGPSIyqC4rxcHP/4Xc40cgt7bG3XPjMezBaZDJe+cvQiKi2/Hz80NpaSny8/N7dQ+fQqGASqVq8xiGPSKCtrkZJ3duxS9bvmkZsh0xBmN/8xRn1hKR2ZJKpfDy8kLfvn0hCEKvDHwSiaTNHr1rGPaILFx+ykkc+GwdKooK4ertgwlP/hb+QyPFLouIqFtIJBKzvw+ZYY/IQlVfLkXiF58g52gy5NbWGDPnCQx7cDrkvfTeFSIiujWGPSILo9M248TO7/DLlq+hbWyZZTv28afg1MdT7NKIiMgEGPaILEj+mdM4sH4dKgo1LUO28xfBPzxK7LKIiMiEGPaILED15TL89MUnyD76M+RW1hgz+3EMm/IQh2yJiCwAwx6RGdNpm3Fy1zYc2bwR2sZGBA0fhXFPLOCQLRGRBWHYIzJTF9NS8OO/16KiUAMXL29MmP9b9I8YJnZZRETUzRj2iMxMTfllJP7nU2QfSYLcyhqjH/0NoqfO4JAtEZGFYtgjMhM6bTNO7d6OI5s2ormxAYExIzH+iafh5MEhWyIiS8awR2RiOq0Wl9X5KM7LRl11FQS9HnqdHoJeB72+9U+dvuWx7rrX9HrodbrW43UQhF891+uhv9aGToerlZWoKS+DS19vTJi/CP0jo8W+dCIi6gEY9oi6kCAIqCotQXFuFopys1Gcm43SC3nQNjd1rmGJBFKpDFKpFBLZjX9KpVJIpDLIFHKMeuQxxEydCbmVVddcEBER9XoMe0SdUF9TjeK8HBTlZKE4ryXc1ddUG963treHT2gYvAOD4RUYAkf3PpDKZJBIpZBeC22tIe761yVSKaTSa39KITFi70MiIqJbYdgjMpK2qQml+edRnJdtCHeVxUWG92VyOTz8AxAy6h54B4XAa0AwXL37mf2ei0RE1LMx7BHdgqDXo6K40BDqinKyUXbxAvQ6reEYV28fhN49Hl4DguEdFAwPvwDOeCUioh6HYY8ILeHuUtZZ5KeebhmOzctG49WrhvdtnZzhHx4Jr8BgeAeGoO+AINg6OIpYMRERkXEY9siilV28gHOHE5H58yHUlJcBAOQKK3gGBLbeZ9cS7pw8PDkcS0REvRLDHlmc6rLS1oD3Ey4XXAQAOLp7IGbawwi+azQ8/PpDJud/GkREZB74jUYWoa66Ctm//IxzhxNRmHUWAGDj4Ijw+yZh4Oix8AkZxBmvRERklhj2yGw1NzQg9+RRZB5ORH7qKeh1OsitrBEy6h6EjhkL//AoyOScUEFEROaNYY/Mik6rhTotBecOJyL3+C9obmyARCqF39BIhI4ei8CYEbCytRO7TCIiom7DsEe9niAIKMzORObPicg6chj11VUAAO+gEISOGYfgEWNg7+IqcpVERETiYNijXqtco8a5w4k4d/gnVJeVAADc+ikRFTsVA8eMg0tfL5ErJCIiEh/DHvUqNeWXkfnzTzj3808oyz8PAHBwdcOwKQ8hdMw4ePoHcIkUIiKi6zDsUa8gCAKObv0WP3+7ARAEWNvZY/D4+xE6ZhyUg8IglcrELpGIiKhHYtijHk+n1WL/Jx8i/eAP6KPyx6hZc9E/MoZbkxERERmBYY96tMa6Oux47y1cPHMa/uFRmLLkJVjbcTYtERGRsRj2qMeqKb+MrateRZk6H0Mm3I+JTz3LnS2IiIjaid+c1COV5p/H1rdfQ+2VcoyZ/TiGT5/FiRdEREQdwLBHPU5+yknseH8VdM3NmPz7FxE6ZpzYJREREfVaDHvUo5z5cS/2f/JPWNva4aEVf4Zy0GCxSyIiIurVGPaoRxAEAT9/8x8c3fotnDz6YsYfX4W7j6/YZREREfV6DHskOm1zM/Z+9D4yf/4JXgOCMH3ZSm5vRkRE1EUY9khU9bU12P7uX6E5l44B0SPw4O9fhMLGRuyyiIiIzAbDHommqrQYW956FVcKNYicNBXjHl/AnTCIiIi6GMMeiaI4Nxtb33kdddVVGP/E04iaPE3skoiIiMwSwx51u9zjv2BXwt8AQUDc0j8iaPgosUsiIiIyWwx71K1O7dmBg59/DFtHJzy0bCW8g0LELomIiMisMexRtxD0evy04VOc3LUNrt4+mPHSq3Dx8ha7LCIiIrPHsEcm19zUiD1r/o6cY8nwGTgI0158GbaOTmKXRUREZBGkpj5BTk4ORo0aheDgYMTExCAjI+OmYw4cOIDhw4dj0KBBCAsLw7Jly6DX6wEA+fn5kMlkiIiIMPzk5eWZumzqInVVlfjv639CzrFkhIy6Bw+v+AuDHhERUTcyec/eokWLsHDhQsTHx2PTpk2Ij4/H8ePHbzjG1dUVX3/9NQICAtDQ0IB7770XX3zxBeLj4wEAjo6OSElJMXWp1MWuFF7CllV/RlVJMYZPexhjZj8OidTk/39BRERE1zHpN29paSlOnDiBefPmAQBmzpyJgoIC5Obm3nBcZGQkAgICAAA2NjaIiIhAfn6+KUsjE9NkZmDjKy+iuqwU9z39HO6eG8+gR0REJAKTfvsWFBTA29sbcnlLB6JEIoFKpYJarb7tZ4qLi7Fp0yZMmTLF8NrVq1cRExODqKgovP7669DpdKYsmzopM/kQNv3lZeiam/HQspUYem+s2CURERFZrB7V1VJdXY2pU6di2bJliI6OBgB4e3vj0qVLOH78OPbv34+kpCT8/e9/v+XnV69eDaVSafipra3tzvIJwPEdW7Drg3dg4+CIR197G/0jo8UuiYiIyKKZNOz5+vqiqKgIWq0WACAIAtRqNVQq1U3H1tTUIDY2FtOmTcPSpUsNr1tbW8PT0xMA4ObmhieffBJJSUm3PN/SpUuh0WgMPw4ODia4KrqdK4UaHNrwb7grVZj7l3fRt/8AsUsiIiKyeCYNe56enoiKisKGDRsAAJs3b4ZSqURgYOANx9XW1iI2NhaxsbF4+eWXb3ivtLQUzc3NAIDGxkZs2bIFkZGRpiybOig/9TQA4O65T8Cpj6fI1RARERHQDcO469atw7p16xAcHIxVq1Zh/fr1AIAFCxZg+/btAIAPPvgAx44dw5YtWwzLq/z1r38FABw+fBiRkZEIDw9HVFQUvLy8sGLFClOXTR2gTk+FRCqFMnSI2KUQERFRK4kgCILYRZiKUqmERqMRuwyLoNfp8OGCuXDzUWLuX259TyUREZEx+P3dtXrUBA3qvUou5KKx7ipUgyPELoWIiIiuw7BHXUKdlgoA8BsSLnIlREREdD2GPeoS6vQUyK2s4R0cKnYpREREdB2GPeq05qZGXMo6B5+BgyBXKMQuh4iIiK7DsEedVph1DrrmZqgGcwiXiIiop2HYo05Tp7fcr8ewR0RE1PMw7FGnqdNSYGPvAM/+AWKXQkRERL/CsEed0nC1FiXn8+AbNhRSqUzscoiIiOhXGPaoUwrOpkEQ9BzCJSIi6qEY9qhTrq2vp+L6ekRERD0Swx51ijo9FQ5u7nD19hG7FCIiIroFhj3qsNor5bhyqQCqweGQSCRil0NERES3wLBHHXZtyRW/IdwPl4iIqKdi2KMO4/p6REREPR/DHnWIIAi4mJ4Kt35KOLi5i10OERER3QbDHnVIRVEhassvcxYuERFRD8ewRx3CIVwiIqLegWGPOkSdlgKJRArfQUPFLoWIiIjawLBH7abX61CQcQZ9AwbAxsFB7HKIiIioDQx71G5l+RfQcLWWQ7hERES9AMMetdvFtBQAgGow19cjIiLq6Rj2qN3U6amQKRToNzBU7FKIiIjoDhj2qF20zc24lHkWPiGhUFhZi10OERER3QHDHrVLUfY5aJsaOYRLRETUSzDsUbtwfT0iIqLehWGP2uVieiqsbO3QNyBQ7FKIiIjICAx7ZLTGujoU52bDN2wIpDKZ2OUQERGRERj2yGiac+kQ9HoO4RIREfUiDHtkNHXr+np+Qzg5g4iIqLdg2COjqdNTYe/qBjcfX7FLISIiIiMx7JFRrlZW4HLBRajChkIikYhdDhERERmJYY+Mos44A4BLrhAREfU2DHtkFHVa6/p6Qxj2iIiIehOGPbojQRCgTk+Bq3c/OPXxFLscIiIiageGPbqjqpJiVJeVcgiXiIioF2LYozviFmlERES9F8Me3dHF9FRAIoFv2FCxSyEiIqJ2YtijNgl6PQrSU+HpFwBbRyexyyEiIqJ2YtijNpWp81FfU81ZuERERL0Uwx61ybBFGu/XIyIi6pXaHfaqqqqQnp5uilqoB1Knp0Iqk8NnYJjYpRAREVEHGBX2YmNjUVlZidraWoSHh2PKlClYuXKlqWsjkem0zdCcy0C/4IFQ2NiIXQ4RERF1gFFhr6SkBC4uLti9ezemTZuGnJwcbN261dS1kciKcrPR3NjAJVeIiIh6MaPCXnNzMwDg0KFDuO+++6BQKCCXy01aGInPsEUawx4REVGvZVTYGzx4MCZNmoSdO3diwoQJqKurM3Vd1AOo01OhsLGFV2Cw2KUQERFRBxnVPffZZ5/h+++/R3h4OOzs7HDp0iW89dZbpq6NRNTUUI+inEz4h0dBxl5cIiKiXsuonj0bGxv4+vri8OHDAAA7OzuEh3Noz5xdOpcBvU7HIVwiIqJezqiw9+GHH+LJJ5/Eq6++CgC4cuUK5s6da8q6SGQXuR8uERGRWTAq7H388cf45Zdf4OTUsl3WgAEDUFZWZtLCSFzq9FTYOjmjj6+f2KUQERFRJxgV9qytrWFra3vDa5yNa77qqqtQln8eqrChkEi5yQoREVFvZtQ3uYeHB7KzsyGRSAC0TNhQqVQmLYzEU5BxBgCgGhIhciVERETUWUaFvffffx+PPfYYMjMz4evri7/97W/44IMPjDpBTk4ORo0aheDgYMTExCAjI+OmYw4cOIDhw4dj0KBBCAsLw7Jly6DX6w3v79y5EwMHDkRQUBBmzJiB6upqIy+POuLa+np+Q3i/HhERUW9nVNgLDAzE0aNHcfLkSezduxdnzpxB//79jTrBokWLsHDhQmRnZ2P58uWIj4+/6RhXV1d8/fXXOHv2LE6ePInk5GR88cUXAIDa2lo89dRT+O6775CTk4N+/frhjTfeMP4Kqd3U6alw9uwLZ08vsUshIiKiTjIq7KnVamg0Gtjb28PBwQGXLl2CWq2+4+dKS0tx4sQJzJs3DwAwc+ZMFBQUIDc394bjIiMjERAQAKBlmZeIiAjk5+cDAPbs2YPIyEgMHDgQAPDss89i48aNRl8gtU91WSkqS4o4C5eIiMhMGDXLYtiwYZBIJBAEAQ0NDairq4O7uztKS0vb/FxBQQG8vb0NkzkkEglUKhXUajUCAwNv+Zni4mJs2rQJO3fuBNASNP38/jcj1N/fH0VFRdBqtZwkYgIX01MAcMkVIiIic2FUWvr1MitbtmxBampqlxdTXV2NqVOnYtmyZYiOjm7351evXo3Vq1cbntfW1nZleRaB++ESERGZlw6tqzFjxgzs2rXrjsf5+voaeuEAQBAEqNXqW87krampQWxsLKZNm4alS5caXlepVLh48aLheX5+/g29hddbunQpNBqN4cfBwaEjl2exBEGAOj0VHip/2Dm7iF0OERERdQGjwl51dbXhp6KiAt9//71RM2I9PT0RFRWFDRs2AAA2b94MpVJ50xBubW0tYmNjERsbi5dffvmG92JjY3Hq1ClkZmYCaNnNY/bs2UZdHLVPecFF1FVVQsVZuERERGbDqGFcFxcXwz17MpkMQUFBSEhIMOoE69atQ3x8PN588004OTlh/fr1AIAFCxYgLi4OcXFx+OCDD3Ds2DFcvXoVW7ZsAQDMmjULK1asgKOjIz755BNMnz4dWq0WgwcPxueff97By6W2qA1bpHF9PSIiInMhEQRBELsIU1EqldBoNGKX0Wtsfed15KecxO8+3QgrWzuxyyEiIgvF7++uxb2wCACg1+mgOZsGrwHBDHpERERmpM1hXFdXV8MWadcTBAESiQRXrlwxWWHUvYrzstFUX88t0oiIiMxMm2EvJSWlu+ogkRm2SOOSK0RERGalzbB3/WLGZN7U6amQW1vDOzhE7FKIiIioCxk1G7e0tBR//vOfkZqaioaGBsPrp06dMllh1H2aGxtQmH0OvmFDIZMrxC6HiIiIupBREzSeeuop+Pv74/Lly3jttdfQr18/PPjgg6aujbrJpaxz0Gm13DWDiIjIDBkV9goKCrB8+XJYW1tj6tSp2LJlC/bv32/q2qibGNbX4+QMIiIis2NU2LOysgIA2NjYoLy8HHK5HJcvXzZpYdR91GkpsHF0gqdff7FLISIioi5m1D17wcHBKC8vx7x583DXXXfByckJw4YNM3Vt1A3qa2tQciEPwcNHQSLlsotERETmxqiwd21v2z/84Q+Ijo5GRUUFYmNjTVoYdQ9NRhogCNwPl4iIyEwZ1ZWzZs0aVFZWAgBGjx6NKVOmQC43KidSD3fRsB8uwx4REZE5MirsnTp1CoGBgXjkkUewZ88emPF2uhZHnZ4KR3cPuHj1E7sUIiIiMgGjwt769etx8eJFTJo0CW+//Tb8/PywYsUKU9dGJlZTfhkVhRqohoTfcls8IiIi6v2MviPf3t4e8+fPx9dff43Jkydj1apVpqyLusG1JVe4RRoREZH5MirsabVabNmyBVOmTEF4eDjkcjmOHj1q6trIxNRpLXsf+zLsERERmS2jZln4+PggMjIS8fHx2Lx5M6ytrU1dF5mYIAhQp6fCXamCg6ub2OUQERGRiRgV9k6ePAmlUmnqWqgbXSnUoLbiCoLuGi12KURERGRCRg3jMuiZH26RRkREZBm4ZYKFUqelQCKRwnfQYLFLISIiIhNi2LNAer0OBWfT4DUgCNZ29mKXQ0RERCZ0x3v2du3ahaysLAwbNgxjx47tjprIxErP56Hx6lVukUZERGQB2uzZe+WVV/Dcc8/h6NGjmDt3Lj7++OPuqotMiFukERERWY42e/Y2bdqE06dPw8XFBRqNBjNnzsTChQu7qzYyEXV6KuQKK/QLDhW7FCIiIjKxNnv2bG1t4eLiAqBlRm5zc3O3FEWmo21qQmHmWfQbOAhyKyuxyyEiIiITa7Nnr7KyEtu3bzc8r6qquuF5XFyc6SojkyjMPgdtcxOHcImIiCxEm2FPpVLhvffeu+VziUTCsNcLcT9cIiIiy9Jm2EtMTOymMuiaxro6qNNToG1shE6rhU7bDF1zM7TNza2PW19rfd3w/Nr7Wm3r683Q3uL1xqtXYW1vD8+AAWJfKhEREXUDo7ZLu15FRQU2bNiATz/9FCkpKaaoyaIl//dLnNq9rUOflSkUkMkVrX/KDc+tbO0gv/a6QoGg4SMhlcq6uHIiIiLqiYwOe/v378cnn3yCnTt3YurUqXjjjTdMWZfFuqzOh8LaBpOeW9oSzmQKyBTyG4KcvPWxVC5veaxQQCqTQyKRiF0+ERER9TBthr2CggL8+9//xvr169GnTx/Mnz8fR44cwcaNG7urPotTWVIEF+9+CBo+SuxSiIiIyAy0ufRK//79cfjwYWzbtg0nTpzA7373O8hkHP4zFW1TE6ovl8G1r7fYpRAREZGZaDPsrVixAjk5OVi0aBH+9a9/oaamprvqskhVpSWAIMDFu5/YpRAREZGZaDPsvfbaa7hw4QJef/117N+/HyqVCpcvX8a+ffug1+u7q0aLUVFcCABw8WLPHhEREXWNO07QkEgkuP/++3H//fejvLwc//nPf/DCCy+grKwMRUVF3VGjxahsDXuuXuzZIyIioq7RZs/er7m7u2PJkiU4c+bMDTtpUNeoNPTsMewRERFR12izZy8hIaHND8fExHRpMZauorgIChtb2Lu4il0KERERmYk2w97p06cNj3fs2IGpU6cannNNt65XWVwIFy9v/rMlIiKiLtNm2Fu/fr3hcWRk5A3PqWtdW3YlOCBI7FKIiIjIjBh9zx57m0yLy64QERGRKbRrggaZDpddISIiIlNocxj3+hm3VVVVN83AjYuLM01VFojLrhAREZEptBn23nvvPcNjlUp1w3OJRMKw14W47AoRERGZQpth7+DBg91Vh8WrKC6CwtqGy64QERFRl2rXPXvnz5/H+++/j507d5qqHovFZVeIiIjIFNrs2bv33nvx7rvvIiIiAoWFhYiOjsZdd92FtWvXIiMjA8uXL++uOs2atrm5ZdmV4Vx2hYiIOk9bXo/L/06HtqoREqkUErkEkEogkUkBuQSSa49lEkhkktY/pa2Ppbd47cbjJVIJIJfCbqgH5G42Yl8u3UGbYe/SpUuIiIgAAHz11VcYO3Ystm7dioqKCowdO5Zhr4tUlRS3LLvCmbhERNRJgl7Alf9mQ1veAJsQVwgCAK0egk6AoBdaHusFCFo9hIaWx9C1vA+dAEGnB/TGnctK6ciw1wu0GfZsbW0Nj5OTkzF58mQAgKurK+TyNj9K7VBZ0jo5g2vsERFRJ9UevoSm/GrYj/SG67TADrUh6AVAL7QGwNageIvH8j62d26MRNdmYpNKpdBoNHBxccFPP/2EVatWGd6rq6szeXGWoqKoddmVvgx7RETUcc0lV1G1Nx/yPrZwntS/w+1IpK3DvnIAkHVZfSSONsPen/70J0RGRkIul2P8+PEIDg4G0NLL5+/v3x31WYTK4iIA7NkjIqKOE7R6XPkmC9ALcH0kGFIrhjRq0WbYmzFjBkaNGoWSkhIMHTrU8Lq/vz8+/vhjkxdnKSqKC7nsChERdUr1ATWaC6/CcbwvrFVOYpdDPcgdb7zz8vKCl5fXDa/168ceqK5UWVzEZVeIiKjDmgpqUJNYAIW3PZwmqsQuh3oYk++Nm5OTg1GjRiE4OBgxMTHIyMi46Zj8/HyMGzcOzs7Ohtm/1yQmJsLW1hYRERGGn/r6elOX3W20zc2ouVzGmbhERNQhQrMOV77NAiQSuD0aAomc297TjUz+N2LRokVYuHAhsrOzsXz5csTHx990jJOTE/7yl7/gq6++umUbISEhSElJMfxcP0u4t6sqLYYg6LknLhERdUjV9/nQltXD+X4/KLzsxS6HeiCThr3S0lKcOHEC8+bNAwDMnDkTBQUFyM3NveE4Nzc3jBkzBvb2lveXlHviEhFRRzXkVqL250JY+TnB4W6l2OVQD2XSsFdQUABvb2/DmnwSiQQqlQpqtbpd7eTl5SEqKgoxMTH48MMPTVGqaK7NxGXPHhERtYe+QYuKTdmQWEnh9khwy3IpRLfQ41dGjoqKgkajgbOzMzQaDSZPnow+ffrgkUceuenY1atXY/Xq1YbntbW13Vlqh1xbY4/37BERUXtU7jgPXWUjXB4KhNzdfG5voq5n0p49X19fFBUVQavVAgAEQYBarYZKZfxMIScnJzg7OwMAlEol5syZg6SkpFseu3TpUmg0GsOPg4ND5y/CxCpLiiC3toa9q5vYpRARUS9Rn1GOupMlsA52hf1wrzt/gCyaScOep6cnoqKisGHDBgDA5s2boVQqERho/PYtRUVF0OtbNumrqanBzp07ERkZaZJ6xVBRVAjXvlx2hYiIjKOrbULF1hxIbOVweziI3x90Ryafjbtu3TqsW7cOwcHBWLVqFdavXw8AWLBgAbZv3w6gZes1pVKJWbNm4ezZs1AqlfjjH/8IoCUgDhkyBOHh4RgxYgTuu+8+zJ8/39RldwvDsivcOYOIiIwgCAIqtuZCX9sM1+kDIHOyFrsk6gUkgiAIYhdhKkqlEhqNRuwybqv8UgE+W/oMYqY9jHvmxotdDhER9XBXT5Wg4tts2A7tA/e5oWKXYzI9/fu7t+HKiyLiTFwiIjKWtrIRldvzIHVUwGWa8bdDETHsieh/a+xxJi4REd2eoBdQsSkbQoMOrjODIbNXiF0S9SIMeyK6tuwKe/aIiKgtV48WoTG3EvbDvWA7kKs3UPsw7ImIy64QEdGdNJfVoWr3BcjcbOD8YH+xy6FeiGFPRFx2hYiI2iLoBFT8NxuCVg+3h4Mhte7xeyFQD8SwJxLDsiscwiUiotuoOVSAJnUNHMb4wDrAWexyqJdi2BNJVWkxBEHPNfaIiOiWmgprUb1fDbmnHZzv9xe7HOrFGPZEcm3ZFZe+nIlLREQ3ErR6XPkmCxAAt0dDIFHw65o6jn97RHJt2RVX9uwREdGvVO+7CG1JHZwmqmDl0/P3eaeejWFPJBVcUJmIiG6hMb8KNYc0UPg6wnGcr9jlkBlg2BNJZXEhl10hIqIb6Bt1uPLfbEAmhdusYEhkXK2BOo9hTySVxVx2hYiIblS1+zx05Q1wnuQPhaed2OWQmWDYE4FO24zqMi67QkRE/9OQXYGrR4thHeAMh5H8fqCuw7AngqrSEi67QkREBvq6ZlzZlA2JtQyujwRDIuWoD3Udhj0RXNsTl8uuEBERAFRsz4O+ugkuUwdA7mIjdjlkZhj2RHBtjT0uu0JERHVnylCfUgabQe6wG+Ypdjlkhhj2RFDRusaeixd79oiILJmuugmV3+VCai+H64xATtojk2DYE0FlcSHkVtZwcHUXuxQiIhKJIAio2JIDfZ0Wrg8FQeZgJXZJZKYY9kRQWVwIFy8uu0JEZMnqjpegIfMK7CI9YTu4j9jlkBlj2Otm15Zd4c4ZRESWS1fbhMrd5yFztoJL3ACxyyEzx7DXzQzLrvB+PSIii1W97yKEBh2cpwRAaisXuxwycwx73ezaTFwuqExEZJmaiq7i6rFiWPV34vAtdQuGvW52bY09V/bsERFZHEEQULUzDwDgMmUA792mbsGw180qS1qXXeEae0REFqfhbDka86pgN6wvrHwcxC6HLATDXjerKGpddsXFwqxIWQAAIABJREFUTexSiIioGwlaPSp3X4DEWgbnB/zFLocsCMNeN6ssKWpZdkXKf/RERJak9udC6Mob4DjeFzJHrqlH3YeJoxvptM2oLi3lnrhERBZGV9OE6gNqyNxs4DjGR+xyyMIw7HWjqtJSCIKee+ISEVmY6n0XITTq4DK5PyRyfvVS9+LfuG5UyT1xiYgsTlNhLa4eL4Z1gDNswrhNJnU/hr1u9L9lV9izR0RkCQRBQOWO8wAA5/9n787j26ruvI9/7tUuy5b33c6Cs+8hbEmggXSnBTpAHzqlZW2hQzsLfdrOTDdoCw9MW6adzpS9lJbSmRLaklKmC5QtYSkhJCTO5qy241W2ZUvWrnueP6TIdnbbkmUrv3erl6Sre6Ujxeh+dc69v/ORmVJqRWSFhL0JlCq7ImFPCCHOCKHGHiIH+sk7pxJrtZRaEdkhYW8CpcquFEnZFSGEyHXDS60UvH9atpsjzmAS9iaQt7OdwopKKbsihBBnAN+Gw8R7QxSsrcfkklIrInskdUyQVNkVGcIVQoicF/dF8P2lBVOJHddK+d4X2SVhb4IcKbsiZ+IKIUTu6//jQVQkTuGHZ0qpFZF18hc4QY6UXZEae0IIkdsih/0E3u7EdpYb+3w5Rltkn4S9CZKqsVchYU8IIXKVUgrvs/sAKPzoWVJqRUwKEvYmSJ/07AkhRM4LbvcQOTBA3rmVWCrzst0cIQAJexPG29GO2WKVsitCCJGjVNSg/7kDaHYTBe+TUiti8pCwN0H6OtoorKySsitCCJGjfBsOE+8LS6kVMelI8pgAUnZFCCFyW3wggu/FZsylDlwXyHe9mFwk7E0AKbsihBC5LVFqxcD94RlSakVMOvIXOQGOzIkrJ2cIIUTuibT6CGzuxDarEPs8OS5bTD4S9iaAt13KrojMMkIxen6+A//rbShDZbs5QpwxlFJ4f7cfgMJLZ0qpFTEpmbPdgDNBX0c7ID17InOC2zwEG3sINvYQ2NJN0ZWzsJQ7s90sIXJecJuHyKEB8s6vklIrYtKSnr0J4O1ok7IrIqOCjT2ga+SdV0mkeYDOH25m4IVmVMzIdtOEyFkqGk+WWjFLqRUxqUnYmwDejnYpuyIyxgjHCDX1YZvppuhjsyi7dQnmEgcDfz5E13++Q6TFl+0mCpGTfK8eJu4NU/Deekx5lmw3R4gTkvSRYfFYjP7uTjkTV2RMaHcfxBWOBSUA2KYVUPH3y8hfW0+0O0jXj7fgfXY/RiSe5ZYKkTviA2F8L7VgLnPgukC+38XkJmEvwwa6O1GGITX2RMYEG3sAcMwvSS3TzDru902j4gvLsNbm499wmM4fbCbU1JetZgqRU/r/kCy1culMNJPsSsXkJn+hGZaaE1fCnsgAFTMI7erFWpePyW075nFLZR5ln1uC+yMzMXwRPI9up/epPRiBaBZaK0RuiLT4CGzuwja7CPucomw3R4hTkrNxM8ybPBNXevZEJoT2elHhOPYFJSdcR9M18lfX4JhfQt9vmgi83Ulody+Fl5+FY2GplIoQYhSUUnif3Q86FF46Q/77EVNCxnv2mpqaWLlyJbNnz+acc86hsbHxmHUOHjzImjVrcLvdLF269JjHH330UWbNmsVZZ53FZz7zGaLRqdMr0Xekxp4csycyIHRkCPckYe8Ic7Gd0hsXUnT1bDAUvb/YRc/PdxIfCGe6mULkjOC73YlSK+dVYamQUitiash42Lvlllv47Gc/y549e/jKV77C9ddff8w6BQUFfOc73+HJJ5885rEDBw7w9a9/nVdffZW9e/fS2dnJQw89lOlmp423sx2zxUp+8al3xkKMhjIUwR09mCucWMpOr6aepmnknV1BxT+djWNxKaEdPXR8/238b7ZLMWYhTsGIxOl/7iCaw0zBe6XUipg6Mhr2urq62LRpE9deey0AV155JS0tLezdu3fEesXFxaxevZq8vGN/Ja1bt47LLruMyspKNE3j1ltv5Ze//GUmm51W3vY23BWVUnZFpF3k0ADGYPS0evWOZsq3UvK38yj59Hx0mwnvb/bieWQbMU8wAy0VIjf4X2kl3i+lVsTUk9EE0tLSQlVVFWZz4tBATdOor6+nubn5tJ+jubmZadOGfkFNnz59VNtn05GyKzJzhsiE1Fm4C0rH/ByO+SVU3H42eedVEt7fT8cPNuN7uQUVl14+IYaL9YfxvdyaKLVyvhyWI6aWnOpuuu+++6itrU1d/H5/VtsjZVdEpiilCDZ6MBXasFSP77gh3W5OFGP+7GLMhTb6//cgXT/eQqQtu//9CDGZDPzhICpq4P6IlFoRU09G/2Lr6upob28nFosBiR1Uc3Mz9fX1p/0c9fX1HDp0KHX/4MGDJ9z+9ttvp7W1NXVxuVzjewPjJGVXRKZE2weJ94VxLChJ29mAtpluKv5hOfkX1xFt99P1n+/Q/4cDqKgUYxZntnDzAIF3urDPKcIxR6a9FFNPRsNeeXk5y5cv54knngDg6aefpra2loaGhtN+jiuvvJL169fT0dGBUooHHniAa665JlNNTquhsivS5S/SKziKs3BHQ7PouD8wnfLPL8NS5cL3UiudP3yH8P7+tL6OEFOFUor+ZKkV96Uzs90cIcYk433RDz74IA8++CCzZ8/mnnvu4bHHHgPg5ptvZv369QAEAgFqa2u5+uqr2bFjB7W1tfzLv/wLADNnzuTOO+9k1apVNDQ0UFZWxi233JLpZqfFUNkV6dkT6RVq9KDnmbFOd2fk+a3VLsr/binuD88g3h+m+6F36ftNEypqZOT1hJisglu6iTT7cJ1fjaX89M56F2Ky0ZRSOXskdm1tLa2trVl7/af/3zdpbdzG3/9snZyNK9Im1hOk47ubcK6ooPiq2RPyen1PNxHe349tdhGln5qHZjFl/HWFyDYjEqfze5tQMYPK/7sC3Sln4E6UbO+/c43MoJFBUnZFZEKmhnBPxFzioPTmRXh/s5fBtzrwPL4jUbLFKoFPTF7KUKhoHBWOY4SS1+HY0P3IyZYn7wdiGINRCi8/S4KemNIk7GXIkbIrM5efm+2miBwTbOxBs5qwN0zcnJyarlH4sQYwaQy+0U7PTxspuX6BBD6RVTFPkIHnDxH3RzHCcdSRkBZOhDZGO26lg2Y1o9tNaDYT5hI7lsWl5J0rx12LqU3CXoYcKbsiNfZEOsV9ESLNA4k5bS0T22Os6RqFl5+Fpmv4X2vD85PtlN6wAN02+b9Gop2D6E4Lpnxrtpsi0iTaMUj3I9sw/FE0uwndZkazmTAV2jDbTOg2E5rNnLw2pQLckfUSy8zJZYn7mkWXuW5FTpr839JTVOpM3Ar5RSjSJ7ijB9TEDeEeTdM03B+dCbqGf8NhPD9pTAQ+++T8KlFK4X/lMP1/OAAmHdd5leSvqZPQN8VFWnx4HtuOEY5Tcu08HAvHXlhciDPB5PyGzgGpGnvSsyfSKNjYAyYN+9zs1frSNA33pTPApOF/uRXPo9spvXEhumNyfZ0YkTh9TzcR3NqNudyJZtbwb2xj8K8d5J1XRf57aiX0TUHh/f14Hm8EQ1F63QLssyfucAYhpqrJ9e2cQ6TGnkg3IxQjvM+L7azCrPekaZqG+4PT0XQN34stdD+6jbIbF06ag9hjvSF6fr6DaPsgjgUlFH18NprVRGhHLwPPH8K/4TCDb7aTd34y9Lkk9E0Fod299DyxE3SN0hsXYpuRmdJDQuQaCXsZ0tfRhsliIb9YhhdEeoR29UJcZW0I92iaplHw/mmga/heaKb7kW2U3rQo6xPEh/Z66X1yJ0YwRsH7ppF/cR2anjgOy7GgBPv8YkI7ehh4vhn/q4cZfKOdvAuqyL9IQt9kFtjmofe/d6HbTJTeuBBrbX62myTElCFhL0O8HW0UVlRJ2RWRNsHtHtDAMX9yhD1I9vC9bxqarjHw50N4Ht5G6c0LsxKalFL4N7bR/9x+NIuJkk/NP+5npWkajgWl2OeVDIW+Vw4z+Ho7eRdUk39RjYS+SWbw7U761u1Bd1kou2kRlsrxzQctxJlGwl4GxGMx+ruk7IpIHxWNE9rdh7W+YFIeZ1awth5MGgN/OEj3w9sou3nRhLZTReP0/XovgXe6MJc5KPn0fCxlJ5/tQNM1HAtLsc8vIdjYg++FQ/hfaWXwjTZcF1Tjuqg2672UAvyvt+F9Zh+mQhtlNy/CXOrIdpOEmHIk7GXAgKcLZRhyvJ5Im1CTFxU1cCycPL16RytYU4dm0uj//QG6H3qXss8sxlSQ+cAX84YTx+cd9mOfV0zx/5kzqmMaNV3DuagUx4ISgo0eBp5vxvdyK/7X23CtrMZ1oYS+bBl4qYWBPxzEXJoo7G0utGW7SUJMSRL2MsCbnBO3SObEFWkyNGvG5D4GNP/CWtA1+n+3Pxn4FmFyZ24HHd7fT88vdmIMRslfW0/B2vrU8XmjlQh9ZTgWlBLc7mHghWZ8L7Xif609GfpqJPRNEKUUA386hO/FFiyVeZTetHBS9mgLMVVI2MuAPjkTV6SRiitCO3uwVOVhLrZnuzmnlL+qBk3X8D6zj65kD1+6e2SUUgy+0Y73d/vRzHpaa61puoZzcRmOhaUEt3kYeOEQvpda8L/WhmtVNfkX1kyas45zkTIU/c/ux/9aG9a6/EQdR/m8hRgXCXsZ4JUaeyKNwgf7MQIxXCunzt+T64Jq0DW8v9mb6OG7eVHagqqKGfT9di+BTZ2YS+yJ4/Mq0n/AvqZrOJeU4VhUSnBbd2J498VhoW+1hL50U4ai7+kmAm93YpvppuS6+VNihhYhJjv5rygDvFJ2RaRRKDmEa5/kQ7hHc51XhaZr9P26KTWkay4Z38H18f4wPU/sJNLiwza7iJJr5mQ8cCVCXzmORWUE3+1ODO/+pQX/xvGFPqUUKmqgwvHEfK6RxHyuRvL+8GWa2YRzeXlODyOrmEHv/+wmuM2DfU4RJdfOQ7PI3MtCpIOEvQzok7IrIk2UUgQbezAV27FUnvzs0sko75xK0DX61u1JnbQx1rMpw4cG6HliB4YvSv6aWgreP33Mx+eNhaZrOJeW41hcRnDrUaFvZTWmAuvIsBYxMMKxEdcqnHw8GeJQp//6A386SN65lbguqsWcweMgs0FF4/Q8sZPQ7j4ci0op/j9z0Mzy/SlEukjYS7N4LMZAdxczlp2T7aaIHBA97CfeH8Z1Yc2UnaA97+wKNJNG7//sTh7Dt+iUZVGO5n+zHe/6fWi6RvHfzsW5uCxDrT01TddwLivHsaSMwNZufC8khndPyKyj23Q0qwndZsLstKHZTOhWE5rNNHQ7eV+3DbudvI55AomTRTa24X+jHeeycvLX1GHJgTIkRjhGz+M7CO/vx3l2BUVXzprQEC/EmUDCXpoNeLow4nE5OUOkxdBZuJO35MrpcC4tB12j9793pXr4LOWnDnwqZuD93T4G3+zAVGyn5FPzsVZNjoK6mq6Rt6wc5+Iywgf7QZEIasmwlghtOppp/D1U1hoXjsVlhHb34XuxhcCmTgJvd+JYVEr+mjqs1a40vKOJZwSidD/WSLTFh2tlNe6PzJSgJ0QGSNhLsyNz4krZFZEOwUYPusuCtb4g200ZN+fiMtA0en+5K3XSxslmQoj7Ionj8w4NYGsopPgTcyflMWuaScN+VmHmX0fTcMwtxjG3mPCBfgZebCH4rofgu4lj3PIvrsM2ferMFRv3RfA8up1oxyD5F9dR8P5pU7b3WojJTsJemvUla+xJz54Yr2h3gFhXkLxzK3Omt8O5qBRNn0fPkzvpfvhdSm9adNxeqUiLD8/Pd2AMRHBdWIP7gzPQTLnxGaSDbYabshluIof9+F5qIbjdk5hhZXoBBRfXYZtdNKmDU8wbxvPINmKeIAUfnE7BmrpsN0mInCZHwKaZt1PKroj0yJUh3KM5FpRQ8qn5GKE4nke2ETnsH/H44KYOuh7YihGIUXzNHAovnSlB7wSsNS5KPjmPitvPxrmigkizD89jjXT96B0C73ajjFGcATJBYp4g3Q9sJeYJUnj5WRL0hJgAEvbSzNsuZVdEegQbe9BsJmwTMEQ40Rxziyn99HyMSJzuh7cRafWh4gZ9z+ylb10Tpnwr5Z9bkjjWT5ySpcxJ8VWzqfzyObhWVRPrDtL75C4673ubwbc6UDEj200EINoxSNeDW4n3hym6enaiHqMQIuNkGDfNpOyKSId4f5hoiw/HkrKcLUFhn1NM6XUL6PnZDrof3oalwkmk2Ydtppviv52LySXTY42WudBG4UfPIv/iOvyvteF/rY2+p5sYeP4QrgtryTu3Et2andp1kVYfnp9sxwjHKf7beTgXyQ9iISZKbu5FsuRI2RU5Xk+MV3BHbg7hHs0+q4iS6xeAoYg0J87ILL1poQS9cTK5rLjfP52qfz6Xgg9OR8UTU5B13PtXBl5oxghEJ7Q94QP9dD+8DSNiUPrp+RL0hJhg0rOXRkNlV2RoQoxPcLsHzBr2OcXZbkrG2c8qpPy2pcQHIthnF2W7OTlFt5spWFNH/qpqBjd14nu5lYE/H8L3Sit551eRv7oGU356g7WKG4mZQSIGKhon0uan71d7QNcou3EhtplT54xhIXKFhL00Giq7Ij17Yuzig1HCB/qxzy5Gt50Z00VZKvNOWoZFjI9mMeG6oJq8cysThaBfasH/civ+jYfJW1GJY34JKpYMadF4MqzFMSLDlkWOuj6yzrBgp6IGxI89KURzmCm7cSHWuvwsvHshhIS9NBoquyI9e2LsQrt6wcj9IVwx8TSTTt7yCpxLywnt7GHgxRYG32hn8I32038Ss4ZmMaFbkrOCWE1oLiuaRU9crCY0i55YnrzvWFKWE7N9CDFVSdhLo1TZFQl7YhyCjT2ggX1e7g/hiuzQdA3HglLs80sI7+8n1h1AsyRn/LCY0JPXR+4PD3G5UvNRiDOJhL00SpVdKZGDj8XYGJE4oT19WKe75SQFkXGalpz9IwfL+wghhsjZuGnk7WzHXV4pZVfEmIX39EHMwLFQhnCFEEKkh6SSNDHicfq7OmXmDDEuuTprhhBCiOyRsJcmA93JsisVciauGBsVNwju7MVS48JcaM92c4QQQuQICXtp0tchc+KK8Qnv70eFYtKrJ4QQIq0k7KWJNxn2Cisk7ImxkSFcIYQQmSBhL02kZ0+MhzIUwcYezKUOzOXObDdHCCFEDpGwlybejnZMZjOuEumVEaMXafVh+CI4FpSgaVLHTAghRPpI2EsTb0cb7ooqdP3MmN5KpNeRIVy7DOEKIYRIMwl7aXCk7EqhzIkrxkApRaixB73AirVW5g4VQgiRXhL20uBI2RWZJk2MRawrQMwTxDG/RKaiEkIIkXYS9tIgdSauhD0xBsHtchauEEKIzJGwlwapM3El7IkxCO7oQXOYsc10Z7spQgghcpCEvTTwdrQDyDF7YtRivSGih/045hajmeQ/RyGEEOkne5c06Otow2Q2k19amu2miCkmuEOGcIUQQmSWhL008Ha04y6vlLIrYtSCjR40i45tdlG2myKEECJHSdgbp0TZlQ4KZeYMMUpxf4TIwQFss4rQrfJDQQghRGZI2BunAU+3lF0RYxLa2QtKhnCFEFOTPxwjGjey3QxxGszZbsBU520/DEjZFTF6wcYe0MExrzjbTRFCiBTDUPQMRujoD9ExkLh09odo7w/ROey+LxzjqVsv4Jzp8h022UnYGycpuyLGwgjHCDX1YZtZiO60ZLs5QogJFokZvNPcR89gBLtFx242YbOYErctpsTFPHTblKaC66FonK6BMB0DIdr7g4nw1h+mM3U/TJcvRDSujrt9ntVEpdvO4jo3FQV28u0SI6YC+VcaJym7IsYitLsP4kqGcIU4QyilOOAZ5NUmD6/s6eb1/T0EIvHT3t5i0o4TCBMh8chtm8WUvJ943GrW8QaO9NAlAl3vYOS4z69pUJJno9JtY15VPpVuO5UFdioK7FS5HVS6bclwJz9OpyIJe+MkZVfEWAQbkyVX5kvYEyJX9QejvL7Pw8t7PLza1E1rXxAAs66xvL6Ii2aXUl+SRygaJxyNE4oahKJxQrFht6MGodjxH/cGooSi4cRjsfgJe+OsJp0Kt42GMhcVDXYqC2zHhLjyfDtWsxzGn6sk7I2TlF0Ro6ViBqFdvVjr8jG5bdlujhAiTeKGYmurl1f3eHilqZstLV7iRiKATStx8qnzp3HhrFIuOKskIz1ksbhBOHYkECaui5xWipwWNE3m3T6TSdgbhyNlV6YvPTvbTRFTSGifFxWOY5chXCGmvMPeIK/u6eaVpm42NHkYCMUAcNnMXDK3nItml3HRrFKmleRlvC1mk47ZpJNnk127GCnjfxFNTU1cd911eDwe3G43P/3pT1mwYMEx6z366KPcc889GIbBJZdcwo9//GMsFgsvvfQSH/rQh5gzZ05q3ddffx2Hw5Hppp/SUNkVOV5PnL5Qo8yaMZWEY3G8gSh9gQi9gxG8gWjyOkLvYBRvIMJAKEZZvpXaIie1RQ7qip3UFTkpdVmlRyXHBCIx3tzfy8t7unm1qZt93YNA4pi3xTVuLppdxoWzylhWX4hFpkAUk0TGw94tt9zCZz/7Wa6//nrWrVvH9ddfz1tvvTVinQMHDvD1r3+dzZs3U1FRweWXX85DDz3EbbfdBsCcOXPYsmVLpps6akNlV2qy3BIxVShDEdzRg7nciaXMme3mnHFC0fgJQ1tfIJK8ROkbTN4ejDB4GgfRm3QtNVw3nN2iU1vkpK7IkbgudlBX5EzddjsmZngtbqjU++kZTLz/nsHE/d7BCDHDIM9qxmk1k2cz4bSacVpNOK0m8mzmkddWM06bCatJPyOCrFKKHe0DqRMrNh3sI5KsLVdZYOfjK2q5cFYZqxtKKcqzZrm1QhxfRsNeV1cXmzZt4k9/+hMAV155JZ///OfZu3cvDQ0NqfXWrVvHZZddRmVlJQC33nord999dyrsTVZ9nXImrhidSPMAhj9K3jmVaXvO/d1+fruljed3dBKIxEY8drzDtdVRC9Vx1jp6nSNMupa4aFrqtlnX0JPXqcd1HZNG4loHs66n1tG15Lqmkc+jATFDETdU8togZiiM1P2jrw1icYWhhi2PH3ncOGZ9fyhGMHrq4Oa0mhLHOeVZmFGalzrmqSjPmlyevJ+8Xey0Yrfo9A5GaOkL0toXoKU3ed0XpLU3wMa9PUTi3ce8Vr7NTM2wnsBUr2BxIhy6TjAcdyS09vgTYW14eEuEuXBqee9gBG8wesJ/07Ey69pJwqCZPKspFR7TVTbkeJQCQyX+nVOXYfcNlfy7UEN/S0PrQ9wwiCuSjxkYBsSTf1OGoWjvD+Hxh4FEeL/grBIunFXKe2aX0VDuOiMCr5j6Mhr2WlpaqKqqwmxOvIymadTX19Pc3Dwi7DU3NzNt2rTU/enTp9Pc3Jy6v2/fPpYvX47JZOKGG27g7/7u7zLZ7NPmbZcae2J0gtvTM4TbNRBi/dY21m9t493WfgBK8qyUF9hHrHf0bujo/dIx94/aYvjjSjG08zQU4ZgxYgcbS+1YjePscMfzboeYjxsw9aHlpqHHLSY9eT8RPF12C8VOC4VOK8V5Q6Gt2GlNLSt0WrBbxnayVYnLRonLxtK6wmMeMwxFtz9MS2+A1r4gLb0BWvqSt/sC/GVX13F7BoucFmqLnBTnWfEGo4kQ5z91b6OmQVHyPc2qyKck+X5L8hLLhl9K8myYTRrBSJzBSIzBcJzA8OtInED4qOvjPN7jj9AciREIx1M9X5OZriV+vBz58aEP/9sa9iOkosDG3yyv4aJZZayYXjTmvw8hsmnSH8W5fPlyWltbcbvdtLa28uEPf5jS0lI+/vGPH7Pufffdx3333Ze67/f7M9o2b2e7lF0Rp02pxBCuqdCGpcY16u0HQlH+uL2DZ7a08do+D4ZKHAR+5fJarlhWzQUzSzBP0mOElBrZMxdXinh8KBDGDIVSCotJP25voVnX0TWmbC+KrmtUJGuWrZh+7OOxuEHHQIiW3mAqBLb2DoXBpi5fsofRyvSSvKGw5rRS7EqEuCKnlRKXleI8G26HJaO9aacSiRmp8BiIxMh09jPpyV5kTUNP9SSDSdOGbh/VKz1V/5aEGIuMhr26ujra29uJxWKYzWaUUjQ3N1NfXz9ivfr6evbt25e6f/DgwdQ6BQUFqeW1tbV84hOf4NVXXz1u2Lv99tu5/fbbR6yfSX3tbVJ2RZy2aPsg8d4QrpXVp72jCcfivLS7m2e2HOb5nV1EYgYWk8baeRVcsbSGtfPKp0RPg6Ylet3Mk7+pWWE26cmTO5xcwNQ/ccdq1rGaddwyO4wQk0JGw155eTnLly/niSee4Prrr+fpp5+mtrZ2xBAuJI7lW716NXfccQcVFRU88MADXHPNNQC0t7dTUVGBruv4fD6effZZbrrppkw2+7Qkyq50Mn3Jsmw3RUwRRwopn6rkimEo3jzQyzNbDvPctvZUKYfzZhRzxbIaPrSwkkKnHAguhBDi9GR8GPfBBx/k+uuv5+6776agoIDHHnsMgJtvvpnLLruMyy67jJkzZ3LnnXeyatUqANasWcMtt9wCwNNPP83999+P2WwmFotx9dVXc8MNN2S62aeUKLsSo6hKjtcTx6cMRawrQKTFR6TVR3C7B91pxjbdfey6yTP+ntnSxvotbXQMhACYV1XAbUur+eiSaqoLs19uSAghxNSjKZXuc7Qmj9raWlpbWzPy3Ae3bubpu7/B2hs/x9IPXJqR1xBTh1KKeE+ISKuPSKufSKuP6GE/Kjp0sJLuspC/po781UOlelp6Azyz5TDPbGmjqStxjGlNoYPLl1ZzxbIaZlfkT/h7EUKIbMvk/vtMNOlP0Jis+joSZ+IWSs/eGSneHx4R7CKtflRwqOyJZjdhnVaAtdaFtTYfS20+JneiwG6PP8zvt7XzzJY23j7UByTOurysMj6RAAAgAElEQVT2/HquWFrD2dOK5OBxIYQQaSNhb4y8HYkaezJ7Ru6LD0aJHhXsDF8k9bhm0bFUu4YFOxfmEgda8mzIUDTOjk4fjXs6+VNjB682eYgZCofFxGVLqrliWTUXziqTavtCCCEyQsLeGHk72tBNZvJLy7LdFJFGRjhG9LB/RLCL94aGVtA1LFV5OOYXp3rsLOVONFMi2HX5Quxs97GjsZ2d7QPsbB9gv2cwVUPNpGtcOKuUK5bW8L75FTKHpRBCiIyTPc0Y9XW0466Qsiu5QhkK7/p9DL7ZPjTthAbmMgfO5eWpHjtrlQvNohONG+zvHmRnu5ed7zSzIxnsPP7IiOetK3awdm4586oKmF9dwIppRZS4bBP/BoUQQpyxJOyNgRGP09/ZIWVXcoQyFH1PNxF4uxNrfT6OBaWJYFfjQreb6Q9E2dY+wM7mHna+eYAd7QM0dfpHzBJgM+vMrcznvfMqUsFuTmU+BXapMyaEECK7JOyNga8nUXalUKZJm/KGBz3bnCL8H5rGpm4/O/d1snNDEzvaBmjrD43YpqLAxsqGEuZVFSSCXVUBM0rzsjpjgRBCCHEiEvbGoE/mxJ3ylFIc7gvQu66J4v0+djo0vnLgML0/OJRax6xrNJS7OH/mULCbV5Uvw7BCCCGmFAl7Y3DkTNxCORN3yuj2hXm31cvW1n7ebfWyvaWfzwR0LsXKa0S5KxZhTo2bhTVu5lcnQl1DuQubzO8lhBBiipOwNwZHauzJ7BmTU38wyrbWfra2enm31cu21v4RQ7E2Xec7dhcXAH2VDpZctZjN1QUyDCuEECInSdgbg1TZlRIpu5JtgUiMxrYBtrZ42Xa4n3db+zngGUw9rmswuyKf1bNKWVxbyOLqAqpe7yT8Tjf2ucUsvHYemlnq2wkhxGjF+/sxuY+d/lFMPhL2xiBVdsUkQ3wTIW4o/OEYvlCUHn8kGeq8vNvaz55OH8awCf9mlOZx+dJqFtW4WVJXyILqApzWxJ956mSMZNArkaAnhBBj0r9+PR3f/g71Dz+EY+nSbDdHnIKEvVEyDCm7crrihmIwEsMXiuEPxfCHo/hCyfvhxDJfKIovfOTxxGOJ+9HUeoFI/LjPX+228775FSyuLWRJbSGLaty4nccvdTL8rFsJekIIMXa9Tz5J57e+jbmqCr1AevamAgl7o+TzTO2yK0YkjqZraQ060bjB5kN9vLi7m417PXT5QvhDMQZPENJORNPAZTXjspvJt5upLXLgslvIt5vJt5lx2cwUOi3MqypgUa2b8nz7aT2vMhR96/YQ2NwlQU8IIcZIKUXPgw/R/YMfYJ0+nfqfPIqlemruC880EvZG6UjZlal4Jq6KGnT95zuoiEHZzYswlzrG/FxdvhAv7+7mpd3dvNLUjS8UA6A4z8q0EicuWyKwJa4tqfuJZZZUoMu3JcKdy2Ymz2pGT/NJEhL0hBBi/JRSdH3ve/Q++hNsc+dS/8jDmEtLs90scZok7I2SyWKhftFSyuqnZ7spo+Z7pZVYVxCArge3UnbzIiwVeae1bdxQbGnx8tLuLl7c3cX2wwNAojduSW0hF88pZ82cMhbVuNMe2MZKgp4QQoyfisfpuONOvE89hWPZMuoefABTQUG2myVGQVNKqVOvNjXV1tbS2tqa7WZMCjFviM7vv42p0EbB2np6f7UH3W6i9KZFWGtcx92mdzDCK3u6eXF3Fy/v6cYbiALgdlh4z+wyLp5bxkWzyiZlkWEJekIIMX4qEqHtn/+Zgef+l7xVq6j90X+gO50Zf13Zf6eX9OydIfp/fwAVNSj86FnYZxeh2Uz0/GIn3Q+/S+kNC7FNK8AwFI1tA7yY7L3b0uLlyE+BhTUFfOr8aayZU8bSuqJJXZNuRNCbV0zJJyXoCSEmF2UYRNvaMBUWYXKd3gjLRDOCQVr/4R8YfOVV8t//fqq/9110qzXbzRJjIGHvDBDa20dwmwf7ghLss4sAcMwrofT6BXge30HHw+/y6xkOftHeh8cfBiDfZuZDCytZM6ecNbPLKC84vZMhsk2CnhBisjHCYcJNewnv2klo5y5Cu3YR3rULY3AQk9tNxVf/lYKPfhRNmzw/ouM+Hy2f+xzBTW/j/pu/oepbd6KZJTJMVfIvl+NU3MC7fj+YdQovnYlSil0dPl7c3cVLu7oJx/zcqxxc1jRIa5GVsvfUcvGcMpZPK8JimlohSRmKvqf2EHhHgp4QIjtifX2Ed+1KhrqdhHfuIrx/P8SHqhPo+fnY58/H2nAWvj/9mbYvf4X+3/+eqjvvxFJZmcXWJ8R6e2m5+TOEduyg+LpPU/6Vr6Dp8l06lckxezmu96UWAn84yO6zXPzSGmdLSx8efwQAp9XEqoZSPlruZvkb3RA1KPnEXBwLp94ZVhL0hBATSRkG0dbWEaEutGsXsY6OEetZqquxzZuHfe5c7PPmYps7D0tNdaoXL9bXR9c999D/zHp0l4vyL32Jwo9fnbVevmhHB8033kRk/35Kv/B5Sv/u77LSFtl/p5eEvRyilOJgT4DNh/p4p6WPpv1e7uxSDKC4Fj+GrjG/uoBzphdzydxyVkwvwmZOzAIS7Rik+5FtGIEoxVfPwbmsPMvv5vRJ0BNCZNLJhmFTLBZsDQ1DoW7OXOxz55z2dGL+l1+m/Zt3EOvowHn++VR9+1tY6+oy9I6OL3LwIM033kS0rY2Kf/1Xij/9qQl9/eHOtP13pknYm8J8oSjvtvYnw52Xd5r76EueMQtwpzmPtTETm5YVUX1uYgoxh/XEU7xFuwN4HtlGfCBC4ccacJ07+WsJStATQqRb3Odj8I03GNy4keDbm48dhi0oGNFTZ583F9vMmWjjPHkh7vPR9d3v4f3Vr9AcDsr/6R8p+uQn0SZgas7Q7t0033Qz8d5equ66i8KPXZHx1zyZXN9/TzQJe1OEYSj2e/xsPuTlnZY+3mn2srvTlzpb1mLSWFDtZll9Icvri1hmMsMTu7HNKqT0xoWn3Q0f6w3R/cg24r0h3B+ZSf7qmgy+q/GRoCeESAcVjxNqbMS/YQODGzYS3Lo1Fe7M1VXY58/Hngx19rlzMVdXZ3Roc/CNN2n/+teJtrTgWLqUqrvvwjZzZsZeL/DOO7TccisqGKT6+9+j4P3vz9hrna5c2n9PBhL2Jqn+QJQtrd5Ur92W5j4GkrNUAFS57UPBrr6IBdUF2C2JX3/KUHT91xai7YNU/ONyLOVDNZF62wfJL7FjOUkPX7w/TPcj24h1Byn4wDQKLq7P3BsdIwl6QojxiHZ0MLhxI/4NGwi89jrx/n4AdKcT5/nnk7d6Fa7Vq7HWZ+f7zwgE6P7hD+n92c/RLBZKb7uNkptuTPsZsYOvvUbLbZ8HoPY/f4Rr1aq0Pv9YTeX992QkYW+SWb+1jf94oYm9Xf7UMqtZZ1GNm2V1hSyfVsSy+kKq3Cee6sz/13a8v96L66IaCj889GtQKcUv73yTwECEuSurWPSeGtxlxy+OGfdH8DyynWjHIPkX11Hw/mmTpiyAMhR9v9pNYEu3BD0hxGkxgkECmzYxuGED/o0biezdl3hA07AvWEDeqlW4Vq/CsWTJuIdj0ynwzju0f/VrRPbvxz5/PlV334V97ty0PPfAn/9M2+1fRHM4qHvwAZzLlqXledNhKu6/JzMJe5PIr95q4Su/fpdCh4XVs8pS4W5eVX7qRIpTMQJROr6/CXSNyi+uQLcP/Qo04gaNr7ax7aVW+joCoMG0BSUsfE8N0xaUoB1VKNkIROl+rJFoiw/XqmrcH5mZ9cAnQU8IcTqUUoT3NDG4YQODGzcQ2PQ2KpKoRGAuKyNv9WryVq0ib+UFmIuLs9zakzPCYTw/vp+eRx4BTaP0s5+h5NZbx1Xg2Pub39L+1a9iKiqi/tFH0hYg02Wq7b8nOwl7k8Qv3jzEV3+znbpiB0/efD51xWObjqbvmb0Mvt5O0cdnk7e84rjrKKU4vLuPbS8d5sDWbpSCgjIHCy+qYd7KKux5ltS6RiiG56eNRA4OkHduJYVXNBwTCieKihmJgskS9IQQxxHr7WVw42sMbtzI4MaNxLq7AdCsVpwrVqQCnm32rKz/cB2LYGMj7V/9GuFdu7A2nEX1XXfhWLJk1M/T+7Of03n33Zirq5j2k59gnT49/Y0dp6m0/54KJOxNAj97/SDfeKaRaSVOfvmZ86kuPPEQ7clE2vx0/egdrPUFlN26+LS+zHy9IRpfOcyOjW0EfVHMFp3Z51awcE0tZXX5ABiROD0/30G4yYtzaRlFV89BM03MF6VSikiLj8DmLgJbu1HBmAQ9IdJERSJ4Hn6Y0I6dmEtLE5ey0tRtU2kZ5tISdPvkmkFHxePEBwYw+vsTx9699jqDGzcS2rGDI2et2WY1kLdyFXmrV+M8Z8Wkew9jpaJReh59FM9//RgVj1N83XWU/f0X0B2n3m8opfDcfz+e//gR1hkzqP/Jo1iqJmfVhamy/54qJOxl2aMbDvDtZ3cwszSPJz9zPpXusX0hKaXofvBdIocGKP/8Mqw1rlFtH48a7N3cxbaXWuk8MABA1VluFq2pZeayMnSg58ldhHb04FhQQvEn5mY0bMW8YQLvdBLY3EWsOwiAucJJ3ooKXBdUS9ATYpwiLS0cvv2LhLZtO+W6uss1FADLSjGXlh0TDk2lpZiLi0d1AoEyDAyfj3h/P3Gvd+ja23/ssv5+4v2Jx4yBgVSoO8LkdpO3amViaHbVqkkxE0Umhffupe2rXyW09V0s0+qp+va3yTv33BOur5Si65576X38cWzz51H/yCOTevh6Kuy/pxIJe1n04Mv7+H//u4uGchdP3nzeuOafDWzpove/d5N3XiVFH5s1rnZ1HRpg20utNL3VRTxm4CywMv/CahasrCL8p0MEt3Zjn1NEybXz0Czpq/9khOMEt3sIbO4kvL8fFOh5FpxLy3Aur8BSnTclh16EmGz6n/09Hd/8JsbgICWfuZnSL3wBw+8n1u0h5ukm7vEQ83iS949cuol3e1JnrR6XpmEqLh4KgqWlmEpLUNEoRn8/Ma8XwzssvA0MgGGcsr2a3Y7J7U5cCguT18nbRcU4V5yNfcGCCalHN5moeJzen/2c7h/+EBUKUfS3n6Ds9i9icuUds177N75B/9O/xnH22dQ9cD+m/Pwstfr0TPb991QjYS9L/uvFvXz3j7uZU5HPEzefR1m+bczPZYTjiZMyYgYVX1yBadgxd+MR9EfYubGd7S8fxtcbQtc1Zi4tZYEJaPJim+mm5LoF6Laxf8EqQxHe309gcyfB7R5UxACThmNeMc7lFdjnFKFNsTl6hZisjECAjrvuov/pX2MqLaX63ntGXWrDiESI9/QQ8/QQ83QT83gS4XBEMPQQ6+5GBYMjttWs1mFhrRBToRvd7cZcWIg+IswVph43ud05MwSbKZFDh2j/2tcJvPUW5uoqqu78Fq4LVwOJf6+2L30Z3x//SN6FF1L7Hz88rSHfbJvM+++pSMJeFvzw+Sb+/fk9zK3M5xc3n0eJa+xBD6D/fw/ge7mVwisacJ2f/uMvDENxaJuHbS+10rKzD4BzSm1UxwwsdfmU3bgQ3TG62k/R7kDiOLzNXcT7wwBY6/Jxnl2Oc3EZujM9gVUIkRDatYvDt3+RyP795K1eTfU9/w9zaWbnwTYGB4n19CRCntuNZrdL73yGKMPA+6tf0fVv38UIBHB/7GOU/cPf0/7VrzG4cSP5H/ogNffeO6nKypzMZN1/T1US9iaQUop///Me/uMve1lQXcATN51HUd74/sOLdgfo/MFmLBVOyj+/LONnyvZ1DLL95cPser2dBhSz7CbCDjPF1y2gaHrBSbc1AlEC7yaGaSPNPgBMbhvO5eU4l5WPKP4shEgPpRR9v3iSrn/7N5RhUP5P/0TxDdej6dJjnouibW20f/MOBl99FXQdDIPCq6+i8o47ptQw92Tbf091EvYmiFKKf/vjbu5/aR+La938/MbzcI+z90opheexRsJ7+ii7dTG26ac34XY6REIx9rzZgff5ZqZH4wzEFYeqXcxbW0/9whL0ZOhUcYPQnj4Cm7sI7uiBuEKz6DgWluI8uxzbzMKslXIRItfFvV7avvo1/C+8gKWujprvfw/H4sXZbpbIMKUU/b99hu4f/AD3ZZdRdvs/Tbke1cm0/84FEvYmgFKKu5/bycOvHmBpXSGP33gubsf4hymDO3ro+dkOnMvKKf4/c9LQ0tFTSnH413vhrQ78ccVr/hjTL6hi9XvrCLzdSWBrN4Y/CoBtphvn2RU4FpaO6zg/IcSpBTZt4vD//RKxjg4KLr2UyjvvwOQa3Vn6QmTLZNl/54r0TrInjqGU4lvP7uCxjQdZMa2Ix244h3z7+IOeihp4n92PZjPh/tCMNLR0bDRNo/bKWfhrXPDbvVxSZsfc7qfrP94BwFzqwLWyGueycsxFcpC1EJmm4nE89z+A58c/RrPZqLrrLtx/87Ep17MjhEgfCXsZZBiKb65v5OdvHOLcGcU8dv055NlO/JErpRj43e8IvruN/Pe+F+d5557wC9r3Sivx3hDuD8/AVJD9A25d51ehWXT61u1BC2o4z6vEeXYF1rp82ckIMUGiHR20fenLBN56C9ucOdT8+33YZs489YZCiJwmYS9DDEPx1d9u45d/bWHlWSU8ct0KnNYTf9yh3Xvo+Na3CL79NgB9TzyBZVo9hVdehfuKy7GUl6fWjfWF8L3Ugrks0Ws2WeSdXYG9oRDdaUGzyMHfQkwk31/+Qvu//Cvx/n6KPvlJyr/8JXTb+M70F0LkBjlmLwPihuKfn36Xp95u5cJZpTz0qRU4rMc/Ri3u9+P50X/S+8QTYBjEL38vzRc1ULBhO+4XNmPuH0TpOoPnzqX/A+cSOHs2M14soOiAlb0fHmSwNg6AhoamaWgke9G05LLk8tQ6iQdS6x293Yh1T7adpg0tP9l2w9t0gu1O1aZTbTe8jSPey7DnPfq9DO9tPNlncPRrj9juRG06+vETteno55ceUDEGRjhM13e/R98TT2Byu6m6+y7y167NdrOEGBc5Zi+9pGcvzeKG4ktPbeXX7xxmzZwyHrj2bOzHmWVCKcXA75+j6957iXV3Y1+8mMpvfIPv+3/D/+x+GOaAqUGxoknnkq2KJW/swPXGDgbr51K0/HY22d7h6wcehgNZeJMi404WEo9+/KTB9SQh/XjbpW6PdbvTCOnDlw/fzml28viHHk/XR3hGCO8/wOEvfpHwzp04VpxNzXe/O2nnOhVCZI+EvTSKxQ1u/9VW1m9t473zyvmvTy7HZj426IX37aPj298h8MYbmNxuKr91J4VXXYWm61zTZ+M9te9BkexwfX8iGPZ0enD84Q3Ku85DxSPM+d1T/GxuA4EPriR4wUKU1cyRTlqV/F/i/8llSqWWD+/MHX7/6Nsn3S65fPh2qTYP3+4Ezzea7Y55L0dtd6L2n+wzOOl2J2pTGrY71Xs51Wcw/P7Rn+WR5x3Vdif59x6x3Uk+g6O3O97fiUFiSixlnGA7NXI7cXJKKfp/81s6vvMdVChE6W23Ufq5W0c1L60Q4swh3wxpEo0b/ON/b+H329r5wIIKfvSJ5VjNI49bMwIBPPffT89PH4dolMKrr6bs9n/CXFSUWqehqIGGooZjX6AOfMHz6f/9fuyzQAstgxdfxL55F6aiItyXX07h1VdhO+usTL9VIUQWxf1+Ou64k4Fnn8VcUUH1A/eTd+652W6WEGISk2P20iASM/jCLzfzx8ZOPryokh9eswzLsPlclVL4/vRnOu+5h1h7O7b586j65jdxLFly2q8R90Xo+N4mdKeZytvPRrOYiHk89D/zDN6n1hE5eBAAx/LlFF51FQUf/AC6U2akECKXBLdt5/AXv0i0uRnXxRdTdfddI34sCpEr5Ji99JKwN07hWJzbfrGZ53d28dEl1fz7x5dgHhb0IgcP0vGduxjcsAE9P5+yf/wHiq65ZtTT1vT+ajeBzV2UfGoejgUj57NUShHctAnvunUM/OGPqHAY3eWi4COXUnjV1dgXzJeD/4WYwpRh0PvTx+n6939HA8q//GWKrv2k/HctcpaEvfSSsDcOoWiczz3xNi/u7uZjy2r47lWLU0HPCIXoeegheh5+BBWN4r7iCsr/7xfHNPF4+NAA3fdvxTarkNIbF570Cz4+MED/736H96l1hHftAsA2bx6FV12J+6MfxVRw8vlrhRCTh1KKyP79dN57L4OvvIp1xgxq7vs+9nnzxvqEYMTAiCevY6CModtHlp/uMiMOKj6GZUeec1g7DOM0lg3bNrU8Dmv+BWZcmN4PX2SVhL30krA3RqFonM/8bBOvNnm46uxa7r1yMabkHK++v7xI5113ET18GNvs2VR+4+s4V6wY0+soQ9H1X1uItg9S8Y/LsZSf3tCsUopQ4w68Tz3FwLPPYgwOotlsFHzwAzgvuADd7kB32NHsDnS7beS1w45mt6NZLNJzIKYepY4fCI67LHZU8BjjshFh6HRDz7DwMmyZisUItvjw7fLi3z1ApDcCgHthHpWXFKCbj7zOsPB10vc4bBlT/Ote00E3g2ZKXOumxOWyH8HcS7PdOpFGEvbSS8LeGAQiMW5+fBOv7evhE+fWcdcVi9B1jUhrK5133Y3/xRfR8/Io/cLnKf7kJ9EsY58ezf9mO97f7MV1US2FHx7btGhGIMDAH/6Id906gps3n/6Guo5uTwQ/3W5HczjQbbaR13Y7mt02FB5t9pHXR4fI5PIRz2u3j3pY+4w3PNCctMfkeMuODgmj2fY0AsUxoedUy44feo6/7HhtP2qZMrL9rzNqRhwGO234W+34DtuJhxP/PZgdBq76OAUzFXk1pmEhJxl6UsHnyOXoZeaha23YNqNaNvx1zSNf6+jQlell8uPzjCFhL73kbNxRGgzHuOGnb/HXA71ce34937psIcSieB59FM8DD6LCYQouvZTyL38ZS0X5qZ/wJIxAlIE/HkTPt1Kwtm7Mz6M7nRT+zcco/JuPEd5/gMjBAxjBICoUxggFUaEQRiiUvA6jQkGMYAgVDmEEQ8l1EuvGBwYwurpQwSBGKATx+Lje4xGaxXKCEGlHs1nRbbZEaLRa0e3WxDKrBc1mSVxbLeg2C5rFhG41J5ZZdHSbOXFtMaFZTGgm0DCODQ/pGk467SB0suGx0wg9UzDQjKDpp9ixDws0mglMNrA4TyMUHB1wjrNsRKAZHiSOXu9Uy04Rhk6xLO4fxL/xDXwvvop/4+uoYBAAa8NZFK59L/lrL8G+cCGaLrPRCCHGR8LeKK3f2sZfD/Ry/crpfPOj8xncsJGO73yb6KFmrDNnUvmNr5N3/vmn/4RKHdvrkQwJ/c+1YQRiFH2kBH2wGXxjOTZm5DKbEceWH4O8EwUKwLCAoYFhBSPvpD0rKhrFCMdQkQhGOIoRiaGiMYxwPHEdiaGicYxoHBUxEtdRAyNmYERV8rZCxWIYsSAqBoYPVB/EY1ryo9BQ8XTt8BSaSaGbk9cmhWZWyX2xQjMbyX3y8MdOtM0p1tG10QUAkw0sRw9TjaYX53SXpaO35wTLTic0aabEezoDRdva8L3wAr4XXiDw1luJH0uahmPZMvLXriV/7SVYp0/PdjOFEDlGhnFHSR16jb7nvo0rquh6vhvfngCaWaP0fCclS21o2qmGzI7T+3McEWMGXZEfYNV2UWb9yuQevdAto+whGf0wjtJMKENLfmQaRpREMIyDiqlEcExdGxjRxEVF4xiR+LDAGU8E0Ehs2HUUIxxFRaIY4QjE0tNbicWS7Km0ow8f1h7Re3nUkHZyXc1hTwyNH2cIfGiIPLmNzSa9P5OUUorwnj34nn8e3wsvEN6xEwDNaiVv5Ury37sW15o1YzpxS4hcJsO46SU9e6Pl60Vt2MP+rRZUTCN/hqLiAh1LQSQR7HQTmCyg28c43KOjNDPe7WsgolO4vA/N/YWjemcy3dszmmUTEzK05GUiXk3FYkPD2acY1k5dB0MY4VDiOjUkHko9x5F14z2e5LrhxLBdmn5raXb7CY6nPH6ITB1nebwweXQAtdvRk8+LnLRzSioWI7B5M/4XXsD3wl+IJndYutuN+/LLcK1di2vVKvS8vCy3VAhxpsh42GtqauK6667D4/Hgdrv56U9/yoIFC45Z79FHH+Wee+7BMAwuueQSfvzjH2NJnthwsscmmne7n663rVim1VP5ta/hujD9p/sHt3QReW03eedXYb3iX9L+/OLkNLMZk8sMrszujJVSqEjkqGMmQ8njIY9zPOWwwHncMDk8cAYDxHt7U9uqSCQ9jTaZThwijxcmT9YjmQyRms2WCJNHn7QzhXorjWCQwY0b8T3/Av6XXiLu9QJgrq6i6FOfIn/tWpxnLx/XyVpCCDFWGR/GveSSS/j0pz/N9ddfz7p167j33nt56623Rqxz4MABVq1axebNm6moqODyyy/nAx/4ALfddttJHzuVjAzjRiL0/c+vKPz41YmejjQzwjE6vv82xAwqvrgCU57sHMT4qXg8ERCTPYrG0QHzmDAZHNG7edLey2BwxPNipOfkEc1qHdZTOTJEDvVMHhU8U/ePXRdlJNoZjqAiYVQ4POJ+6nY4fIL7ydvhMEZk2GPh8Ij3bZs7N3H83XvXYps7V3pChRgDGcZNr4yGva6uLhoaGujt7cVsNqOUoqqqig0bNtDQ0JBa77vf/S779u3jgQceAOC5557j7rvvZsOGDSd97FSm4h+L938P4H+5lcKPNeA6ryrbzRFiVJRSEI2ODJMnCZFHhsiHQuSRofETh8nU8nB4Qt6TZrUmjou02dCt1kSv45HbR5bb7TjPWYHrkrVYa2smpF1C5LKpuP+ezDI6jNvS0kJVVRVmc+JlNE2jvr6e5ubmEWGvubmZadOmpe5Pnz6d5ubmUz6Wi6y1Luzzisk7pzLbTRFi1Kc1u2YAAAoFSURBVDRNA6sVk9Wa8dlalGGketVSoXJEmBxaboSCaCYTmtWWKuWj2WxoVhu6bSi0HXPfYplSw8lCCHE8OXWCxn333cd9992Xuu/3+7PYmrFxLirDuags280QYtLTdD0xzOtwZLspQggxqWX0J2tdXR3t7e3EYjEgMcTT3NxMfX39iPXq6+s5dOhQ6v7BgwdT65zssaPdfvvttLa2pi4ulyvdb0kIIYQQYkrJaNgrLy9n+fLlPPHEEwA8/fTT1NbWjhjCBbjyyitZv349HR0dKKV44IEHuOaaa075mBBCCCGEOLmMH4zy4IMP8uCDDzJ79mzuueceHnvsMQBuvvlm1q9fD8DMmTO58847WbVqFQ0NDZSVlXHLLbec8jEhhBBCCHFyMoOGEEIIISYV2X+nl5xmJoQQQgiRwyTsCSGEEELkMAl7QgghhBA5TMKeEEIIIUQOk7AnhBBCCJHDJOwJIYQQQuQwCXtCCCGEEDlMwp4QQgghRA6TsCeEEEIIkcMk7AkhhBBC5DAJe0IIIYQQOUzCnhBCCCFEDpOwJ4QQQgiRwyTsCSGEEELkMAl7QgghhBA5TMKeEEIIIUQO05RSKtuNyBSbzUZZWVlGntvv9+NyuTLy3OLE5HPPHvnss0c+++yRzz47uru7CYfD2W5GzsjpsJdJtbW1tLa2ZrsZZxz53LNHPvvskc8+e+SzF7lAhnGFEEIIIXKYhD0hhBBCiBxmuuOOO+7IdiOmqgsuuCDbTTgjyeeePfLZZ4989tkjn72Y6uSYPSGEEEKIHCbDuEIIIYQQOUzCnhBCCCFEDpOwN0pNTU2sXLmS2bNnc84559DY2JjtJuWkUCjEFVdcwezZs1myZAnve9/72Lt3LwBdXV188IMfZNasWSxcuJBXXnkly63NXY899hiapvHb3/4WkM9+IoTDYT7/+c8za9YsFi1axLXXXgvId89EeO6551i+fDlLly5l4cKFPP7444D83YscoMSoXHzxxeqxxx5TSin11FNPqRUrVmS3QTkqGAyq3//+98owDKWUUj/60Y/Ue97zHqWUUjfccIP65je/qZRS6q9//auqqalRkf/f3v2GNNX2cQD/bs2MbFIWmqVzZRHURktdKeZU1ApfmCRlNAopkBQhIyF8kRTF6EVZkS8kiP4nBtreVMi0CF+YTHKKVCPX/vRnGVmYRnk7dz0v4jn3LXTzPN5Pc3me7+fVzrl2zvU71y6u/c51tnP++CNMkcqX2+0WGRkZIj09Xdy9e1cIwbafCdXV1aKqqkrq+36/XwjBsSfUgsGgWLRokejr6xNC/Oj/kZGR4suXL+z3NOsx2ZuGoaEhoVarxcTEhBDix+AQFxcnXr58GebI5M9ut4ukpCQhhBBRUVHSF6AQQhiNRmGz2cIUmTxNTk6KvLw80dPTI7Kzs6Vkj20fWmNjY0KtVouRkZEp6zn2hF4wGBQxMTHi8ePHQggh+vr6xLJly8T4+Dj7Pc16vIw7Da9fv0Z8fDxUKhUAQKFQQKPRwOfzhTky+btw4QK2b9+O4eFhTExMYOnSpVKZVqvlZ/CL1dfXIzMzE6mpqdI6tn3ouVwuxMTEwGKxIC0tDVlZWejo6ODYMwMUCgWam5uxY8cOJCUlYfPmzbh27RpGR0fZ72nWU4U7AKL/xGKxYHBwEB0dHfj27Vu4w5G9gYEBtLS08HdJYRAIBOD1erF27VqcPn0avb29KCgowL1798IdmuwFAgGcOnUKra2tMJlMsNvtKCoqgsPhCHdoRP8zzuxNQ2JiIvx+PwKBAABACAGfzweNRhPmyOTrzJkzaG1txYMHDzB//nwsXrwYKpUK79+/l97j8Xj4GfxCnZ2d8Hg8WL16NbRaLZ48eYLy8nLcuXOHbR9iGo0GSqUSZrMZALBhwwasWLECXq+XY0+IORwOvHv3DiaTCQBgNBqRkJCA/v5+9nua9ZjsTUNsbCxSUlJw8+ZNAEBLSwsSEhKwatWqMEcmT/X19WhqaoLNZsPChQul9Tt37kRjYyMAwG634+3bt8jOzg5XmLJTUVEBv98Pj8cDj8eD9PR0XLp0CRUVFWz7EFuyZAny8vLQ1tYGAHC73XC73cjMzOTYE2L/Ppl//vw5AGBwcBAulwtr1qxhv6dZj0/QmCan04mysjIMDw8jOjoaV65cgV6vD3dYsvPmzRskJiZi5cqVUKvVAIDIyEh0d3djaGgIe/fuhdvtxty5c9HQ0IDc3NwwRyxfOTk5qK6uRnFxMdt+Brx69QoHDhzAx48foVQqUVdXh5KSEo49M6CpqQkWiwVKpRLBYBC1tbXYs2cP+z3Nekz2iIiIiGSMl3GJiIiIZIzJHhEREZGMMdkjIiIikjEme0REREQyxmSPiIiISMaY7BFRyGi1WjgcDly9ehUvXrwISR3Hjx/H9+/fpeW6ujrcunUrJHUREc1GvPUKEYWMVquF1WpFdXW1dK++6QgGgwAApfLvz0sVCgU+f/485cbbRET0J87sEVFItbe3o6enB4cPH4bBYMD9+/cB/HgU3saNG5GSkoJt27bB6/UC+DFTV1JSgq1bt0Kn08Hv96OmpgZGoxEGgwEmkwlOpxMAcPDgQQBAVlYWDAYDPnz4gLKyMpw/fx4AMDY2hv3790On00Gn0+HEiRNSXDk5OaipqUFWVhaSk5OlfRERyQ2TPSIKqfz8fKSlpeHcuXNwOBwoLCzE7du34XQ60dXVhadPn8JsNqOyslLapqurC9evX8ezZ8+wfPlyHD16FHa7HQ6HA5WVlTh06BAASI+w6uzshMPhQGxs7JS6T548ifHxcfT396O7uxtWqxXNzc1SucvlwqNHjzAwMIC2tjZ0dXXNQIsQEc0sVbgDIKL/P1arFXa7HampqQCAycnJKeWFhYWIi4uTlm02Gy5evIjR0VEEg0F8+vTpv6qnvb0dZ8+ehVKpRFRUFPbt2webzYbS0lIAQGlpKVQqFVQqFQwGA1wuFzIyMn7RURIR/R6Y7BHRjBNCoLa2FuXl5T8tX7BggfTa5/OhqqoKdrsdycnJ6O/vh8lk+kf1KhSKKcvz5s2TXs+ZMweBQOAf7ZeI6HfGy7hEFHLR0dEYGRmRlouLi9HY2CjN0E1MTKC3t/en246MjCAiIgLx8fEQQqChoWFKuVqtnrLvv8rPz8fly5chhMDXr19x48YNbNmy5RcdFRHR7MBkj4hCrry8HBaLRfqDhtlsRllZGXJzc7F+/XoYDAY8fPjwp9vq9Xrs3r0b69atg9FohEajmVJ+5MgRFBQUSH/Q+Ktjx44hIiICer0emzZtQlFREXbt2hWy4yQi+h3x1itEREREMsaZPSIiIiIZY7JHREREJGNM9oiIiIhkjMkeERERkYwx2SMiIiKSMSZ7RERERDLGZI+IiIhIxpjsEREREckYkz0iIiIiGfsXTWOzlEamVw0AAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "plot_collections(trial, \"/average_shap\", \"SHAP values\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Data Analysis - Automatic\n",
- "So far we have conducted a human analysis, but the real power of Tornasole comes from having automatic monitoring of training runs. To do so we will build a SageMaker-based system that monitors existing runs in real time. Data traces deposited in S3 are the exchange mechanism: \n",
- "- the training system deposits data into s3://mybucket/myrun/\n",
- "- the monitoring system watches and reads data from s3://mybucket/myrun/\n",
- "\n",
- "In this example we will simulate reading from that."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "metadata": {},
- "outputs": [],
- "source": [
- "from smdebug.rules.generic import LossNotDecreasing\n",
- "from smdebug.rules.rule_invoker import invoke_rule"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[2019-09-04 20:29:13.371 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:65] LossNotDecreasing rule created with num_steps: 5, diff_percent: 0.0, mode: GLOBAL, tensor_regex: , collection_names: metric\n",
- "[2019-09-04 20:29:13.372 38f9d36a2c42.ant.amazon.com:42958 INFO rule_invoker.py:76] Started execution of rule LossNotDecreasing at step 0\n",
- "[2019-09-04 20:29:13.376 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 1 loss is not decreasing over the last 5 steps at step 20\n",
- "[2019-09-04 20:29:13.377 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 2 losses are not decreasing over the last 5 steps at step 25\n",
- "[2019-09-04 20:29:13.380 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 2 losses are not decreasing over the last 5 steps at step 30\n",
- "[2019-09-04 20:29:13.382 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 1 loss is not decreasing over the last 5 steps at step 35\n",
- "[2019-09-04 20:29:13.384 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 1 loss is not decreasing over the last 5 steps at step 40\n",
- "[2019-09-04 20:29:13.385 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 2 losses are not decreasing over the last 5 steps at step 50\n",
- "[2019-09-04 20:29:13.388 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 2 losses are not decreasing over the last 5 steps at step 65\n",
- "[2019-09-04 20:29:13.389 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 2 losses are not decreasing over the last 5 steps at step 70\n",
- "[2019-09-04 20:29:13.391 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 2 losses are not decreasing over the last 5 steps at step 75\n",
- "[2019-09-04 20:29:13.392 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 2 losses are not decreasing over the last 5 steps at step 80\n",
- "[2019-09-04 20:29:13.393 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 1 loss is not decreasing over the last 5 steps at step 85\n",
- "[2019-09-04 20:29:13.395 38f9d36a2c42.ant.amazon.com:42958 INFO loss_decrease.py:180] 1 loss is not decreasing over the last 5 steps at step 90\n",
- "[2019-09-04 20:29:13.396 38f9d36a2c42.ant.amazon.com:42958 INFO rule_invoker.py:90] Ended execution of rule LossNotDecreasing at end_step 94\n"
- ]
- }
- ],
- "source": [
- "loss_not_decreasing = LossNotDecreasing(\n",
- " trial,\n",
- " use_losses_collection=False,\n",
- " collection_names=\"metrics\",\n",
- " num_steps=5)\n",
- "invoke_rule(loss_not_decreasing, end_step=95)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This concludes this notebook. For more information see the documentation at \n",
- "- https://github.com/awslabs/tornasole_core\n"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.7.4"
- },
- "pycharm": {
- "stem_cell": {
- "cell_type": "raw",
- "source": [],
- "metadata": {
- "collapsed": false
- }
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/examples/xgboost/sagemaker-notebooks/xgboost_classification.ipynb b/examples/xgboost/sagemaker-notebooks/xgboost_classification.ipynb
deleted file mode 100644
index c7d7cfdb4..000000000
--- a/examples/xgboost/sagemaker-notebooks/xgboost_classification.ipynb
+++ /dev/null
@@ -1,726 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Debugging SageMaker XGBoost Training Jobs with Tornasole\n",
- "\n",
- "This notebook uses the MNIST dataset to demonstrate a classification task using Tornasole with XGBoost.\n",
- "For a regression problem, see [xgboost_regression.ipynb](xgboost_regression.ipynb).\n",
- "\n",
- "## Overview\n",
- "\n",
- "Tornasole is a new capability of Amazon SageMaker that allows debugging machine learning training. \n",
- "Tornasole helps you to monitor your training in near real time using rules and would provide you\n",
- "alerts, once it has detected inconsistency in training. \n",
- "\n",
- "Using Tornasole is a two step process: Saving tensors and Analysis.\n",
- "Let's look at each one of them closely.\n",
- "\n",
- "### Saving tensors (and scalars)\n",
- "\n",
- "In deep learning algorithms, tensors define the state of the training job\n",
- "at any particular instant in its lifecycle.\n",
- "Tornasole exposes a library which allows you to capture these tensors and\n",
- "save them for analysis.\n",
- "Although XGBoost is not a deep learning algorithm, Tornasole is highly customizable\n",
- "and can help provide interpretability by saving insightful metrics, such as\n",
- "performance metrics or feature importances, at different frequencies.\n",
- "Refer to [DeveloperGuide_XGBoost](../DeveloperGuide_XG.md) for details on how to\n",
- "save the metrics you want.\n",
- "\n",
- "### Analysis\n",
- "\n",
- "Analysis of the tensors emitted is captured by the Tornasole concept called ***Rules***.\n",
- "On a very broad level, a rule is a python code used to detect certain conditions during training.\n",
- "Some of the conditions that a data scientist training an algorithm may care about are\n",
- "monitoring for gradients getting too large or too small, detecting overfitting, and so on.\n",
- "Tornasole will come pre-packaged with certain rules.\n",
- "Users can write their own rules using the Tornasole APIs.\n",
- "You can also analyze raw tensor data outside of the Rules construct in say, a Sagemaker notebook,\n",
- "using Tornasole's full set of APIs. \n",
- "Please refer to [DeveloperGuide_Rules](../../../rules/DeveloperGuide_Rules.md) for more details about analysis.\n",
- "\n",
- "This example guides you through installation of the required components for emitting tensors in a \n",
- "SageMaker training job and applying a rule over the tensors to monitor the live status of the job. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup\n",
- "\n",
- "We will also install the required tools which will allow emission of tensors (saving tensors) and application of rules to analyze them. This is only for the purposes of this private beta. Once we do this, we will be ready to use smdebug.\n",
- "\n",
- "You'll probably have to restart this notebook after running the following code cell."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "! aws s3 sync s3://tornasole-external-preview-use1/sdk/ ~/SageMaker/tornasole-preview-sdk/\n",
- "! pip3 -q install ~/SageMaker/tornasole-preview-sdk/ts-binaries/tornasole_xgboost/py3/latest/tornasole-* --user\n",
- "! chmod +x ~/SageMaker/tornasole-preview-sdk/installer.sh && ~/SageMaker/tornasole-preview-sdk/installer.sh"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### If you running this notebook for the first time, please wait for the above setup to complete and restart the notebook by selecting *Kernel -> Restart Kernel* before proceeding."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We have built SageMaker XGBoost containers with smdebug. You can use them from ECR from SageMaker. Here are the links to the images. Please use the image from the appropriate region in which you want your jobs to run."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import boto3\n",
- "from sagemaker import get_execution_role\n",
- "\n",
- "# Below changes the region to be one where this notebook is running\n",
- "REGION = boto3.Session().region_name\n",
- "ROLE = get_execution_role()\n",
- "os.environ[\"AWS_REGION\"] = REGION\n",
- "\n",
- "TAG = \"latest\"\n",
- "docker_image_name = \"072677473360.dkr.ecr.{}.amazonaws.com/tornasole-preprod-xgboost-0.90-cpu:{}\".format(REGION, TAG)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Training XGBoost models in SageMaker with Tornasole\n",
- "\n",
- "### SageMaker XGBoost as a framwork\n",
- "\n",
- "We'll train a few XGBoost models in this notebook with Tornasole enabled and monitor the training jobs with Tornasole Rules. This will be done using SageMaker XGBoost 0.90 Container as a framework. The [XGBoost algorithm](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html) can be used as a built-in algorithm or as a framework such TensorFlow. Using XGBoost as a framework provides more flexibility than using it as a built-in algorithm as it enables more advanced scenarios that allow pre-processing and post-processing scripts to be incorporated into your training script.\n",
- "\n",
- "Let us first train a simple example training script [xgboost_mnist_basic_hook_demo.py](../scripts/xgboost_abalone_basic_hook_demo.py) with XGBoost enabled in SageMaker using the SageMaker Estimator API, along with a LossNotDecreasing Rule to monitor the training job in realtime. A Tornasole Rule is essentially python code which analyzes tensors saved by tornasole and validates some condition. LossNotDecreasing rule is a first party (1P) rule provided by smdebug. For other 1P rules that can be used in XGBoost, refer to [FirstPartyRules.md](../../../rules/FirstPartyRules.md)\n",
- "\n",
- "During training, Tornasole will capture tensors as specified in its configuration and LossNotDecreasing Rule job will monitor whether you are running into a situation where loss is not going down. The rule will emit a cloudwatch event if it finds that the performance metrics are not decreasing during training.\n",
- "\n",
- "### Enabling Tornasole in the script\n",
- "\n",
- "You can see in the script that we have made a couple of simple changes to enable smdebug. We created a SessionHook which we pass as a callback function when creating a Booster. We passed a SaveConfig object telling the hook to save the evaluation metrics, feature importances, and SHAP values at regular intervals. Note that Tornasole is highly configurable, you can choose exactly what to save. The changes are described in a bit more detail below after we train this example as well as in even more detail in our [Developer Guide for XGBoost](../DeveloperGuide_XG.md). \n",
- "\n",
- "```python\n",
- "from smdebug.xgboost import SessionHook, SaveConfig\n",
- "\n",
- "save_config = SaveConfig(save_interval=frequency)\n",
- "hook = SessionHook(save_config=save_config)\n",
- "\n",
- "bst = xgboost.train(\n",
- " ...\n",
- " callbacks=[hook]\n",
- ")\n",
- "```\n",
- "\n",
- "### XGBoost for Classification\n",
- "\n",
- "We use the [MNIST data](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass.html) stored in [LIBSVM](https://www.csie.ntu.edu.tw/~cjlin/libsvm/) format.\n",
- "\n",
- "Refer to [XGBoost for Classification](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/introduction_to_amazon_algorithms/xgboost_mnist)\n",
- "for an example of using classification from Amazon SageMaker's implementation of\n",
- "[XGBoost](https://github.com/dmlc/xgboost)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "entry_point_script = \"../scripts/xgboost_mnist_basic_hook_demo.py\"\n",
- "\n",
- "hyperparameters={\n",
- " \"max_depth\": \"5\",\n",
- " \"eta\": \"0.5\",\n",
- " \"gamma\": \"4\",\n",
- " \"min_child_weight\": \"6\",\n",
- " \"silent\": \"0\",\n",
- " \"objective\": \"multi:softmax\",\n",
- " \"num_class\": \"10\", # num_class is required for 'multi:*' objectives\n",
- " \"num_round\": \"10\",\n",
- " \"save_frequency\": \"1\"\n",
- "}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from sagemaker.xgboost import XGBoost\n",
- "\n",
- "estimator = XGBoost(\n",
- " image_name=docker_image_name,\n",
- " base_job_name=\"demo-tornasole-xgboost-classification\",\n",
- " entry_point=entry_point_script,\n",
- " hyperparameters=hyperparameters,\n",
- " train_instance_type=\"ml.m4.4xlarge\",\n",
- " train_instance_count=1,\n",
- " framework_version=\"0.90-1\",\n",
- " py_version=\"py3\",\n",
- " role=ROLE,\n",
- " \n",
- " # These are Tornasole specific parameters, \n",
- " # debug=True means rule specified in rules_specification \n",
- " # will run as rule job. \n",
- " # Below, we specify to run the first party rule LossNotDecreasing\n",
- " # on a ml.c5.4xlarge instance\n",
- " debug=True,\n",
- " rules_specification=[\n",
- " {\n",
- " \"RuleName\": \"LossNotDecreasing\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"RuntimeConfigurations\": {\n",
- " \"use_losses_collection\": \"False\",\n",
- " \"tensor_regex\": \"train-merror,validation-merror\",\n",
- " \"num_steps\" : \"10\"\n",
- " }\n",
- " }\n",
- " ]\n",
- ")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "pycharm": {
- "name": "#%% md\n"
- }
- },
- "source": [
- "*Note that Tornasole is only supported for `py_version='py3'` currently.*"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# This is a fire and forget event.\n",
- "# By setting wait=False, we just submit the job to run in the background.\n",
- "# In the background SageMaker will spin off 1 training job and 1 rule job for you.\n",
- "# Please follow this notebook to see status of the training job and the rule job.\n",
- "estimator.fit(wait=False)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Result\n",
- "As a result of the above command, SageMaker will spin off 1 training job and 1 rule job for you - the first one being the job which produces the tensors to be analyzed and the second one, which analyzes the tensors to check if `train-merror` and `validation-merror` are not decreasing at any point during training.\n",
- "\n",
- "### Describing the training job\n",
- "We can check the status of the training job by running the following command:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Below command will give the status of training job\n",
- "# Note: In the output of below command you will see DebugConfig parameter \n",
- "job_name = estimator.latest_training_job.name\n",
- "client = estimator.sagemaker_session.sagemaker_client\n",
- "description = client.describe_training_job(TrainingJobName=job_name)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# The status of the training job can be seen below\n",
- "description[\"TrainingJobStatus\"]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Once your training job is started SageMaker will spin up a rule execution job to run the LossNotDecreasing rule.\n",
- "\n",
- "### Tornasole specific parameters in the description\n",
- "**DebugConfig** parameter has details about Tornasole related configuration. The key parameters to look for below are\n",
- "\n",
- "*S3OutputPath* : This is the path where output tensors from tornasole is getting saved. \n",
- "*RuleConfig*' : This parameter tells about the rule config parameter that was passed when creating the trainning job. In this you should be able to see details of the rule that ran for training. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "description[\"DebugConfig\"]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Check the status of the Rule Execution Job\n",
- "To get the rule execution job that SageMaker started for you, run the command below and it shows you the `RuleName`, `RuleStatus`, `FailureReason` if any, and `RuleExecutionJobArn`. If the tensors meets a rule evaluation condition, the rule execution job throws a client error with `FailureReason: RuleEvaluationConditionMet`. These details are also available as part of the response `description` above under: `description['RuleMonitoringStatuses']`\n",
- "\n",
- "\n",
- "The logs of the training job are available in the Cloudwatch Logstream `/aws/sagemaker/TrainingJobs` with `RuleExecutionJobArn`. \n",
- "\n",
- "You will see that once the rule execution job starts, that it identifies the loss not decreasing situation in the training job, raises the `RuleEvaluationConditionMet` exception and ends the job. \n",
- "\n",
- "**Note that the next cell blocks until the rule execution job ends. You can stop it at any point to proceed to the rest of the notebook. Once it says RuleStatus is Started, and shows the `RuleExecutionJobArn`, you can look at the status of the rule being monitored. At that point, we can also look at the logs as shown in the next cell**"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator.describe_rule_execution_jobs()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Check logs of the rule execution jobs\n",
- "\n",
- "If you want to access the logs of a particular rule job name, you can do the following. First, you need to get the rule job name (`RuleExecutionJobArn` field from the training job description). Note that this is only available after the rule job reaches Started stage. Hence the next cell waits till the job name is available."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import time\n",
- "\n",
- "rule_descr = client.describe_training_job(TrainingJobName=job_name)[\"RuleMonitoringStatuses\"]\n",
- "print(\"Waiting for rule execution job to start\")\n",
- "while \"RuleExecutionJobArn\" not in rule_descr[0]:\n",
- " time.sleep(5)\n",
- " rule_descr = client.describe_training_job(TrainingJobName=job_name)[\"RuleMonitoringStatuses\"]\n",
- "\n",
- "rule_job_arn = rule_descr[0][\"RuleExecutionJobArn\"]\n",
- "print(\"Rule execution job has started. The job ARN is {}\".format(rule_job_arn))\n",
- "rule_job_name = rule_job_arn.split('/')[1]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we can attach to this job to see its logs"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from sagemaker.estimator import Estimator\n",
- "loss_not_decreasing = Estimator.attach(rule_job_name)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In the above example, the `LossNotDecreasing` rule was completed without producing an alert because both `train-merror` and `validation-merror` decreased steadily throught the training run. To see an example of the rule when performance metrics stop decreasing during training, see [xgboost_regression.ipynb](xgboost_regression.ipynb)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Data Analysis - Manual\n",
- "\n",
- "Now that we have trained the system we can analyze the data. Here we focus on after-the-fact analysis.\n",
- "\n",
- "We import a basic analysis library, which defines a concept of `Trial` that represents a single training run."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "from urllib.parse import urlparse\n",
- "from smdebug.trials import create_trial\n",
- "\n",
- "s3_output_path = description[\"DebugConfig\"][\"DebugHookConfig\"][\"S3OutputPath\"]\n",
- "trial = create_trial(s3_output_path)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can list all the tensors we know something about. Each one of these names is the name of a tensor - the name is a combination of the feature name (which, in these cases, is auto-assigned by XGBoost) and whether it's an evaluation metric, feature importance, or SHAP value. We also have `y/validation` for true labels from the validation set and `y_hat/validation` for predicted labels on the same validation set."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "trial.tensors()[:10]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For each tensor we can ask for which steps we have data - in this case, every 2 steps"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "print(list(trial.tensor(\"validation-merror\").steps()))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can obtain each tensor at each step as a `numpy` array"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "type(trial.tensor(\"train-merror\").value(5))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Performance metrics\n",
- "\n",
- "We can also create a simple function that visualizes the training and validation errors\n",
- "as the training progresses.\n",
- "We expect each training errors to get smaller over time, as the system converges to a good solution.\n",
- "Now, remember that this is an interactive analysis - we are showing these tensors to give an idea of the data. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import matplotlib.pyplot as plt\n",
- "import seaborn as sns\n",
- "\n",
- "# Define a function that, for the given tensor name, walks through all \n",
- "# the iterations for which we have data and fetches the value.\n",
- "# Returns the set of steps and the values\n",
- "def get_data(trial, tname):\n",
- " tensor = trial.tensor(tname)\n",
- " steps = tensor.steps()\n",
- " vals = [tensor.value(s) for s in steps]\n",
- " return steps, vals"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "metrics_to_plot = [\"train-merror\", \"validation-merror\"]\n",
- "for metric in metrics_to_plot:\n",
- " steps, data = get_data(trial, metric)\n",
- " plt.plot(steps, data, label=metric)\n",
- "plt.xlabel('Iteration')\n",
- "plt.ylabel('Classification error')\n",
- "plt.legend()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Feature importances\n",
- "\n",
- "We can also visualize the feature importances as determined by\n",
- "[xgboost.get_fscore()](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.Booster.get_fscore).\n",
- "Note that feature importances with zero values are not included here\n",
- "(which means that those features were not used in any split condisitons)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def plot_collections(trial, collection_name, ylabel=''):\n",
- " \n",
- " plt.figure(\n",
- " num=1, figsize=(8, 8), dpi=80,\n",
- " facecolor='w', edgecolor='k')\n",
- "\n",
- " features = trial.collection(collection_name).tensor_names\n",
- "\n",
- " # to avoid cluttering, we will plot only one out of 20 features\n",
- " for feature in list(features)[::20]:\n",
- " steps, data = get_data(trial, feature)\n",
- " label = feature.replace('/' + collection_name, '')\n",
- " plt.plot(steps, data, label=label)\n",
- "\n",
- " plt.legend(bbox_to_anchor=(1.04,1), loc='upper left')\n",
- " plt.xlabel('Iteration')\n",
- " plt.ylabel(ylabel)\n",
- " plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "plot_collections(trial, \"feature_importance\", \"Feature importance\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### SHAP\n",
- "\n",
- "[SHAP](https://github.com/slundberg/shap) (SHapley Additive exPlanations) is\n",
- "another approach to explain the output of machine learning models.\n",
- "SHAP values represent a feature's contribution to a change in the model output."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "plot_collections(trial, \"average_shap\", \"SHAP values\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Confusion matrix"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import numpy as np\n",
- "from sklearn.metrics import confusion_matrix\n",
- "from IPython.display import display, clear_output\n",
- "\n",
- "fig, ax = plt.subplots()\n",
- "\n",
- "for step in range(0, 9):\n",
- " cm = confusion_matrix(\n",
- " trial.tensor('labels').value(step),\n",
- " trial.tensor('predictions').value(step)\n",
- " )\n",
- " normalized_cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
- " sns.heatmap(normalized_cm, cmap=\"bone\", ax=ax, cbar=False, annot=cm, fmt='')\n",
- " print(f\"iteartion: {step}\")\n",
- " display(fig)\n",
- " plt.pause(1)\n",
- " ax.clear()\n",
- " clear_output(wait=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 1P rule: Confusion matrix\n",
- "\n",
- "As another example of using a first party (1P) rule provided by Tornasole, let us again train the example training script [xgboost_mnist_basic_hook_demo.py](../scripts/xgboost_abalone_basic_hook_demo.py) and use a 1P rule `Confusion` to monitor the training job in realtime.\n",
- "\n",
- "During training, `Confusion` Rule job will monitor whether you are running into a situation where the ratio of on-diagonal and off-diagonal values in the confusion matrix is not within a specified range. In other words, this rule evaluates the goodness of a confusion matrix for a classification problem. It creates a matrix of size `category_no` $\\times$ `category_no` and populates it with data coming from (`y`, `y_hat`) pairs. For each (`y`, `y_hat`) pairs the count in `confusion[y][y_hat]` is incremented by 1. Once the matrix is fully populated, the ratio of data on- and off-diagonal will be evaluated according to:\n",
- "\n",
- "- For elements on the diagonal:\n",
- "\n",
- "$$ \\frac{ \\text{confusion}_{ii} }{ \\sum_j \\text{confusion}_{jj} } \\geq \\text{min_diag} $$\n",
- "\n",
- "- For elements off the diagonal:\n",
- "\n",
- "$$ \\frac{ \\text{confusion}_{ji} }{ \\sum_j \\text{confusion}_{ji} } \\leq \\text{max_off_diag} $$\n",
- "\n",
- "If the condition is met, the rule will emit a cloudwatch event.\n",
- "\n",
- "Note that this rule will infer the default parameters if configurations are not specified, so you can simply use\n",
- "\n",
- "```python\n",
- "rules_specification = [\n",
- " {\n",
- " \"RuleName\": \"Confusion\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\"\n",
- " }\n",
- "]\n",
- "```\n",
- "If you want to specify the optional parameters, you can do so by using `RuntimeConfigurations`:\n",
- "\n",
- "```python\n",
- "rules_specification = [\n",
- " {\n",
- " \"RuleName\": \"Confusion\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"RuntimeConfigurations\": {\n",
- " \"category_no\": \"10\",\n",
- " \"min_diag\": \"0.8\",\n",
- " \"max_diag\": \"0.2\"\n",
- " }\n",
- " }\n",
- "]\n",
- "```\n",
- "\n",
- "For `Confusion` Rule API and other 1P rules that can be used in XGBoost, refer to [FirstPartyRules.md](../../../rules/FirstPartyRules.md)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator = XGBoost(\n",
- " image_name=docker_image_name,\n",
- " base_job_name=\"demo-tornasole-xgboost-confusion\",\n",
- " entry_point=entry_point_script,\n",
- " hyperparameters=hyperparameters,\n",
- " train_instance_type=\"ml.m4.4xlarge\",\n",
- " train_instance_count=1,\n",
- " framework_version=\"0.90-1\",\n",
- " py_version=\"py3\",\n",
- " role=ROLE,\n",
- "\n",
- " debug=True,\n",
- " rules_specification=[\n",
- " {\n",
- " \"RuleName\": \"Confusion\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\"\n",
- " }\n",
- " ]\n",
- ")\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator.fit(wait=False)\n",
- "\n",
- "job_name = estimator.latest_training_job.name\n",
- "client = estimator.sagemaker_session.sagemaker_client\n",
- "description = client.describe_training_job(TrainingJobName=job_name)\n",
- "\n",
- "description[\"TrainingJobStatus\"]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "description[\"DebugConfig\"]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator.describe_rule_execution_jobs()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This notebook showed two examples of using 1P rules provided Tornasole, but you can also write your own rules looking at these 1P rules for inspiration. Refer to [DeveloperGuide_Rules.md](../../../rules/DeveloperGuide_Rules.md) for more on the APIs you can use to write your own rules as well as descriptions for the 1P rules that we provide. [xgboost_regression.ipynb](xgboost_regression.ipynb) also demonstrates how to use a custom rule that monitors the ratio of feature importance values."
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "conda_python3",
- "language": "python",
- "name": "conda_python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.5"
- },
- "pycharm": {
- "stem_cell": {
- "cell_type": "raw",
- "metadata": {
- "collapsed": false
- },
- "source": []
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/examples/xgboost/sagemaker-notebooks/xgboost_regression.ipynb b/examples/xgboost/sagemaker-notebooks/xgboost_regression.ipynb
deleted file mode 100644
index a8c358516..000000000
--- a/examples/xgboost/sagemaker-notebooks/xgboost_regression.ipynb
+++ /dev/null
@@ -1,962 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Debugging SageMaker XGBoost Training Jobs with Tornasole\n",
- "\n",
- "\n",
- "This notebook uses the [Abalone data](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html) to demonstrate a regression task using Tornasole with XGBoost. For a classification problem, see [xgboost_classification.ipynb](xgboost_classification.ipynb).\n",
- "\n",
- "## Overview\n",
- "\n",
- "Tornasole is a new capability of Amazon SageMaker that allows debugging machine learning training. \n",
- "Tornasole helps you to monitor your training in near real time using rules and would provide you\n",
- "alerts, once it has detected inconsistency in training. \n",
- "\n",
- "Using Tornasole is a two step process: Saving tensors and Analysis.\n",
- "Let's look at each one of them closely.\n",
- "\n",
- "### Saving tensors (and scalars)\n",
- "\n",
- "In deep learning algorithms, tensors define the state of the training job\n",
- "at any particular instant in its lifecycle.\n",
- "Tornasole exposes a library which allows you to capture these tensors and\n",
- "save them for analysis.\n",
- "Although XGBoost is not a deep learning algorithm, Tornasole is highly customizable\n",
- "and can help provide interpretability by saving insightful metrics, such as\n",
- "performance metrics or feature importances, at different frequencies.\n",
- "Refer to [DeveloperGuide_XGBoost](../DeveloperGuide_XG.md) for details on how to\n",
- "save the metrics you want.\n",
- "\n",
- "### Analysis\n",
- "\n",
- "Analysis of the tensors emitted is captured by the Tornasole concept called ***Rules***.\n",
- "On a very broad level, a rule is a python code used to detect certain conditions during training.\n",
- "Some of the conditions that a data scientist training an algorithm may care about are\n",
- "monitoring for gradients getting too large or too small, detecting overfitting, and so on.\n",
- "Tornasole will come pre-packaged with certain rules.\n",
- "Users can write their own rules using the Tornasole APIs.\n",
- "You can also analyze raw tensor data outside of the Rules construct in say, a Sagemaker notebook,\n",
- "using Tornasole's full set of APIs. \n",
- "Please refer [DeveloperGuide_Rules](../../../rules/DeveloperGuide_Rules.md) for more details about analysis.\n",
- "\n",
- "This example guides you through installation of the required components for emitting tensors in a \n",
- "SageMaker training job and applying a rule over the tensors to monitor the live status of the job. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We will also install the required tools which will allow emission of tensors (saving tensors) and application of rules to analyze them. This is only for the purposes of this private beta. Once we do this, we will be ready to use smdebug.\n",
- "\n",
- "You'll probably have to restart this notebook after running the following code cell."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "! aws s3 sync s3://tornasole-external-preview-use1/sdk/ ~/SageMaker/tornasole-preview-sdk/\n",
- "! pip3 -q install ~/SageMaker/tornasole-preview-sdk/ts-binaries/tornasole_xgboost/py3/latest/tornasole-* --user\n",
- "! chmod +x ~/SageMaker/tornasole-preview-sdk/installer.sh && ~/SageMaker/tornasole-preview-sdk/installer.sh"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### If you running this notebook for the first time, please wait for the above setup to complete and restart the notebook by selecting *Kernel -> Restart Kernel* before proceeding."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We have built SageMaker XGBoost containers with smdebug. You can use them from ECR from SageMaker. Here are the links to the images. Please use the image from the appropriate region in which you want your jobs to run."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import boto3\n",
- "from sagemaker import get_execution_role\n",
- "\n",
- "# Below changes the region to be one where this notebook is running\n",
- "REGION = boto3.Session().region_name\n",
- "ROLE = get_execution_role()\n",
- "os.environ[\"AWS_REGION\"] = REGION\n",
- "\n",
- "TAG = \"latest\"\n",
- "docker_image_name = \"072677473360.dkr.ecr.{}.amazonaws.com/tornasole-preprod-xgboost-0.90-cpu:{}\".format(REGION, TAG)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Training XGBoost models in SageMaker with Tornasole\n",
- "\n",
- "### SageMaker XGBoost as a framwork\n",
- "\n",
- "We'll train a few XGBoost models in this notebook with Tornasole enabled and monitor the training jobs with Tornasole Rules. This will be done using SageMaker XGBoost 0.90 Container as a framework. The [XGBoost algorithm](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html) can be used as a built-in algorithm or as a framework such TensorFlow. Using XGBoost as a framework provides more flexible than using it as a built-in algorithm as it enables more advanced scenarios that allow pre-processing and post-processing scripts to be incorporated into your training script.\n",
- "\n",
- "Let us first train a simple example training script [xgboost_abalone_basic_hook_demo.py](../scripts/xgboost_abalone_basic_hook_demo.py) with XGBoost enabled in SageMaker using the SageMaker Estimator API, along with a LossNotDecreasing Rule to monitor the training job in realtime. A Tornasole Rule is essentially python code which analyzes tensors saved by tornasole and validates some condition. LossNotDecreasing rule is a first party (1P) rule provided by smdebug. For other 1P rules that can be used in XGBoost, refer to [FirstPartyRules.md](../../../rules/FirstPartyRules.md)\n",
- "\n",
- "During training, Tornasole will capture tensors as specified in its configuration and LossNotDecreasing Rule job will monitor whether you are running into a situation where loss is not going down. The rule will emit a cloudwatch event if it finds that the performance metrics are not decreasing during training.\n",
- "\n",
- "### Enabling Tornasole in the script\n",
- "\n",
- "You can see in the script that we have made a couple of simple changes to enable smdebug. We created a SessionHook which we pass as a callback function when creating a Booster. We passed a SaveConfig object telling the hook to save the evaluation metrics, feature importances, and SHAP values at regular intervals. Note that Tornasole is highly configurable, you can choose exactly what to save. The changes are described in a bit more detail below after we train this example as well as in even more detail in our [Developer Guide for XGBoost](../DeveloperGuide_XG.md). \n",
- "\n",
- "```python\n",
- "from smdebug.xgboost import SessionHook, SaveConfig\n",
- "\n",
- "save_config = SaveConfig(save_interval=frequency)\n",
- "hook = SessionHook(save_config=save_config)\n",
- "\n",
- "bst = xgboost.train(\n",
- " ...\n",
- " callbacks=[hook]\n",
- ")\n",
- "```\n",
- "\n",
- "### XGBoost for Regression\n",
- "\n",
- "We use the [Abalone data](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html) originally from the [UCI data repository](https://archive.ics.uci.edu/ml/datasets/abalone). More details about the original dataset can be found [here](https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.names). In the libsvm converted [version](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html), the nominal feature (Male/Female/Infant) has been converted into a real valued feature. Age of abalone is to be predicted from eight physical measurements.\n",
- "\n",
- "Refer to [XGBoost for Regression](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/introduction_to_amazon_algorithms/xgboost_abalone)\n",
- "for an example of using regression from Amazon SageMaker's implementation of\n",
- "[XGBoost](https://github.com/dmlc/xgboost).\n",
- "\n",
- "Just a quick reminder if you are not familiar with script mode in SageMaker. You can pass command line arguments taken by your training script with a hyperparameter dictionary which gets passed to the SageMaker XGBoost Estimator class. You can see this in the examples below."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "entry_point_script = \"../scripts/xgboost_abalone_basic_hook_demo.py\"\n",
- "\n",
- "hyperparameters={\n",
- " \"max_depth\": \"5\",\n",
- " \"eta\": \"0.2\",\n",
- " \"gamma\": \"4\",\n",
- " \"min_child_weight\": \"6\",\n",
- " \"subsample\": \"0.7\",\n",
- " \"silent\": \"0\",\n",
- " \"objective\": \"reg:linear\",\n",
- " \"num_round\": \"50\",\n",
- " \"save_frequency\": \"2\"\n",
- "}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from sagemaker.xgboost import XGBoost\n",
- "\n",
- "estimator = XGBoost(\n",
- " image_name=docker_image_name,\n",
- " base_job_name=\"demo-tornasole-xgboost\",\n",
- " entry_point=entry_point_script,\n",
- " hyperparameters=hyperparameters,\n",
- " train_instance_type=\"ml.m4.4xlarge\",\n",
- " train_instance_count=1,\n",
- " framework_version=\"0.90-1\",\n",
- " py_version=\"py3\",\n",
- " role=ROLE,\n",
- " \n",
- " # These are Tornasole specific parameters, \n",
- " # debug=True means rule specified in rules_specification \n",
- " # will run as rule job. \n",
- " # Below, we specify to run the first party rule LossNotDecreasing\n",
- " # on a ml.c5.4xlarge instance\n",
- " debug=True,\n",
- " rules_specification=[\n",
- " {\n",
- " \"RuleName\": \"LossNotDecreasing\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"RuntimeConfigurations\": {\n",
- " \"use_losses_collection\": \"False\",\n",
- " \"tensor_regex\": \"train-rmse,validation-rmse\",\n",
- " \"num_steps\" : \"10\"\n",
- " }\n",
- " }\n",
- " ]\n",
- ")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "pycharm": {
- "name": "#%% md\n"
- }
- },
- "source": [
- "*Note that Tornasole is only supported for `py_version='py3'` currently.*"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# This is a fire and forget event.\n",
- "# By setting wait=False, we just submit the job to run in the background.\n",
- "# In the background SageMaker will spin off 1 training job and 1 rule job for you.\n",
- "# Please follow this notebook to see status of the training job and the rule job.\n",
- "estimator.fit(wait=False)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Result\n",
- "As a result of the above command, SageMaker will spin off 1 training job and 1 rule job for you - the first one being the job which produces the tensors to be analyzed and the second one, which analyzes the tensors to check if `train-rmse` and `validation-rmse` are not decreasing at any point during training.\n",
- "\n",
- "### Describing the training job\n",
- "We can check the status of the training job by running the following command:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Below command will give the status of training job\n",
- "# Note: In the output of below command you will see DebugConfig parameter \n",
- "job_name = estimator.latest_training_job.name\n",
- "client = estimator.sagemaker_session.sagemaker_client\n",
- "description = client.describe_training_job(TrainingJobName=job_name)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# The status of the training job can be seen below\n",
- "description[\"TrainingJobStatus\"]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Once your training job is started SageMaker will spin up a rule execution job to run the LossNotDecreasing rule.\n",
- "\n",
- "### Tornasole specific parameters in the description\n",
- "**DebugConfig** parameter has details about Tornasole related configuration. The key parameters to look for below are\n",
- "\n",
- "*S3OutputPath* : This is the path where output tensors from tornasole is getting saved. \n",
- "*RuleConfig*' : This parameter tells about the rule config parameter that was passed when creating the trainning job. In this you should be able to see details of the rule that ran for training. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "description[\"DebugConfig\"]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Check the status of the Rule Execution Job\n",
- "To get the rule execution job that SageMaker started for you, run the command below and it shows you the `RuleName`, `RuleStatus`, `FailureReason` if any, and `RuleExecutionJobArn`. If the tensors meets a rule evaluation condition, the rule execution job throws a client error with `FailureReason: RuleEvaluationConditionMet`. These details are also available as part of the response `description` above under: `description['RuleMonitoringStatuses']`\n",
- "\n",
- "\n",
- "The logs of the training job are available in the Cloudwatch Logstream `/aws/sagemaker/TrainingJobs` with `RuleExecutionJobArn`. \n",
- "\n",
- "You will see that once the rule execution job starts, that it identifies the loss not decreasing situation in the training job, raises the `RuleEvaluationConditionMet` exception and ends the job. \n",
- "\n",
- "**Note that the next cell blocks until the rule execution job ends. You can stop it at any point to proceed to the rest of the notebook. Once it says RuleStatus is Started, and shows the `RuleExecutionJobArn`, you can look at the status of the rule being monitored. At that point, we can also look at the logs as shown in the next cell**"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator.describe_rule_execution_jobs()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Check logs of the rule execution jobs\n",
- "\n",
- "If you want to access the logs of a particular rule job name, you can do the following. First, you need to get the rule job name (`RuleExecutionJobArn` field from the training job description). Note that this is only available after the rule job reaches Started stage. Hence the next cell waits till the job name is available."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import time\n",
- "\n",
- "rule_descr = client.describe_training_job(TrainingJobName=job_name)[\"RuleMonitoringStatuses\"]\n",
- "print(\"Waiting for rule execution job to start\")\n",
- "while \"RuleExecutionJobArn\" not in rule_descr[0]:\n",
- " time.sleep(5)\n",
- " rule_descr = client.describe_training_job(TrainingJobName=job_name)[\"RuleMonitoringStatuses\"]\n",
- "\n",
- "rule_job_arn = rule_descr[0][\"RuleExecutionJobArn\"]\n",
- "print(\"Rule execution job has started. The job ARN is {}\".format(rule_job_arn))\n",
- "rule_job_name = rule_job_arn.split('/')[1]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we can attach to this job to see its logs"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from sagemaker.estimator import Estimator\n",
- "loss_not_decreasing = Estimator.attach(rule_job_name)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Receive a CloudWatch Event for Rules\n",
- "When the status of training job or rule execution job change (i.e. starting, failed), TrainingJobStatus [CloudWatch events](https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html) are emitted. More details on this, see [below](#CloudWatch-Event-Integration-for-Rules). \n",
- "\n",
- "\n",
- "### Making this a good run\n",
- "\n",
- "In above example, we saw how a LossNotDecreasing Rule was run which analyzed the performance metrics when training was running and produced an alert in form of cloudwatch event.\n",
- "\n",
- "You can go back and change the hyperparameters passed to the estimator to `hyperparameters` and start a new training job (e.g., use a smaller learning rate `eta=0.05`). You will see that the LossNotDecreasing rule is not fired in that case as both `train-rmse` and `validation-rmse` keep decreasing steadily throughout the entire training duration."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Data Analysis - Manual\n",
- "\n",
- "Now that we have trained the system we can analyze the data. Here we focus on after-the-fact analysis.\n",
- "\n",
- "We import a basic analysis library, which defines a concept of `Trial` that represents a single training run."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "from urllib.parse import urlparse\n",
- "from smdebug.trials import create_trial\n",
- "\n",
- "s3_output_path = description[\"DebugConfig\"][\"DebugHookConfig\"][\"S3OutputPath\"]\n",
- "trial = create_trial(s3_output_path)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can list all the tensors we know something about. Each one of these names is the name of a tensor - the name is a combination of the feature name (which, in these cases, is auto-assigned by XGBoost) and whether it's an evaluation metric, feature importance, or SHAP value."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "trial.tensors()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For each tensor we can ask for which steps we have data - in this case, every 2 steps"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "print(list(trial.tensor(\"train-rmse\").steps()))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can obtain each tensor at each step as a `numpy` array"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "type(trial.tensor(\"train-rmse\").value(30))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Performance metrics\n",
- "\n",
- "We can also create a simple function that visualizes the training and validation errors\n",
- "as the training progresses.\n",
- "We expect each gradient to get smaller over time, as the system converges to a good solution.\n",
- "Now, remember that this is an interactive analysis - we are showing these tensors to give an idea of the data. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import matplotlib.pyplot as plt\n",
- "import seaborn as sns\n",
- "\n",
- "# Define a function that, for the given tensor name, walks through all \n",
- "# the iterations for which we have data and fetches the value.\n",
- "# Returns the set of steps and the values\n",
- "def get_data(trial, tname):\n",
- " tensor = trial.tensor(tname)\n",
- " steps = tensor.steps()\n",
- " vals = [tensor.value(s) for s in steps]\n",
- " return steps, vals"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "metrics_to_plot = [\"train-rmse\", \"validation-rmse\"]\n",
- "for metric in metrics_to_plot:\n",
- " steps, data = get_data(trial, metric)\n",
- " plt.plot(steps, data, label=metric)\n",
- "plt.xlabel('Iteration')\n",
- "plt.ylabel('Root mean squred error')\n",
- "plt.legend()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Feature importances\n",
- "\n",
- "We can also visualize the feature importances as determined by\n",
- "[xgboost.get_fscore()](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.Booster.get_fscore).\n",
- "Note that feature importances with zero values are not included here\n",
- "(which means that those features were not used in any split conditons)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def plot_collections(trial, collection_name, ylabel=''):\n",
- " \n",
- " plt.figure(\n",
- " num=1, figsize=(8, 8), dpi=80,\n",
- " facecolor='w', edgecolor='k')\n",
- "\n",
- " features = trial.collection(collection_name).tensor_names\n",
- "\n",
- " for feature in sorted(features):\n",
- " steps, data = get_data(trial, feature)\n",
- " label = feature.replace('/' + collection_name, '')\n",
- " plt.plot(steps, data, label=label)\n",
- "\n",
- " plt.legend(bbox_to_anchor=(1.04,1), loc='upper left')\n",
- " plt.xlabel('Iteration')\n",
- " plt.ylabel(ylabel)\n",
- " plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "plot_collections(trial, \"feature_importance\", \"Feature importance\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### SHAP\n",
- "\n",
- "[SHAP](https://github.com/slundberg/shap) (SHapley Additive exPlanations) is\n",
- "another approach to explain the output of machine learning models.\n",
- "SHAP values represent a feature's contribution to a change in the model output."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "plot_collections(trial, \"average_shap\", \"SHAP values\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We also have an example at the end of this notebook that demonstrates how to use a custom rule in smdebug. Before moving further, let's take some detailed look at Tornasole, some of which were touched upon above.\n",
- "\n",
- "\n",
- "## Enabling Tornasole in the training script\n",
- "\n",
- "The first step to using Tornasole is to save tensors from the training job. The containers we provide in SageMaker come with Tornasole library installed, which needs to be used to enable Tornasole in your training script.\n",
- "\n",
- "To enable Tornasole in the training script, you need to create and pass SessionHook, a construct Tornasole exposes to save tensors. Here's how you will need to modify your training script.\n",
- "\n",
- "First, you need to import `smdebug.xgboost`. \n",
- "```\n",
- "import tornasole\n",
- "import smdebug.xgboost as tx\n",
- "```\n",
- "Then create the SessionHook by specifying what you want to save and when you want to save them.\n",
- "```\n",
- "hook = tx.SessionHook(include_collections=['metric','feature_importance'],\n",
- " save_config=smdebug.SaveConfig(save_interval=5))\n",
- "```\n",
- "Now pass this hook as a callback function to the Booster object's train method.\n",
- "```\n",
- "import xgboost\n",
- "\n",
- "bst = xgboost.train(..., callbacks=[hook])\n",
- "```\n",
- "\n",
- "Refer to our example script [xgboost_abalone_basic_hook_demo.py](../scripts/xgboost_abalone_basic_hook_demo.py) for examples of using Tornasole with the XGBoost interface.\n",
- "\n",
- "Refer [DeveloperGuide_XGBoost.md](../DeveloperGuide_XG.md) for more details on the APIs Tornasole provides to help you save tensors."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Enabling Tornasole with SageMaker\n",
- "\n",
- "#### Storage\n",
- "The tensors saved by Tornasole are, by default, stored in the S3 output path of the training job, under the folder **`/tensors-`**. This is done to ensure that we don't end up accidentally overwriting the tensors from a training job with the others. Rules evaluation require separation of the tensors paths to be evaluated correctly.\n",
- "\n",
- "If you don't provide an S3 output path to the estimator, SageMaker creates one for you as: **`s3://sagemaker--/`**\n",
- "\n",
- "This path is used to create a Tornasole Trial taken by Rules (see below).\n",
- "\n",
- "#### New Parameters \n",
- "The new parameters in Sagemaker Estimator to look out for are\n",
- "\n",
- "- `debug`: (bool)\n",
- "This indicates that debugging should be enabled for the training job. \n",
- "Setting this as `True` would make Tornasole available for use with the job\n",
- "\n",
- "- `rules_specification`: (list[*dict*])\n",
- "You can specify any number of rules to monitor your SageMaker training job. This parameter takes a list of python dictionaries, one for each rule you want to enable. Each `dict` is of the following form:\n",
- "```\n",
- "{\n",
- " \"RuleName\": \n",
- " # The name of the class implementing the Tornasole Rule interface. (required)\n",
- "\n",
- " \"SourceS3Uri\": \n",
- " # S3 URI of the rule script containing the class in 'RuleName'. \n",
- " # This is not required if you want to use one of the\n",
- " # First Party rules provided to you by Amazon. \n",
- " # In such a case you can leave it empty or not pass it. \n",
- " # If you want to run a custom rule \n",
- " # defined by you, you will need to define the custom rule class in a python \n",
- " # file and provide it to SageMaker as a S3 URI. \n",
- " # SageMaker will fetch this file and try to look for the rule class \n",
- " # identified by RuleName in this file.\n",
- " \n",
- " \"InstanceType\": \n",
- " # The ML instance type which should be used to run the rule evaluation job\n",
- " \n",
- " \"VolumeSizeInGB\": \n",
- " # The volume size to store the runtime artifacts from the rule evaluation \n",
- " \n",
- " \"RuntimeConfigurations\": {\n",
- " # Map defining the parameters required to instantiate the Rule class and\n",
- " # parameters regarding invokation of the rule (start-step and end-step)\n",
- " # This can be any parameter taken by the rule. \n",
- " # Every value here needs to be a string. \n",
- " # So when you write custom rules, ensure that you can parse each argument \n",
- " # from a string.\n",
- " #\n",
- " # PARAMS CAN BE\n",
- " #\n",
- " # STANDARD PARAMS FOR RULE EXECUTION\n",
- " # \"start-step\": \n",
- " # \"end-step\": \n",
- " # \"other-trials-paths\": (';' separated list of s3 paths as a string)\n",
- " # \"logging-level\": (can be one of \"CRITICAL\", \"FATAL\", \"ERROR\", \n",
- " # \"WARNING\", \"WARN\", \"DEBUG\", \"NOTSET\")\n",
- " #\n",
- " # ANY OTHER PARAMETER TAKEN BY THE RULE\n",
- " # \"parameter\" : \n",
- " # : \n",
- " }\n",
- "}\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Rules\n",
- "Rules are the medium by which Tornasole executes a certain piece of code regularly on different steps of the job.\n",
- "They can be used to assert certain conditions during training, and raise Cloudwatch Events based on them that you can\n",
- "use to process in any way you like. \n",
- "\n",
- "Tornasole comes with a set of **First Party rules** (1P rules).\n",
- "You can also write your own rules looking at these 1P rules for inspiration. \n",
- "Refer [DeveloperGuide_Rules.md](../../../rules/DeveloperGuide_Rules.md) for more on the APIs you can use to write your own rules as well as descriptions for the 1P rules that we provide. \n",
- " \n",
- "Here we will talk about how to use Sagemaker to evalute these rules on the training jobs.\n",
- "\n",
- "\n",
- "##### 1P Rule \n",
- "If you want to use a 1P rule. Specify the RuleName field with the 1P RuleName, and the rule will be automatically applied. You can pass any parameters accepted by the rule as part of the RuntimeConfigurations dictionary. Rules constructor take trial as parameter. \n",
- "A Trial in Tornasole's context refers to a training job. It is identified by the path where the saved tensors for the job are stored. \n",
- "A rule takes a `base_trial` which refers to the job whose run invokes the rule execution. \n",
- "\n",
- "**Note:** A rule can be written to compare & analyze tensors across training jobs. A rule which needs to compare tensors across trials can be run by passing the argument `other_trials`. The argument `base_trial` will automatically be set by SageMaker when executing the rule. The parameter `other_trials` (if taken by the rule) can be passed by passing `other-trials-paths` in the RuntimeConfigurations dictionary. The value for this argument should be `;` separated list of S3 output paths where the tensors for those trials are stored.\n",
- "\n",
- "Here's a example of a complex configuration for the SimilarAcrossRuns (which accepts one other trial and a regex pattern) where we ask for the rule to be invoked for the steps between 10 and 100.\n",
- "\n",
- "``` \n",
- "rules_specification = [ \n",
- " {\n",
- " \"RuleName\": \"SimilarAcrossRuns\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"other_trials\": \"s3://sagemaker--/past-job\",\n",
- " \"include_regex\": \".*\",\n",
- " \"start-step\": \"10\",\n",
- " \"end-step\": \"100\"\n",
- " }\n",
- " }\n",
- "]\n",
- "```\n",
- "List of 1P rules and details about the rules can be found in *First party rules* section in [DeveloperGuide_Rules.md](../../../rules/DeveloperGuide_Rules.md) \n",
- "\n",
- "\n",
- "##### Custom rule\n",
- "In this case you need to define a custom rule class which inherits from `smdebug.rules.Rule` class.\n",
- "You need to provide Sagemaker the S3 location of the file which defines your custom rule classes as the value for the field `SourceS3Uri`. Again, you can pass any arguments taken by this rule through the RuntimeConfigurations dictionary. Note that the custom rules can only have arguments which expect a string as the value except the two arguments specifying trials to the Rule. Refer section *Writing a rule* in [DeveloperGuide_Rules.md](../../../rules/DeveloperGuide_Rules.md) for more details.\n",
- "\n",
- "Here's an example:\n",
- "```\n",
- "rules_specification = [\n",
- " {\n",
- " \"RuleName\": \"CustomRule\",\n",
- " \"SourceS3Uri\": \"s3://tornasole-test/rule-script/custom_rule.py\",\n",
- " \"InstanceType\": \"ml.c5.4xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"threshold\" : \"0.5\"\n",
- " }\n",
- " }\n",
- "]\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### CloudWatch Event Integration for Rules\n",
- "When the status of training job or rule execution job change (i.e. starting, failed), TrainingJobStatus [CloudWatch events](https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html) are emitted.\n",
- "\n",
- "After GA, you can configure a CloudWatch event rule to receive and process these events by setting up a target (Lambda function, SNS) as follows:\n",
- "\n",
- "- The SageMaker TrainingJobStatus CW event (https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#sagemaker_event_types) will include rule job statuses associated with the training job\n",
- "- A CW event will be emitted when a RuleStatus changes\n",
- "- Customer can create a CloudWatch event rule that monitors the Training Job customer started\n",
- "- Customer can set a Target (Lambda funtion, SQS) for the CloudWatch event rule that processes the event, and triggers an alarm for the customer based on the RuleStatus. \n",
- "\n",
- "Refer [this page](https://docs.aws.amazon.com/sagemaker/latest/dg/cloudwatch-events.html) for more details. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Custom rule: Single Feature Importance\n",
- "\n",
- "In this case you need to define a custom rule class which inherits from `smdebug.rules.Rule` class.\n",
- "You need to provide Sagemaker the S3 location of the file which defines your custom rule classes as the value for the field `SourceS3Uri`.\n",
- "Again, you can pass any arguments taken by this rule through the RuntimeConfigurations dictionary. \n",
- "Note that the custom rules can only have arguments which expect a string as the value except the two arguments \n",
- "specifying trials to the Rule. Refer [DeveloperGuide_Rules.md](../../../rules/DeveloperGuide_Rules.md) for more.\n",
- "\n",
- "In the following code cell, we write a custom rule named `SingleFeatureImportance`\n",
- "that checks if any feature importance in a given collection goes out of the\n",
- "specified range."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "%%writefile /tmp/custom_feature_importance_rule.py\n",
- "\n",
- "from smdebug.rules.rule import Rule\n",
- "\n",
- "class SingleFeatureImportance(Rule):\n",
- " def __init__(\n",
- " self,\n",
- " base_trial,\n",
- " collection_name,\n",
- " num_features=None,\n",
- " min_importance_ratio=0,\n",
- " max_importance_ratio=1\n",
- " ):\n",
- " \"\"\"\n",
- " This rule checks the following statement:\n",
- " - In a given collection, each feature should have importance\n",
- " satisfying the following conditions:\n",
- " a) min_importance*(1/feature_no) <= feature importance\n",
- " b) feature_importance <= max_importance*(1/feature_no)\n",
- "\n",
- " :param base_trial: the trial whose execution will invoke the rule\n",
- " :param min_importance_ratio: the minimum allowed importance (as a proportion of 1/feature_no)\n",
- " :param max_importance_ratio: the maximum allowed importance (as a proportion of 1/feature_no)\n",
- " \"\"\"\n",
- " self.collection_name = collection_name\n",
- " self.tensor_names = base_trial.collection(self.collection_name).tensor_names\n",
- " self.num_features = len(self.tensor_names) if num_features is None else int(num_features)\n",
- " self.min_importance_ratio = float(min_importance_ratio)\n",
- " self.max_importance_ratio = float(max_importance_ratio)\n",
- "\n",
- " super().__init__(base_trial, other_trials=None)\n",
- " \n",
- " self.logger.info(\"FeatureImportance rule created.\")\n",
- "\n",
- " def invoke_at_step(self, step, **kwargs):\n",
- " \n",
- " min_importance = self.min_importance_ratio * (1 / self.num_features)\n",
- " max_importance = self.max_importance_ratio * (1 / self.num_features)\n",
- "\n",
- " failed = []\n",
- "\n",
- " for name in self.tensor_names:\n",
- " \n",
- " if step not in self.base_trial.tensor(name).steps():\n",
- " importance = 0\n",
- " else:\n",
- " importance = self.base_trial.tensor(name).value(step)\n",
- "\n",
- " if importance < min_importance:\n",
- " self.logger.debug(f\"Step {step} feature {name} has importance {importance}<{min_importance}\")\n",
- " failed.append((name, importance))\n",
- " elif max_importance < importance:\n",
- " self.logger.debug(f\"Step {step} feature {name} has importance {importance}>{max_importance}\")\n",
- " failed.append((name, importance))\n",
- "\n",
- " self.logger.info(failed)\n",
- " self.logger.info(f\"Step {step} had {len(failed)} features with out-of-band values\")\n",
- "\n",
- " return True if failed else False"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We need to upload this to a bucket in the same region where we want to run the job. We have chosen a default bucket below. Please change it to the bucket you want. We will now create this bucket if it does not exist, and upload this file. We will then specify this path when starting the job as `SourceS3Uri`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "ACCOUNT_ID = boto3.client('sts').get_caller_identity().get('Account')\n",
- "BUCKET = f'tornasole-resources-{ACCOUNT_ID}-{REGION}'\n",
- "\n",
- "CUSTOM_RULE_PATH = '/tmp/custom_feature_importance_rule.py'\n",
- "\n",
- "PREFIX = os.path.join('rules', os.path.basename(CUSTOM_RULE_PATH))\n",
- "\n",
- "import os\n",
- "s3 = boto3.resource('s3')\n",
- "bucket = s3.Bucket(BUCKET)\n",
- "if not bucket.creation_date:\n",
- " s3.create_bucket(Bucket=BUCKET, CreateBucketConfiguration={'LocationConstraint': REGION})\n",
- "s3.Object(BUCKET, PREFIX).put(Body=open(CUSTOM_RULE_PATH, 'rb'))\n",
- "SOURCE_S3_URI = f's3://{BUCKET}/{PREFIX}'\n",
- "print(f\"Upload to {SOURCE_S3_URI}\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator = Estimator(\n",
- " base_job_name=\"xgboost-tornasole-feature-importance\",\n",
- " hyperparameters=hyperparameters,\n",
- " image_name=docker_image_name,\n",
- " role=ROLE,\n",
- " train_instance_count=1,\n",
- " train_instance_type=\"ml.m4.4xlarge\",\n",
- " debug=True,\n",
- " rules_specification = [\n",
- " {\n",
- " \"RuleName\": \"SingleFeatureImportance\",\n",
- " \"SourceS3Uri\": SOURCE_S3_URI,\n",
- " \"InstanceType\": \"ml.c5.xlarge\",\n",
- " \"VolumeSizeInGB\": 10,\n",
- " \"RuntimeConfigurations\": {\n",
- " \"collection_name\": \"average_shap\",\n",
- " \"num_features\": \"8\",\n",
- " \"min_importance_ratio\": \"0.0\",\n",
- " \"max_importance_ratio\": \"2.0\"\n",
- " }\n",
- " }\n",
- " ]\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator.fit(wait=False)\n",
- "\n",
- "job_name = estimator.latest_training_job.name\n",
- "client = estimator.sagemaker_session.sagemaker_client\n",
- "description = client.describe_training_job(TrainingJobName=job_name)\n",
- "\n",
- "description[\"TrainingJobStatus\"]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "description[\"DebugConfig\"]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "As we can visually verify in the [Data Analysis](#Data-Analysis---Manual) section above,\n",
- "the SHAP value of feature will become significant, and we expect our rule to throw a\n",
- "`RuleEvaluationConditionMet` exception."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "estimator.describe_rule_execution_jobs()"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "conda_python3",
- "language": "python",
- "name": "conda_python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.5"
- },
- "pycharm": {
- "stem_cell": {
- "cell_type": "raw",
- "metadata": {
- "collapsed": false
- },
- "source": []
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/examples/xgboost/scripts/xgboost_abalone_basic_hook_demo.py b/examples/xgboost/scripts/xgboost_abalone_basic_hook_demo.py
deleted file mode 100644
index ec5551d1e..000000000
--- a/examples/xgboost/scripts/xgboost_abalone_basic_hook_demo.py
+++ /dev/null
@@ -1,125 +0,0 @@
-# Standard Library
-import argparse
-import os
-import random
-import tempfile
-import urllib.request
-
-# Third Party
-import xgboost
-
-# First Party
-from smdebug import SaveConfig
-from smdebug.xgboost import Hook
-
-
-def parse_args():
-
- parser = argparse.ArgumentParser()
-
- parser.add_argument("--max_depth", type=int, default=5)
- parser.add_argument("--eta", type=float, default=0.2)
- parser.add_argument("--gamma", type=int, default=4)
- parser.add_argument("--min_child_weight", type=int, default=6)
- parser.add_argument("--subsample", type=float, default=0.7)
- parser.add_argument("--silent", type=int, default=0)
- parser.add_argument("--objective", type=str, default="reg:squarederror")
- parser.add_argument("--num_round", type=int, default=50)
- parser.add_argument("--smdebug_path", type=str, default=None)
- parser.add_argument("--save_frequency", type=int, default=1)
- parser.add_argument(
- "--output_uri",
- type=str,
- default="/opt/ml/output/tensors",
- help="S3 URI of the bucket where tensor data will be stored.",
- )
-
- parser.add_argument("--train", type=str, default=os.environ.get("SM_CHANNEL_TRAIN"))
- parser.add_argument("--validation", type=str, default=os.environ.get("SM_CHANNEL_VALIDATION"))
-
- args = parser.parse_args()
-
- return args
-
-
-def load_abalone(train_split=0.8, seed=42):
-
- if not (0 < train_split <= 1):
- raise ValueError("'train_split' must be between 0 and 1.")
-
- url = "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression/abalone"
-
- response = urllib.request.urlopen(url).read().decode("utf-8")
- lines = response.strip().split("\n")
- n = sum(1 for line in lines)
- indices = list(range(n))
- random.seed(seed)
- random.shuffle(indices)
- train_indices = set(indices[: int(n * 0.8)])
-
- with tempfile.NamedTemporaryFile(mode="w", delete=False) as train_file:
- with tempfile.NamedTemporaryFile(mode="w", delete=False) as valid_file:
- for idx, line in enumerate(lines):
- if idx in train_indices:
- train_file.write(line + "\n")
- else:
- valid_file.write(line + "\n")
-
- return train_file.name, valid_file.name
-
-
-def create_hook(out_dir, train_data=None, validation_data=None, frequency=1):
-
- save_config = SaveConfig(save_interval=frequency)
- hook = Hook(
- out_dir=out_dir,
- save_config=save_config,
- train_data=train_data,
- validation_data=validation_data,
- )
-
- return hook
-
-
-def main():
-
- args = parse_args()
-
- if args.train and args.validation:
- train, validation = args.train, args.validation
- else:
- train, validation = load_abalone()
-
- dtrain = xgboost.DMatrix(train)
- dval = xgboost.DMatrix(validation)
-
- watchlist = [(dtrain, "train"), (dval, "validation")]
-
- params = {
- "max_depth": args.max_depth,
- "eta": args.eta,
- "gamma": args.gamma,
- "min_child_weight": args.min_child_weight,
- "subsample": args.subsample,
- "silent": args.silent,
- "objective": args.objective,
- }
-
- # The output_uri is a the URI for the s3 bucket where the metrics will be
- # saved.
- output_uri = args.smdebug_path if args.smdebug_path is not None else args.output_uri
-
- hook = create_hook(out_dir=output_uri, frequency=args.save_frequency, train_data=dtrain)
-
- bst = xgboost.train(
- params=params,
- dtrain=dtrain,
- evals=watchlist,
- num_boost_round=args.num_round,
- callbacks=[hook],
- )
-
-
-if __name__ == "__main__":
-
- main()
diff --git a/examples/xgboost/scripts/xgboost_mnist_basic_hook_demo.py b/examples/xgboost/scripts/xgboost_mnist_basic_hook_demo.py
deleted file mode 100644
index 6221a7320..000000000
--- a/examples/xgboost/scripts/xgboost_mnist_basic_hook_demo.py
+++ /dev/null
@@ -1,132 +0,0 @@
-# Standard Library
-import argparse
-import bz2
-import os
-import random
-import tempfile
-import urllib.request
-
-# Third Party
-import xgboost
-
-# First Party
-from smdebug import SaveConfig
-from smdebug.xgboost import Hook
-
-
-def parse_args():
-
- parser = argparse.ArgumentParser()
-
- parser.add_argument("--max_depth", type=int, default=5)
- parser.add_argument("--eta", type=float, default=0.05) # 0.2
- parser.add_argument("--gamma", type=int, default=4)
- parser.add_argument("--min_child_weight", type=int, default=6)
- parser.add_argument("--silent", type=int, default=0)
- parser.add_argument("--objective", type=str, default="multi:softmax")
- parser.add_argument("--num_class", type=int, default=10)
- parser.add_argument("--num_round", type=int, default=10)
- parser.add_argument("--smdebug_path", type=str, default=None)
- parser.add_argument("--save_frequency", type=int, default=1)
- parser.add_argument(
- "--output_uri",
- type=str,
- default="/opt/ml/output/tensors",
- help="S3 URI of the bucket where tensor data will be stored.",
- )
-
- parser.add_argument("--train", type=str, default=os.environ.get("SM_CHANNEL_TRAIN"))
- parser.add_argument("--validation", type=str, default=os.environ.get("SM_CHANNEL_VALIDATION"))
-
- args = parser.parse_args()
-
- return args
-
-
-def load_mnist(train_split=0.8, seed=42):
-
- if not (0 < train_split <= 1):
- raise ValueError("'train_split' must be between 0 and 1.")
-
- url = "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass/mnist.bz2"
-
- with tempfile.NamedTemporaryFile(mode="wb", delete=False) as mnist_bz2:
- urllib.request.urlretrieve(url, mnist_bz2.name)
-
- with bz2.open(mnist_bz2.name, "r") as fin:
- content = fin.read().decode("utf-8")
- lines = content.strip().split("\n")
- n = sum(1 for line in lines)
- indices = list(range(n))
- random.seed(seed)
- random.shuffle(indices)
- train_indices = set(indices[: int(n * 0.8)])
-
- with tempfile.NamedTemporaryFile(mode="w", delete=False) as train_file:
- with tempfile.NamedTemporaryFile(mode="w", delete=False) as valid_file:
- for idx, line in enumerate(lines):
- if idx in train_indices:
- train_file.write(line + "\n")
- else:
- valid_file.write(line + "\n")
-
- return train_file.name, valid_file.name
-
-
-def create_hook(out_dir, train_data=None, validation_data=None, frequency=1):
-
- save_config = SaveConfig(save_interval=frequency)
- hook = Hook(
- out_dir=out_dir,
- save_config=save_config,
- train_data=train_data,
- validation_data=validation_data,
- )
-
- return hook
-
-
-def main():
-
- args = parse_args()
-
- if args.train and args.validation:
- train, validation = args.train, args.validation
- else:
- train, validation = load_mnist()
-
- dtrain = xgboost.DMatrix(train)
- dval = xgboost.DMatrix(validation)
-
- watchlist = [(dtrain, "train"), (dval, "validation")]
-
- params = {
- "max_depth": args.max_depth,
- "eta": args.eta,
- "gamma": args.gamma,
- "min_child_weight": args.min_child_weight,
- "silent": args.silent,
- "objective": args.objective,
- "num_class": args.num_class,
- }
-
- # The output_uri is a the URI for the s3 bucket where the metrics will be
- # saved.
- output_uri = args.smdebug_path if args.smdebug_path is not None else args.output_uri
-
- hook = create_hook(
- out_dir=output_uri, frequency=args.save_frequency, train_data=dtrain, validation_data=dval
- )
-
- bst = xgboost.train(
- params=params,
- dtrain=dtrain,
- evals=watchlist,
- num_boost_round=args.num_round,
- callbacks=[hook],
- )
-
-
-if __name__ == "__main__":
-
- main()
diff --git a/tests/tensorflow/hooks/test_training_end.py b/tests/tensorflow/hooks/test_training_end.py
index 541b6be56..ce59c14dc 100644
--- a/tests/tensorflow/hooks/test_training_end.py
+++ b/tests/tensorflow/hooks/test_training_end.py
@@ -16,15 +16,13 @@ def test_training_job_has_ended(out_dir):
subprocess.check_call(
[
sys.executable,
- "examples/tensorflow/scripts/simple.py",
- "--smdebug_path",
+ "examples/tensorflow/local/simple.py",
+ "--out_dir",
out_dir,
"--steps",
"10",
- "--save_frequency",
+ "--save_interval",
"5",
- "--script-mode",
- "y",
],
env={"CUDA_VISIBLE_DEVICES": "-1", "SMDEBUG_LOG_LEVEL": "debug"},
)