From 69b3943e72a63b23271af49fa6735cac21909609 Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Mon, 20 Apr 2020 15:14:09 -0700 Subject: [PATCH 01/35] Update README.md --- README.md | 142 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 105 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index 650dc1f16..6bd1ff01e 100644 --- a/README.md +++ b/README.md @@ -2,20 +2,23 @@ [![codecov](https://codecov.io/gh/awslabs/sagemaker-debugger/branch/master/graph/badge.svg)](https://codecov.io/gh/awslabs/sagemaker-debugger) [![PyPI](https://badge.fury.io/py/smdebug.svg)](https://badge.fury.io/py/smdebug) +## Table of Contents + - [Overview](#overview) -- [Examples](#examples) +- [Support](#support) - [How It Works](#how-it-works) - [Docs](#docs) +- [Examples](#examples) - [SageMaker Debugger in action](#sagemaker-debugger-in-action) - ## Overview -Amazon SageMaker Debugger is an offering from AWS which help you automate the debugging of machine learning training jobs. -This library powers Amazon SageMaker Debugger, and helps you develop better, faster and cheaper models by catching common errors quickly. -It allows you to save tensors from training jobs and makes these tensors available for analysis, all through a flexible and powerful API. -It supports TensorFlow, PyTorch, MXNet, and XGBoost on Python 3.6+. +[Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) automates the debugging process of machine learning training jobs. It allows you to save tensors from training jobs and makes these tensors available for analysis, all through a flexible and powerful API. +`smdebug` library powers [Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) by calling the saved tensors in S3 during the training job. `smdebug` retrieves and filters tensors such as gradients, weights, and biases. + +Amazon SageMaker Debugger supports TensorFlow, PyTorch, MXNet, and XGBoost frameworks. +The following list is a summary of the main funcionalities of Amazon SageMaker Debugger: -- Zero Script Change experience on SageMaker when using [supported containers](docs/sagemaker.md#zero-script-change) +- Zero Script Change experience on SageMaker when using [supported containers](#support) - Full visibility into any tensor part of the training process - Real-time training job monitoring through Rules - Automated anomaly detection and state assertions through built-in and custom Rules on SageMaker @@ -24,12 +27,104 @@ It supports TensorFlow, PyTorch, MXNet, and XGBoost on Python 3.6+. - Distributed training support - TensorBoard support +## Support + +### [Latest release v0.7.2](https://github.com/awslabs/sagemaker-debugger/releases) + + - Experimental support for TF 2.x GradientTape - Introducing experimental support for TF 2.x training scripts using GradientTape. With this change, weights, bias, loss, metrics, and gradients are captured by SageMaker Debugger. These changes work with vanilla version of Tensorflow 2.x (not with the zero-code change version). + + Note: Training scripts using GradientTape for higher-order gradients or multiple tapes are not supported. Distributed training scripts that use GradientTape are not supported at this time. + + - Support SyncOnReadVariable in mirrored strategy - Fixes a bug that occurred because SyncOnRead distributed variable was not supported with smdebug. Also enables the use of smdebug with training scripts using TF 2.x MirroredStrategy with fit() API. + + - Turn off hook and write only from one worker for unsupported distributed training techniques – Fixes a crash when distributed training in PyTorch framework is implemented using generic multiprocessing library, which is not a method supported by smdebug. This fix handles this case and ensures that tensors are saved. + + - Bug fix: Pytorch: Register only if tensors require gradients – Users were observing a crash when training with pretrained embeddings which does not need gradient updates. This fix checks if a gradient update is required and registers a backward hook only in those cases. + +#### Setting up SageMaker Debugger + +- `smdebug` library runs on Python 3.x. Install `smdebug` through: + + ``` + pip install smdebug + ``` + +#### Zero Script Change + +| Framework | Version | +| --- | --- | +| [TensorFlow](tensorflow.md) | 1.15, 2.1 | +| [MXNet](mxnet.md) | 1.6 | +| [PyTorch](pytorch.md) | 1.3, 1.4 | +| [XGBoost](xgboost.md) | >=0.90-2 [As Built-in algorithm](xgboost.md#use-xgboost-as-a-built-in-algorithm)| + +#### Bring your own training container + +| Framework | Versions | +| --- | --- | +| [TensorFlow](tensorflow.md) | 1.14. 1.15, 2.0.1, 2.1.0 | +| Keras (with TensorFlow backend) | 2.3 | +| [MXNet](mxnet.md) | 1.4, 1.5, 1.6 | +| [PyTorch](pytorch.md) | 1.2, 1.3, 1.4 | +| [XGBoost](xgboost.md) | [As Framework](xgboost.md#use-xgboost-as-a-framework) | + +#### Support for Distributed Training and Known Limitations + + + + + + + + + + + + + + + + + + + + + + +
+ Distributed Training +
HorovodSupportedTF 1.15, PT 1.4, MX 1.6
Not supportedTF 2.x, PT 1.5
Parameter Server-basedNot supported
+ + +## How It Works + +Amazon SageMaker Debugger uses the construct of a `Hook` to save the values of requested tensors throughout the training process. You can then setup a `Rule` job which simultaneously monitors and validates these tensors to ensure +that training is progressing as expected. +A rule might check for vanishing gradients, or exploding tensor values, or poor weight initialization. Rules are attached to CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. + +Amazon SageMaker Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following: +- **SageMaker Zero-Script-Change**: Here you specify which rules to use when setting up the estimator and run your existing script, no changes needed. See the first example above. +- **SageMaker Bring-Your-Own-Container**: Here you specify the rules to use, and modify your training script minimally to enable SageMaker Debugger. +- **Non-SageMaker**: Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. See the second example above. + +The reason for different setups is that SageMaker Zero-Script-Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add our Hook to the training job and save requested tensors automatically. +These framework forks are not available in custom containers or non-SM environments, so you must modify your training script in these environments. + +## Docs + +| Section | Description | +| --- | --- | +| [SageMaker Training](docs/sagemaker.md) | SageMaker users, we recommend you start with this page on how to run SageMaker training jobs with SageMaker Debugger | +| Frameworks | See the frameworks pages for details on what's supported and how to modify your training script if applicable | +| [APIs for Saving Tensors](docs/api.md) | Full description of our APIs on saving tensors | +| [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by our APIs which allows you to perform interactive exploration of tensors saved as well as to write your own Rules monitoring your training jobs. | + ## Examples ### Notebooks -We have a bunch of [example notebooks](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger) here demonstrating different functionality of SageMaker Debugger. +Example notebooks demonstrating different functionalities of SageMaker Debugger are provided [here](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger). ### Running a Rule with Zero Script Change on SageMaker -This example uses a zero-script-change experience, where you can use your training script as-is. Refer [Running SageMaker jobs with Amazon SageMaker Debugger](docs/sagemaker.md) for more details on this. +This example uses a zero-script-change experience, where you can use your training script as-is. Refer to [Running SageMaker jobs with Amazon SageMaker Debugger](docs/sagemaker.md) for more details on this. ```python import sagemaker as sm from sagemaker.debugger import rule_configs, Rule, CollectionConfig @@ -71,10 +166,7 @@ event which tracks the status of the Rule, so you can take any action based on t If you want greater configuration and control, we offer that too. Head over [here](docs/sagemaker.md) for more information. ### Running Locally -Requires Python 3.6+, and this example uses tf.keras. Run -``` -pip install smdebug -``` +Requires Python 3.6+, and this example uses tf.keras. To use Amazon SageMaker Debugger, simply add a callback hook: ```python @@ -97,30 +189,6 @@ print(f"Saved these tensors: {trial.tensor_names()}") print(f"Loss values during evaluation were {trial.tensor('CrossEntropyLoss:0').values(mode=smd.modes.EVAL)}") ``` -## How It Works - -Amazon SageMaker Debugger uses the construct of a `Hook` to save the values of requested tensors throughout the training process. You can then setup a `Rule` job which simultaneously monitors and validates these tensors to ensure -that training is progressing as expected. -A rule might check for vanishing gradients, or exploding tensor values, or poor weight initialization. Rules are attached to CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. - -Amazon SageMaker Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following: -- **SageMaker Zero-Script-Change**: Here you specify which rules to use when setting up the estimator and run your existing script, no changes needed. See the first example above. -- **SageMaker Bring-Your-Own-Container**: Here you specify the rules to use, and modify your training script minimally to enable SageMaker Debugger. -- **Non-SageMaker**: Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. See the second example above. - -The reason for different setups is that SageMaker Zero-Script-Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add our Hook to the training job and save requested tensors automatically. -These framework forks are not available in custom containers or non-SM environments, so you must modify your training script in these environments. - -## Docs - -| Section | Description | -| --- | --- | -| [SageMaker Training](docs/sagemaker.md) | SageMaker users, we recommend you start with this page on how to run SageMaker training jobs with SageMaker Debugger | -| Frameworks | See the frameworks pages for details on what's supported and how to modify your training script if applicable | -| [APIs for Saving Tensors](docs/api.md) | Full description of our APIs on saving tensors | -| [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by our APIs which allows you to perform interactive exploration of tensors saved as well as to write your own Rules monitoring your training jobs. | - - ## SageMaker Debugger in action - Using SageMaker Debugger with XGBoost in SageMaker Studio to save feature importance values and plot them in a notebook during training. ![](docs/resources/xgboost_feature_importance.png?raw=true) - Using SageMaker Debugger with TensorFlow in SageMaker Studio to run built-in rules and visualize the loss. ![](docs/resources/tensorflow_rules_loss.png?raw=true) From 90ab484dd396edd07ba68cbf5bba59150b108b99 Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Mon, 20 Apr 2020 17:33:57 -0700 Subject: [PATCH 02/35] response to the comment from vandanavk --- README.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 6bd1ff01e..9cd9aa780 100644 --- a/README.md +++ b/README.md @@ -30,8 +30,11 @@ The following list is a summary of the main funcionalities of Amazon SageMaker D ## Support ### [Latest release v0.7.2](https://github.com/awslabs/sagemaker-debugger/releases) - - - Experimental support for TF 2.x GradientTape - Introducing experimental support for TF 2.x training scripts using GradientTape. With this change, weights, bias, loss, metrics, and gradients are captured by SageMaker Debugger. These changes work with vanilla version of Tensorflow 2.x (not with the zero-code change version). + + - Experimental support for TF 2.x GradientTape - Introducing experimental support for TF 2.x training scripts using GradientTape. + + SageMaker Debugger GradientTape now captures tensors such as loss, metrics, weights, biases, and gradients by modifying training scripts in TF 2.x framework. A sample training script is provided [here](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_gradienttape.py). + GradientTape does not work with Zero Script Change experience at this time. Note: Training scripts using GradientTape for higher-order gradients or multiple tapes are not supported. Distributed training scripts that use GradientTape are not supported at this time. @@ -39,7 +42,7 @@ The following list is a summary of the main funcionalities of Amazon SageMaker D - Turn off hook and write only from one worker for unsupported distributed training techniques – Fixes a crash when distributed training in PyTorch framework is implemented using generic multiprocessing library, which is not a method supported by smdebug. This fix handles this case and ensures that tensors are saved. - - Bug fix: Pytorch: Register only if tensors require gradients – Users were observing a crash when training with pretrained embeddings which does not need gradient updates. This fix checks if a gradient update is required and registers a backward hook only in those cases. + - Bug fix: Pytorch: Register only if tensors require gradients – Users were observing a crash when training with pre-trained embeddings which does not need gradient updates. This fix checks if a gradient update is required and registers a backward hook only in those cases. #### Setting up SageMaker Debugger @@ -121,7 +124,7 @@ These framework forks are not available in custom containers or non-SM environme ## Examples ### Notebooks -Example notebooks demonstrating different functionalities of SageMaker Debugger are provided [here](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger). +Example notebooks demonstrating different functionalities of SageMaker Debugger are provided [here](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger). ### Running a Rule with Zero Script Change on SageMaker This example uses a zero-script-change experience, where you can use your training script as-is. Refer to [Running SageMaker jobs with Amazon SageMaker Debugger](docs/sagemaker.md) for more details on this. From 55d735ec92283a1316444507a7b5f2dacf1d210a Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Mon, 20 Apr 2020 17:49:17 -0700 Subject: [PATCH 03/35] fixing a typo --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 9cd9aa780..fb63d181c 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ `smdebug` library powers [Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) by calling the saved tensors in S3 during the training job. `smdebug` retrieves and filters tensors such as gradients, weights, and biases. Amazon SageMaker Debugger supports TensorFlow, PyTorch, MXNet, and XGBoost frameworks. -The following list is a summary of the main funcionalities of Amazon SageMaker Debugger: +The following list is a summary of the main functionalities of Amazon SageMaker Debugger: - Zero Script Change experience on SageMaker when using [supported containers](#support) - Full visibility into any tensor part of the training process @@ -32,7 +32,6 @@ The following list is a summary of the main funcionalities of Amazon SageMaker D ### [Latest release v0.7.2](https://github.com/awslabs/sagemaker-debugger/releases) - Experimental support for TF 2.x GradientTape - Introducing experimental support for TF 2.x training scripts using GradientTape. - SageMaker Debugger GradientTape now captures tensors such as loss, metrics, weights, biases, and gradients by modifying training scripts in TF 2.x framework. A sample training script is provided [here](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_gradienttape.py). GradientTape does not work with Zero Script Change experience at this time. From fd39891d6ebef46d3894cd4dc0607bb0fff8bc58 Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Thu, 23 Apr 2020 10:51:41 -0700 Subject: [PATCH 04/35] Fixing the example links --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index fb63d181c..e28c30945 100644 --- a/README.md +++ b/README.md @@ -105,9 +105,9 @@ that training is progressing as expected. A rule might check for vanishing gradients, or exploding tensor values, or poor weight initialization. Rules are attached to CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. Amazon SageMaker Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following: -- **SageMaker Zero-Script-Change**: Here you specify which rules to use when setting up the estimator and run your existing script, no changes needed. See the first example above. +- **SageMaker Zero-Script-Change**: Here you specify which rules to use when setting up the estimator and run your existing script, no changes needed. See [the first example below](#running-a-rule-with-zero-script-change-on-sageMaker). - **SageMaker Bring-Your-Own-Container**: Here you specify the rules to use, and modify your training script minimally to enable SageMaker Debugger. -- **Non-SageMaker**: Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. See the second example above. +- **Non-SageMaker**: Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. See [the second example below](#running-locally). The reason for different setups is that SageMaker Zero-Script-Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add our Hook to the training job and save requested tensors automatically. These framework forks are not available in custom containers or non-SM environments, so you must modify your training script in these environments. From f69a2067bed00a311c12d6282f1d107ef478d515 Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Mon, 27 Apr 2020 11:49:25 -0700 Subject: [PATCH 05/35] Staging for preview --- README.md | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index e28c30945..80f04dd5d 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,11 @@ ## Overview [Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) automates the debugging process of machine learning training jobs. It allows you to save tensors from training jobs and makes these tensors available for analysis, all through a flexible and powerful API. -`smdebug` library powers [Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) by calling the saved tensors in S3 during the training job. `smdebug` retrieves and filters tensors such as gradients, weights, and biases. +The `smdebug` library powers [Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) by calling the saved tensors in S3 during the training job. +[Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) supports `Hook` and `Rule` features to designate saving directory in S3 and monitor the tensors. See [How it work](#how-it-works) for more details. +[Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) provides built-in (Zero Script Change) `Hook` and `Rule` configurations and supports customization of configurations. +`smdebug` retrieves and filters the tensors generated from [Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) such as gradients, weights, and biases. `smdebug` helps you develop better, faster and cheaper models by tracing the tensors and iterative model pruning. +Through the model pruning process, you can iteratively identify the importance of weights and cut neurons below a threshold. This process allows us to train the model with significantly less number of neurons, which means lighter, efficient, after, and cheaper without compensating accuracy. Amazon SageMaker Debugger supports TensorFlow, PyTorch, MXNet, and XGBoost frameworks. The following list is a summary of the main functionalities of Amazon SageMaker Debugger: @@ -105,21 +109,13 @@ that training is progressing as expected. A rule might check for vanishing gradients, or exploding tensor values, or poor weight initialization. Rules are attached to CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. Amazon SageMaker Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following: -- **SageMaker Zero-Script-Change**: Here you specify which rules to use when setting up the estimator and run your existing script, no changes needed. See [the first example below](#running-a-rule-with-zero-script-change-on-sageMaker). -- **SageMaker Bring-Your-Own-Container**: Here you specify the rules to use, and modify your training script minimally to enable SageMaker Debugger. -- **Non-SageMaker**: Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. See [the second example below](#running-locally). +- **SageMaker Zero Script Change**: Here you specify which rules to use when setting up the estimator and run your existing script, no changes needed. See [Running a Rule with Zero Script Change on SageMaker](#running-a-rule-with-zero-script-change-on-sageMaker). +- **SageMaker Bring Your Own Container**: Here you specify the rules to use, and modify your training script minimally to enable SageMaker Debugger. +- **Non-SageMaker**: Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. See [Running Locally](#running-locally). The reason for different setups is that SageMaker Zero-Script-Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add our Hook to the training job and save requested tensors automatically. These framework forks are not available in custom containers or non-SM environments, so you must modify your training script in these environments. -## Docs - -| Section | Description | -| --- | --- | -| [SageMaker Training](docs/sagemaker.md) | SageMaker users, we recommend you start with this page on how to run SageMaker training jobs with SageMaker Debugger | -| Frameworks
  • [TensorFlow](docs/tensorflow.md)
  • [PyTorch](docs/pytorch.md)
  • [MXNet](docs/mxnet.md)
  • [XGBoost](docs/xgboost.md)
| See the frameworks pages for details on what's supported and how to modify your training script if applicable | -| [APIs for Saving Tensors](docs/api.md) | Full description of our APIs on saving tensors | -| [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by our APIs which allows you to perform interactive exploration of tensors saved as well as to write your own Rules monitoring your training jobs. | ## Examples ### Notebooks @@ -191,6 +187,15 @@ print(f"Saved these tensors: {trial.tensor_names()}") print(f"Loss values during evaluation were {trial.tensor('CrossEntropyLoss:0').values(mode=smd.modes.EVAL)}") ``` +## Docs + +| Section | Description | +| --- | --- | +| [SageMaker Training](docs/sagemaker.md) | SageMaker users, we recommend you start with this page on how to run SageMaker training jobs with SageMaker Debugger | +| Frameworks
  • [TensorFlow](docs/tensorflow.md)
  • [PyTorch](docs/pytorch.md)
  • [MXNet](docs/mxnet.md)
  • [XGBoost](docs/xgboost.md)
| See the frameworks pages for details on what's supported and how to modify your training script if applicable | +| [APIs for Saving Tensors](docs/api.md) | Full description of our APIs on saving tensors | +| [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by our APIs which allows you to perform interactive exploration of tensors saved as well as to write your own Rules monitoring your training jobs. | + ## SageMaker Debugger in action - Using SageMaker Debugger with XGBoost in SageMaker Studio to save feature importance values and plot them in a notebook during training. ![](docs/resources/xgboost_feature_importance.png?raw=true) - Using SageMaker Debugger with TensorFlow in SageMaker Studio to run built-in rules and visualize the loss. ![](docs/resources/tensorflow_rules_loss.png?raw=true) From d1543aa42efad20b53350a628b942956320dfe8f Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Mon, 27 Apr 2020 17:53:26 -0700 Subject: [PATCH 06/35] fixed the doc responding to the comments --- README.md | 47 ++++++++++++++++++++++++++++++----------------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 80f04dd5d..8e849b639 100644 --- a/README.md +++ b/README.md @@ -12,12 +12,16 @@ - [SageMaker Debugger in action](#sagemaker-debugger-in-action) ## Overview -[Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) automates the debugging process of machine learning training jobs. It allows you to save tensors from training jobs and makes these tensors available for analysis, all through a flexible and powerful API. -The `smdebug` library powers [Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) by calling the saved tensors in S3 during the training job. -[Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) supports `Hook` and `Rule` features to designate saving directory in S3 and monitor the tensors. See [How it work](#how-it-works) for more details. -[Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) provides built-in (Zero Script Change) `Hook` and `Rule` configurations and supports customization of configurations. -`smdebug` retrieves and filters the tensors generated from [Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) such as gradients, weights, and biases. `smdebug` helps you develop better, faster and cheaper models by tracing the tensors and iterative model pruning. -Through the model pruning process, you can iteratively identify the importance of weights and cut neurons below a threshold. This process allows us to train the model with significantly less number of neurons, which means lighter, efficient, after, and cheaper without compensating accuracy. +[Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) automates the debugging process of machine learning training jobs. From training jobs, Debugger allows you to +run your own training script (Zero Script Change experience) using Debugger built-in features---`Hook` and `Rule`---to capture tensors, +have flexibility to build customized `Hook`s and `Rule`s for configuring tensors as you want, +and make the tensors available for analysis by saving in S3, +all through a flexible and powerful API. +The `smdebug` library powers Debugger by calling the saved tensors in S3 during the training job. + +`smdebug` retrieves and filters the tensors generated from Debugger such as gradients, weights, and biases. `smdebug` helps you develop better, faster and cheaper models by tracing the tensors and iterative model pruning. + +Through the model pruning process using Debugger and `smdebug`, you can iteratively identify the importance of weights, and cut neurons below a threshold you define. This process allows you to train the model with significantly fewer neurons, which means a lighter, more efficient, faster, and cheaper model without compromising accuracy. Amazon SageMaker Debugger supports TensorFlow, PyTorch, MXNet, and XGBoost frameworks. The following list is a summary of the main functionalities of Amazon SageMaker Debugger: @@ -31,19 +35,22 @@ The following list is a summary of the main functionalities of Amazon SageMaker - Distributed training support - TensorBoard support +See [How it work](#how-it-works) for more details. + ## Support ### [Latest release v0.7.2](https://github.com/awslabs/sagemaker-debugger/releases) - - Experimental support for TF 2.x GradientTape - Introducing experimental support for TF 2.x training scripts using GradientTape. - SageMaker Debugger GradientTape now captures tensors such as loss, metrics, weights, biases, and gradients by modifying training scripts in TF 2.x framework. A sample training script is provided [here](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_gradienttape.py). + - Introducing experimental support for TF 2.x training scripts using GradientTape - + With this update, weights, bias, loss, metrics, and gradients are captured by SageMaker Debugger. + GradientTape in TF 2.x captures these tensors from custom training jobs. An example of GradientTape implementation to a custom ResNet training script using TensorFlow's Keras interface is provided at [tf_keras_gradienttape.py](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_gradienttape.py). GradientTape does not work with Zero Script Change experience at this time. - Note: Training scripts using GradientTape for higher-order gradients or multiple tapes are not supported. Distributed training scripts that use GradientTape are not supported at this time. + *Note*: Training scripts using GradientTape for higher-order gradients or multiple tapes are not supported. Distributed training scripts that use GradientTape are not supported at this time. - - Support SyncOnReadVariable in mirrored strategy - Fixes a bug that occurred because SyncOnRead distributed variable was not supported with smdebug. Also enables the use of smdebug with training scripts using TF 2.x MirroredStrategy with fit() API. + - Support `SyncOnReadVariable` in mirrored strategy - Fixes a bug that occurred because the `SyncOnRead` distributed variable was not supported with `smdebug`. Also enables the use of `smdebug` with training scripts using TF 2.x MirroredStrategy with the `fit()` API. - - Turn off hook and write only from one worker for unsupported distributed training techniques – Fixes a crash when distributed training in PyTorch framework is implemented using generic multiprocessing library, which is not a method supported by smdebug. This fix handles this case and ensures that tensors are saved. + - Turn off hook and write only from one worker for unsupported distributed training techniques – Fixes a crash when distributed training in PyTorch framework is implemented using generic multiprocessing library, which is not a method supported by `smdebug`. This fix handles this case and ensures that tensors are saved. - Bug fix: Pytorch: Register only if tensors require gradients – Users were observing a crash when training with pre-trained embeddings which does not need gradient updates. This fix checks if a gradient update is required and registers a backward hook only in those cases. @@ -57,6 +64,9 @@ The following list is a summary of the main functionalities of Amazon SageMaker #### Zero Script Change +You can use your own training script while using [AWS Deep Learning Containers (DLC)](https://aws.amazon.com/machine-learning/containers/) in TensorFlow, PyTorch, MXNet and XGBoost frameworks. The AWS DLCs enable you to use Debugger with no changes to your training script, by automatically adding SageMaker Debugger's Hook. +The following table shows currently supported versions of DLC. + | Framework | Version | | --- | --- | | [TensorFlow](tensorflow.md) | 1.15, 2.1 | @@ -64,6 +74,9 @@ The following list is a summary of the main functionalities of Amazon SageMaker | [PyTorch](pytorch.md) | 1.3, 1.4 | | [XGBoost](xgboost.md) | >=0.90-2 [As Built-in algorithm](xgboost.md#use-xgboost-as-a-built-in-algorithm)| +*Note*: For the TensorFlow framework, Zero Script Change experience is supported +when using [AWS Deep Learning Containers (DLC)](https://aws.amazon.com/machine-learning/containers/). + #### Bring your own training container | Framework | Versions | @@ -109,11 +122,11 @@ that training is progressing as expected. A rule might check for vanishing gradients, or exploding tensor values, or poor weight initialization. Rules are attached to CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. Amazon SageMaker Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following: -- **SageMaker Zero Script Change**: Here you specify which rules to use when setting up the estimator and run your existing script, no changes needed. See [Running a Rule with Zero Script Change on SageMaker](#running-a-rule-with-zero-script-change-on-sageMaker). -- **SageMaker Bring Your Own Container**: Here you specify the rules to use, and modify your training script minimally to enable SageMaker Debugger. -- **Non-SageMaker**: Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. See [Running Locally](#running-locally). +- **SageMaker Zero Script Change**: Here you specify which rules to use when setting up the estimator and run your existing script without no change. For an example of how to [Running a Rule with Zero Script Change on SageMaker](#running-a-rule-with-zero-script-change-on-sageMaker). +- **SageMaker Bring Your Own Container**: Here you specify the rules to use and modify your training script minimally to enable SageMaker Debugger. +- **Non-SageMaker**: Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. For an example, see [Running Locally](#running-locally). -The reason for different setups is that SageMaker Zero-Script-Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add our Hook to the training job and save requested tensors automatically. +The reason for different setups is that SageMaker Zero-Script-Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add the `Hook` to the training job and save requested tensors automatically. These framework forks are not available in custom containers or non-SM environments, so you must modify your training script in these environments. @@ -193,8 +206,8 @@ print(f"Loss values during evaluation were {trial.tensor('CrossEntropyLoss:0').v | --- | --- | | [SageMaker Training](docs/sagemaker.md) | SageMaker users, we recommend you start with this page on how to run SageMaker training jobs with SageMaker Debugger | | Frameworks
  • [TensorFlow](docs/tensorflow.md)
  • [PyTorch](docs/pytorch.md)
  • [MXNet](docs/mxnet.md)
  • [XGBoost](docs/xgboost.md)
| See the frameworks pages for details on what's supported and how to modify your training script if applicable | -| [APIs for Saving Tensors](docs/api.md) | Full description of our APIs on saving tensors | -| [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by our APIs which allows you to perform interactive exploration of tensors saved as well as to write your own Rules monitoring your training jobs. | +| [APIs for Saving Tensors](docs/api.md) | Full description of the APIs for saving tensors | +| [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by the APIs that enable you to perform interactive exploration of tensors saved as well as to write your own Rules monitoring your training jobs. | ## SageMaker Debugger in action - Using SageMaker Debugger with XGBoost in SageMaker Studio to save feature importance values and plot them in a notebook during training. ![](docs/resources/xgboost_feature_importance.png?raw=true) From 23aab4ea5ba33ba72823c7b8ea64ed1c16921b5b Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Mon, 27 Apr 2020 13:26:54 -0700 Subject: [PATCH 07/35] Update README.md Co-Authored-By: Aaron Markham --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index 8e849b639..60f1b45e2 100644 --- a/README.md +++ b/README.md @@ -206,8 +206,13 @@ print(f"Loss values during evaluation were {trial.tensor('CrossEntropyLoss:0').v | --- | --- | | [SageMaker Training](docs/sagemaker.md) | SageMaker users, we recommend you start with this page on how to run SageMaker training jobs with SageMaker Debugger | | Frameworks
  • [TensorFlow](docs/tensorflow.md)
  • [PyTorch](docs/pytorch.md)
  • [MXNet](docs/mxnet.md)
  • [XGBoost](docs/xgboost.md)
| See the frameworks pages for details on what's supported and how to modify your training script if applicable | +<<<<<<< HEAD | [APIs for Saving Tensors](docs/api.md) | Full description of the APIs for saving tensors | | [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by the APIs that enable you to perform interactive exploration of tensors saved as well as to write your own Rules monitoring your training jobs. | +======= +| [APIs for Saving Tensors](docs/api.md) | Full description of our APIs on saving tensors | +| [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by the APIs that enable you to perform interactive exploration of tensors saved, as well as to write your own Rules monitoring your training jobs. | +>>>>>>> 15ed2dc... Update README.md ## SageMaker Debugger in action - Using SageMaker Debugger with XGBoost in SageMaker Studio to save feature importance values and plot them in a notebook during training. ![](docs/resources/xgboost_feature_importance.png?raw=true) From d6b74cb8c87e5465cb4226f3ea74921082ed4ed0 Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Mon, 27 Apr 2020 18:25:46 -0700 Subject: [PATCH 08/35] fix minor things --- README.md | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 60f1b45e2..5630dd0d5 100644 --- a/README.md +++ b/README.md @@ -13,11 +13,11 @@ ## Overview [Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) automates the debugging process of machine learning training jobs. From training jobs, Debugger allows you to -run your own training script (Zero Script Change experience) using Debugger built-in features---`Hook` and `Rule`---to capture tensors, +run your own training script (Zero Script Change experience) using Debugger built-in features—`Hook` and `Rule`—to capture tensors, have flexibility to build customized `Hook`s and `Rule`s for configuring tensors as you want, -and make the tensors available for analysis by saving in S3, +and make the tensors available for analysis by saving in an [Amazon S3](https://aws.amazon.com/s3/?nc=sn&loc=0) bucket, all through a flexible and powerful API. -The `smdebug` library powers Debugger by calling the saved tensors in S3 during the training job. +The `smdebug` library powers Debugger by calling the saved tensors from the S3 bucket during the training job. `smdebug` retrieves and filters the tensors generated from Debugger such as gradients, weights, and biases. `smdebug` helps you develop better, faster and cheaper models by tracing the tensors and iterative model pruning. @@ -54,18 +54,18 @@ See [How it work](#how-it-works) for more details. - Bug fix: Pytorch: Register only if tensors require gradients – Users were observing a crash when training with pre-trained embeddings which does not need gradient updates. This fix checks if a gradient update is required and registers a backward hook only in those cases. -#### Setting up SageMaker Debugger +### Setting up SageMaker Debugger -- `smdebug` library runs on Python 3.x. Install `smdebug` through: +`smdebug` library runs on Python 3.x. Install `smdebug` through: - ``` - pip install smdebug - ``` +``` +pip install smdebug +``` #### Zero Script Change -You can use your own training script while using [AWS Deep Learning Containers (DLC)](https://aws.amazon.com/machine-learning/containers/) in TensorFlow, PyTorch, MXNet and XGBoost frameworks. The AWS DLCs enable you to use Debugger with no changes to your training script, by automatically adding SageMaker Debugger's Hook. -The following table shows currently supported versions of DLC. +You can use your own training script while using [AWS Deep Learning Containers (DLC)](https://aws.amazon.com/machine-learning/containers/) in TensorFlow, PyTorch, MXNet and XGBoost frameworks. The AWS DLCs enable you to use Debugger with no changes to your training script by automatically adding SageMaker Debugger's `Hook`. +The following table shows currently supported versions of the four frameworks for Zero Script Change experience. | Framework | Version | | --- | --- | @@ -74,8 +74,6 @@ The following table shows currently supported versions of DLC. | [PyTorch](pytorch.md) | 1.3, 1.4 | | [XGBoost](xgboost.md) | >=0.90-2 [As Built-in algorithm](xgboost.md#use-xgboost-as-a-built-in-algorithm)| -*Note*: For the TensorFlow framework, Zero Script Change experience is supported -when using [AWS Deep Learning Containers (DLC)](https://aws.amazon.com/machine-learning/containers/). #### Bring your own training container @@ -179,7 +177,7 @@ If you want greater configuration and control, we offer that too. Head over [her ### Running Locally Requires Python 3.6+, and this example uses tf.keras. -To use Amazon SageMaker Debugger, simply add a callback hook: +To use Amazon SageMaker Debugger, simply add a callback `hook`: ```python import smdebug.tensorflow as smd hook = smd.KerasHook(out_dir='~/smd_outputs/') From 4821dedc7fc3827c6768a715735acd1a3e92c007 Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Tue, 28 Apr 2020 18:12:41 -0700 Subject: [PATCH 09/35] re-arange and edit README and sagemaker markdown files --- README.md | 51 ++++++++++++++++++++++++++++++++--------------- docs/sagemaker.md | 42 -------------------------------------- 2 files changed, 35 insertions(+), 58 deletions(-) diff --git a/README.md b/README.md index 5630dd0d5..5f14c3ca7 100644 --- a/README.md +++ b/README.md @@ -62,6 +62,9 @@ See [How it work](#how-it-works) for more details. pip install smdebug ``` +### Supported Frameworks +There are two ways in which you can enable SageMaker Debugger while training on SageMaker. + #### Zero Script Change You can use your own training script while using [AWS Deep Learning Containers (DLC)](https://aws.amazon.com/machine-learning/containers/) in TensorFlow, PyTorch, MXNet and XGBoost frameworks. The AWS DLCs enable you to use Debugger with no changes to your training script by automatically adding SageMaker Debugger's `Hook`. @@ -69,14 +72,19 @@ The following table shows currently supported versions of the four frameworks fo | Framework | Version | | --- | --- | -| [TensorFlow](tensorflow.md) | 1.15, 2.1 | +| [TensorFlow](tensorflow.md) | 1.15, 1.15.2, 2.1 | | [MXNet](mxnet.md) | 1.6 | | [PyTorch](pytorch.md) | 1.3, 1.4 | | [XGBoost](xgboost.md) | >=0.90-2 [As Built-in algorithm](xgboost.md#use-xgboost-as-a-built-in-algorithm)| +For the full list and information of the AWS DLCs, see [Deep Learning Containers Images](https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html#deep-learning-containers-images-table). + #### Bring your own training container +`smdebug` supports frameworks other than the ones listed in the previous Zero Script Change section. You can orchestrate your training script by ading a few lines. +Currently supported versions of frameworks are listed in the following table. + | Framework | Versions | | --- | --- | | [TensorFlow](tensorflow.md) | 1.14. 1.15, 2.0.1, 2.1.0 | @@ -112,19 +120,26 @@ The following table shows currently supported versions of the four frameworks fo - ## How It Works Amazon SageMaker Debugger uses the construct of a `Hook` to save the values of requested tensors throughout the training process. You can then setup a `Rule` job which simultaneously monitors and validates these tensors to ensure that training is progressing as expected. -A rule might check for vanishing gradients, or exploding tensor values, or poor weight initialization. Rules are attached to CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. - -Amazon SageMaker Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following: -- **SageMaker Zero Script Change**: Here you specify which rules to use when setting up the estimator and run your existing script without no change. For an example of how to [Running a Rule with Zero Script Change on SageMaker](#running-a-rule-with-zero-script-change-on-sageMaker). -- **SageMaker Bring Your Own Container**: Here you specify the rules to use and modify your training script minimally to enable SageMaker Debugger. -- **Non-SageMaker**: Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. For an example, see [Running Locally](#running-locally). - -The reason for different setups is that SageMaker Zero-Script-Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add the `Hook` to the training job and save requested tensors automatically. +A `Rule` checks for vanishing gradients, exploding tensor values, or poor weight initialization. `Rule`s are attached to CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. + +Amazon SageMaker Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following three cases. +- **SageMaker Zero Script Change**: +Here you specify which rules to use when setting up the estimator and run your existing script without no change. For an example of this, see [Running a Rule with Zero Script Change on SageMaker](#running-a-rule-with-zero-script-change-on-sageMaker). +- **SageMaker Bring Your Own Container**: +To use Debugger with your own script on your own container, make some minimal modifications to your training script to add SageMaker Debugger's `Hook`. +Refer to the following instruction pages to set up Debugger in your preferred framework. + - [TensorFlow](tensorflow.md) + - [PyTorch](pytorch.md) + - [MXNet](mxnet.md) + - [XGBoost](xgboost.md) +- **Non-SageMaker**: +Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. For an example of this, see [Running Locally](#running-locally). + +The reason for different setups is that SageMaker Zero Script Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add the `Hook` to the training job and save requested tensors automatically. These framework forks are not available in custom containers or non-SM environments, so you must modify your training script in these environments. @@ -133,7 +148,8 @@ These framework forks are not available in custom containers or non-SM environme Example notebooks demonstrating different functionalities of SageMaker Debugger are provided [here](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger). ### Running a Rule with Zero Script Change on SageMaker -This example uses a zero-script-change experience, where you can use your training script as-is. Refer to [Running SageMaker jobs with Amazon SageMaker Debugger](docs/sagemaker.md) for more details on this. +This example shows a how to use SageMaker Debugger with Zero Script Change. + ```python import sagemaker as sm from sagemaker.debugger import rule_configs, Rule, CollectionConfig @@ -152,7 +168,7 @@ rule = Rule.sagemaker( # Pass the rule to the estimator sagemaker_simple_estimator = sm.tensorflow.TensorFlow( - entry_point="script.py", + entry_point="script.py", #replace script.py to your own training script role=sm.get_execution_role(), framework_version="1.15", py_version="py3", @@ -169,13 +185,16 @@ print(f"Saved these tensors: {trial.tensor_names()}") print(f"Loss values during evaluation were {trial.tensor('CrossEntropyLoss:0').values(mode=smd.modes.EVAL)}") ``` -That's it! Amazon SageMaker will automatically monitor your training job for you with the Rules specified and create a CloudWatch -event which tracks the status of the Rule, so you can take any action based on them. +That's it! When you configure the `sagemaker_simple_estimator`, +you designate the `entry_point` to your training script python file. +When you run the `sagemaker_simple_estimator.fit()` API, +Amazon SageMaker will automatically monitor your training job for you with the `Rule`s specified and create a `CloudWatch` event that tracks the status of the `Rule`, +so you can take any action based on them. -If you want greater configuration and control, we offer that too. Head over [here](docs/sagemaker.md) for more information. +If you want greater configuration and control, see [SageMaker page](docs/sagemaker.md) for more information. ### Running Locally -Requires Python 3.6+, and this example uses tf.keras. +Requires Python 3.6+ and this example uses tf.keras. To use Amazon SageMaker Debugger, simply add a callback `hook`: ```python diff --git a/docs/sagemaker.md b/docs/sagemaker.md index 94d93b837..2ff46ec41 100644 --- a/docs/sagemaker.md +++ b/docs/sagemaker.md @@ -1,9 +1,6 @@ ## Running SageMaker jobs with Amazon SageMaker Debugger ## Outline -- [Enabling SageMaker Debugger](#enabling-sagemaker-debugger) - - [Zero Script Change](#zero-script-change) - - [Bring your own training container](#bring-your-own-training-container) - [Configuring SageMaker Debugger](#configuring-sagemaker-debugger) - [Saving data](#saving-data) - [Saving built-in collections that we manage](#saving-built-in-collections-that-we-manage) @@ -17,45 +14,6 @@ - [TensorBoard Visualization](#tensorboard-visualization) - [Example Notebooks](#example-notebooks) -## Enabling SageMaker Debugger -There are two ways in which you can enable SageMaker Debugger while training on SageMaker. - -### Zero Script Change -We have equipped the official Framework containers on SageMaker with custom versions of supported frameworks TensorFlow, PyTorch, MXNet and XGBoost. These containers enable you to use SageMaker Debugger with no changes to your training script, by automatically adding [SageMaker Debugger's Hook](api.md#glossary). - -Here's a list of frameworks and versions which support this experience. - -| Framework | Version | -| --- | --- | -| [TensorFlow](tensorflow.md) | 1.15 | -| [MXNet](mxnet.md) | 1.6 | -| [PyTorch](pytorch.md) | 1.3 | -| [XGBoost](xgboost.md) | >=0.90-2 [As Built-in algorithm](xgboost.md#use-xgboost-as-a-built-in-algorithm)| - -More details for the deep learning frameworks on which containers these are can be found here: [SageMaker Framework Containers](https://docs.aws.amazon.com/sagemaker/latest/dg/pre-built-containers-frameworks-deep-learning.html) and [AWS Deep Learning Containers](https://aws.amazon.com/machine-learning/containers/). You do not have to specify any training container image if you want to use them on SageMaker. You only need to specify the version above to use these containers. - -### Bring your own training container - -This library `smdebug` itself supports versions other than the ones listed above. If you want to use SageMaker Debugger with a version different from the above, you will have to orchestrate your training script with a few lines. Before we discuss how these changes look like, let us take a look at the versions supported. - -| Framework | Versions | -| --- | --- | -| [TensorFlow](tensorflow.md) | 1.13, 1.14, 1.15 | -| Keras (with TensorFlow backend) | 2.3 | -| [MXNet](mxnet.md) | 1.4, 1.5, 1.6 | -| [PyTorch](pytorch.md) | 1.2, 1.3 | -| [XGBoost](xgboost.md) | [As Framework](xgboost.md#use-xgboost-as-a-framework) | - -#### Setting up SageMaker Debugger with your script on your container - -- Ensure that you are using Python3 runtime as `smdebug` only supports Python3. -- Install `smdebug` binary through `pip install smdebug` -- Make some minimal modifications to your training script to add SageMaker Debugger's Hook. Please refer to the framework pages linked below for instructions on how to do that. - - [TensorFlow](tensorflow.md) - - [PyTorch](pytorch.md) - - [MXNet](mxnet.md) - - [XGBoost](xgboost.md) - ## Configuring SageMaker Debugger Regardless of which of the two above ways you have enabled SageMaker Debugger, you can configure it using the SageMaker python SDK. There are two aspects to this configuration. From b1170d53113d9c3dd4d8572f7da09e717d31d732 Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Tue, 28 Apr 2020 18:28:19 -0700 Subject: [PATCH 10/35] fixing few typos --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 5f14c3ca7..c4b66db6b 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ The following list is a summary of the main functionalities of Amazon SageMaker - Distributed training support - TensorBoard support -See [How it work](#how-it-works) for more details. +See [How it works](#how-it-works) for more details. ## Support @@ -82,7 +82,7 @@ For the full list and information of the AWS DLCs, see [Deep Learning Containers #### Bring your own training container -`smdebug` supports frameworks other than the ones listed in the previous Zero Script Change section. You can orchestrate your training script by ading a few lines. +`smdebug` supports frameworks other than the ones listed in the previous Zero Script Change section. You can orchestrate your training script by adding a few lines. Currently supported versions of frameworks are listed in the following table. | Framework | Versions | @@ -186,12 +186,12 @@ print(f"Loss values during evaluation were {trial.tensor('CrossEntropyLoss:0').v ``` That's it! When you configure the `sagemaker_simple_estimator`, -you designate the `entry_point` to your training script python file. +you simply specify the `entry_point` to your training script python file. When you run the `sagemaker_simple_estimator.fit()` API, Amazon SageMaker will automatically monitor your training job for you with the `Rule`s specified and create a `CloudWatch` event that tracks the status of the `Rule`, so you can take any action based on them. -If you want greater configuration and control, see [SageMaker page](docs/sagemaker.md) for more information. +If you want a greater configuration and control, see [Running SageMaker jobs with Amazon SageMaker Debugger](docs/sagemaker.md) for more information. ### Running Locally Requires Python 3.6+ and this example uses tf.keras. From 4c5cbaf03ca055a15c20c0a23042229875223709 Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Wed, 29 Apr 2020 17:54:38 -0700 Subject: [PATCH 11/35] update README.md / add BYOC example --- README.md | 71 ++++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 57 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index c4b66db6b..7135a52b3 100644 --- a/README.md +++ b/README.md @@ -63,11 +63,11 @@ pip install smdebug ``` ### Supported Frameworks -There are two ways in which you can enable SageMaker Debugger while training on SageMaker. +There are two ways in which you can enable SageMaker Debugger while training on SageMaker—Zero Script Change and Bring Your Own Training Container. #### Zero Script Change -You can use your own training script while using [AWS Deep Learning Containers (DLC)](https://aws.amazon.com/machine-learning/containers/) in TensorFlow, PyTorch, MXNet and XGBoost frameworks. The AWS DLCs enable you to use Debugger with no changes to your training script by automatically adding SageMaker Debugger's `Hook`. +You can use your own training script while using [AWS Deep Learning Containers (DLC)](https://aws.amazon.com/machine-learning/containers/) in TensorFlow, PyTorch, MXNet, and XGBoost frameworks. The AWS DLCs enable you to use Debugger with no changes to your training script by automatically adding SageMaker Debugger's `Hook`. The following table shows currently supported versions of the four frameworks for Zero Script Change experience. | Framework | Version | @@ -80,9 +80,9 @@ The following table shows currently supported versions of the four frameworks fo For the full list and information of the AWS DLCs, see [Deep Learning Containers Images](https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html#deep-learning-containers-images-table). -#### Bring your own training container +#### Bring Your Own Training Container -`smdebug` supports frameworks other than the ones listed in the previous Zero Script Change section. You can orchestrate your training script by adding a few lines. +`smdebug` supports frameworks other than the ones listed in the previous Zero Script Change section. You can use your own training script by adding a minimal modification. Currently supported versions of frameworks are listed in the following table. | Framework | Versions | @@ -93,7 +93,9 @@ Currently supported versions of frameworks are listed in the following table. | [PyTorch](pytorch.md) | 1.2, 1.3, 1.4 | | [XGBoost](xgboost.md) | [As Framework](xgboost.md#use-xgboost-as-a-framework) | -#### Support for Distributed Training and Known Limitations +For more information and examples, see [SageMaker Bring Your Own Container](#sagemaker-bring-your-own-container). + +### Support for Distributed Training and Known Limitations @@ -124,31 +126,47 @@ Currently supported versions of frameworks are listed in the following table. Amazon SageMaker Debugger uses the construct of a `Hook` to save the values of requested tensors throughout the training process. You can then setup a `Rule` job which simultaneously monitors and validates these tensors to ensure that training is progressing as expected. -A `Rule` checks for vanishing gradients, exploding tensor values, or poor weight initialization. `Rule`s are attached to CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. + +A `Rule` checks for vanishing gradients, exploding tensor values, or poor weight initialization. `Rule`s are attached to CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. +You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. Amazon SageMaker Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following three cases. -- **SageMaker Zero Script Change**: + +### Using SageMaker Debugger with Zero Script Change of Your Training Script + Here you specify which rules to use when setting up the estimator and run your existing script without no change. For an example of this, see [Running a Rule with Zero Script Change on SageMaker](#running-a-rule-with-zero-script-change-on-sageMaker). -- **SageMaker Bring Your Own Container**: -To use Debugger with your own script on your own container, make some minimal modifications to your training script to add SageMaker Debugger's `Hook`. + +### Using SageMaker Debugger on Bring Your Own Container + +You can use Debugger with your training script on your own container making only a minimal modification to your training script to add SageMaker Debugger's `Hook`. +For an example template of code to use Debugger on your own container in TensorFlow 2.x frameworks, see [Running on Your Own Container](#Running-on-Your-Own-Container). + Refer to the following instruction pages to set up Debugger in your preferred framework. - [TensorFlow](tensorflow.md) - [PyTorch](pytorch.md) - [MXNet](mxnet.md) - [XGBoost](xgboost.md) -- **Non-SageMaker**: + +### Non-SageMaker + Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. For an example of this, see [Running Locally](#running-locally). The reason for different setups is that SageMaker Zero Script Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add the `Hook` to the training job and save requested tensors automatically. These framework forks are not available in custom containers or non-SM environments, so you must modify your training script in these environments. +** not on SageMaker environments like DLC + ## Examples -### Notebooks -Example notebooks demonstrating different functionalities of SageMaker Debugger are provided [here](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger). -### Running a Rule with Zero Script Change on SageMaker -This example shows a how to use SageMaker Debugger with Zero Script Change. +### SageMaker Notebook Examples + +To find a collection of demonstrations using SageMaker Debugger, see [SageMaker Debugger Example Notebooks](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger). + +### Running a Rule with Zero Script Change + +This example shows a how to use SageMaker Debugger with Zero Script Change of +your training script on a SageMaker DLC. ```python import sagemaker as sm @@ -193,6 +211,31 @@ so you can take any action based on them. If you want a greater configuration and control, see [Running SageMaker jobs with Amazon SageMaker Debugger](docs/sagemaker.md) for more information. +### Running on Your Own Container + +in TF 2.x framework using GradientTape + +```python +import smdebug.tensorflow as smd +hook = smd.KerasHook(out_dir=args.out_dir) + +model = tf.keras.models.Sequential([ ... ]) + for epoch in range(n_epochs): + for data, labels in dataset: + dataset_labels = labels + # wrap the tape to capture tensors + with hook.wrap_tape(tf.GradientTape(persistent=True)) as tape: + logits = model(data, training=True) # (32,10) + loss_value = cce(labels, logits) + grads = tape.gradient(loss_value, model.variables) + opt.apply_gradients(zip(grads, model.variables)) + acc = train_acc_metric(dataset_labels, logits) + # manually save metric values + hook.record_tensor_value(tensor_name="accuracy", tensor_value=acc) +``` + +To see the full script of this, refer to the [tf_keras_gradienttape.py](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_gradienttape.py) example script. + ### Running Locally Requires Python 3.6+ and this example uses tf.keras. From ca3e7a9aa21c548957b9330e3d9af3a2c23706d7 Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Wed, 29 Apr 2020 18:11:06 -0700 Subject: [PATCH 12/35] minor fix --- README.md | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 7135a52b3..688c7819f 100644 --- a/README.md +++ b/README.md @@ -132,29 +132,26 @@ You can configure any action on the CloudWatch event, such as to stop the traini Amazon SageMaker Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following three cases. -### Using SageMaker Debugger with Zero Script Change of Your Training Script +#### Using SageMaker Debugger with Zero Script Change of Your Training Script Here you specify which rules to use when setting up the estimator and run your existing script without no change. For an example of this, see [Running a Rule with Zero Script Change on SageMaker](#running-a-rule-with-zero-script-change-on-sageMaker). -### Using SageMaker Debugger on Bring Your Own Container +#### Using SageMaker Debugger on Bring Your Own Container You can use Debugger with your training script on your own container making only a minimal modification to your training script to add SageMaker Debugger's `Hook`. For an example template of code to use Debugger on your own container in TensorFlow 2.x frameworks, see [Running on Your Own Container](#Running-on-Your-Own-Container). - -Refer to the following instruction pages to set up Debugger in your preferred framework. +See the following instruction pages to set up Debugger in your preferred framework. - [TensorFlow](tensorflow.md) - [PyTorch](pytorch.md) - [MXNet](mxnet.md) - [XGBoost](xgboost.md) -### Non-SageMaker +#### Using SageMaker Debugger on a Non-SageMaker Environment -Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger. For an example of this, see [Running Locally](#running-locally). +Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger on a non-SageMaker Environment such as your local machine. For an example of this, see [Running Locally](#running-locally). The reason for different setups is that SageMaker Zero Script Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add the `Hook` to the training job and save requested tensors automatically. -These framework forks are not available in custom containers or non-SM environments, so you must modify your training script in these environments. - -** not on SageMaker environments like DLC +These framework forks are not available in custom containers or non-SageMaker environments, so you must modify your training script in these environments. ## Examples @@ -163,7 +160,7 @@ These framework forks are not available in custom containers or non-SM environme To find a collection of demonstrations using SageMaker Debugger, see [SageMaker Debugger Example Notebooks](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger). -### Running a Rule with Zero Script Change +#### Running a Rule with Zero Script Change This example shows a how to use SageMaker Debugger with Zero Script Change of your training script on a SageMaker DLC. @@ -211,7 +208,7 @@ so you can take any action based on them. If you want a greater configuration and control, see [Running SageMaker jobs with Amazon SageMaker Debugger](docs/sagemaker.md) for more information. -### Running on Your Own Container +#### Running on Your Own Container in TF 2.x framework using GradientTape @@ -234,9 +231,9 @@ model = tf.keras.models.Sequential([ ... ]) hook.record_tensor_value(tensor_name="accuracy", tensor_value=acc) ``` -To see the full script of this, refer to the [tf_keras_gradienttape.py](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_gradienttape.py) example script. +To see a full script of this, refer to the [tf_keras_gradienttape.py](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_gradienttape.py) example script. -### Running Locally +#### Running Locally Requires Python 3.6+ and this example uses tf.keras. To use Amazon SageMaker Debugger, simply add a callback `hook`: From 5e549aaceb7c30451c7a1f82b5a365a9499f886c Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Wed, 29 Apr 2020 18:21:33 -0700 Subject: [PATCH 13/35] fixed links --- README.md | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 688c7819f..4f1faa77e 100644 --- a/README.md +++ b/README.md @@ -72,10 +72,10 @@ The following table shows currently supported versions of the four frameworks fo | Framework | Version | | --- | --- | -| [TensorFlow](tensorflow.md) | 1.15, 1.15.2, 2.1 | -| [MXNet](mxnet.md) | 1.6 | -| [PyTorch](pytorch.md) | 1.3, 1.4 | -| [XGBoost](xgboost.md) | >=0.90-2 [As Built-in algorithm](xgboost.md#use-xgboost-as-a-built-in-algorithm)| +| [TensorFlow](docs/tensorflow.md) | 1.15, 1.15.2, 2.1 | +| [MXNet](docs/mxnet.md) | 1.6 | +| [PyTorch](docs/pytorch.md) | 1.3, 1.4 | +| [XGBoost](docs/xgboost.md) | >=0.90-2 [As Built-in algorithm](xgboost.md#use-xgboost-as-a-built-in-algorithm)| For the full list and information of the AWS DLCs, see [Deep Learning Containers Images](https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html#deep-learning-containers-images-table). @@ -87,13 +87,11 @@ Currently supported versions of frameworks are listed in the following table. | Framework | Versions | | --- | --- | -| [TensorFlow](tensorflow.md) | 1.14. 1.15, 2.0.1, 2.1.0 | +| [TensorFlow](docs/tensorflow.md) | 1.14. 1.15, 2.0.1, 2.1.0 | | Keras (with TensorFlow backend) | 2.3 | -| [MXNet](mxnet.md) | 1.4, 1.5, 1.6 | -| [PyTorch](pytorch.md) | 1.2, 1.3, 1.4 | -| [XGBoost](xgboost.md) | [As Framework](xgboost.md#use-xgboost-as-a-framework) | - -For more information and examples, see [SageMaker Bring Your Own Container](#sagemaker-bring-your-own-container). +| [MXNet](docs/mxnet.md) | 1.4, 1.5, 1.6 | +| [PyTorch](docs/pytorch.md) | 1.2, 1.3, 1.4 | +| [XGBoost](docs/xgboost.md) | [As Framework](xgboost.md#use-xgboost-as-a-framework) | ### Support for Distributed Training and Known Limitations @@ -141,10 +139,10 @@ Here you specify which rules to use when setting up the estimator and run your e You can use Debugger with your training script on your own container making only a minimal modification to your training script to add SageMaker Debugger's `Hook`. For an example template of code to use Debugger on your own container in TensorFlow 2.x frameworks, see [Running on Your Own Container](#Running-on-Your-Own-Container). See the following instruction pages to set up Debugger in your preferred framework. - - [TensorFlow](tensorflow.md) - - [PyTorch](pytorch.md) - - [MXNet](mxnet.md) - - [XGBoost](xgboost.md) + - [TensorFlow](docs/tensorflow.md) + - [PyTorch](docs/pytorch.md) + - [MXNet](docs/mxnet.md) + - [XGBoost](docs/xgboost.md) #### Using SageMaker Debugger on a Non-SageMaker Environment From 8f3a17110b1dd8991de7af2fa8fdee498a834f47 Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:02:02 -0700 Subject: [PATCH 14/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 4f1faa77e..415317467 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,7 @@ - [Overview](#overview) - [Support](#support) +- [Install](#install-sagemaker-debugger) - [How It Works](#how-it-works) - [Docs](#docs) - [Examples](#examples) From 52a2a90ac9709c0e290d70c305de50b5ffb92e79 Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:04:42 -0700 Subject: [PATCH 15/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 415317467..360649e08 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ The `smdebug` library powers Debugger by calling the saved tensors from the S3 b Through the model pruning process using Debugger and `smdebug`, you can iteratively identify the importance of weights, and cut neurons below a threshold you define. This process allows you to train the model with significantly fewer neurons, which means a lighter, more efficient, faster, and cheaper model without compromising accuracy. -Amazon SageMaker Debugger supports TensorFlow, PyTorch, MXNet, and XGBoost frameworks. +Debugger supports TensorFlow, PyTorch, MXNet, and XGBoost frameworks. The following list is a summary of the main functionalities of Amazon SageMaker Debugger: - Zero Script Change experience on SageMaker when using [supported containers](#support) From fedefc05b85c1fc4951c8ef249e2f114e9c79cdb Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:04:53 -0700 Subject: [PATCH 16/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 360649e08..55557ab3a 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ The `smdebug` library powers Debugger by calling the saved tensors from the S3 b Through the model pruning process using Debugger and `smdebug`, you can iteratively identify the importance of weights, and cut neurons below a threshold you define. This process allows you to train the model with significantly fewer neurons, which means a lighter, more efficient, faster, and cheaper model without compromising accuracy. Debugger supports TensorFlow, PyTorch, MXNet, and XGBoost frameworks. -The following list is a summary of the main functionalities of Amazon SageMaker Debugger: +The following list is a summary of the main functionalities of Debugger: - Zero Script Change experience on SageMaker when using [supported containers](#support) - Full visibility into any tensor part of the training process From ef0707d1195d8ca88f0c2cb0332ff0f183d8818a Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:05:59 -0700 Subject: [PATCH 17/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 55557ab3a..42c94df67 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ See [How it works](#how-it-works) for more details. - Bug fix: Pytorch: Register only if tensors require gradients – Users were observing a crash when training with pre-trained embeddings which does not need gradient updates. This fix checks if a gradient update is required and registers a backward hook only in those cases. -### Setting up SageMaker Debugger +## Install SageMaker Debugger `smdebug` library runs on Python 3.x. Install `smdebug` through: From 77e0d349d7ff2a03db3a352e3059ebde413410cb Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:06:37 -0700 Subject: [PATCH 18/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 42c94df67..269c31a53 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ pip install smdebug ``` ### Supported Frameworks -There are two ways in which you can enable SageMaker Debugger while training on SageMaker—Zero Script Change and Bring Your Own Training Container. +There are two ways in which you can enable SageMaker Debugger while training on SageMaker—Zero Script Change and Bring Your Own Training Container (BYOC). #### Zero Script Change From 3c37a480be54b48dbc5e147a4ba0d96363238f8e Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:07:32 -0700 Subject: [PATCH 19/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 269c31a53..8d916bebc 100644 --- a/README.md +++ b/README.md @@ -126,7 +126,7 @@ Currently supported versions of frameworks are listed in the following table. Amazon SageMaker Debugger uses the construct of a `Hook` to save the values of requested tensors throughout the training process. You can then setup a `Rule` job which simultaneously monitors and validates these tensors to ensure that training is progressing as expected. -A `Rule` checks for vanishing gradients, exploding tensor values, or poor weight initialization. `Rule`s are attached to CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. +A `Rule` checks for vanishing gradients, exploding tensor values, or poor weight initialization. `Rule`s are attached to Amazon CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. Amazon SageMaker Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following three cases. From 5dceb533057bbdf1f527937a856a40a0d2a60cef Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:07:44 -0700 Subject: [PATCH 20/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8d916bebc..ed2d04ad3 100644 --- a/README.md +++ b/README.md @@ -129,7 +129,7 @@ that training is progressing as expected. A `Rule` checks for vanishing gradients, exploding tensor values, or poor weight initialization. `Rule`s are attached to Amazon CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. -Amazon SageMaker Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following three cases. +Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following three cases. #### Using SageMaker Debugger with Zero Script Change of Your Training Script From 05039cc42025cf91de90ee6838d2f5a3c86457f9 Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:07:53 -0700 Subject: [PATCH 21/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ed2d04ad3..1e6bd3d0d 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ Here you specify which rules to use when setting up the estimator and run your e #### Using SageMaker Debugger on Bring Your Own Container -You can use Debugger with your training script on your own container making only a minimal modification to your training script to add SageMaker Debugger's `Hook`. +You can use Debugger with your training script on your own container making only a minimal modification to your training script to add Debugger's `Hook`. For an example template of code to use Debugger on your own container in TensorFlow 2.x frameworks, see [Running on Your Own Container](#Running-on-Your-Own-Container). See the following instruction pages to set up Debugger in your preferred framework. - [TensorFlow](docs/tensorflow.md) From 309c4a03f90710cfa471f1a85f729b3fd52d69ea Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:08:21 -0700 Subject: [PATCH 22/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1e6bd3d0d..acfeaa385 100644 --- a/README.md +++ b/README.md @@ -147,7 +147,7 @@ See the following instruction pages to set up Debugger in your preferred framewo #### Using SageMaker Debugger on a Non-SageMaker Environment -Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable SageMaker Debugger on a non-SageMaker Environment such as your local machine. For an example of this, see [Running Locally](#running-locally). +Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable Debugger on a non-SageMaker Environment such as your local machine. For an example of this, see [Running Locally](#running-locally). The reason for different setups is that SageMaker Zero Script Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add the `Hook` to the training job and save requested tensors automatically. These framework forks are not available in custom containers or non-SageMaker environments, so you must modify your training script in these environments. From 97a1fb3e600d57e334d3ffa23273f639b267881e Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:08:44 -0700 Subject: [PATCH 23/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index acfeaa385..b1618100a 100644 --- a/README.md +++ b/README.md @@ -149,7 +149,7 @@ See the following instruction pages to set up Debugger in your preferred framewo Here you write custom rules (or manually analyze the tensors) and modify your training script minimally to enable Debugger on a non-SageMaker Environment such as your local machine. For an example of this, see [Running Locally](#running-locally). -The reason for different setups is that SageMaker Zero Script Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add the `Hook` to the training job and save requested tensors automatically. +The reason for different setups is that Zero Script Change (via AWS Deep Learning Containers) uses custom framework forks of TensorFlow, PyTorch, MXNet, and XGBoost which add the `Hook` to the training job and save requested tensors automatically. These framework forks are not available in custom containers or non-SageMaker environments, so you must modify your training script in these environments. From a29c6c9504b63c230001cd9ccc1d3b79e5ff0379 Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:08:55 -0700 Subject: [PATCH 24/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b1618100a..8d9f7a121 100644 --- a/README.md +++ b/README.md @@ -157,7 +157,7 @@ These framework forks are not available in custom containers or non-SageMaker en ### SageMaker Notebook Examples -To find a collection of demonstrations using SageMaker Debugger, see [SageMaker Debugger Example Notebooks](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger). +To find a collection of demonstrations using Debugger, see [SageMaker Debugger Example Notebooks](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger). #### Running a Rule with Zero Script Change From 4733be3462047c51da38b6cb4889410269b0ab34 Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:09:04 -0700 Subject: [PATCH 25/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8d9f7a121..4a9ae3850 100644 --- a/README.md +++ b/README.md @@ -159,7 +159,7 @@ These framework forks are not available in custom containers or non-SageMaker en To find a collection of demonstrations using Debugger, see [SageMaker Debugger Example Notebooks](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-debugger). -#### Running a Rule with Zero Script Change +#### Run a Rule with Zero Script Change This example shows a how to use SageMaker Debugger with Zero Script Change of your training script on a SageMaker DLC. From 7781628a05e6319a59558c924c53b49edcc49085 Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:09:13 -0700 Subject: [PATCH 26/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4a9ae3850..0a992629f 100644 --- a/README.md +++ b/README.md @@ -161,7 +161,7 @@ To find a collection of demonstrations using Debugger, see [SageMaker Debugger E #### Run a Rule with Zero Script Change -This example shows a how to use SageMaker Debugger with Zero Script Change of +This example shows a how to use Debugger with Zero Script Change of your training script on a SageMaker DLC. ```python From 1e87842cc7c1dcb96770a2b0401a7c300b01dd92 Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:10:38 -0700 Subject: [PATCH 27/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0a992629f..1c7c4150e 100644 --- a/README.md +++ b/README.md @@ -202,7 +202,7 @@ print(f"Loss values during evaluation were {trial.tensor('CrossEntropyLoss:0').v That's it! When you configure the `sagemaker_simple_estimator`, you simply specify the `entry_point` to your training script python file. When you run the `sagemaker_simple_estimator.fit()` API, -Amazon SageMaker will automatically monitor your training job for you with the `Rule`s specified and create a `CloudWatch` event that tracks the status of the `Rule`, +SageMaker will automatically monitor your training job for you with the Rules specified and create a `CloudWatch` event that tracks the status of the Rule, so you can take any action based on them. If you want a greater configuration and control, see [Running SageMaker jobs with Amazon SageMaker Debugger](docs/sagemaker.md) for more information. From 9e3e14c1d03110c20eeec9e46aa9a76e1f6ab8e4 Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:10:50 -0700 Subject: [PATCH 28/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1c7c4150e..67f293fa3 100644 --- a/README.md +++ b/README.md @@ -205,7 +205,7 @@ When you run the `sagemaker_simple_estimator.fit()` API, SageMaker will automatically monitor your training job for you with the Rules specified and create a `CloudWatch` event that tracks the status of the Rule, so you can take any action based on them. -If you want a greater configuration and control, see [Running SageMaker jobs with Amazon SageMaker Debugger](docs/sagemaker.md) for more information. +If you want additional configuration and control, see [Running SageMaker jobs with Debugger](docs/sagemaker.md) for more information. #### Running on Your Own Container From ed1b75b9575be8b14645d211987a3f8106b7dc6e Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:10:58 -0700 Subject: [PATCH 29/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 67f293fa3..9507aa07d 100644 --- a/README.md +++ b/README.md @@ -207,7 +207,7 @@ so you can take any action based on them. If you want additional configuration and control, see [Running SageMaker jobs with Debugger](docs/sagemaker.md) for more information. -#### Running on Your Own Container +#### Run Debugger in Your Own Container in TF 2.x framework using GradientTape From 5059f997218db4ceef9ea08114c3105d6b88316d Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:11:10 -0700 Subject: [PATCH 30/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9507aa07d..514867094 100644 --- a/README.md +++ b/README.md @@ -232,7 +232,7 @@ model = tf.keras.models.Sequential([ ... ]) To see a full script of this, refer to the [tf_keras_gradienttape.py](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_gradienttape.py) example script. -#### Running Locally +#### Run Debugger Locally Requires Python 3.6+ and this example uses tf.keras. To use Amazon SageMaker Debugger, simply add a callback `hook`: From 7935b871923299e6ce9dbbdc2626b79d180359b6 Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 09:11:20 -0700 Subject: [PATCH 31/35] Update README.md Co-authored-by: Aaron Markham --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 514867094..c77e40646 100644 --- a/README.md +++ b/README.md @@ -235,7 +235,7 @@ To see a full script of this, refer to the [tf_keras_gradienttape.py](https://gi #### Run Debugger Locally Requires Python 3.6+ and this example uses tf.keras. -To use Amazon SageMaker Debugger, simply add a callback `hook`: +To use Debugger, simply add a callback `hook`: ```python import smdebug.tensorflow as smd hook = smd.KerasHook(out_dir='~/smd_outputs/') From fd467b4ca640b751a4189b7561f7939a5231aa05 Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Thu, 30 Apr 2020 10:04:11 -0700 Subject: [PATCH 32/35] a few changes and re-ordering --- README.md | 51 ++++++++++++++++++++++++++------------------------- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index c77e40646..9d2462cad 100644 --- a/README.md +++ b/README.md @@ -5,17 +5,18 @@ ## Table of Contents - [Overview](#overview) -- [Support](#support) +- [SageMaker Debugger in action](#sagemaker-debugger-in-action) - [Install](#install-sagemaker-debugger) - [How It Works](#how-it-works) -- [Docs](#docs) - [Examples](#examples) -- [SageMaker Debugger in action](#sagemaker-debugger-in-action) +- [Further Documentation](#further-documentation) +- [Release Notes](#release-notes) + ## Overview [Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html) automates the debugging process of machine learning training jobs. From training jobs, Debugger allows you to run your own training script (Zero Script Change experience) using Debugger built-in features—`Hook` and `Rule`—to capture tensors, -have flexibility to build customized `Hook`s and `Rule`s for configuring tensors as you want, +have flexibility to build customized Hooks and Rules for configuring tensors as you want, and make the tensors available for analysis by saving in an [Amazon S3](https://aws.amazon.com/s3/?nc=sn&loc=0) bucket, all through a flexible and powerful API. The `smdebug` library powers Debugger by calling the saved tensors from the S3 bucket during the training job. @@ -38,22 +39,11 @@ The following list is a summary of the main functionalities of Debugger: See [How it works](#how-it-works) for more details. -## Support - -### [Latest release v0.7.2](https://github.com/awslabs/sagemaker-debugger/releases) - - Introducing experimental support for TF 2.x training scripts using GradientTape - - With this update, weights, bias, loss, metrics, and gradients are captured by SageMaker Debugger. - GradientTape in TF 2.x captures these tensors from custom training jobs. An example of GradientTape implementation to a custom ResNet training script using TensorFlow's Keras interface is provided at [tf_keras_gradienttape.py](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_gradienttape.py). - GradientTape does not work with Zero Script Change experience at this time. - - *Note*: Training scripts using GradientTape for higher-order gradients or multiple tapes are not supported. Distributed training scripts that use GradientTape are not supported at this time. - - - Support `SyncOnReadVariable` in mirrored strategy - Fixes a bug that occurred because the `SyncOnRead` distributed variable was not supported with `smdebug`. Also enables the use of `smdebug` with training scripts using TF 2.x MirroredStrategy with the `fit()` API. - - - Turn off hook and write only from one worker for unsupported distributed training techniques – Fixes a crash when distributed training in PyTorch framework is implemented using generic multiprocessing library, which is not a method supported by `smdebug`. This fix handles this case and ensures that tensors are saved. +## SageMaker Debugger in action +- Use Debugger with XGBoost in SageMaker Studio to save feature importance values and plot them in a notebook during training. ![Debugger XGBoost Visualization Example](docs/resources/xgboost_feature_importance.png?raw=true) +- Use Debugger with TensorFlow in SageMaker Studio to run built-in rules and visualize the loss. ![Debugger TensorFlow Visualization Example](docs/resources/tensorflow_rules_loss.png?raw=true) - - Bug fix: Pytorch: Register only if tensors require gradients – Users were observing a crash when training with pre-trained embeddings which does not need gradient updates. This fix checks if a gradient update is required and registers a backward hook only in those cases. ## Install SageMaker Debugger @@ -63,7 +53,7 @@ See [How it works](#how-it-works) for more details. pip install smdebug ``` -### Supported Frameworks +### Debugger Usage and Supported Frameworks There are two ways in which you can enable SageMaker Debugger while training on SageMaker—Zero Script Change and Bring Your Own Training Container (BYOC). #### Zero Script Change @@ -126,7 +116,7 @@ Currently supported versions of frameworks are listed in the following table. Amazon SageMaker Debugger uses the construct of a `Hook` to save the values of requested tensors throughout the training process. You can then setup a `Rule` job which simultaneously monitors and validates these tensors to ensure that training is progressing as expected. -A `Rule` checks for vanishing gradients, exploding tensor values, or poor weight initialization. `Rule`s are attached to Amazon CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. +A `Rule` checks for vanishing gradients, exploding tensor values, or poor weight initialization. Rules are attached to Amazon CloudWatch events, so that when a rule is triggered it changes the state of the CloudWatch event. You can configure any action on the CloudWatch event, such as to stop the training job saving you time and money. Debugger can be used inside or outside of SageMaker. However the built-in rules that AWS provides are only available for SageMaker training. Scenarios of usage can be classified into the following three cases. @@ -141,8 +131,8 @@ You can use Debugger with your training script on your own container making only For an example template of code to use Debugger on your own container in TensorFlow 2.x frameworks, see [Running on Your Own Container](#Running-on-Your-Own-Container). See the following instruction pages to set up Debugger in your preferred framework. - [TensorFlow](docs/tensorflow.md) - - [PyTorch](docs/pytorch.md) - [MXNet](docs/mxnet.md) + - [PyTorch](docs/pytorch.md) - [XGBoost](docs/xgboost.md) #### Using SageMaker Debugger on a Non-SageMaker Environment @@ -256,7 +246,7 @@ print(f"Saved these tensors: {trial.tensor_names()}") print(f"Loss values during evaluation were {trial.tensor('CrossEntropyLoss:0').values(mode=smd.modes.EVAL)}") ``` -## Docs +## Further Documentation | Section | Description | | --- | --- | @@ -270,11 +260,22 @@ print(f"Loss values during evaluation were {trial.tensor('CrossEntropyLoss:0').v | [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by the APIs that enable you to perform interactive exploration of tensors saved, as well as to write your own Rules monitoring your training jobs. | >>>>>>> 15ed2dc... Update README.md -## SageMaker Debugger in action -- Using SageMaker Debugger with XGBoost in SageMaker Studio to save feature importance values and plot them in a notebook during training. ![](docs/resources/xgboost_feature_importance.png?raw=true) -- Using SageMaker Debugger with TensorFlow in SageMaker Studio to run built-in rules and visualize the loss. ![](docs/resources/tensorflow_rules_loss.png?raw=true) +## Release Notes + +### [Latest release v0.7.2](https://github.com/awslabs/sagemaker-debugger/releases) + + - Introducing experimental support for TF 2.x training scripts using GradientTape - + With this update, weights, bias, loss, metrics, and gradients are captured by SageMaker Debugger. + GradientTape in TF 2.x captures these tensors from custom training jobs. An example of GradientTape implementation to a custom ResNet training script using TensorFlow's Keras interface is provided at [tf_keras_gradienttape.py](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_gradienttape.py). + GradientTape does not work with Zero Script Change experience at this time. + + *Note*: Training scripts using GradientTape for higher-order gradients or multiple tapes are not supported. Distributed training scripts that use GradientTape are not supported at this time. + - Support `SyncOnReadVariable` in mirrored strategy - Fixes a bug that occurred because the `SyncOnRead` distributed variable was not supported with `smdebug`. Also enables the use of `smdebug` with training scripts using TF 2.x MirroredStrategy with the `fit()` API. + + - Turn off hook and write only from one worker for unsupported distributed training techniques – Fixes a crash when distributed training in PyTorch framework is implemented using generic multiprocessing library, which is not a method supported by `smdebug`. This fix handles this case and ensures that tensors are saved. + - Bug fix: Pytorch: Register only if tensors require gradients – Users were observing a crash when training with pre-trained embeddings which does not need gradient updates. This fix checks if a gradient update is required and registers a backward hook only in those cases. ## License This library is licensed under the Apache 2.0 License. From d08795b1e2ae74bc9ea03cfdd640f60c6f8db91b Mon Sep 17 00:00:00 2001 From: mchoi8739 Date: Thu, 30 Apr 2020 14:02:50 -0700 Subject: [PATCH 33/35] model pruning resnet image --- docs/resources/results_resnet.png | Bin 0 -> 99567 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 docs/resources/results_resnet.png diff --git a/docs/resources/results_resnet.png b/docs/resources/results_resnet.png new file mode 100644 index 0000000000000000000000000000000000000000..614b92b990b0d9e0c6d29267b55c94fc6a69bcaa GIT binary patch literal 99567 zcmeEucQl;c+BYFev>+k6C=or;jUb};M2T)BdL4E2o<#KCYY2kqeT)_noea?#y&I!< z<{QuHIp;iQegD7K!tI`2uYL9Fw}rk`lP9`EeFp;rgGf<9Mgs!_M;8O*RvaD<`b|s! zSOf+Jp^~+<^h-r)X~vh%APZ|da|{fH&_o?vUCkbfECUTit3W&j{N_-mW(E9DvUg)R zCh!y)KYdTRE3@5Qb*IcortL#n*kcIAbGwg_G1$T*iEfo9(krTW)**A=Nbj|a zRk)<(;F)^QjJGAm&!Ppz{KhXl7@l!HdOk9Ewl8d`!w(-Y2H+`}+^bmLThRa;8VUsP zd7zr0&6uU14BcKiU%6bjhGji}yoMFY%)tSp1O|rjx8&Y^Ovdx)%|{VFMP%wh z6VGwT-{}^A*RwIq{Z3_LSX8ALvGyEFfWa5p6>pB2GfQ~%^_CySGOXoxWr+-95=M^n z2ay=wMSRAncOEjvZ~@j2;I@ZLG+AF;oh%5Q_C55Zb&bN6x<9>b6i606K-qPnlAwm2 zCWK-lR!n+k&?<|GdIQms+=LVyr)U=DM-*>;KhA~U=J5Xbs#t+E^>dxP>X|d~HkHYX zZI1c(a#D78oElh!AL-`}b>9nddH8`7oBQ6H3%OuR_`CfIY$D4@JAI#R#=!)q3F?7r!ni5>)g&pg)Y0*ciT28nBSLmD~aB#W3Md45NI>>By51=iTnM@}Km6~&WzxwKjsFfG$u3sf>#do3#$(t+KJN~g^76*Gpj zhL|t?~~pKgV!4F`4hyG5Ylee%-RqE%!?FNB!gx_P@FE@L|rZ#;KkCW&!azQ0e} z`I6cuAng-){qZkc^SoD9OaQ8Z7lnDUR``&6r82fKBP(XFTg?H+@?x+xD+Zh=+!3LA zZ+TKkezd=>6yPF{f8$cg_w`muCv^qIsU)xyP)WayrS_dlgup$JQ-n2BLuM(KjQg zq7lu=kV}dQ@43I6xoo=3xm>vH7sk_>v?w2%kN23Lz3;IBJ9mO|0*{iUcC?K;QO+Zc z>LP+KiNA0*tv8J}ZMnI*Ik|zQd-`un&-4@Zy=w<;#DAVYA1M@tiPR_x+_q=87b{H! zryr-O6;8u~H>?kR&NaN~u2`-lu9}6Uy*~@-803hnH|aOMx(GWapbLvCB{r}OQ;Su5 zKj@dK3=oIhGZHh3;!$@mbEw^UIGYaf^&)JBc~yFOdMO`*H#0}P3)F|}M@U8}*OS*l zFv%j+$H1pwsc3PsD`P4Xozh_D&jl)k=c&zT5@@!Ctc2GEe+eHxQ4!)2_$u_wW}@eN z3Rs1QKe;iYG1@cu%qeB^$?=ngip~z|4@OkSgH{c%Dq??r>9>l8x#dOw;97dJM7wnV zhw6_L6-RQm_`OH`Dncn4LK{>LPiFi@{5>XjeLMVxWmPj(mQ*cM9iBE@O<7tvP`r60 zXkZB*{@RscO>dWF-!gTX;#q&YgX!lByUMA>hE4`9X^Qm3zGAb?w@3PE`f~alT*3x* z2FC`I2CRDb4G@(OOV!%r%E20A`&o-wThllEgBEqF%>&4~dj<>!u?YQX{yOCLHbP_D zug12@+X7@ZRLfgcUf;U)+#Sz3W#{FV<3Q~C`cleL1AMQFd~8 zi@8TIM(ol(qf!sG?aWOmWUW`>`21Zm-B?{See&Nq+K}*sq#KEnZ zU*G?50@j&lAlK^b;=I(P-DKoa*c8$<Wg0x9Xya95*ivZh8y#h zkCXJc| ztiB4+#^5z^d2Y$b`9$1N)5J2y{4Vboq4hTvZWW9k2D_Sd%wp-y;a5!G1!X9lpDfdL zMe~P@P=~#jdzbXSE@UHw^}Y7{S(5rYLwCwakdeC)DfeO2iXS7w3`srXZGWt|Y&A5S zHg^qDd2@tOB;M7&fAnOEU7uo>kC;vvJX7rCZfa-B@>I2wF^Q&$4#3VKba)J4}6)5E%)u$yG(o#xincaD_qf8L4&21b&Ew^sZ7}~nJle}FW=?c7krmT zh6;j;_X~vzV~68L>hh0N3DX-a-j1^jDdxvJsMKW}ik!{di)U}tj8Yp-?M;VBX2xX2 zweVk&^Cs!^ZS~Dt{4zV-7rm^#TnjR3w~?xU^1MOgRDhq+#eB8;Yrt1+z1dvGSMR^b z_mk>`0xje%K>dEnhl;|wzJZ+=7 z>hYwIC{s2UXizwS<~aNgZYN0??(M+aqqmWkEJG8(FFKFep&}#JtBa}X;orjjvfd9Z z4%wvJi+Zzm7B*ZAhzy*d3({YK~`=?0%VONP@hstgGgO zRC#6&KvuRSv_p%_bV9U{MNdoh3=bTdO|BA&+G(hT((JkBPMzXCMoI@r(`jrGHk@^L z>+jilRtb7;rDps3l}%ezmHJGtA>t63VvFt>#Rp|2)7B6dJ#qc=FG#qc_4vx{r>1lo zLuw)9n;rH!f@6;3>pkUjUg2X|gzYr4x^*D{5ttSn5o{i*u+;wp{^*sarHhSA*Qd&m zDg8?WVtRT{j|+@z>6F^B%ChmeEqo)e$W<^C8mTNVa)N8Y& z0KUiLyFm4gmRM;I;9tH!AbY9pAlgX9Hf+ z0P<=*21cQ#Mm9!052es~y*@Hx%%LU^_yFRd}{Dd%Zh$PXm>x8BzIeXVI^aExD%)<} zyP#w6k;php+26B9MW}yPPt2y1toI2$cXnq4nLO`}-_tU{BqSE#ClAn>#>^?a)z(0G zLP9>e_AHTcv%2U`AnM(yrBtE^=1&YzF9J|_KcLxhqP94S<$0sFIw8)Nri-6HKRz3o zRUMEGW~2Xh5g0gbff1vRamq#WU>zv=WY?2ZrlJPRE(Qi>mbI3ytFEdFz!YT9^TrJH)||(~ z-Vxmz1LK(o0R3oh?)rw&!`{xp1>hmh^hXN-`uXNHFB9V*ON!(#%WPIjqW&zNUk^5_R^k3pkR<5p&0A5~qcXu9lejbpsCGS&FQBmF}e7t;o z+~^kEE}jmqZ#=jiT$ukd$lu11F?TU_wsv&2201X^jQi#-$jw!piRosdzkmH@o#r0a z|Cq_a<*(a9Z;Ux(W%3{BzHL`sA-YpYh)8;GcH%7jpga7ELdSJI{FkPJ4+vui89%(Z@k+ zEu*f5{(bA_Y|tmCi2m{TFTZb|iJEqVpTL76>u0_5E2J=P-3`FNX2ih6d;T9ESyXZYFqFT2?Z*3WE&kfi z=N7i^6YT#o%HPJ0VZpq`rFe^3>`!LAY0KCaMD({l|7B@`Ecka#6v-Y;KKw5Y{I5*SPOKaTsZNjuJeXNEsW_B;+7uYv_XV*7tk zA^?Lx=zkmyb{tC#C1w(8+Isr=b|R-vuVruucGu*@r1Jq!)ilpEZuIraxa7)gG!9PJ zHhOZ0yH;a0p&6dNu)vCuf`J%6_2vh(Ya_%6M+s|kz5l#!GITlvGig-EmZuqnB<+US zojHr{9EJxk*=-*}_it^~z#;8F*Dvm)mWCFF>)e#B zXBytbpym`c&42XkwL5vpi!B(`+L21I>dSf9{;CN4I$5WEnh-%P{+fbEpVo!P=Wf{n z6Yrb?ucUzmKCNeWi}S9ZH#SM8kZ(bTxAoAx2oP))av&Su9ZQH=dr>=X!(3`m`q0g0 zJjwH5wJ$=i`l`;ew`RjU>`r};S7ntDR0R``?_1#a9rjg^+BSgSbizz`L z=4V}(sWT^FIlacqm-@Q{?#HonABZ`{+l|#oC%`2GD?!z2Dh-3lN9(NhhwD7GHurCz z_#AZbIu8X$<901%5)6sz)lI$~ng11+ue|b(zX`Vs;7904s*qVt9#V;m&IEMg7SLzshv82H5fbimj00&2IIKJV zr`Bit=?uNx%sx_B)P>oOpY1CALQ>`^7oHtZawSCL`mqnS?t@QVD;RwJ92Z(M^Oci( zD5d>&cl?!C`!gwvMjw!xNsb0l~hG{c1a;6H6#?jAoSBvxP9*fR#h?=!B!Ga`KrxI$&Xzx2z*A z-1G3-AYbartG$T9@7ic*eU?{bEHebztVk`)K}VXm#z9R)T6%CA8lQh5h^?DoP*4{s zR?e&AF3ILGa|d+(jexiHRot1fgv zU-D61i+##-ENj>~b`IvUt+nT_pL6s~o#v!2?Gq~^3| z^-4WQa(<1&uU{H_`9A-HA|Jto+K&QpbOG+0jBIo+Rw`p2%&7oxAoEGXB{aq7sJjb8897~kP_WKH7yFuOMGITk)vLP+)ChQl2b^>{ zI-f6ZjmNzZr->Id4lv(~%XgAr^f_4qIgevS|Cj;ccF}ppY_CPmhdK~Ya>vCFvD0|K ze4z2UH6kDWX~Q>4YD$#yX#6H8WaF140ASre zv6oo&5pLV{LriCCL1qdZ&pj!S3KlV}i&Fc^9`_?E`+>4`+_0<&CS|D?<3_mn<*mIFf9UV7ZnjTaXl zm{GeQYpnv9V@bJS>@2mn09O;_SHRfT<~5F1`??vGDDlO%lw-MrS&uHPr(2$~+5Xq& z!ly?g$EWX@Nsc86e_ptzFHH_otpRH{8TUc1n;p9=Rxf+XRz}2%AlKF^3cvdujFNa- z>}|Q3l=Add+1t=1A>*7}@bkB0+l_FB>m_J8(mzQG=trJ0o{6a6;qXRQR@WQ@z;- zXBtTdL1JqRO6>a~5A?gu%87$Re0Z_oD1ic(gC8Zl`xG%of;#6<%A=wsj)F2A4=n}OGAw(ro;?&bVL zQT~GNrPo|^>-EEE$%Cr<57`*JQ(TWyCdW?puBGG$c(ULm;wHSP;YC*E%UW?_F)Ppp zq(`pGBK|n&%tG+e>oS*IqAdw88#{5Dj(M1amn0qx^ zLBR_13eAjj^W5mYl+|T7{vFr!$(tjKgTANK|dIo4GY;wjbF~*AQ+G zhD&+5Tqj=*1E_#7FT=&ry!BR-q)olTM^%hjJOZstbp|+0*7EDM!n1U=A?^YbwI>d< zL7L2R(d_&Ku-fEIPTLw??(PGaC9pdk=|@YuHAs)D&l4`4dDGr_n}D*19mRQ7WiVZD zN+Q_?O8%OuNul{{4mLRAE=m2sv`LL!NXc-WE!!%Qj?n{!%bgG`^YuCAndHdrdcZr0|x;Z6*hPUl6uX0fapn=c0%^R8D3;^b*d&!b< zekZbLKFg)emCl`aCi|-Ab{ChX#w7Na3HV%A?L@cdoAX_|h>hDIuNl)?bZYY?Y4Iw; zskKg99j3!-+r9iI_gD5Wezsom9p&NHu3(9`?)<83t}$%>#hb7+gBshbIC+;?Qs+Gt zDu0GBz4BrO>mOzX8sF6j#;ucG`Pxk^(Cw0NGW|L8J(3>9%O{wS1kw-&mR zuRmwirsXeY=zSoLfBVbVnHvel=o=xKW3O!{A+)1e>2Vq*a-Vt=Q>>15--1%+RAH~U zU~2{evIH=YR>1)M(=yh>iCo-Kg5oi0hM31_*|T{rBgSDL?q}~aR64G} zi6k(Yc2&KL!2Zb%$bB(>3tv54sAH>JY%Pv{cL z_yiyD%#_Cqw?jU1Ppta_?qaIa1$2eE6Okbrx1!n+Q%=4sNAB2Np*v_9vI?=Uk8x3# zlvsVbK+KI0AyhBQqsklkXl;Uk@~h+ZzHHi>XfYy0ig`(HZmiZbMx#3oK7Fs{TroX3 z-Dk~4`%1CwJ5NaAa7kFXOyN$Ynu0yU4&i*Q4q6xgI?r<_2-!E^j2drzw0rEfF!$@OcYaw&;7uzk>C!{^?;1Wv5@K(Q!GTs7A>}fdd zu2Vn6r7wv)d1R=ysy~FV@ls7`?Bob5HXZ67aN9m{w26Tzt}xqQHMf-@AsMHij84>Y zM)7x>le2nP0nkPU8u<7{WC26Drh_j{Cpw;}B7zQ8ZaB6i)cVD0E!?lvYi5HIH(If# z`ZY-8ETb<-Ezl}}_3>_Nrd5FXkdgbooG&f4S-4-N@u(L|2X890h_C|t1*$$?B|sL0 zUoM(UM=lKXR($g9p?)2`K>#bDIVmckVR_Ra$RIa!VhFnXVf|yOS-n+POceFmWgn>T zUY_R6dUxND>d1&tRwcrS0)hK4r#2B;nLgV%b9AMvbxe zr2X13F%R8SFlde~=oy5()CEw0r(94CC~$Tn{+2nZy?383*3In%)#=lGbz05a_kqSW zp|r_??AhM9}|80F67 zW1%)tOo|5XVWaR{QBe|h{{C7@=JlHi?sKTabVIM}jr$fvxVo;c>t66LKYE4IZ8*-O zqTREn+UwvS6^$;6b4!acDd=jc=!uF((mBe}vHNAnej^SJ+%`W3yAlFh%jR00cffac=d`AVBdX6#ecPB+RXdW(sSg@b^491KY-5LqM`cH z)EPS1SQiJxeQHa0u$y`3Z{FX4e+@lAjOl6eT+bCfbx8^lljd-mpdpAW(8N9 z{1GUxNe8omvFCxf=gNRK9pSjAX0+YBs?^(bbbFiY-4?YmH{Gc^W9cLhp!62>WGdG8 zWI)FoQai(REbHLvs^AG6O@ixP8`qeAfo3c)2Cjrg9pkIh6at;x_X;|yxgm1OHPt}9 z3_p0{wbru*S!Cjcq9xF#^VGU#Fcq)j3&`hswEo)FUlnRSt0sQEn*!L-@}BcUg-0BW z8BO*PC^*pHq1_nYE%dCKO!`{B?px@UXAM4)8%?O+;OA-CTvaMOZ2Z23G3|U18T|I! z){IQ+UH_v}lI0a)IMM#5n!qk)CiM;A`aPSD57GOgKkm^2 zx@*C@_KgtxrqFWDqJa$Y>2zY-{F$qH)2Ga29JV2Om`Rf^t6}@&FcP21tUDhPJ4P%7&Adr242isBvbAuDegHq-y~0$ zT~MTb$>qE9-mbH*ro0ba^mlcMew-W>%>9by%WcO#Xj&J8J48(d))g*n{IL6#556pK zwxF?Ixf(nDIpU^WfESLF@eNU=#^8puAJ?yaJG=l-NUDdA3X4z@Iciaugw|Ovtl_-> z0A5s`8@)<12zGPH9U_^ft}cNztqkG@lc;Uys?ksr7fgQnxt@EnCnvMAFIL3SQr^!& z(`m~P{s8#zvA*gpMO@unIu^kDsIpJFBa;S(-IoXDg-` zBRnd{WA6ylX-=y!Vrhu1JQKf2D7O5xE&KdS8R zBz1&iu6@&f%jDJ}ouHSUp;~B7N8;r6Yq(WmILciq;?6f_jJG`Rh&5QHzjT`j!>aO- zE&S>da^Oj)t(=T?0ku@-pZs@gRy>A)UhW5K?1F9@f+^2v4?G(ejuh{ z7e3rXBo`g|eJ4=poX%xd`>PdilZBGHEUdj=#X?(eBoT7Us<$i4ud)AEJd6a#l_-xA zNHbp^dN1pRePcbv&q+!nngefk;GE*hS>gV#ja`QgnD~#Oh-$ybuChbZc}xoMLcdHS;e?iodb(&;fViR)n?0}Q90ch;7I47K-GK)>S=5#U7B*l&ecYSP#C%3 z+?tu0q?vUcWACzh?SHXUzTmusAW?8lc?9X8)3VwqHhv8QfBul+pc3{WrR#b3LHM=251k8coG+{D#zt+?4B z8y&RD!Rb2u5e|LjeVooURc7ruB~(N>)goJOH7@~JG56@L{cB#&NQdp*Nk#%1AGfD& zss~SqQI|w{^*^@iQ@``Cfk&I$EkJytTIAeG<+b(;r{kio-k)!n&0>W6c zc^eVdeyONX5kD${u0h=%x!S}qlT7Lm!^>TXH|IWFBm4smR|`;T>kMZ)p1{GfCDpZ8 zlNOnPP}dfjRtQ)5b}^1FJ!GvQ<+e9)w`JX@-2nA z);S@B4(^3K;uYAqz{0R-_t66|SY~fXMU?L6gKt5Q+$lg0enF--&E>yB2i=`txH!1ITP++1`NVxI1sTeb9y@{n? z1j68uht8zP$kWR85im@9DTYFqjwH;#Y(05jm`#{mIFh<4CPBMR2rRdfv=M&Oc+{dQ z1ZQk83a%KF&~e9BcI9YeRZi~3w*-@9-JcDmq^+J)N)x6wsbm*CN>ghr3iQtWFj5z5 z9HSik7}9AlCoB0?uShz+dBE(L1nO^h{zPnW#a%=Da%6ez@O>AOfZ_Zi(f66gD$egI zRmXq;9GyXl6!5OqZYQy0GNx*2u>QS1R(4*jg-^O4Tg{J4J+* z9=w?o2gfc&10FX}Tm_3Y?9QpV`eKQfZav~U0YuYyWwO2h)G73JB&TYSGSkv{O`Qa! z-+BfIJUy3pY0%8!z@*6ZzK(NICkl%`>4X@6Rr;GIfnae#=p)NQ?!prWGDHtPw+N3K zOjvNW<&Id-_;su-7<@mZb>VY%L}!kAP>33P$75+walSZs^3&mjL5D2mZ0Y9f*{Fwf zo#a^(vSOzRvel0wuQ;qxziubbF=`g5k`zBc=o0W~RUF8mbb>o1M~tS=8NRD(!CuqaJ#>9q~y z_{CVthKXy<@c7*I10<RU8?|4#WbnfngIi|LdDUUmg~bW3oa0p zog`?pHkS$IT9f$^M}#AtNd-$-$Jq6CKQjmQ*?sg z42({-i7O_8Yc-u1cw=s=Dc`v)%7GEQM+H)M(4l{3FC`FdkL0To^ETprksniWxM+79 z4hl-%0#v>(iBFy7W=eHsc)fXP7u`1)KbQ%??gDv-kDT~#sG?3?L6t{FP4<+A1d_ZE z9A!_zaqWKa%ARWCRv3zxL!p(Nnam7*+V;ZN#%6E z+-FWIB)`4AC~fNXw9GQG`hJVcYq7CwCvwBi?=*DnsumQ~?*z^rQI)Np_$3TG2&FH~ zHC>kgkuG~^R#?&H+S0B!dgMJh^pQ2t4JrEKB0Dr}d!t_d!aR`8wbAQc_ zbKB-`MCt%pwrYWF4lV1h%4yc9n<$7xlCfLT9ezvn>T;qsH7NWOoqY!}++Tgt`hq){ zRC#l2fmHF+wv)T9msmzvolz>S`+4z^MDATO`w-6RNvoVkGwm5_RKu^p+E`pLHD4M< zGIM(IC@>4^ayr|2TAZZQOzik0mN}~VnpcF|Z?EuJF0Z5O z)n1=w?H`Y}o_jHCZ;chvnZb&>9QNddBeO-=MGDasmh7rmEJVK_ey z;g}4>36{nlM($Uo=peJFi~f=K4kXX~1U?=PwFdA;(>x>Q)2=vJ#o*03+AI!u&C?oR zyqFy(JN_N!?&;)I+K{lber1*F=CEk7%<@|)JOodR^`%Clw??SI2HImipO~egdjh_c zq2dR>*p+YL*)Q`~nJNH|_@6A5@3&=&n*#dXcAPU`*k#mX*z+U9cLk?w_5mrV*xLt` zO2}#N351`G=HvD554JpFI@E$+7!2z%l&cF=KMxt@1FdXu=t^@4%C{5(CERDx^}RA` z_MGwU@K=Z@jo%E+ynR8gf%XfGFPJbXxbx<&;FMb^-_qihndy_BhVaeT`SctHxtFeH z3b~$c509)YQ^5MOesjzc)uqkvvreW;)8~rFqE(sD2Ep(moA6xeT>K~(IK$q>K+pe; zg%)klnOyCoD4`pvey9Ub(UO}(5jU4%Lw7xQNqPZ{*Bz0@oT7bB1 zY8Bb5*O-c<3pF;?n|(;=Nzvv0MYcw{ap)!Ta4GneQtxH?Lf+-zww2MLJ`=KV-FGht z3-+vtKR^AE3) zoOFRDdFS8Kajp-Wi;(ZUSXrwY>jx)q>Vtz!%E>(OePvtszqaO5=3!D?4e*cBtPEGY z@tyREkAzxt)`y_vgkLc26NzE5N1MRI?LbP6$qYUBI zT<8vck}iWzO9lji!&FaPHI{GZy|#r56dXFDlhPjOTFQFL7A)9fL>xh$I`(4+W@79{ zAw1^1=tih_wm018b-0fevtYXy^vKJoQAVCaX8id_@UULvUV)Z3%mB6K2D-{N+p-&B z)SC?b3>)0kQYC$T9|+6obkoUTh``uOV`*&B0l|1z#|_*O&og!xf3^4A0q1TxS&ey^ zf2WbvhIg}_msY*TLE-yZW<1dRy zOQ@FpdP_^b?a61&53UJDqZr{;kC8^XbX&FI%7xgYZmMO?XY>p}*F*J$qUP$nJZ8b{ zD&y5i4Pc~yC;EI(ctOaum*c7j5tYdx%gMdx8gd7Ml% zwgdesRa(JakvgXgc@woV8s3!e4jvu&G!Mze$c+!ON1F(I>Ndp3J(;5g)-}pMmt`gmwVtb;KRdT`>?i(Blj414icy$niG7wj zuZO@;hAI(X*}j``ytuk%y<*ZXRQP#B&^wKe%#0Wz&`U2kz}6L|4|47MkU2IoYcj%% zHPqM;k}$qUZ$^lC7mODgVG;`V^4aY%@ytbWej-zLFVX*tAK$7mHkiAs|2V&vyNS{w zUIBgn86e2&;=D4viZuy+iA48sTRS;}*+|H+RY2N|*Em2H%ZFsv+a==J)WZ^?96FlO z$vDJ=Fm#S#tt6Wz%o5O#;S0V<>C;tdymf z#|f1xQCg0X+MRf1F4=@kebUy_wfcA&kPe=CR3&R~gbWofg%z>UK5RlleP4dwcql?T z6^f(#A~|SQ2kr$l_FQV77lDuyRmhZYIjR0p0Arh~%u*4{P(i1l^Z(+PjiPIzO`ntN z>0fIirQd}K(JYvFa%{HMyVt2njm;XUh|{W6UUY1T!SRUbX5+rl$%wUgm~XVSka*1= zR9p(V>K(WoIhTc8^cyB#otOToV&1!k!((x`X+q>zeoNhEDBQNU7H0X?PR#F$I(X{)3_t`tDM~NqvTF(W;e6i zs*gCIw1D-q7crNMszqHm{~G0C+^WRpxRayh%3YUm$iPZxGU*t61TEh`jwjwKW3F3; zf(_U4$W?K@flK07m$v2w&qNUKn9VQjAxTslxtd_;<$2TfJ=rNmzl}4?@BOJ|ct6xL zxF9Jzk7&UAq3_l@$H}KQoM0%mSg^5WaTHS#QWb6G#x|d=K&;mad4lM7%trZYmh|PF zCn&BLpUqH}`C|Zmjs~7;+RGHu!}xl)bnEv^G)S6+N&R|OwI8w3yI7^>e5VBZBmMIk zjMbU?A5Cr08XL}T<`gBBXZkFZ3y#W3&>hxpLk_@!5{r?1FtuYWsP(Hf0fz@X!`T{g zlpw_MJ3Xznm-%An%EJ_!Si5|^xZjXKT zsax6PUjA0CerIRYI)n$Z|H%K1vpAKJZO_zLbbJ7sb{Wrvwt?zX1U+jE?WJptyrwUlfs{*SR(-YL1`his_(8XCJvQCmUV*z( z9bFV!Gy?2BX6ZAJ6{jDh5C1%j;i0OzbWNZ`bk~wlsbqE0H*j0rvuud02W(ut=|U)t z<5YPIj=stT5W&;2B<`qfDe(E~l#bRV&b{8cO}B84{LJorQ=dDl!P1xK0Ph#{hSzUA zewszTF0NQ$2qW5<98N#1l}QnUw=G3QkL$pov01Cui4)zf-BujT+(X?7_=94FFrV@yF4NxD?ODd@6Z~5qqO$g2&iZ| z`^_BG>ZT)#>?aS>oRTUJpYE~^2~vSaf%$B^{3?Z4{ybN|OH(=0)QGpV+g8ldEBvd@7=>+< z(PAl4t^w2?Dh@o@@Zu>Q)1A#`vV)v>o8(d&7OAC|ND14Et$HyStRv(7+0;pmP@Q#J z^={`}bmH~D(ic^*IPdq^x_xmd2In#)^#d!~)DzQtb?F>NOErs;<)O&Uw9V`y3w6)m zqDmX;K&u#5!&W<7of;l=NcrxzlN0I{$jLi%Znkc}pub`Zt<`ff4F;M~?vvIu{@)|S z(MiYKer)x0wvYje^xpSt8MG4G8!b`YKg&%HJV(?9+0JLO5Q2+`sq0v|ty}d7gJkeA<`g7iv24G?Lq~^-PWZPhYq!Czy&x{ZhYh)H z_!V01+quB6TQw?{hK}0rf#Ny~PiHjn13fg8C%DU-uRcK!&oRUfH39Z-9kFp;0XUE2 z;ILQ{NLip7(tMh(nc=8l{cM0(aYQDE1MvhiNr0VMJ}keqk4Sc}Y>-I&mGW=Ap33q( zfHl5`f0rvWO?oZAUUiLo{38__37SA@lFDcQ!4SSw$A?)y0C7F0yrWi-)V$-X>$kr; zJakf8bvrk2fBM!h*Z~hc19XmFLsxe_yyb1@9nY&w5~J@iPN^ls#N;MD4QVzrSk_xM z8A`ocAKl%kSTcUY63+_+dbHoi(jb?sd=k`IBbDZYEi~zYwc)p8-Oe<@(e@E$dXv4- zYIvHs*2$28(Fk z@4{byyAVku*gtWrBWg*`=Qn!%6W3U?-sNo=MvMP7KPMzIlhjxFY&_Q`=!w>YO@WfU&!ZR z&SOKODndFTCe!x&zC4%|dB5k6|3<#QFP-d$F{NIb{rTQ7Zpj5;gvTV)fDKyBJ3|m&{&$*7#1U>14f5!;_#%nQ*=qnKT zK}wv;|%f12m_FGBL zi%&zkV?(yrYz5GZHg0l@p9VF|w1V2dz321l_t5hm5E{<$6Gs_l%V#Hs(EAS(DN8B+ zxyHD4%ag_g*4A4xbQl{K2MMfzpb=Lq2Peic=p;{1zh3pq$N0hc9?gPKHOhh&A<&qN z#XuIb7{XUe<&5@U4b1OEi9uoS1KRd?IHQwcCg?mn4-GiZ@?sV-=RbZG-K#m&N|E3; zTp7~Si4yCSlt3d1_dCbuaK|SC|5WKSqVSjP17x)e^HlA;&x$VcJ8cHY>D!&>Tbjbl zT63o*LNb|_*wm_WKbwYt_j^Z3&b!ZM+ZPKk>iXG?MhaoJ?MUOQar7qor^1G&Dc-bp=s??%9ap>b=52**tps2%#w}UfpH{xjKI03 zh+jVK9`Bu~$v>xZj?b+S2EPVc*O|r@ONEa6x42fGBb?4WohJgl`z~)2m`~3VNA_s9 z8i)E^?Uh_NqOUM8lSuCQ^dHG!ae2k#=*jU|JqmyQ>rSt3y%{9o9yiJVobm0gt2tUN zrw{jltZf>^KV8(s)CK`74^TF)u@~IJNTz6U$8>)L#9$|JeD16wL!(#HUMi*ZSX7(3 z4nZ65w293Aj=qQ0UWp4n=LwCXs zR(@g~$K6~NPcBYOY*)?=9*Sq1-Ike|;`3!|Wzjo?*Z6JztxmnP=Y;u#cXy;aVy+Wye z?J@#=y#RfCrA_jpXL?8}e09KVW7ymomMGk_2K42;Kpy!!UujdGd-^i~RH8%?xf~np z=!+zi-x~Q`mg}`Scy%0{Yv1r+HJ~?t@PePg@nI*t(tZ0MlhW~>iuy%m|D76ZzERmr zYpJ^~x47t6EL79G+c1aDpgJ)LJ|=irEAyJ%%{mpt`wyn2^Ml5{XA@@n%}5nV<)z#cN=+7-M%IsdS|(ailD zW|B`f1y}!%(GLR;{{IPpR^g-X8@0z(%}VP^R|>W5r1YLt5`RV1Vtr? z^o=p}6w{?AX(i`BhCH*=i5g-S*Cv_>zz%#CeQ^AW6db+`G83n*kdjoLI@;N+?9zLU zvsLaoNA}k%#h2ZY+36$=F>jF*V&E(k_U;YKm1gixqj~R!|8JqsKMS7|4Sc}wb;^Ohsx*t&dDqJ#W+6%9wY#i;=$yh6+1KDg1Su-_nwNHr#<#wD${d;6fO zoDO1EyAdXaNJsCJ-7l)S$z<(OUdi!J@yvSW@qis8IXh;O%FCz&!dqMpZX)UU4d-vA z9qPAdwb9GE4B)3hGxJcTJLUF_nd>OrD3PXlq$TyAXali9mi0!$t>^l2_+rWaPV?6x z)&IAs|D)9ZH&1y>OnNGo-mh}hd(+6&c*9%vALh~Gl{x7ra*J!#iAoK>!ffm(|5vy( zgFMz+5*$`4wD!MetNT@AZUia%sUb?*<8=8HBFPow)Bl>%6+KV-c* zO1coC7O6jsYMC9b7nSh(yXc;cJT!%vcup07eGMg zGoB8#aQ0JseHb;^S+bg+85TPSE6=T-uFrBe=Yi@EBX8D-mQc4^U;bGLdW@zcV@~zC zt$UP8g@E_NRbLdwii{ptCzZ!2{y+BKGpfmSZ5y_XI3ptCfFK|;$^g=bCQ>sNKtVw1 zy%*`dm!OCR6r=@_rXnCU^bjBfr4u0nLZ}G>p%NsR1Sti z?ap})HOI#9VUk1kgb$aN-r-eYRvKgNc_;?68;#PM&l3))*wD#^<>czz7I*!+<_>2* zRvG#$5uB&kAU>d_54T;=7<7$qr0=sk%4?wi3x$2=xocSv=0O^Bw-C%`uK)VzMThIC z%%^V(EA*J!UkK9Lw#T!iH({!I#qtd7)Q!7k+!JcBq#RR~@@!q{e2q_&+=7NeE+6#y zVqx$Xk-YwL36UKE*?5l|E;0ooDkqF`9jh*RxJ8TSm&g`e@Y%rwW2q(m4(q~l#iOOQ zUxHJUf=#qTKfr1LMWk)nT^)5G!tjLic#NPDV#1bRVXxXHL6LuAmw@g5095&r&ce{^ z`Tj@l@*?!p6B*%e1tCF8aDmflWcKj8X7RHX_7e}dG?ygxP>FeEUd-9X=v2ANo6@Tb z9imgDlf{88ouW96p!1sUS(W&1L3*Ta*$j_F!Dy7;jr$X^B}`;agP&RqHJ4wK1_RP& zKAGwIurTByUYD;@N@)?SiyLhF!uTUj6i!PkwL+Q9%3G zR!Nl##kzI z)Ui9CHm!C<0gW}L7@jtfnx@+%T9V-ca6pHF#5~t-yUw=4bj)q^DMt?o2KZl{S zd`YanIm{UJ{a}Qy6{4?G@*pg{G^WwvJH0`7SFsF7sY*GBA+LLtH5DjtVF21-gGh2T zd-vyTxZ8CHW{5jgXl$pL3{y-mh)wv*UF97tByAOwXl5RX&!2JN*&RNxgNw5^twDx8R(EaoNzQ#jHbJ*>(MnJ^ISk z68i54J|ZBoT+7T4)H6IGL+)9n#k>L^ypTKV!?aQt#QlkBT>N7#eG!-+@kkTLX9_q= zzdPeGq8skS`!oJn(g93g3b!)(LBt`CGXwU}M3X|+gRqcCY%+;U%kM`jHMFV5 zW4urk79rM{?(xl;+~@MKEbo?dF`QQB=0S%9Xt2@ZDZwVUv{YD2jnt>b?av)i$l%=4 z5i;eAn7A457yAIyO6+4HCa57Dil?kpoDT`;7auz$u(A5k@3l>wpptrN@Ko!=!M*0O z;|5S*Bev~mar9aAgvC^~_g3Uf*C6~xj$X|cY9%V|OPSSL5oxQ4H1gnl?t-4Qzk*W1 zO;$ZtquW)#20>*fJhHVletg~<_m1CcYH;}11@D{lMzbL*VWOS9Xd1PP@>ZmmFDiX0 zF7O~mu#?rm`O{_apU9>7dAs+Wygol)qZ@QHo*%h;)q)8DzjKfr%Zj^k$%RhK>t?(W zESllF6 z<)&Ffs$DjX!T6IMP1uJN`z%pdrV7}S^+vmPwHT^uY zx_J(Us>*R$X4fTR;|AHpT{u@!uW+g}7oX9T%``tzShgO2zaeTrjuv&X>&q6nDLkbt z#^qBvZQAYb5HnciWqiKMTXhRl7`*T)`ZhzZWK0Hki>Wrd4zVUMuF7QK5Y#PTw8(5L zM`3nOPcxQ0)R7i3j!&3VsmsZ+#CA=3S*nkV|E#2}aNFh435zIk72Z^eRirY@C@fKbf4*rewA zBe@SmTzWlg9EndovkkR2BhSiaM~nAv-7@7UK;hht3Ef_=J)2E2WW!RshxrTz{dpab z=FW`_hofA9mJ*_wSLf}ucq|b%)ForlbejzY_`mzJV?P#_KCoO8e;#27fPGCH+9)2R zx8#Ka&Y_;ZTQ9m-42A zTxas`%wG_A6!(!+b^aR8Y)Y%@R^*amVedt|Yy=HPpvO$)EHdT=-lHtY#kg2c2!COb zuqB2%@L}jGF7E2uP=oI6f>vW~a;sO6uC{ZPF#D3E6FOcpM`~@M)0%doHT)FjO<|!< zq5~#DATE&;-8SiIKJ{$idC!=*p*xriW3+!XbIYyk`ygN#>5Rqdg#Vs*?%La)twW{G zL46)Ws&6id!}Av-GdXS5V&IMR5vgheZqh_G!wk(^uBy@GFDcBJadwp~8aC<7>!}+r zOGlLy1hY#t+GVVwIUGegyyRdU+XdDSZ)8a*Y`@_*X=7WuJ>TDU&A?GpTA{c22)}i) zq=KHW7&2Y+zAAh171)RGCqBS~Vh741FLYT{5>264b_=FLVHcm5;6((cwH{$( zQk4)X1aY)QUv)-#=zZtU#X_DC*LNC;rGa7MycfP?9^sEMp2o#U<7=F*R?jS8)dLx` zhjcYHKqJGG#2m)uY;omjhOSa_<8K=2@vNJjPaqbAF91N($k)A;Aa1J3TFRGF z=@Eu#Y)vjg*FU+{aZ(6uMyw0BPbkorSRkOSKd-pyh1HP*0~*>idKUdyE@Ss30SSqGUU!0jZboFm;~I8?PFHR8IJCYWIU;KAK9x|snE6vK zH=flepk`X;_Q%9$;E<4v;>vi*){J6u3^MfxWNfy2=F)jm4q;A7qChaEI+`!uDMBYN zj>lGTO%V|nJa(&3PGV8oA4@^wl3yORP@sf-CLGc?R|nqAN&8kitwwtb_Sh4 z|42+|*=dI34fs0*UkGc0s?O!3RrA8zNOO$4Hbx)wme*gs?9&na!~EQZTf=P$#y=t^ zFFN=xKk`dDq6ZT+;?=Ctz&>KuO~fwZsrOT<)J-yvNYj+;^|HaKJO@5ZJ8TdfP?X*9OY8aC zD|M;&kz;{9juNhYG!F;Y{mPwwjm4@bpg7QOQ@Rj7tQfihkt>u7pNp~*NyO0B_pWMX zOvT7(ud3)uNhr)uGMZxpZy7#_qG+0`sf2%kcUDM`-#eZr8m)s*jtZnl_K1wYO!Tjj=UndKhx z>!_UaG>Z-95YpV)=JDqhY)Xug5sZf`zG`MClXI+Z-}TJV`~?iKHyB)g!w9(0WZuBQ z$ecRCCX7(&O#5MMfBvML&Vs(?wFhUPP)G4Z?VMG7bJALFq$DN5>9N|$a;xo|iE^7t zRf2PNIYhO9FwK<^#}?HPE`?(^%TFH?5T9-7zbQ0<)j6EtWCLhB+W{J}jV;e!GV3ZE z=Io3!>mw1rZGjEfPr~5*FP_Be&*Yq{v=hi&!>pTwrkZ1sLhk5hLmOf=Vx}40rIIBH zpgS_TXvZR(Ej+8RIjB4DP39!Zz6`!4#NN_bpCpuLGH~zt2PQL??K;oIRom?-BazR` z#sqF#Kf6q#OHV=mi7*-BA+4g{#>RaDPO0+n zxYXhI2*>tB`;D1=yxKIprr?w?KWZx5_R0E18^#DV`b^EoVa!O=OHO$olEh^zgok86 zfgV~_)!yifNjjjCBmUe<3JfIvuM++KdBy^_>2;UPDdXaDM**kfXMShP&q4AIUa|bn z(79#|Xxj<~-kq-&aQ1?DpNn|ooYGsRq$&^>rgH4fn>A&9gxBmD?j1G$XLxvg zJ^tC)ZPjQA zX0|b|w7SgTy`Kg$%^tvcgeNS1w~lJ&o{CcqX~<_rCSLjH!FVc$JiLm4(Uhq$bMCl) z!ixHBJJzwk3COFu@!<+7@7+%?Z(!6aSB&FZ6|;77@}A(~jghX|0?_Iii5uI#t139D zvGJf(p@E#oL_quosFq^1a4ld4jr>#Rm?|X2j5{cY>aa63NKZ%`F=w)T7=y_ z>$S<05GNIm+(6i1c+&tWAuKKy`j-e*e;^w4`AXs?pVWj#)6+tR-iF;1 z+P%09kV77)gM@)#bNa;5 z<2xqfg2rc(Rd8yRV`{oVn%d5)u>ptz6)Ri;hL6!D+V+Gn{S&mJN| z-l)uL;l#R$y8u7C$Lb#OB;e1(Y;67l4N&KU5*TqYqZbE%tIRX0ypY_NL1VDFvz>K3 zO$wt%v@(d^lH$E}FJ3KS5LEdZ_R7X57TI~q1oboLplWHi^vmOn)Ds4IvyE73vmd+9 z81O&zF+88IW2mpV^vW{)8qQe1qT=O;vYh0%kC${6BCQrKgWK6H08RKJl2JL~+RQwd zR5)~rO3wX~%Lkb|J^KTkT#CpKoMf7AGTYqL_2r)=C zYRo+~Ev&_gq1^a>e!4BQh)BJ}BG*(Z6ynCF8LRTfpye3cc}#Ox2%mEX6{*@z5^0Y> z`Uibb0mSkyQS4(gAo*>AnZy?a$hj<+a<^e-TCl|gPuMxlAPeyDgiBGPI6vVe-|=n7`@dW0+C;tKeh7?e&~0bo`Z-2pX` z8T1X6^B~jj(&|4eZzyHLcY1k8nim)YvVt%!v{CUVz=~K3o`*zHpFk$H_Rl5#O}`a0 z2MhJP%taiwIV3(yrJUiNKAz!)B*n-WAEj+e=aX+Ln^BQwMNSVS6?(mHp0v4S9~cf) zs%~FSRr4c2hm{K(*RqUo2LklDKMV6;<>~uig}!;o)=Ws#axMQLN(z{R&U?^7o<@3_ zN0U#>N^Bnh_mR7(_~QLBYUw}Lmfgri%t=YQVieDT1Y+1xqwC<8*=7E%^jD=t&~+@y zD7E|a(deBzG(}Xq5u3u(Hr*4y`a&XloQq2Y7Z60rhsupxojCK7q=A|Usmw7}pvstS zW`A<-3*(%VJDG<5P1vDMN$|v}(q(0TO`@|gxE3+831_Xv*@Vw=yUcyvv>?fCkGW5n zu5zlQq2_!13|j{+QI(UaO_w2xN~}A)4%RT{+5Mg1KYu4{wA-;Z>{Itzo97Iu{&PJ6 zZB&u48r${{(q9j4+s)c;W;QC*Nt`mwIws?sPInDwkbMW!o&D}+=F%8qX5+3CY zIlH zx+IEt=veD}OiOFtM}iWPEIzxqYGS1Iv%^Ipn4%YLhKbuNM&`#KOiHhe`|d;~p|oH^ zrgquklAp|Ozk4C!bRK842?57l&5gE7 zxpgukx68?6GcM5Z06CmG)k+X%vo<{)k$)bUgyWBCGr-0Bm!aM&Cp*21Aue^*D~JZY zTe_s{u$CP@R?sH_H<<%U;ajRm1ysJ>X9#`!FO+zr>x&+kIgI#G6^OEY)GqM5O=0^tiT`9102T_cDt;)asU5-DDHip9W#MEx z@jbtD+#&;y_H$vJ5Ll30&emD6JwlFEecr#}Nbz#zYKG2j^DEs3<$+;=8`wjMC|`vu z8xB`8ecf^Q{WGdgfNL?L*0~{nI5G#wzmQvPG=JW{ahZ=UQE{Qv<*z=9zG70|cY49w zBkaJ-HK%1_~Lc`n^HJOl6@1nbo_KB{%IMk-SG1`I}>R+`k z+sx>?C#Y9><$1*EUvR&*- z&RtR>_9)DeOwg~)7nzGV#O+fB^eRQcK5^;RjDbqJYJ~;bd&>)SNl9W6cqQVA`^{7e z{bCulye`*u{|h|w7o!T+AVlHKZ{RTBkU@9dv6NReJMLvpMj0On!&}rm+4yY# zpQvc9{8Kbx9zfB(C- z#hc(3?ROi;!`#0CNnivVq69Q}Mo{Mv+Kps@%`-K@BbfhC?gAP|pyU_2Y|Vh~7soIO z!U?Zmcfxzo>ze^ngV~|rX!Uwh7}&<25ZhjJ)8<36bxpKqCC+)sDxFTcwR&Tj2+pvL z=YVd=Ws8Dy!F{HBgW2th48(wE^ucj9c@2EaNeR3Aqiv^>&o*kbjpac#|c z)RPxAdckSc6|V&(=26RY6cls@jL0_2iL^|+jYNdDFI zVY`NcEgrzw7)xU`>4&BUHbxp$tT4tWVvf$w^2#_N;-SuIb!yRxB{pO~Y41Bc-~gxC z!;#Wg!S+$BnrK)5`gi2!47`Cb(}yB9DB1G)nSf0NZ$0PA#_8OatE9nS<_7Y$xv`IK zTaJ}37v#AQKcpf_)RRLcp7&@9^6)q(9f29dTmDMCa<$XcQoN|+^m!Hu%kEF+i%XwA zc}!O2)i~r7Xy^<7-aX69521c{<9SG%Q0uj^6XE8rJJcD4GopIf$b-O{X4eS3j@LUv z!kr@$!1$d)9|Mk9H=>$_cLrAxt1#f`g^Um8`;g)%O#KGXsgn>VLqwczs^6{Y__#-r zF)SkfPsANMd)dM|2Zb$~eIPC1>>^`xHwrje(a@zRJ54F{GV7Qf!rPEnz^H7vJhEI0 z7^F|ZZsQeznF96k!wI9HB5iIT)W{)yguauV76OJCZ_PRB)HN=gAQb`gyBp6XRq_@! z)z{fNRvx$ktK>sWb^OIfxN)@&AZA8L6W+McBep{um>HZ`)W|jfI)tZLE(rA^Tb>IB zV12tJRrHU3ob$L?GlQVKU&ei_ik75WrTZYvAtWukW6w1#LHi0{%m)*f3eDCBQ z&&s+J+KSs?Nnph1-&&GqldVV3EglVMVM=nEUEuIr+E|(>JV|6CEi)gkHdDU4wTN?- znj7S|+P)XI{mEJw;)UjT3s`_`$BwyL_XYUmA;4q+wT`KnIJvgvnDM*xD*Uf1Ro=UM zAe(@vb{Cr6)bj%mi|-RD6d%{$&~P^poJRdR*?T1Votx(bDvG9+h#NX9XZ(5bnYziU z68!k)2%FSG@l0s7^jt5$I%JZ=!F6K(=vmr=Fdt8YLL3<~mgYZIcF?^mC3n7qfkf!g ztONodbG1J2z=GRz4&NULNzL-$iq8x>-h1$e8*1758E`rN2RG+GI2LWVYTbzqwq#DN zo4=(TntgkfdNiRyms-Oc6o<$WY_XKD?)}G+ z_Q>HSHj9EVdU?MY7U-MSPj+3z`Vyhltna%dtTyo;=^WJ^9T%zR=&Zx;lYZDWY_dG~ zFASS@I)?=Ko)^>=Q+kXu-oCvqUMp%oVuhT6<+ci3hj?usml|&Fie=?0PSKseYVzYX z1Os+j2ttwmL_k1Ie5(_>zSy93+q@lckDXJFKzwYNn4+-k?O($ z2F|IWuVc;bSadWQEbTW8q~DBhFhDsRWI9z4jQXk12WxEgF@exJ#@l6;TaeUDJqtBO zx;()e1H*tf5VWSCIO18vY`Ge);9NRxi&oA-KTV0+%-(-a<_H+RNPRQNLJbA!r znh-*L7l4uXJC@)@@)_@NNSW@szB(i@!Lf3YW(h~&q;AxpH9fZG^`a<>5NKfEokL7x zlW~3xw&z``bo_L~0AVx)k#yNU)VJ`ctM$7O0pFh(b>6ALO;CIltz7?p3&T(6KJK)A z2%bzXI}o8Z>NPB1wds%?7xyXt&;mMslh7KUd&w_*4nIKpVPn3@l;QKlM4tn~*_}+6 zKqglzh8M*DgvSSNn?q3v=LnQvz!q4qb&W#k*g0}b_Yih5DlK3`5IL8sO^cs-yo{B- zttPX0eW#xtMQ#xjuAW~Qfc(d8Xg`a6m;8OgkuJ9P5J)k?FETW1aM{m zF*nL2nr3w-F<N`3{HE3zFEFe40fWt%+J?ra<}t9s&kPeaaht2wn1?>v5o; z?)TQ#h~2nV|F1}rKgSX+5=k|FfWLhKJ8#TWL55PV@ptRy&JsRMQ zQuLlHPl2}F`+0Mg+Q|i9NX9BeFk!%c2u|>Rla=L_nV<+XzI9+B=seimeD_RH@Ahsj z!)>qhnD|XDPK<$fKUh)UuW_$LeMzZhvcW@W>YQmH1(hb82&Fp5>$)8||H%OHZ7|%P zF!$Fz4EN)0WotWYh76YccRE*Zo(&%H^DExrzxtsiyU-dKBRN(B{9JTjIvcYSgt-?MuELC_>z{#OLe zpZ#+(2UW!<>7NxO#@2l8#UjWY!Yc8hu!_0)D`7%;75j^&;qbd%9o(d$o~tj!?X0q_ z@LNYZcQPcYyQ06FC{;5J{1U@>>eR-&)GCg^L39sz3lMTT#={C6mHz!wf4fpz&^bBh znygkY9>Dc*ytR=msOf#S=Sy6df((E7Pt)c^UsTKTk){ z?OEuarY#?C|4&HcA6F{=fQoPZqaUOk*3Re`S%k4SzDY0%FpNLMDH8Un@bI2*djt7j z>;(Xw^>vpsyVqp!4;UOcn3q_mev~H}!YoQ1^y~T@JO7*J0@yjYyuSkLCwM|(AlS33 zs68syS=d#($NBD-^^09p4;}ZjZ;U%9b>{|mDa<_qi%Kyi7A`*l=X!tkGY;jdZhJ1J zvYm!n4k>wE)4vnV_wr6_cx~mIF!4H`-A?Sq1@T$Z@@Dp-T}ixG=gQlj^D-wlt4JJTzy{%Rlrv+Mdx7 z@1r~PDP^1c8=q*@hacBdlm9%qPB;uro?(-mDq0`?ST)jyAQ-;IpFd%Cy3p6ym1(em zmaP)y>ys}jKk=Q~H}3XZ_kW6|P;2Kq6{+)45TcKWTk9xsw%=;aymrLvoxWM6kmKaq z$Pjh;7%qtn|F|esE#oTmw4l0bI&*$S@*AH#JFjC^yEGej^tFD8jrGh7#LV@{SxMw9 zCFyU$$NgWU5#TL0!ienM=b?SxXYxjFa7X!EjP*>_(gR3Ck_*akZClk`iAKPL zIl1A@%=-takIQdKpo1NHD=1n!9K!O6$0HFnyAe$^h0X;_2N$q&@h5CJ>M7X;>j) zmY}{=4hwu|#C*{+U~%5SC9wU6WKgu$&*0fFB`E6RjlDE|Hj}>b$@P;gg>RbO1-}Nw z_Z9!o21L>A6M8xZZW|TLZtv`q~-r(nXm_4RRI*vI3p zSClfF`sCJqT7GsjW_$ZEP1B@XtO_>bOUP~_h|K(NBJ6gaeO2WDJ`rM%>-Cp8G2|N} zgbdXaIK0EzggA`)(K{jUMEnQGI+xGO?iI}s`{`pB*F&r`P9;Q%knq)d2B!-JbT(PL zlk8^2D-@4d9o`Y4HbEw}g3XS$pS91rKKgQE!PR8z_^?NyNnk6dk(92J@oIX~HaGpH zYrYRjjLDMgAsL!Y+Lr5uGt4TqhhYuMrs@7A4wjlHjljPfO(P3U6$!q#@4Oh|D9FdK?|tW8;(oq!l19L|$}>`q?8qUr zhUcSJ4cViCEkVZ^JI{nRs)edtH5k1VPKAXEDPaA?uE!rebH2JxCYc#>7_2*UntRO_ zYuYGyxVw)y_1==Ah*=8_Z@@y0ro5Iu%A@IsrmhkyLc zj{jV}&Qd$EUcDaY@%hxB-vPozpZbK=KQD*V?Z$DyT&3k~u_av3Ctt$iF&9kltqF|) zt*UD5z;J)B)&*V4q|N2nzh9S7GYGjzY(&~Y>YwUTGg9(R6TE?`fwjftc2qhxa-cDj z%`1vY&_FRQ0*7kUfxJj^A<9U*kJ=?|!CgIQsb=E4~p=^XM0J@0Z7!RN5 zC_`tLgXo?sD?kFMT9O*^&t3X1qex3g1`d|gSfG71J|EG0slVJ#6IP&7uzB^#X*v1r z#di=)KRSD^9v}~nwk=()G_WKeo$`m$Ck z9?|c?!n*wg>?yXwz>RL*#ZO|`ruCMnhDWzYEo09Pjdm^v8+LvO$%qTTBKNYWYxvla z6~rSq08bFEu3)C1$zBS%df&(Q;qr*>fwa-(5zTfVvA;YB&08i#2QivmsZmGIvNETi zO=T>Bq%jOQm$88SJq7u=Jjq4BMX?a-pS-qmO%v;~VOai) zX3-V1J!a|O7azTZ=d!od8u9Z>I*%HNLqTo_}_HfwGBZ?WujXE*2RBjZ~r5NCgyfajbA9e zp6E$R6hsq5v}{y=Oje3)bK%sPiBw$y zNPN|2q*9+GVKK+yW$?=nrW7(kWUa$?bb%wdxS7K0Z<+A~|FN=Wj+9=t?YmQMB>7;j zU{z-xlxem!%-`2w$65yD5lC8U>dGD=-Z(1IePpgTAh*{m%PD;#lTC3(CqKu(%vgnyGjVps*oxNrWPpg!?=EOSU zqu~Pgk2p8|NBM>xyg+m2wF6X>+BOCpj6J{*XDi42#~7l=i>C&_7jJDsd^DYbcb|Du zuNkV0I%V~1f$2~=`_2;mG{XuJrZ2tL{~1jfh~?s`qU%-Kd2y=52KzXpzwGgBk9jwA zd;4<}uzhHDn<{Ts$glEH>X?_!CdWDKBOc$1^_Qux3y3|mCbL~skj!rk(q9$&PZdW# z`vC~dV`)4rg=Y#$&LXSXc8lYN3ql&2)TV+Dm}Qi4L3nx2bs|fJ55qM0UTlSKIyw%J zj3Cna#TL9gWkERQi)MYn%d6=iWHTs=YJtTTh-10x6t2sv!v`z%KfXASxw^*lUZxty zcJt{vGt#{vrUDdH%0D$pmO_wosqZI1+xZ=)8DVn8Ork_aGwXx23E4|g^xMemLAgHj zrvv87Em!(2VlR1ov0nV(AuQm&lUW{ZcweyQ9euBL?v^HB4#kO+r`w+lcL`R)lqT5J ziwUC&5L{2i*Bamfo%aOC-@OI(F^txWlq0jcmIJ12CR)&7E7v5ud$b{ys2&OGq}Su| z=U5y+A~zMHU@?JB;}EV{O4y>EU0@TrD8Fr$d9e@7v=KfdFM`e!p`_mOdKGVnLc@6u zQ-2f6(D z9HK#O0S(v#)JAXt-W-xqSqNUA-L%4N+Jdg%PW_y2u{Pn6<03R`2`<6*%Mls@ZBsKS zCg`{xYp%K8N2VxJqX90AVhplF9)Oc?WXEBqzm%wBt!#m{F+q>PkuN8nc;QSq4NC76 zU00=+>a4T?5svaFzITjXJrq2y$3z(Jj&)_hr`htlGghP# z-}7gj?>o#?+AtxfUo`#sH*f*d|I-UNUx$;HK5&5d`HgFrwNCcEw9ZXUs_!v$Jd@;H z8k|>Xbkr$FBABv@TdWuBvRtW)9i7inFK3*4cn~wfCR;V9>rED>1d1hRzC9sp@eB-e9H%3sR@#K)MMUU z@IR$+ES=sjT15k_#ufwl7b8-WoKHKzH`4FmtC1H4eM@jZ2Q4a4Xcv0=T>1k-2HrdJC;R{q2EdM-2#owt@z6!X`p-w_*dYd#qur9kD3(Hj zXEEI5D?yUwZI$qo4m)o(`!@7?o$6b+k3=9XgJw)WKzRBFI@<2Rj+ZOC{tN)FVu*R@ z&JUny6n(pB&}8rR6osP#eP+9m0hZ*#nB-J?SpYorqWG|Y~HQ2-~uvhr$)A) zf(vl@7Z)I~cL7TWshQ@K^MKFLv1ioKg$~9w9GkckZn`y{_xvIygK^`!_i^WwI%pa! zjX>YajLigm?sMqQm)PekW6K*(bt?uL=bxg2XH2rCVF5-|?Co}W2sN#SLQOx#A|ICh zTjJ3&!&hC@Y^r4R;zpKM>xMg7;M~sUO3CwI{p_ExN=jwQuEn`)y#*O-u4a^AXfrkW zF?dR^awul8m6GM*HS>his=GWUgR$)V>eK$e(%J4^oF#QstyUAmX1 zY)dRU)79&*T5=N?4GACXeZaJ2?fO~S@{jt4KK|4H6bTl+J`4lRs3|U6y&=?+yKy~7 zJ?|1_k45d>CyTlf*q*#qZc6_&h0&5nHs*@uY;_P^FUt`3Tq3|*c5!9JcO1z)?K#*j zX{;G5fXRD^=|w=@h3aXiNgmUXe~?LQW43q zzaTRGxrNu0*wnkz6f`mx9F*>rt0NJ)u@wPtJg1bVH(C}ZC$;pEGIHONJ_-z2-MvJN zFgm;|m_%NX{l#LJ$NRD$+6IiI#Hw&Ty9Q#qbp+GK0Ksp8BE9pXTSs4NzM9y&#lS-? zx-8mh(o-LU6ZOnDh)V>C)fT%B1Ys&-3neExcT(3>{8yNs#pCs*T=>Y8t?47CwxlL* zxyp$J^5ke@(VX*9;UymdN6uZEZe%|+9Tl+8^_M@!v_m*KZd%==Nrn1#3zp;g>@elF z5CDTU{RP<~pHzEXpYlU%9_`tXF({~7neBBPrL(|S`?-zSg(@#_xD)+~I%sCb(d3Y{ zGoEZ8*t@}MMX((m*xya%8}X- z!80DaMB$P9`ydJ}0Q8miGLm>eUsdylxULz@%#bCU3W0ilNGZ~)D&H6tYe)Vv?`P(@ zy`Ho%waanqA`Bx`gVqbSy7>EwiY=$J91_|$1Tn=El5{A3p~?|OCxf5mWr%mTddz(2 zIZoN0s3|R#tD5Vp?Se)_b|}_3Xx4(o9__xb*wFE;UHVi{eyxpxA=^uYEBKtN$iT%i zazd~8VDtTxSC-kXhb7gf)b%7i z|LpQ2Z$!by@+`w8($mwL0|TwWPj@ru?&p0B-+IjlRn~@haYlhUspz_>ISzo{wKx;% z9vCq>C<{yJYMVCz%N3TXkL_reuEz$0&dn7LRelKMu1T_*D z9JIW(+DB?`rs3>{t`Jw@)oZxkokyd&E_`y^t2ZDKhe>^c(CxJ~$d%M&AG{xJ-+PQz$nj?fMSr7dKEU$x;cx%X zZ%CGk)tb_sS03j;TmHhycFziObZ3P*r`8+x@fD! z&f0X=&vkocjBq^iHobaJ=45|{$DrSG{Kj(C0*LF>06ROwwUC1&f1!-dF;POvAR>7Pb)9W}XSnUp$&Mg|`-`6^czltPzxhY!T&(Lcu!2;<3r+otmv7l6$9lTm^m4y0n0 z@LAcP%V)a9ARoFV@vMD2mH8c998Um7ez-moA#4dw(Twa%!?uu++r=KQfrUb3>)k_6 zwnNqs`oZ9gbfoZcB&HD~qV8a^)O!XDv`yCpo%+jM#wf2XXHrS@WK!L)dW)!CQ z6{_rNU>H*X%${P22OoaF@a4qU`J^E$d8e5DA5iIgP_Y(nAg6McEweNFyI(W`&ZAH8 zeR0|eOgVhB9`kX_&0D5uw%npmeRaXT#q{ka8tE~ixIEsroCT1AmRCTbM_IO7^g*Br zA36G`CcGnh9Y~_M?acnlX(a@%a-6EU_zQhgZlzwsm{TF%z-!`^`$lC zBM8j8CQy~BvRO@DO~s*9dNt?D)%J?X)^iz7eS!^Y3p>yv&$7#^Yc26I^DHc(LhD%O zp#<-Oy2E>w^cr>bvy&S)~77P556~jjFu< zug>~adHugbXU%I4A>JKud4#j6KFO)1{s?p*UX=-Dej$$BU?`C~|Frg_ix#K^`@Irh z&wV5#_yvW!u$#kKX>H+;HP1$!)UJamaX_WR#ueR;6U|VmL97jg*y$)L3$9Vrb2@UA z>WC1T0PNFojlWlO+4u1tyJXmu%O0<-{?4xfP^SvP>m5xZ&Liodm$->g8gz49_yz&} zWJ1GSEAIU}gF@5%@9DHZd?&ju6H%w7FMvAPIKPTo#*;;C2IDGr-W4t4B=H7gHL7mZ z5eE7){PuoI9w1byEgV%oC|=&{nBH5eMACB$BQj-6kZWK^6`w|d;(ij0G9}GHNv%}9 z4ERSf_A_Tt{UeO!{u7a4M|V=gI3R@ND&eS)>WIL_h6Fc4vxwW;5#69=U2){ZZUv{k zwV&IeYE|+b#mj3biy6U-j|Fq(yE}`mBNXh>F7?mX+xnP=-P0OtK*t28mBsy-R`iMv zsvsmPuPx+DMhQw1E>wEW{nYkC@OC*qQ4I1lpL|AdZTqR)SEJ3#eJ&I|0jiQ$xb+(C z!^N;O{RC(EsrQD+R0q%Km(aDyA)h`WYzsgk>13v=%x3k_-{r!w{Y)%iDIU<4{SZhV zLQ@t0QGCuo2x_CJD#KV>U_hSx$!P=s8WXI1Mw9B|UH6Zlygo^g#JyRKj1%oz$yLDJ zW0(gphGeMLzXF!dn@M}xpr$MS0*%1V)?}=&?+EAb$-$IE1``UiieR@8A$YTvmZ{QI zv$r;+0W)k7^QYo_K>vS?X8_X&cnEfevwpH?eeVRyZmxInmR`)EnoxkrZP;UfWy5;4 zoqZJUQXM}tP@s!l&$I7cs_()QvARHA)w3p^vg}_;(3phxer@Y__S4pZJ(mEA0`Rdk8YE;L~78T$GCzbSPMTS zNeZ9XoDv=(>e|Mw3BRXeL{Lcb_LCn=2Ql`!d1p8m&S-S5ZA0XrusI{;oY4W-w)@AE z9G?;79qZ<2;?|JsIp-E1LWcDHP3|Av!-V?#sVHNIVU8j0b6mCwr76DsFG298XH59J zV)JqJ(Kv>xylNYB$bPLy&AYAXrPn@U3>YcFeuk$bIkZP%rPLYOcLGIH!LANZ zN3@3vUYv=Bq->6rP7W-1I`Lhz+U3(m&3NF z%V+IpiGVI|HA1D}HqW}<+WK;IhatD1T4O6<9w!3DZ}}poRlugLt*MInOmklMThU+b zbj|x=V)&)O+bhg)LMfwVEG}^+4zc}v_RV+uS{ESuQ_&f!y!Xv?7e71ImnKnaLvTaa z|I~JVz;-5;t?5Qq8Ux_7H7wIIx9{&Y-TfG4^nf99Ejx}4eo2xVIo5WwU(L380F}Rh>ifn{fhpzU7Lrgp_-xvV<{n!CIWUx)`5r)o1L2G1$s<$5Wg8gD6h^(_| zXzwmy34hJ+EP%@TZ>tlPzi(>9Yj>_K@`zbZvXh#yufJehM+wdY`))& z#21eoD$R8x)`dKs{jltBUG~&c#(OqonbU}M+0yid+#&$ ziw*D6p%!Ce%J~bbqzbC+^rn!QuYjd&e}F{=pgD`Bs|dg_3c3f?aRgpcgUq~VG+^ug zir@nG(>Qi}EAWAyNm`jqoguH3=(~;Wq)@v#XBb}5W6wQ)By~TR3@RDV84uqxUh4`T}@mm|jOsMA&YO7R~OjuY%Eoc%gezRoH`S5Tv8e+=D3<+Ck10iVtCruA?3CBOUp6T3#xF2dL@-}_tpm)X#jBLjP$&%LqS?2Xg!@J&$IPEn_H5q~O8dzy&;gyRM+Wbd zSj`l#rg(hrUVM*%$D^Q{y?YlXvM*?`J5~PQ`)GkknC>-m#N{L`XfY@kCg?DkD&ymP zCU~}mO4!&}<3FYC&5OSK!SaA+vcCOZ$IzO1i7I;yQ`VY&0%lT376?j|!6(BPV;6U) zpCj4($;@=N!s{!A=MowW9K2i7Qr+6Q+^~n!%d*?uCguVfq=zaS^ZH6gA7^DXiD6N< zCJ3)2sm`?yOG?>j--K!8E1yAl!ps>E#b6iJ&-}2oYmG!9_?*x{BU~h73WVi^CoX&E7Rk6Ei>EI+ zohP47U~#EtVec1a;1@s}ZLejkt3}jM)PyH=iR4m;0q*&l)femAK>WnA6vm}gvdVP! z+PO-ETv%tIZc(ANrJ9nT@%`Otbh02hmuepV-8~p^tes!ftRqXZ_0)Q<7&a<8ohPm; zpoZX;kjkFqRDGE`2~+oOJV}$4fCe?kPc$g9aGb9t=HMk|%V=6F* zp_+1~j2LKEMOGHK%Ly5c#@_m5s}1kg~n!z(DM(Gqyb)I0Nz znrR)YzAOVULKQZayU0@=jTLkB3Bw;3xpCN6af(k(o3iU36Ma5(ChnL zaCT>R<~RE;`~LC%vFC7(9_8fb-sgF~+jn z;c;G4EHb1d4r2Mr=wg&`h!`%5hu3rY9@XvcV3E<6?Bh9}qPLt~r+d5K{`?=w)D1yN z=4{*(UI$m`L03#t$nTLZA#Er({Y?r&VP@Qwj&`wLpM+W*P|}xy7ckh@88Y#zo9Zp_ z@x?hjUZ*`($XewIUvcacVftvQb|)Z1L^@!P^1|tk*<9eQi2Ogi6(_v^%v<4B!zc)< z2A;b0(f8xIo6@s$)?Pv0!PIby+{ALC39aAd#l?}-kG3u$=hl?RGx#Xrm3uGLR~MRK zlNj2oS)dfq8sw6Ic^GHn?FwC!CyUm29X*3=-CrYrAOQ&DCdOR(W1JmMf`Fed@l*#+ zj){=Y(lZe#xb18yoscnQW_arVAnJcl;ZMGNsz+0n(M|`BI!;5V9f5MEtXMX|Rc=v+ zN?sX(#m!?R9pd`u>R?_Q&g-x((K)r?_LKg(0092F{xp?jFSVSC*B+A5pI)OEv=p~( zeQf7TRUAa6cCBXk54hgSY(ra@Pff2WDJ~X1RFIh;xD;lKzw>aARTS82T&J?_1!uTL z4ryQpL@#4K%@reaGa z-BBH8wo`%X9Lcb?O)rIRemJ(E8NYm9&R&CcvEM_D)5&_vB6onnucyCly4k&fj!DrO z`%LL+^ye+il4E*UMpBzTF7pwA&V8dGbg_JsB-2^yTz42IGU3}5i@e~TrrGazM(F-a zdQ{w9kybHOE%n#-!t?jiS6k3EIIHrqO%34qkzV4syJH$!X1PmlcNOZi+()`aT0 z!BWPfzqX4svsQIEktFo1smMuG!_dnTi$)qb7($SGK0J!R*@ zvNe+V3MJ7WhPgVs+&GKvNr}3_b0=!$my$hiviT(zihDP8vhDJh(TTNs%Z0=&F`G|M zrm-t28dK!B@ZKMTB70u*dc6ocesS4d4<6%eL%NO;%*;-*yoJAV#Hq`u}%`r5v>?gx@|0& z{27;SmkKSf)Xb#V`yFG@4EK9N8oeqtTG{6wxn0WHaAz+Ze#8zgb565g8m-)pV#>eNa6?x|=Ej__Ide50(eyH$OsIv6}`vV*kD}0lb`1G^qr5gV|{5D=CDYr%vD`IJ3}xh{oO2 z7|?Kj<5DXDzjE?Ec_yo(*!sbkfrf+|+|!Bi^|RzwqvG|~oe0u#zOiIl7c9JIzY4KC zK~15-dZobZB!{3bQ*?}S2Qz00v}M7)utfja7WmY~v<8Z+B;{Fk4@biL0!lbISFp;FCLyE{9u`Ez3p$qT}I*hW%{ z@R-;rHg~ah-KC9wU}DeqbqHCzyN;Fj8TBE8jAmha)PvkD)S_ag0gIq|szOR-f^$U= zJ!^Llz0+ua(_2wh%ZEx_i1z?r{+ZH;wofXwd{eu9Fp;c;yft1o+5N1y&A2aX=d(!4 zJWS^le{4OQ1)V}}$v<1mKQ8!*=i29)!)yz)%oF_=F;;qZJeHp1jjir9A(3Dnn6nP? z!|dT$VB_lQq?QIRuCT)tvyZiU?OYGH>+3!3aw%B#=Z)^vy6=5PaAwnst-D@G_>PF) zIaUx6xg8L88WJ|+onH;a2HT}%9Kdl&A`BZmb?W*69rZcFGU?#C8&Oi8t;xW}!h*qR1Cz8VpWSR+9?vGXrW`0Iq+Cp4fzSBb$ekpJXX{^Wj7 z|9Hec$A`0ifPuLi1fe!dTpt z$&s7=(Vd-`K_PLo+UpbK;j2Y~-Tf|oCLM)D>By~ijMLOt)&9gf-#Ol$ql4bn(K$O{ zv_F9)Cnt(z4I66+CsN_Dg@uD_AsaalpW7CUu^}Fom82E9>2AdwN$m+L47{Tt;={DF z`7pcQe2ooWW2fda4@mcb{QVi;Qeb|3JA`rH*>wo(P)YC3cdQotsG-s}?eVc>i90{N zrn98Wa(gph+0{zqKHw0%+8w56V;*kc8Ox0vjn^#qEH=?)7ug)Bz4Oj0v94d8T5#>% zRc}Sg6O3raZljE&V|>efr|=l`DprqJ%V}!?^IQl*<{Q0gLQZJFl1`l3iNU$MTHr8u z?=#__m^-t%ng+}%RtQImsO_g`IBNfrLSwZE! znzYg|Z5>EzOI@7d*dyzSf5h{=wk!zcZxQ-{RHLY%1fw^D%dU_90?u5 zs63ze$21h9Fk6$Ch@T*U0>7 zVr{B$F!P%450>a=*`2TgF*L7+6s)z&KCLNVA1$fc%Eweucw&OknFM+~pSB)wu!}KFq-% zAAdL{8K1-<=pSc+){VB}?u|XtL}_u!Q7NMhXSyL72ds{yHU=7syxsb5GY3{r3&WHe zsA|nAYMhRo8NP01ZyqkrKoTysf1mi57(LNk?l5GY5f0PD0=4#$`g0T z#e*3|(oN@UMlq>DQd z6uaB!+aM(r*}a#18S^=`OCSZlGB>vPnL@V(t(wi}K>rtIhM%B55>!oLyV@jW5mGbVg!P!nD$BRTe zgC^Bm{qW@B7Nu@YJ-cZP<||)XgP*VRvwkPq^_DW7JVU{kd^^z_yQ}C89WC|ttIk%$ zTS39tYF4Q)oqE~#Ny;rFgI+K@pb7C?7?X|k z7C{{AZHogK^IYX;&Lt%wazF?c5sSu< zXxVyi&Iu5|zIQo$yt`mdwY*?XCC14w7Bu)8brQ{JP{3H*c0GVP89A3f0Tg!n%hNug z3(jART2>i4+#c+z)w*w+V}o-RTU#T@XamqCFF#BKHt;!Al$hQd)|sw&g!0H%!NYF4 zG}Eke@muW3-c{^?+F?+y+MM#AFOaj^loJW+8U`n*Q}^+q&L8e>C5WU!=d14K zpP#QkEDwNBCLA@GWYSt%+$f2auM0KIP*=eU9ZB593JvTp>`|4m2``lA5OmYiAhAE- z?v}j>aHJ;2ChIFZUaP+!KcQjo6s94^8>;W4(b zg7s+P&~)r~P9%H#hvZ7HP1!Q>+?8>c--=n9ns*Yu1*OMIIqm!(`i4!6x<1RSbbba% z&zVEuSpEkHpz~I495a5RBB9mzcUTTbS zlIaGemr7O0$4)3Mvp!%_-qk4sv4`tHDz+ctMEiwE!*AI!@gmc?2Xo2|+$MZmO0O0M z=iYNs!GiKGmCRg>%{%Ofv1VwyIS5 zQv;70UWqK4=!_=Rf*~r(3m#dqG=+Y7$e;9lCoXF3YQ;dibl9iB$}yeWrSZqbr5KkS zh%msDODz)p7BOgSpRDpy0Nt=qlij&KQfr^uPVuulyd$@rLr~a`h|cH%#ackD-x{V} zp)_mCkWRze^w=+U5@|j85h2S7dynnpao1ht@!WP!!CTJ+11LModwh2>Pz<0Y@Z>wk zW!ld_IEU|bmp16HgAm>hl$Va@PMc16zw07%v7<0I;?2!Pbcqu*j1nedA7FzQ7Gfj z`Yd0_U^Rf-;p21hzo)qBvuG-@W>B#Ks=f8FHWA!jH#zm@C*Zm3-3W@6_*hmv7B%*n zEmD%^RaB9#+0!jPXwx?CL@K?|+?!7;X>FzsVgh_Iu&SdR=OMZW(c`A1XJ2-^rIfa{ zV__3W~3tE5$)&t6{Dep-@KA!=f2lx{@G zOl#>87(Q$`CB5_u{!H{SQ8Vh7aX0Oyij#@0*=F+H-#A1I%_Vtno8CsHRuSGDdm&D- zB>Q`q1ReQyI$+k_b-IED$`K4L7+OV1w7`O z-JPePUdbZQXJSEOFqeeZ((kn~qctq-FqVWTyGp~`j3p5W3=n-6mAa{3&5(BD^CY#f zP)sWRX-A(KTn|f?@BoAQZ7POdEjASfn7Xxmoj-zhS`%;a;1IVogOFFshN(TS1;TMW z*7{kg`52;ESs3M35wEgEcRLzmzdGiYU2e&pJsCsCxzSIT&D`dhr?AroE(xrhGpyRf zRq#)Fm*J=J|AZ#cx$sV-!Smy$v7Y3aIEI%?&c40QZF^ms+|lE2+SKeG0(K({U1*p*o?J7Cw7;$th{2`y z!ydQHJ8=kF3b;ODko?P^=aMJdjHI@rQxFLH*%lo^TIm+AOu<;qt4b|{PVo}n2Bib+ zJ#tI(4_#Cud5Fsn-m4c27efZRJSuWbnqHFODiErMiC|T*Gw@(JH%q_nlx&@fZK2?| zHgPJko9~%R7J{|I1q3Fl%l4~-{`xf=|EHDc$WQ$4d-o&uCp>@>@J}ECyX4_tIu6_x zNGB0DGyyy^+4ryQSaHO8a;%=l#<(iNQy^4SJ2KAwSLcc4x-zv#V%nvL?069VK|C%Y z3QYoj${}(q6*n>-q?tTEd#9+SOpQ52zHb#lAHfWRo1O1n>YehK%%HtnD9mo zf#y(92E<%7VLJWC0UWRw=}e#3!iVfp6`+mk2@J{`zI6Fbw6T8o<6ooyrk1l1&!Cht zXsKsgTx?w!DyCf(aY{u!d?j*=ur@J9cXLpW%Sdbp@invO7L#B7Gg(P;^UdFDN}vEq zeS);^`EU}Ms39k_WX9>TR3@dh1`2F)DXTKUSJb@KsHN9&p*>tPer;7*AYq%1^+2Y9 z+QxbM52_9r7648EqUr!Sb8+Zxi=}NGax*!o)H+WkcJaz(+I#5&A$@s4eNsbelwS~5 zrdDM()vX5ND@^?s3YjJPPLSt@TaAVf8ylyW+zM?!E^;KHZg}AQaGnW!EKzoqy-B~` zD}J`s>&ujY&d*EA0Ga%ukKgDY1&fIHe<*`_`iR(6*48U+*wbL%gc9HAiNvvy*h977aF%yDvsJ-M0>j#14(9Q6abA#-`XT<4$%txvT5~SfJN# zQduWvf>vm{Ak#w3;T?(`&J8@9lw>q+TDqmwpEoK-+B+Gxx^T_)%74R#K9|t`^+OM< zYCceFMJL-&h~T@`6gF%?Hw{7J`kmHtt1Q^GB{`7MY`u1P3;OPO3xEkp4HqR+OR~7> ztxIEFgWa>Kh1fF+n{T`3tQ)TFfZyS}>X;w^>jX6qLDmUqQ+;M92z&h{8?R$Cg`3as z8t}v=DR%a*?0o)*-F?0l_IHHbQ6%S&V#LGO>Y=^QyPw z73Kf$wJGV$VWDpT-eo~1)`>H)c)S}Rb8MAxWS=jg?h^E*nJgIzXCtj zwm#bCM19?k)c(OXVY@6}*&W&g^hzYVHZ(u3ij?A#vZ};fMLhU94L@)`+WE&UdsLSK#Oa0k4yiFd6=?27A|BN>?r*gk-KclqW zHi4&D4w2Y@oHW?R)lAC);-|A54*q&UCY0JF*l1wRU}$fWpZh2r-r8;=f+o zFccW@+(#$l>E$sf!kO}4DsOaYgR<<`mbW2?7!iy`o|61;&9Poyy<-8YyWhPCsS(^r z#ChM5BR5m9htfxouu?9@tgC{0e-AFj*T?Z!i#och}0a8r`ZILoF%0zyK2HMb50LG3jP)Y_U)<#{TZY&eM0WHtg>fUCn2fwwNHc8 zHW%^)bL#!BV){3xa0g8J+y6Gl|2D_(WZwVV9RJY4pLqVi$nj4efd6fd|80){(;)J{ z&GEm@@xRUS2j|qExo!S`l2#IKcqd=oaL1DEaNVrRVcmV6`dCvKp?@(YtGI|9liNh@-c)eEZ<@IwuVF&Fvi4NWiiEh<5gE7O2e3|p2 zx7~LH2jty#5(luAkJjD~Acpo7Of3z16+*IdzS!6!D^h>tM&%V2zMO_UiHlQTX%LA6 zrYU%u4L>qgr*W7^X`)R6@M{AsGK+zS2l08|=ux&^RrI+=oPti~DnSIph>b>iXO>e) zemJhZyCo%%n?sHlXyKW;E}h@dC3!>rvEwPmn;5%Nv!s40N8QyiOq4{fvWV3&IeQNs z2@fzbwe7#9JP+hm9wuCJn|pdJe0tKk`cRbYMx~(0M&nzDzcZP9TrO{Y*Rp-mjNfQt z$J?6M&j2d3?}DpOznG=_YJn&34x*vIt5(hAs<`%VWaTTb-^j|a@IR21VqUF)0*9z@ z?0h(k`lMQIzGq^b`E|?4bZ(S?|Et(I8iSV?x`dgIRK6-$S3&qr>(B`@|24>=80-sl zlWo5&{HyWf&wR@4551Ei)8AM!`^`-;`NRSt?d4lx;fa3p1q_R5v#phJQ<>bJ6`kyr zH=G_YGbL`vXUK(xLd3%nlDrHzED{+9q2%HV<^E`;!ODr16?q^+(ikG_-K8NFJRjX? zRD!eJ;MuwCaX^hW<`(u_tWgW1$~LB%TAC@nu=SP>}7|cY!}odG2K6B%hI0! z{SM)5pHtjInWbwdWk9w*vi`_lvyEI8PsP4o!J{^8)&n1U{iN}7TWF%Q6>E@iT1=+U zyX@jtf2L2-{D({EA5g&}isXab3#I5dYw! z20NUoNhjs9b{=(TKg-Z}fAy2;$n?7Wj1su--yH1cbjp|xLNsMuXwkngh@ae>;_~IP zeGR^(>&)km>s%O~Ga0^QYFx6o8fCk>5ppwRWSy+?F6jslStjebnT9hi95MkrXP<#o z&a5}tatxdf{~!Z$iD9=7JVPDyXN7PWP@4AG|BI{r@LRAm&O{r}0slJse)5YJ=IQF7 zX{43w{IFNAvFC5I06Ggj$m0``q6Nt+PZBY!xz5r%=-1y31`9r-KID}3ytBGson7=1 z67&hd%KL&X+WX2^5RT9}x;q4MOpzjGnV{k_RP~i zLI*jJ?(xB=t}vK`cY|o##Q{lQh@Nffi}Yq?^`=+mEoUmKE}de8*;6tx!A+hPVJg75 z{>}_z9vlz`-X~?&Is4gg5qJ&Jj^3xxbsF3WW`nL2Zfk^LokdU4RvL>#dPxi`AXG_D z0{SLJLRf+%ieI0!rZ`vM_4qm=B`<(@z&-T9S>^nu$8K%% z%MQkwOC;TQ-*=sNfIlg<9*gm`9*dGDvtRmuuhRnc#+m)6MzXFuKT+?P@`MCyOb`E{ zO{WL^AHFx$Z!jM86_j@Q6@J6uI`*h)6rOqywAWlOXa6+-6GRNt{!}t81tECm+<&b_ z0ROvp>^tVLUvKl7IDE(S*FUcvoY%sV$Qfdhc-c} z=v4^(;l^=ZE1_juaeKu_o_f`+<^iFHGO|{9vHCTn1L>lUX$_v61@Agbvt6TM!qIw; z1|Ack64aV-7zUy@omM0pNUKkZOs{cqJBBqa^uP0GNFJ*=#x0VQ9%W;rZY~?0-lav~ zhJvtZY zCi(I$gE4+%bcC~ek!Wj5$p*%-B|NN~o;KwgZR@t21(_X=Vlw#@m>^vZHwq}t@Imk< z6!xP;OEei=;m|l^v_(J(i7lEGAC_#4vg`cLD9W@<@jkdwBtc+pIO# zp}YRdAZdi1>g=!?eObP(w&|Iz^goczu;p-O{i@+(UQ&pbn-4?s$Q6tCi^YqNh^~qFV{6}s$ZzHYlayNSiX!|j z*-P?T%+~PUklTY#Wq=}MW3i|0cKVoPf!^y#ep7$l?j=GrG@b>QSNEOAa&AuqMu@QK z9BqctoZC|Sx+!-^NeOz65~NUOCSg0LlmaPS8WKz_SZp3?7FbKfcMevj_nwJ}(~s5y^(y7V6-c9q3OhFZc^H&{rzwe+*IBc*rZP zeRLxaUj2Z4>CU%);6}&_E0cqfFGI*3d>gj+5q-Qqb!(%q91Om}QU+aai}5!o(IE~g zCe+IF?PXuarnfl1-qlb6y>2NFg^ssrx@M+CPmD@pVWXRIY=PyV*v4XG(~?pdyV@Q9!?+b3uYLAqtB zKDF|2_|6jPGu|f`>l?==FaX1>DKO4*++tPG-AANN676io5WVt|?5Hvq$JFUFy4Pc} z-FZ`P&|UX19d)#=?Ilb1dGoo-)A#AQ=P?pNVDL2jI1>SWGGRehr z@9s^q_>{w^+#uQMtS7b2XVPks}jj3-01R{7>C(SrBny2=bCyMbDzoP8^9 zKRbL&nvhpquJw+cOnWLR{Z&w})x{JwA$9;pW6s_@Zm1^;RmT{F3f=X*L;q+#tPafS%sh)hxaqw`#2LOM*Nko ziMotl>%a4p+Z)@mj<6PMJ~90Jj}6p*`45HMu2-ho{irS!S2GbP{kC}AWSxSqbqxr( zd#*}_cdxq)6w;e5t=J0;XRW<+kE9Zx=-s@LU8ubAy4-n^eZhE6;oyiMXW7L%FUMLR z=>DTdtS=s5y4~^A!Qrg_wYF4G4#98XZ-J(F`94{Zd$>4(VK}6@ZkiK*67{-->l(c? zJVj*b;v$yQHm(d2z!FQ(RK$i{ zL@ol|Z`+8LxH0lkAiKFRj0|Hd;}WiSrc_W8c(u$o51raRj-vQC3c=rk;sWiOpdO2l zsmAa-4bof6!7$Ng$$DXYl5@Ux+|$C|kNl#EbwH1r>N|3O_y?uoORoOBU4rz)Jf6B? zNmaR89dXKmjjcD^3+d^})l>|YGFk~NWeM&@a~4pOb4r*P<+p=@5cx8Es>=8gDoKUf z8lbD#Of>8iDyen0+B!@vI>M~Y9r0l|@AuuoFrpm3tk=F+ZSh{X3o7_|%8kVc> zhb)%Dq}+sOAC;TB6iwnpJwKds@kmN9(D>RcHKwNTDiXlJpb$h|$ z)4<4#%Y~obNN+cj2bzZ2eCFW-FyG~9$Hr%2-WxGnjuX%H*DhFIcYDH+JbHz+C+Tj( z@QF;dX4y+U;-FAnT*UF-P`Et5qIe3D_nZAnP9|Tm$dQ#adz>rs5}dwNL;%vr>LzLI zX;gmcrvcP_$#)5}rHT;7mr2W@#PgIH#EEkkasmHM59nxO~iDS&CGEnU=~92lELq zQ>n6iYmgOk|0FjH3)P>5OmxqgN#Wc~mzrh0AZ198Mp+_*VuZrG&$8D`$0Z;~Wx}1A z(s}NE&J%5NE(3=R%yAapXY8g{$!4_!UHV}qTC)mMm}Y!_qMEb-J$bx#7>uM5iDb{x z&m~XeNJtwYDtx_X#KCcMIwbLwzPK2y3DjX18ji%iWoD|1=m|Zq@uIx3Kuh05iOb!H zpzPy|k*w}ehy2;NOv-`|3Nc!KX9WI7IJbrJ4A zL-kK2sCMILl=j?V|h!xQlwOkmM~!Nzapx?O8HbP)EdrF z@2S-fy!_Sb4Zi$+sb}?*mBtLfWI-mH+6S&w*3l^4k{oHw(aH42A0Q;7t<4A&AA0Lv0pxY0%cBcbs~uAGx_W~HKRSKB-o~}v|Fk4@>==0pOQMwqq|DvO9bc`9?2X+ z+^9H@7uXQ# zy4u<<@2ce9R#gsNKwI^^R;34up7l{r+PH6-(dxLAzeYF2CNaC74%~7$ah0hyq^N$e z*R<=yJ@-ce*n5qI(6jfnNhBQkTZBgF_?05mY9-TjI7)8)Hm!Q^fkar9Ujw7>1Ss%x zfSV@T@GN}~2t@F?ijh3Z6`<78Nks@`iNfafP)i$Ig8Hj`eg2obZh*$KxX8$`$MJm< zqjGewkR|?cXQSix1iD;iev(-v@xx>ZW2#MU(}+oE8e_d1hyGlx5P7 z0ec5@SUoI0*7^RN-VmbHa-bnxq>x{fCy%n4UG&s^u~i_}S5&nsg3tK8yJh)>X57@U z`xmXnqRJ|j&sFW#6?da#r=E`IZjtdE`Q{vIGVHEzJevKpLZ|^HBi@y?K8oYhBrBL3DaBsGo)`G+740C#V@nXx7N-z{1p3^pJ~oE z$ESf5>!sQ@UjFdvm7*n!t*eDfb=cd7AB2T)qxnHLbj#`$s&Bws$uo!HKi79GN81_n z>VUBmL0Q*1rHK*h%OjV#e)*vd%-8F&b-B~%8Br&Bf_9~QWnZeApE_UlQI_)g8(8CE$y=DjLlx>~OX?bYVou#ysziKV$j$z`=BkjZSNyCu>@UNt7X z_Tpm>!Q)TPZd{b9S!`o!c>Ix(8FSc| z4^fitFQ@JziQ}9icen>9kRx}Y2?V7_-;S9oS-)5TwcUKGwS~eqp;PgzP-Ex$~9pfl55odF&mggY+1YD}5=Z!(( zr>G!Gxg(i9&KfSE=a?K$mwDVIJ>yC9Ap-p`7ULTrUvM)u2JW~SJWysHq=4iSCS3e= zyJcz+C4~wxX?7YF7d1tst?kQR9=@8jb`X9078mJ_JmTn1JI_V_X*j)CvmPs$P#xDkpmgbW@{2=%oMruZ1JQ*A; z*>Dr)v*qyqp!Cc)W2DV5lbHp?x;wYMu1|W_xGKh^z8b339jbt4RXNOvQE8jx#1Jrw zuj^*`^?k&V8HOa#(dI%%Dt~TV!v5So zc-!)xl*NW@eDCIh>a&4c;FlE-eYHI90;THZUdKgc)^v3lWR6syhsmZ4O)32IoJTac zKQHu@S-(wXl^XT4vPeEO7mLs*zEp)fUq5F#(>(cs7 z{IZ6Dr>QfiGGlewf%W-J12;b825C+~aHS>+`Ck>#Asg}E?!O*pt63>}ys zj~bqP!;+aNIx3gbgTDDqOAsx_8Zoyz)5#IJ&b z;8l>AG3z#bGj^&s`j|o9r$L^q;%Ar+xs-rz^M z=TEus&yT4*91DW~*YRfbt+7J{vg7<`}PIT@%LR~3=OV0Ya3O{&5;CQw>U?`q7J@!pWm@0qHy^<)a| z->izQF6s=5>sJm3&n_hqZf}_4qolX0e_kYR&HSol*7X!Z5iOGXqDmU!h3#YV0jevH zJgXd)dYlck^o!4cuPC4G)hx_5-7~tjfB4cW4LS45;H@-$fi6=1S=S6acDg|)>udjG zZr*2P2}`ui9GiF04N+gHk+`fYrPJ<5nXYAZ4nga_uj^WeP@yo0Yw3zZwpA^RK|R1`DTDFI z!c4mib+8$K)W9!$oX?ND3Zmcl@_HQ>d-$R}c18BG;v`kC0ku9sj+Xcuh|3)86OCq+ zo1XR%9psS<7Crm11)Mb#*o)(mcNxJfrQEtCK5u_526x2k2A1sQ zTA5hT_0&_cyrfML(OX~FbE`?@970x*%E#q;#g=H~y&$CjgQv!l)-$@xWBCSE9;X1? z+?5mr$hI%eoctEX0b+MCafs_&LtXG@dT&sJr+h)w>3u*?_BhHJbfl-JpKPU{)sGU> zPb|p1v&r_pMO5083^p`0M0KEzZHn z*5&=?FcYzYf-~>;KxbN%Oy*$`wfsQ|4I%7h?ut4q!JZ*i2&h>&-%t>B`53q?B6}19Aj}kqrqd)&UlF8wOC6h(Y=_yG{^Z>KxBrOXBX8-K~#Z?jyjEMJUn^*VO?{~HsrWKJizB{Q_z$?yN|@(6||gH@1C{J7v*hvW(`!r^R~k)CW(#eMv2Pn@ z!}WR^0e)$YOX|d3pOmoyp6f+CZ&-I#h(ZB~4JLp;dkE2Y-9%CAwC;ek<52&qrm$88 zc!(6_W{p-cTYY39q=W!7`r%C;Hj8*$kjV2$D{t`E3cc8V{3p_K&JovhN zN@<6MF?OOQpV%FW%;d)~H|}*6ys;Rsth`gHEq3B;3(h|z)(CJUdQEDrNIl8#2Jo2`4;Brs9`ZA_=GB^sI> zGH;=(o`?So)8_1T8BSQRq$c@r=5MM8tk}>jVAYgzmQcih@7`s3d((T2RNmG7%Afn_iX^b* zPn9CH*xshzdd~k5IPId&Ux<*}$Qf1gSO5dRM(cRh!>4*LI`?GRio&o`B8?dFDO?H} zZI?JaoaWNqsb?Fp89g&p#wve1=2$)$K+^va8gSToTr6Iydon=);(DJAsOJO0*5j4g z)VPGM^IC?@=d*Bj^OAJDlSp|LRL(W(06B$QJ{T=@G>X^UM-&Du&)9D3j0{{i(?y1V z-fxe=ev7@C`_u3CTLp|#Q6ChN8-w^@*MaR5bUw659zPhIs=gHJ3Z{EIWu4=4==X{% zOJHcEWU8Ok-2_JA9)Pm~rWp(|;}+5bxizs1riPsEYk%68u!Pdq@HQiJXAtE*rtD;F z@Mv2g3|gFCj zZ%ueNLERS&xq;rWtzJl6D&H9JN>G;f75%B|_+BtNAgU~)b>(B5(rNJ}H^E4@u6b=T32oW6M8uJzew6KMP8x;_*a1566 zY0DVQaQEURqHwke4fl|sWi!>RoIeK8&!%TH$6$9QHLH+Uog(aG2jF+E-dC}3xq9Id{p;vq_kme+E z_N3#?I2f*C`{q1nDm!ze$V97X0C9VIa-3Rxi{ng7YD!m`bda9|f0n~7;W(7?7@cJ7 z^05ESxpAJiRxFytM@}%o5)tLZw&s9XcH5lGK-uN1{B-5$Tq%* zuc)uV%StBdK#O?xLFf0fmA^}d%WWV@thBMs{Zrm^+|?Hy}I^F9R(npU~Cu*&&QHd zW8Ue~wp`9!lArKzSHau=BwJM_ce3g+c%6cd_CP9lA$)JJ^w*d38&}ume~M`EZT=7? zePOfe$kI)wt!j_zr>iRJ@B=pUh0Gg?3Cn z^cWgo;whl4vOYKbK_vi?n5pasn4`X&)&}qONN{6+Tzb|@v!;b>`bcZ8ZN@y@k2B7H z;i2l)k(%(OXOHgBm-A#N5k42?jC4%YUU7w`BGBlX3BJ)rXNynz9i~_2XO~oqggV|6 zg`G4e>cFgw956G3hdYo(@TlC#DZ!tF#f}w4EcNQOtj#Tudj*7Z-*9lp(ARt!2KwCx zZB8y_h!#k0+@SQ8tE{3Gwo2PMq93Vtj^Wp9_MfuPlXDsDh||Tr4f`_mL)+wKj_J|` z#*DHho{450d4Q>nngmbb;rcdRha@PoUS!_lqOak7rgUQ6DY91)y;UUG{Q-A2H7X>6 zmxDtk{&|@759RGrb4ey5=x_VNON{PVn}NB%#95Nr|H52zITA{Z7?Dz0lcMnrt7{t) zvIQENrgq0#U621RC>e=yfvSijWtQ#tS)#+~#QLfSKpYb>oSKXKhwu1UyAK`}I*gh{ zf%;MtTjYDTpbjzmiXWbOgT|?7$0sM)ccdCkDNR93h zFUfT`A35C>1&up^J@-(T#w9;C7uSAN)wW>y)ar2Osj%3N#XP%LgNmGjWsC<43h%>VzWk%c*$=LNa7%ZCIDs~rkb9W2 zYdoNwrwGFW9G1gwl0^Q?AFFg!QFI?NE|5H|2|IUqo(8=lWaLe z0fRb|+6JljH+A(mJ2n0!X1YXxkhhO^K%&;rCB_*ICRVMRCXB<4WY@;!CK#II_smUl z+Uphhw8b-Vzm!)#T^bLh$t_oBUULW*vQEa7uWM+_N5}j)t&}?`@6k<)vgI&WR0kj` zb=q=?UBb2IKbIdWvTJ>ptr~MPZ_}+vL z!PgKk&kR^m7o9%Hay|TED^!|;n@Un`%~x}{1men+BDES|n_Owj7G_)bzFsLvY4uBp zq83!g+>d4fHRnR7pl>{F-{`D!l4fwaSe?NFhlRWIr%jdJ9suvr?Glz%)|e^^GTnG} zq&}^e20#t3*lc5CyVO~23iP{hyj&LAm(M%CE^oU~9L%&Fqp9hnY4bp7+j+bjD*tg2 zBo}vmzScaW2F2QqNfeuLYUut61l6`p^(tjxc`nRC$H1^=|0QT3E8tc9dc0fmtdiL~@!V>>P?vKQYu$ufn{82mf z)xCR5Q5S|W;A*OPUaf19^jasSX0HUpS!v5e(tQr2t^c4XM;>~5QUKTy#xArCisSSR zWAoY6qM0dV`8x&BD>#bBT8k`WR zUcQaQMsVOLEoAo$@8tSdQzpPvj{Ozj66IE~#)h#pg|Gc{4IBnzuqN!iToli-$^ZHJ z-~xxMjh(?UDh@~eug@{r^X3a1C0UzxqZpF!3*MT>>G;fsHUYG67PeVj%Wh2blc`4G zu|;RRMH#=&juy{+&VGq=wX4BT>B2_NAtH==NUy1`kpQ*s z2x=3%pH&Yv3;b)BKwQ;qv@hYG~+5It1w#F7N~;*lz$H49q13->%qEvmUd0R9mN1WO?TSOXjHHqd=M!bZ_u4}OnhW~R zjNG*G4mmHEf2AhjyM1B6dgBq&8kFC1V#^6WY-Y+Qm7~t>s}9C4`K?^9{9K2B1}7h_ z@Aj0rLloCVvBPgLZca9rh|~TDJX6KWRb>8fd}XXfQdu_(W~bFF<16Klp4_VLP078z z4!vIx-50m&jS^seyWZR}&%`xd{xKpEyC)<`!Aiibvqt;7yC$OkUwalQW^it+1a`W)3XM#P+^TE19zpo16+RUf zoi>Ka7jB(QiWu*QeZ`i8f~fFf z0@xc?)?eV+h-y-SL)d%}1Z1Oxx(t0=K0?hL3J&kT_Vb<3DY;TlxgN?CyT>At;9UxH zBP}|5$Lp>s30DYS1AF`{?A@Qe_Q|u)L*cZM0zi#LLAPim#oZ0V=srC}nIOTs@$3~3 zr(CBJ18Cy67a8kDL)|Jbh$%*+f{-$V^F4k_zFlNGqPf-%H(Lj4Gs<=|RiLadk+RyKr&Zg*Y`Vjz(%}3j3q$mw>EYbT# z$dOfmFRk!Gm~5H#Rs|?(P+FC%K!itW9gBiEQ)nJnpzx&(oJRNC+(Yz?-=gAjFl^%f zAm`Gi||80Hb&)+?Zy)poUa9skE(wJ=6k+3wSUFlzW zmN%`3U!qZflYB8%H8ieRadEbW4ysX^55IfL%RkoLF-@pPCFbf5)9Q+U zi!x&SFdxD95Ixb03xEqno-?0c749%}@KEG!>hbIE|G-uANp3^eN6XJWW)6w@ct-+> z$Ngjm09twr?>Yb0zu`Km6?3ih-GrN00H3=HmxDg}5U8n{7R@mDbnxb(50EQYE-{^r zNzY||N?qsO;AKz*i{VxPeS`!A2fF<`k^HwC5Qzs;;AUc8rV#ee8SlKDFN>{ADeY@4 zJ|Z`_`!hfmHg^9{?pH+wEE1m~7-uAuA)vEV!G(!!!_V0c4B4y=$jp_MT3hprhEjz= zucs>B(LsF7=Ivzu()3Xk@bneb%z#V9zZ^88`i9&CR)5WeE5Ack-s#7v;u>%}C>Y~! zDi+h1*mh*o2R;B^5@*?`l=Rw=;16Y-zd`Op-5T9LX8#t&VKmDx^4x0F+jspQnK|!} zp4K$zLW0YH{C9I6ZCW%&7fEM7%vhjX?X_h$hg?H%pHURyv*}#}jwqdcdR#1$(vchk zK{pBIR9<(wnl@R1R#@6XYlS0Z60GMdW{B=Uj0 z-VY@-sL?g?k&ubmss5a@4+L=C-mzqs>loZzex;=gO2m@lbNBgux*3K+Vtlm5SV0bq z6v+kK#3v#eFD_*8+pv-1tf8$txL3H~XfLFNBjQAq>#Z|DJOeMpQ9BPP7D#37v9y12 z9epGWZs~MfQfV&yl`J@4MOVmWep}|fk(Un{rcL~JcoBy~mj4DP2lwJ5^sfh1;I}-4 zbq^UeI)7Fuwa<$P|9-ZFv^!Cdm@uf_uOobrnH6_H^VJ>PB|kQOFxlZ7-pEc&f3#lSb;o zv>XSRDT#GTU!F=mtW3V+wec~ZKPcbuvs1g^a8g2&Uvu8M%qV&AOdoMQ8Mw+8)Q=Y+ z5YAC8kli%e!`EJbi(8yP>dX+?MwR*=l;P6xm27exA_Mn)U5zHNgK6tL*jb$rqW`cc3`>)n=+ zL@*C7sNMIr(&EUgg^ilJ_A*Ms1@@JyUBNru#~E4)dXBh(97Mr~e7gs=g3&$CEv;$h zWa13o%QPt)ADXCr-usu0VWSpHET0if%GniztuS>w@S1iuiqZCEzQKR^=VvVl7FDiR z`wjnf1hxZ5{u{8lS!g{pJd%2+crd{^7%+q{ouXZd#!q=6$mTXCE z&LB>f4@j9W)2J?yEhnP&j5sckeBgn|Y8ChP`>b$B zaPG70>~r}sH3bSdpJ^ib`o7FJz9lzn5J6mklf3vG`O;HmlLOwr3J$jZmu`KbOvGsVyV;@2x)X- zq>)6J#|lr?DR?g8f&7Yu-$gs9<*gT%+R85cQRHk~5Mx}dOwP-fo`uG~FBT>J+D0N6 zd^W$&v+dK`M51{2882sdNpJ2F7*?h#U2=&{9VNc~G+MZ@~a#ebme zuYGlg)4s7oum`bniMwLz-~J|}hU;vI0=4U3W&d(coqXTp`^mU$!5RXd*#0#E+-{;1 zpr|nNJq5Gn_wp;ATL?>ez{ceSF)%IF1PWtTTjBAmmzl%2U)F7V3BQ+33_1P6VZ9)? z5nH6v7H>9`GFKk-lD6KjWy>Y`izcP)b>H5OF`q|e{`tYeh$nIc1dRyvp}NkXR1itw zaCP=`1so@8RUfTgC zKIFDCVq4UFkp5&C32&BwAlvvdE?PDJt2#Q_`F3fV*60+G$juPhKA&#Wi&aw|O(rL{ ztgUcB+(>!bXUNjokylaaDZ48Z=9(-nlL;i{Y*Ui(uEU?-05qlZis=bRxBR$Ee{=bH z_f%u7IgI>Ja-(PY&>G=X$ADcuDZrQM-wrG)>nX_48#?_p$gI%X0_HeRZ9tc9lHe9L zN=iv3c??1UnShsqrVyJGdlAcVbsX2o|l|PRUIxA*W zV0cE!ZAW=$07^eS9&M@iSf?flydRnOg905nC`idq8nL%(=d4IGQ!U?{)|6HO6wxDF zF>`0zgIHqKb+VOGiJlGbi;Pk)s@jqfKWcj8zp7exVBkj9oWv+>s8zmB(ED3sF&2*1 z7~b>SJtCRcm)af_|I+DUB5mjk$M;4)$K$Yd$W-@}Kcko9)%ZxOLPfj2+B}(ijKQ!_9$ zeLv%7teL7@r!%`95gDVkhAci|*mU=2zAq#J6wqK095Bot8kZ2pSTmXF4IDGhX|4?| zLdQ8bKVoHZ)foTk?+)C(6G6GLN@_|=dAJ3^X9TUu<+r8PlWa zQlMG&>kNLFYYVy~N;D6fp*^q@jfqjbuG2ee&Fe#l5e6M`3gFjqDB;1^_mJK}9Nkw8*qcg9=ZSHZc9Q+U2|Z@*2_ zQ{uB`_)zJUQ){jHSqz%H@hd|C_|Wv2#SFMEp6j9cixvF0=^t+BM_xTv`P z*z0nGkB*M!G#wFcaA9E1!jf3gzJ`1$hQw?Z=jt6%OgphNor$KC_W;>SFa%t7f`!l4 ziX*!0%Dh^Zq6Xs``S&11-oPbRlExOg0>#WqEeDbBm@WrnA@&vnYi`^YBN?%C0OpGK zyf7xOuSLXbEmytxqN;YeDU9*l2A-Qo%cDu=E zMe3`fBJ{y77$PUWJmRo4nMBRPi#3N%zIxs$050?eBHzF6^Pf95c7NRYd2ncZ427p) za=CxxWRcoF)0NX)W$PykmsK%2M{rS?1L6M*EArpb1-YaTO>_pFztv~e^14Zn{1hJF z4l?;DvBxFG<7W(-HBoqV`;)uXf^_ebg^r>IF|?v$3r6}a&hU{tMFRre>?;m#wgGc3 z7MW87niY?ilmWR$KFz)r$~(8-jG#cFSIJNOtQKtqF*$>3ZpF?0ZsKxnVo1@anIDxa zeTHu5B($N4r8PfS5Y*m<6H$zz;P=yQkEAE2e{S}^Z+XZ2Wx=Gj3&lStWOHb@(Q2j~ zbwmqBa=ZQ*?1~Z6Lf3(A(_T9cKc4d>)zQc;u(%>vc|z|45f@+~0lhHF7FK5ZZ8L`@jMckHpK~e0xldx@UqzSyVaG=sJF6XA&el%gRcAYJsO@y_ z-PHD*U1U4?ksM+%S6NtM!U$dRELaLuK7AkgAnH&d9!z*@$j6}0-ev=Q3>dh84GZRS z=Ur1|78nEk(6WR}&C7>CIdY~jpA0Jzc&;HaI4fCNxgTnUn*F|Azu=`ES$4lg8%Sfw zsaW_3TqoG7 zW(Hs~F|Sg(g|SbgZMQT*vg3!OLkZSRaZ&6a$a$0CR4gV}oF^MZ8Z6~obW+@n>__U~ z)?WO^-p z26@hisaDlX+j7KXVPSE)&g&D3ZK}Cm7X9SR1@cu7RW5)4u0#xMcrZHj6pjZ7O`!3; z#@6=rT=LSnX>i1dAQoYT<;V#&mJ|HFffh(VjKEq%0(+|>JfFHpTNG z3lw)ZM=
GeB+)_MQVfx^qdpm;_1TDE5eT1nA^Ow0=ZVhjTwPmv!O0EV?$tx|zF^f)Sb*k%202mmFQ% zjoO+@)JdsHnxGR?HaypfnZmx6N}1_`vm$qQlcorc>?k z_qtLqphvBD+T(e0`)BMmN0T|3cXdvFqr3|e|0VPJZ!WCkX;3p)x_o|AoSE5tus0_r z#V(#Hn`uyZ`HHL{0C~M*O_Zk7OoI4?wxT^2o|Gov$leul(?-or-Hp-`;EB|uUYa0H zd`weuS7Dp5@hf`xD~-~$<=$~~h0~R`4!d(zSMjqY($_+r`Vw)KiTDur9b2s>czmSi zmKwn<{zwi4E>mWs3GaetId9RK%7gbDQ+E}BaH!31l&j1}auc@*qzHq#Y(~?IwaN$f zmS<#wQTi^UPyin>l6gYHya7nW%%pgYM5_+;Vu;C^eOFFz!|U0UoQu?ZBETJr7E!gm8$*kp^SV47Wumg zjSqy56E{dR$Y5MScC_f-j+#z6Jc@s&q#_0oPA)WYU#7rmxKx6o8Mpg6(2xtN?xqQX z4ds%1<5sG#v6eu`Lkk2~#oo4~6Ilr^BtbTweWQRzU#)lHdy#q~PXOermwQ${0=`;g zpxl;%gFif1c0aid^Q~JWe)Mm2*AI-E?$3+Y@U`>hALV=HQ_7&m9P7sb96UAKna=0L#C&}yz5OVw)0H*egz|I;FXUFHz$ z$H-xaX*WNRl)2J%307{tYCX0WN(bl5+zSe-r;t|MAY>Pnp~yI&&ms>KA1HK)bhqfY zefFUl)NgW9aHpV{Buesrym&Qm1(@YYE4Vou_QggTZm6gJXI=e&aPiN9u=aP!vG?2a z#nKLpNTaT9z3P3F>@Mj&aH=t03dL+8yNycS_})jFpfBL?DH4kswJ*22<#k6ltdd0V z#x~6oc+@r1s7DVI0tAkrn#?!#VT={ptUHatC$#!W4>bqKT^Q#CUE=$baOHhRNv&0( z;gn7mv+~_U^Z$N%%Q!QB0qb&{i{nm?ygN^5|D+x1$HKQs6^EXI0~VQ5oBkX>LIqyR z1!G*^@nXs?_!4y1Q$#m2V9JrFB*9_5fFfqd=j&??Kwu6|tC~ScOl3~D4`t-6?lW+- z!vvoDz}Y>veII^_j=lqlxqg%-bvhPCQSY0cpqlie)@EuB>{jnImV%vS?zIH>E{rny zn%eDmj#XT50<^x0_1AKFPfN8#0GuwTibWW^ZlN`o9XW_IAZ^!YbEpsK@^)wYW6nv; zhIQiyTYn1_*Q8+XIqkRyvSI{}FaOb06^siKHzv?tIy_C8EWS;Pm+m57H+=Kds8x5V z$q^O$JVZ4y-cXXS&g+g%&QNp{DF05beaT&+ed%nd=a0o;G8^MwFa0({pe#u|<`Ql& zF73eG_C&92kxk(-Gt(um27NSjHJa5j`9X?bzo=0g^Dx?j~t<=I^AtB5OSL|hY{bQJMEv&HNWFj zo|+3aC=vHu?6uykdDA9h2HrkvjekxhT7!N@Tfq%HZ{`Lb4D^`1k*qk;EHh{-YXLTX z{j`gzy)tEgqM8JIoee?)Oq3v?J=`SzhwezURv98iWjDIq@)Em91tY@nw$)(3=eGCn>DoPOD)ziNdgD{JqJBpzZB ze@;$&g5Ua+h`Q!%g`2Ciifo?cW(*^kFT1fRGaHW4 zAq!j}g^Hf)*AO%NATeu@xhmgW9+)Yzy$Fnq9o?38ozxck!gNT?jLcB6QlqAhRExqG zm(ujwhlY1l4_2-LuxS4{D@m$k(Rql)RtZe`U&4gM6Kks6{2=6v%3SWa%W_;LZb#R^ zXA-{5i+FHry~Gs4++eEGI~RZQRdYieO{?*;X_ zX6pi626g2|`XfAM1rFq`MK}~mdDyGYlmTQ?$wljdvb(B1!^VP7CQvUzX|mp7Pa~{9 zs8yPCTZhVAD82?=HKpB8W55yJp>s|Md*hd!1!y@yWfBmKNM?6Tc<6BR#>+3MW05C3 zt(*M(TTl~DZWVkwbpAa9Q+ww&82j6s=6M-Z?Rr4$d$gKP`*eO+UDEP7?KN0EU0SU{ zI;T1GfMS##{Y>A8#83$^a`pRUA!`ZgDIma3;eeXK)CMf?kn zS_^aOa=$rZ`T(uita-8vq}DMD^10y1f9BZJNsVn|@88Uvx%g?vduawnUVr#Fl3uPS zPb#APQ~YR3A%Dq;SmGyCU`9W{(~k9f9jf8}XD)L(<# z^~2GpdoyX$tNHZpv?w$e#DFvp zAZwA*Ai}^@dvS#5^2ux@ZcN?PfYSK-iXn5)^Mta3UZUvEbV)?J8C_h1^&cpu7R=)D zWIAn+M@p7cC)Nh*tzi=as0*!Wqu@J_w`jjA=Ooj&IHlx$J@`&DGL-MS`#w2&6PH)u9EAOQxuCeH@c};!=>U86IhQ@eG8b)9GlA z&Nz`xZa0R8+FUjgDk~-B|LeiT=M|BN4hZxp=@r+=lVR`DXe2grKTuL`_N&`c(_sMv z39(2uZ`^U>qbCEI$5mzRGU@z~h*K~#(Akm>b7292VtvWYxxz91sCmk8Z^~1X&(9V% zB~^Et!l>oz61Sr@=_s1}643!sZbt2Cod@H4BMH(s8#xk{)zyNUOv7(P*$f10YJT$G z)#{|wy2}9(Z<|ZC+12bfe`QD3Lf5_Wqa$CsS}HRx05oAMLzm2)p!yQ~3*ZvZ{G%{A zUAs@RM<7}6c*G^|zK|sN_h?%bCinV|5n^}n0;k*cY-2q3ERi)}S6RpIKy`bBzNpjJ zn3F8_oIs=PL}i6RxltV%!eP5MX*t*64it}_IXx|}0A7Q%CZDX#SKlY*Y-G5{O2WQM9yc5K zca+};a&4Fg^`+^(9C(s4nmIP{YyVHhv42AD54=y8Pt}7hhZQJ3Is&0q%YSwA`@-8M zfvxSDm3Q&@_1}FP_PDl~S?z1>9ccf{X|OYclVJ~(L|BRNcX;B@P{0ox26iheFaP&5 zyPrf+zrtpH|F%*7U`B~yzm(tB*6)8m(}Wi0T0g%3^0#UFpI{Ij*4o7U_5XjI$?@c? zxTxy>vy1=Z2qFwwq7(lu_4Q}}`6R($$M<9`y?;m0|6jrNo8L18e8h;{%4kmN(%gYG zIY~~l*hxz+*`L3^x*5%Mb#2UVqS{}qSvJ@o)=TNPq;bFerSf#yb{0i)_Z^J}CNK>{ z0)zZ%K^x`Vg#L__qJa!e5Nlra;`YSA*2M0nx=TNcVo2g2{#wRQd^(OYZ#*}Fl7fK- zxQ83V6aQ0r^VbEEfIW-qrD1rKwN}|ca&Rt(VZgUpvAijiCDG=YJ&;gJ5#Z{t4agJ`N$ zr-6&(;^(y%|G3T-O_ZePuMvGoYSZZUEfLm(@F)kNca*V}bNTJqZKKL1?tTTb(?6gx zA?iM!b6j*G&0ZQK2Rh{eyP*)a@w5@6$l*-d zefhJ@j>!k-s+Jd}Rh1Xb&EV@UcPML?>tTNrLfk-_))EQ+~o`6|2p*ssEf9@7|;_mx=oULS5Svjx;z-Fh_Xr=FXD`GFCG z_p-?8^&QZBCHO=prl%guGtZ;J-K@iT$dA-?DE}f?qx$9P(Sx6AxoePAOY5uMiGeOW zvM5gHgs5g$`=K$I0%S()agcA>>&Q{x8JCT5>Zf|l`Cff`6LgTR?OsVSJ>p^EB^Y@5 zMQ3xDj&TSkMH=w&lQtWE7eulax3;alm59FZiWb~5 z?(^9scuzTpZD?bn99&8tIR`T#y>OE7eC!wxro%M;&1-s?=4M%jM>!-WRz6{^uvprh zyVz>yoauDCRSQ-Uc_>_YaAPt~Eboum6mtb{f(}2X&>;`A#+-Q-)Z1^GFDw^qK+6D9gtuUn{_+Go)-|CIVj&O!dZg>rlZW_Tn0tM+6bxsSKU z8|TT@<2BY=eTgxQk;3#;FVEw+v$G@WX0rU z@=1b2^D~tST$Xbst+0A!nPCe|J*F7D>#&u~*fseMRHc`S0#>R~ep_7xG7i<&% zA{4HWyI3b7^KHrq!UU;1qg_C85^8ZGWi{L(%F34wsz^(Hio@hhQ@6(qDu-QUU(Kw}$jjFX_hs~86 z7JH9A!?wh4$|kNu3&yoFj2X-|;AxhVCdnp9N0)0%u8>mIWzcmDZ|TtoTo`{keZ+OR zvEohJ+*i|=V*-%@xK#>I#dEU`W%l%uH1pB?{ zy%Zkdba~SuR8>3a;2Qc3x3%<}*&W;JDlFp~?2fe9zkl$Wqr-w83T37K$i!vzNAfS3 zN7J)XN)X;FEqvm#vIpOZP*SQoJyc7hvb;mVt7#+?w2%39vqoeE2FtcIjt&X|YiF9u z{DZ83q(~aatQ>j6L@gVjTB6*oiAdB}LW6jl4OI?M(t(`%?ZIqAG-U*`)~8x>@l2@A z@p5^6#T%iLNTYEy8eTZmFOpC`u&#-|J%RRoyr7wD4te6Bzx6e9Wc;J;t-g4grPf{r zxicz?up#Z2(WC17@Y(37I@aGO6E=vlj5ZM8%JLep;xx#2F{1p@PSIY*wAeL)JArS| z{NmAlKDoWiQzFeWgC{)q{^%XLON+daJBYea4e>d0QejKgm{>+W zZW%NXO}@xm?_q&HhI#Mpu>~ zOHgdpVz0;WAW-B?E-@Yy(xl&N|Bg{?qTK-EaQiWpZs6J|j!za|Sj86!ieN`ufWnl( z*?qRg-E63MmrQ@Vf5fY+%$Zb^C(;hKJMwa_h7LSYctJ{%!V4o8IkApNiD9ffi zBHl)!`l1fA^XMNpTdT~UVDu?;3ldSG(0f0uS4e#xc|RMpdF$sUlZOHw=p6>IttQXI!%`d%illmE+H>kt<0fQU+v zLjtRYV_S}<1YX+3@eP&xm>bufPPfrerl6}P4qwXp%+81Ki@5`9Kn0D(O-$r8NyaM?ole< z9L{)Qw&T9IkxW-S_>_5T^npq(>TijxbM0E@8(IA8k{!199RY17LqAtTdGai_oisr( zs@Bu}Ym6XIM#z>yTpB{cslce*{SOrEJ5i@PBq{f@K!kzn`C$2BmV?IRc9%cH-2&echgZ&>pK-V z?gO?wL!M*+k1}OA1Ayr$QOxw%i^Foj<=`}Hhysbfcc=p$erC(r^tr(EYMF1z({d%N4|AonHygQYOsikin(H0 zXw*4aWg5Lc%xFM}Nz>Qf&m8^bxKid?%Akef}yg zO}at#teV{61SK+e$fJqaH@rjqKBDy+{68QOk$E!SV~e zoBqTTU4l{SDuIsIJVp8%3anl3l5 zP!CZOiI(=Ku)ai)D(AL;-%ESfuSZX$W%KR!RD`_svkE z-6!0ddn5ImSaFns`dnT^cpRGFp@iZR=i8QmuT72qGuQ`k9ge z7lW=Ac>BzBbr00hj8fWPWWU__u2kBVZ7X;Jwz&;q1W^mylLCc2PUZTiBk}OC89TP` za$d94K_ybTK~w(n>Ei|TLa+pzwq2QBB2;R|a@=K!)}LRk$(TsQ5)Bx&iz6cW{ay;` z%)goRslcZtG%cp;%G9fvVT{ZikzR-90w=7xNcC*?XQ@Wm8EjH;i=muEyi|)pXs@PT zHQe(YlWX_dV;{kK)Ift)=DDhm910tu`VMAj!N*=6 ze`U9Z2x#+`8GaGtp07Nvuq+j(QFe3mRM-;(bd4>woczXN;}^DHxb1khg#SSqP@2$p z+g0P5HxPQdWkqV2^A-f%DC#2}v)Px9AWbaT!`go^2|Il<@#x z7RYY$>Toq44SsI<7pZH{V=-g z>>6jA)SgqG6>+=S1Wka~bIfMKMh_z@5pUia19u)W3%16*J4*y!IQPO<-qQ8%8%d^0 zB^I=of1*~s?p(tY>DVl_C`x7)N5w4BPLEudQUBw{Bc-1~O-`Tk0P0UXI!vu|NeLQo zyci!oY`Hb>Slf(9Ot}}4o^^)7=3M2m*JomcJZ$zmw@BZ^(&rLp&*Rmf{CiTy4g5S= z3`Mb-#S=IwKCS1RSl)nI7cj1L{% zGk?Drnh;$I-zCChA70{}=JywU=E|96H^RMYJ@qDFhl(UV$}!ttxx1X^Hx@Uq*T4Iq z;h+z&RCrtuJ6CyZeim{4Lfg4Ib^=?aO#QxGb+_ZzT=|G~p1-4is&HnUqC#Bz>O(rl z63wOEY5aHA>9kI;grDQO8sY1FP7Hn03CU~5(#>Sle5 z3HkoOoXll8isoXQ1)J2kLNez5`$M|-3!qYaaL(&&SP9bnA6kZCvw44G-#c=oY>myi z!1>r6E4+Co4VcbN8<|w(7CrI8+d4)owV9d(%sW9gXjhgb%AXYug2Tu~rn=qV53m?p z=Sl4^q&WKQWAiji$Q-diJn$HJwH>ymM14HXHfyx7DK5$8h%+zM66mnEzEKvi|iaQCxuOZUKCbpzJFN37ZUPV3-|u zK&t8ecIFfmSHNRs6CR`Jb4Ya?TzRpP4j~2ljQ{ragZoh7zBOqvG4tzGvFN_#+1T+` zljP2?54d5wu>R(>HC^RLE1wa0l&#~QaU|i4}s@U{G!ekhzX7t^O2`K;WX5k&p+e7fh zKv`ik)3*9Ue0|(6Mhv+(@sS9dovS3+PWElfhX-WD%tZRfEAL=_T-gjS&0$(+L{|2ir}|VzXtP6W|N2Ejf5*3)V`^icKjFylUR~}mN;iO zzkmYvj;-JGpU_pz_Jyot{X~sjqsLS*v{BX~|B_AqTm*Y8m9qqZf#+LVwZsh5@rqClWGXQDw4PAV|@4EAW;Gwj73!@-LAwqB=n9ji>ht_%!!4Lcs2{2HXA( zIO4dZ?AFr9>tnd`Dn!?Q>z9qIJsP0^M+)C}19)!vU*E|O71T{)TEvNqCkn$aUNwwb zX0Y6HXiIJt{{Wdgm(0Z^+K4ff}-Po%UWm(8hoa@+FGT_ z{nC90$d!=W{7WyteuKp!h?MfQjQmknALX=yoD6la<3exaDx$@0apsTk@Dw6U$rhZz zQdjL>QlI83-6Kal!|sOYlVZOtoGRSgUzBny#T5B}*up0Dp*i@Jo2E+us{-BLGf*3l zMz-5K$@%Ng7N6YQ{o$$cGePvm&a(&!p?we0a($QGyZ#-Q7T(VgBVMy*4(o03wNgN~r-1%pL}OZvT0gU5%M!gyB5bVuac;XC^y1oyqG*-6VU!fZJLl$g;Y+E-r1oAt z{`W?eO7&nm3n9lNg5gEN-XVEIqLWS*-mGDmigtYai0^C3)168lFZX18wxQ9jDd+STiLXsZ(dOFS*f4_ym4Go5BnQ!J< z*$H}A+&>|Rr;R8l7Sl<{H1O5lW+jw}p_*J} zxpg(D%+kD&C|I4L$1=F%A3j#0$vP}z(Plkh0VW{%Bw>EvI~<|kcu8O!Z9#Y@3mO;x zZeSz;;*uOW;)XW`y{c==HGy$!-G^R%*Yr)~F|(x1OOjlkToy4I?2pD%!1>bG1Din? z`(@=s0~=J7RAwKRZQQ`39EbQc2N`su zDP){1Z`-{~q1G^^LIqR%LB$R`9O~Z^-9QSh%?c+(jOi4EO>)0yPsey6Gp}Ip)~8sD z72NLI4N+n$CvtyM7^N@MV*2z9&yxD0dOWGI(|>!y#O$Y?L*;WyIXGa-us1012{4>; zz3J!V>Gm~l+e#BuM)-XW(VJZY6a5!1((B{jSB}GimNVA|YNd&oIlmmt?0*?aliy#$ zI6PDw%xQK*_sUR9GB0kmPy)=i3kZFLW5cAaBQ^WroLFH%apOplX>-jlQRr#xN!Qoc zEJ*T^Q=dvs5iX)&1?KfnFICD7&(IsumxSKlwB~gUX2Su~Jw+e&2n9dKtiJ{jU9E9D zoFc|hxohsvaRRkxBYoO&f0_`!?MC+3$PXiQ)K@w8Nn?|aAf8!VEIy@@c=oOXF;I@N z@AqJ)cjO7-7-AuSFSY%tQ(z3lz$x=n93RG!Xd>};0<-sw-;1#}$WP88M0Z~rWiQr= zslez4*ooh>78PGNg&*aK2!SS&q6y9l0A76kvFGEg!PIq#s4(hR(w?^VP{KUU@P;)K?^9{5Oci5=F;q3RmH9D;k;IuiFpHOELdp#+aQFv2v zNlOnSDWrqho1%PwYaYjl+qIEIf|$wZok992s5t(3TyRff-w)*vNBgF~*qYk^pZ3oC zo$c@Mhqh^gRilAm_t2JYf5LMJDiqckx(Hf~B_NI1>BC1wV#9kq0rB+&MrdCVM z5LAp5_gg>TU+(*t`wzJ9{G96~*Ez52T<1JrkLUByfMZT#XyEsOcSuCuPmKlt1Gwp7 zaPXfsA(ced%>^odfUz5!i@D$+Q`TZDWNCb`){RbH0@N$oxsGUV>8Eaj^sM~DZmCoEs~TXY!7xSC5+B{q zOES(`v&s}2*2KzA>PiWaPV+Lq(wv4=lg(ZsD&SR^iy+%2$1CsEol}y%bnXPNm_tMWt z0AcNv&(A`yXzU530FrRxTf)6z2AlxqYCgLbqjyO955ttb?KEvg!~tORMisg=r5E}% z-V+Wd6mfI{`Zx^j-`Z#tL^t>T6K=+c|Aq_2VynHO*jyEl-0Dv@tEF!{G%Y+u4nhnER@6q^*QEe|6)mz?MA^aU$&EUz+v6!*1;y9qFhwy#nQp4gVwPi<+{U}4be@}F1-1SM9qN#GrBU_1sXrciPxY(canfKfvF{b%wZFE4!C`qG0ssH%V zH!MDPLkM&o{>0j0*y67!^G9Ww0SmUSWK2?gAc9QAa8l*08QHZjUhL} z?keCK@g)~RIFd2~XoI{lOUcx|1(FrOm+Gxgw|bdQadrXma=ae#w}~rX^u}M^MTjcr zVXxlp=QXtD5YhnE&>H}9g=8+{zFO1oi}^hvo49+T;yrpWnwY8WM;+hpZ`JpX%$2p} z;n%xES3~HBa8#D(X%r{N>6m ztoiGXDh#dW-FF|(#7Ri})aFi3cT_DxsuZ%aV4#P-UZWAPKi+jG89-k0XwLXZA2X|( zjfW$SbWtiEi@n7vA@-UaO`g&%c3VzlKdU`}>fC=4Fld=wrT*1%OKp(pU|f?&b3R0> zzI&XN1fR9rTRRSUTCWmsE<-`Vl@4<9D!Xr2Pq9z}Y}_k}F4f`s(+T$u3dQd!zo9}T^AMgdFi|4HmH#w zDj<-d615Coi%O}sc3kXEBB9SGwZ1FLDP0s?7zfMlCjUp)q0oNMx%C3_`5vamE4=#H zHHEy5_MfKkZV54nGbB>tk;7Krgep z0q-%sqa}hn9NVqCt*0|jj76Q?*){2TO1qm($84K^8h%^mvt>atj@MYtu89H+^Y79A zl1s7_p$u84)>@KnrRvXhB!whJghN9tl=OS*2E2k1R0vh^hy?(DyDNH_Cwow+eX7B= zpl<0|mMaR-C&9cS^7CDk+UXaE`rKhL50-c|WHN;jG%ivm|Eow{{DRZa~Ipj~X zS)%~O2hE0|30V!L9t7J&m$~-F$`nRJv2#`Vm3+|^gowVJvNk&qtO+uMrj zkW_p9R2oG7Qr@n0#2OWumo@3prZH%G@zg2i45*g+y-7-~#)&AiI-%A=ZT`?KrRvrk zrR6fiybYTEHMBO1^ZWtclbbgqAzdQ)2M= zx2KfV<}esva5()va^Hnf5`GYcK^H6`Io=9nLOkYoO5L9jd>Y{s+vXmyPH6V6yhmm| zMS2VSYFz%CHCbjo^d^8vGkZS~?8dj+$qR`Th&YS2`d(0^pW=jNf$WhTGj6RK=yHa# zX$-l~qXYLIVija^XDY@n9k>KgPp8}TL_m^#X?g;39ReMhN6hdtXK5%ydj*qQpH?gM z-hTN2?mc^a^T`iSsk6DH^C;gl`?nGzecK{gNfCX={s%e%r%3;?&r}jx03C!pR|_Nx z`C##nAA((MpMBW2;r&WaO;`o?;fbLFBopaM=&CrC>@mtakjB92&~LqkNVcSu(izOR z2sn1tXrC=Ru_6lm+59fb*HXQC%pt<@a z)jq;QV5To{j^D~of@ zwpUeSa9eApW>RA&0i@BxMyM^KEYp>SwWv^>HSwWS9I^g5f%$@*@gm=KucOpgz#}^u z(<_GZ-U()ix?J3__f$mvDmnGa(PtIg>Zl#bx1aF0Z}^tz;)CCX;m(F_4r|*f)qnw|C3>gV&pIY>@G77ccsxh84lSuGyc8nx=$GD4MK*(+X z(>w6mmf5hjR!tqHB%-Kj^9f-91mj*~x2WhHqk(94{VImZiMi7G#k zhwH-+XW29S_Z*HiD4mM&j^H8Uq<4f0~*wy`+Mr z!z7c1N3sh)DplN<$5o#UuU<#4zfTf){q51q(B2t$IH4)9`vUb$B*uzOk*|C&5HiMH z&~&lA6S>u@a?gZhHR5#?Rzi^Tgn3R9J0z|;4X%B~($6lRHaZQ{p(@3vk_n z;sEkiMlmR~A(~|Q>Im?+L&KSucS0L*->q(4LT&9A0 z${}HHec-v3Gskm6J#^(=4ENjCrG+)@FO)cNZUB6Y(3p|TVjQb^d)CI27fgxy4~jd3 zmBkAsmC_AUuWUsw6b`odi64t0O4=@zUfj19aqi>6$ZIyc=jyYSPdgxLoSO8ike&O{FRy!dVVwjk%^ z19P_sBPGrTU_4TMxd%U~E?L*64%mT>zzRf{NKpX3`8H!%w{ajkxl0~5g=Pr}B&j`8 z=i@SsbDk92+q5_#KTlvMIrSs%NQt268#P4kCSD6*1g-Ng&rFg6jkRB-^TID}KgBdo zR6R~Sq-W`)WH%!R-{@Q-8`_*s1Ql^Wfig>)H6f5^|Sq0C_MFV(vcW$?eME zRFAv??RGV_-+Z0s_N6s}`M$76ZWT)+-Sjn}e`Bm#B>9!r0j3wewjx@(u=AcDB3zzD z+)Z=kUrLx_y}Fgwb=RX6932GdmyOO5qH^Khj1gh2`Y3A@8qayzytM?3Dyxz_4 zkd^&>6+AX9U8rZxTT7H6TRsx}UA}8MfKJ>VJDeW@@dXJAi1%Ckx*LI~aZ72j*z|aS zsPWd?#ul6Cv!kl%NN>iNbk4ncyx#Z{cqOg$-4yV?+Q3L002DUmVBdBLonlRWAOC%W z?Y{J7B{8xd03wR8IbQ!Yn5$xoxk>k<7i~AXOevHu6ehu8p{|36!0#?17tE%-X>xlv zWqJu18*CpstxjfT#Q~$h7%JSYEx^7^`nQsMnoh} zyP+k6vScj@T*FR(FGT%Upfr8wfpoWU8*SkSgBqvbh^%!%8omAEwZvId?PCs4I}epq z&yaN}b@GIi<($vy(BTjjVWKEI4L)MH*3p2Z;mXgU5z?YWK6AfGP&^_AsX7L@rQdU^ zrDKY=vXgYT(J`&V1_K1$>wQ(76Sed)X;c2CYjx>xBDf^eRg^?Ad~JYGkJ7tZ7Oi$y z?!oiobcpy`ENCs zpIb`lsowyw>DLTesrF1HIHt=LO#xPt&@zt#L4wY&IKqGZ5Cb_H8{I*r>v%pi%FiNl zi6ypYxSt!$G8t8ql5KdH*5<)u}5mQuM* z(ff{pt|#ednZmkYcpQf*z_ZYU9!IB$vmdCMYiy;*%^l?%o)VZjVMZ3V&!PsbU*kpo z`aRf`XeKl(dLK+kugKU>XqM|AF{6$CiI5J*7nceIlOJXpIVG6R1sN{wUj#%K@b!8f z6=1Dj1^a@&UTlRmCA+u3q)GcQCk^1yY%qNjZqnw*&|Rx8=e*DTbU zU5Z|&UqoxG39AfK}M z98ZYBbrPQ}sV);=R%_Udd@(arYhmtlKT7U}VieI)|@e~AZ7{Prb9*NK6tU)$m z=a`f(ij8&Ba60(Cxg|CZoZMR1S^it1>HoCJ{~JBic4AzBv|-- Date: Thu, 30 Apr 2020 14:08:52 -0700 Subject: [PATCH 34/35] update README.md --- README.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 9d2462cad..f9997c1ab 100644 --- a/README.md +++ b/README.md @@ -19,11 +19,11 @@ run your own training script (Zero Script Change experience) using Debugger buil have flexibility to build customized Hooks and Rules for configuring tensors as you want, and make the tensors available for analysis by saving in an [Amazon S3](https://aws.amazon.com/s3/?nc=sn&loc=0) bucket, all through a flexible and powerful API. -The `smdebug` library powers Debugger by calling the saved tensors from the S3 bucket during the training job. -`smdebug` retrieves and filters the tensors generated from Debugger such as gradients, weights, and biases. `smdebug` helps you develop better, faster and cheaper models by tracing the tensors and iterative model pruning. +The `smdebug` library powers Debugger by calling the saved tensors from the S3 bucket during the training job. +`smdebug` retrieves and filters the tensors generated from Debugger such as gradients, weights, and biases. -Through the model pruning process using Debugger and `smdebug`, you can iteratively identify the importance of weights, and cut neurons below a threshold you define. This process allows you to train the model with significantly fewer neurons, which means a lighter, more efficient, faster, and cheaper model without compromising accuracy. +Debugger helps you develop better, faster, and cheaper models by minimally modifying estimator, tracing the tensors, catching anomalies while training models, and iterative model pruning. Debugger supports TensorFlow, PyTorch, MXNet, and XGBoost frameworks. The following list is a summary of the main functionalities of Debugger: @@ -40,7 +40,10 @@ The following list is a summary of the main functionalities of Debugger: See [How it works](#how-it-works) for more details. -## SageMaker Debugger in action +## SageMaker Debugger in Action +- Through the model pruning process using Debugger and `smdebug`, you can iteratively identify the importance of weights and cut neurons below a threshold you define. This process allows you to train the model with significantly fewer neurons, which means a lighter, more efficient, faster, and cheaper model without compromising accuracy. +![Debugger Iterative Model Pruning using ResNet](docs/resources/results_resnet.png?raw=true) +See [Using SageMaker Debugger and SageMaker Experiments for iterative model pruning](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-debugger/pytorch_iterative_model_pruning/iterative_model_pruning_resnet.ipynb) notebook for visualization and further information. - Use Debugger with XGBoost in SageMaker Studio to save feature importance values and plot them in a notebook during training. ![Debugger XGBoost Visualization Example](docs/resources/xgboost_feature_importance.png?raw=true) - Use Debugger with TensorFlow in SageMaker Studio to run built-in rules and visualize the loss. ![Debugger TensorFlow Visualization Example](docs/resources/tensorflow_rules_loss.png?raw=true) @@ -199,7 +202,8 @@ If you want additional configuration and control, see [Running SageMaker jobs wi #### Run Debugger in Your Own Container -in TF 2.x framework using GradientTape +The following example shows how to set `hook` to set a training model using Debugger in your own container. +This example is for containers in TensorFlow 2.x framework using GradientTape to configure the `hook`. ```python import smdebug.tensorflow as smd @@ -221,6 +225,7 @@ model = tf.keras.models.Sequential([ ... ]) ``` To see a full script of this, refer to the [tf_keras_gradienttape.py](https://github.com/awslabs/sagemaker-debugger/blob/master/examples/tensorflow2/scripts/tf_keras_gradienttape.py) example script. +For a notebook example of using BYOC in PyTorch, see [Using Amazon SageMaker Debugger with Your Own PyTorch Container](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-debugger/pytorch_custom_container/pytorch_byoc_smdebug.ipynb) #### Run Debugger Locally Requires Python 3.6+ and this example uses tf.keras. From 5fd228180f6d9101407050b56cd2fcbbd60a7c61 Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Tue, 5 May 2020 09:44:52 -0700 Subject: [PATCH 35/35] fix issues --- README.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index f9997c1ab..fae99619a 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,8 @@ ## Table of Contents +## Table of Contents + - [Overview](#overview) - [SageMaker Debugger in action](#sagemaker-debugger-in-action) - [Install](#install-sagemaker-debugger) @@ -257,13 +259,9 @@ print(f"Loss values during evaluation were {trial.tensor('CrossEntropyLoss:0').v | --- | --- | | [SageMaker Training](docs/sagemaker.md) | SageMaker users, we recommend you start with this page on how to run SageMaker training jobs with SageMaker Debugger | | Frameworks
  • [TensorFlow](docs/tensorflow.md)
  • [PyTorch](docs/pytorch.md)
  • [MXNet](docs/mxnet.md)
  • [XGBoost](docs/xgboost.md)
| See the frameworks pages for details on what's supported and how to modify your training script if applicable | -<<<<<<< HEAD -| [APIs for Saving Tensors](docs/api.md) | Full description of the APIs for saving tensors | -| [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by the APIs that enable you to perform interactive exploration of tensors saved as well as to write your own Rules monitoring your training jobs. | -======= | [APIs for Saving Tensors](docs/api.md) | Full description of our APIs on saving tensors | | [Programming Model for Analysis](docs/analysis.md) | For description of the programming model provided by the APIs that enable you to perform interactive exploration of tensors saved, as well as to write your own Rules monitoring your training jobs. | ->>>>>>> 15ed2dc... Update README.md + ## Release Notes