DLH_Team_71_PyTorch.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"view-in-github"},"source":["<a href=\"https://colab.research.google.com/github/ZKingQ/CS598-DLH-SP24/blob/main/DLH_Team_71_PyTorch.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"]},{"cell_type":"markdown","metadata":{"id":"nfyi6FdlRDyg"},"source":["# CS598 Deep Learning for Healthcare - Final Project - DeepMicro: Deep Representation Learning for Disease Prediction based on Microbiome Data\n","\n","### Lotte Zhu, Kaiqing Zhang, Matthew Trueblood (Team ID: 71)\n","#### GitHub Link: https://github.com/ZKingQ/CS598-DLH-SP24\n","#### Project Video Link: https://mediaspace.illinois.edu/media/t/1_r5pnhwtj"]},{"cell_type":"markdown","metadata":{"id":"MQ0sNuMePBXx"},"source":["# 1. Introduction\n","<!-- This is an introduction to your report, you should edit this text/mardown section to compose. In this text/markdown, you should introduce:\n","\n","*   Background of the problem\n","  * what type of problem: disease/readmission/mortality prediction,  feature engineeing, data processing, etc\n","  * what is the importance/meaning of solving the problem\n","  * what is the difficulty of the problem\n","  * the state of the art methods and effectiveness.\n","*   Paper explanation\n","  * what did the paper propose\n","  * what is the innovations of the method\n","  * how well the proposed method work (in its own metrics)\n","  * what is the contribution to the reasearch regime (referring the Background above, how important the paper is to the problem). -->\n","\n","\n","The expanding knowledge of microbiota uncovers its crucial role in human health [1]. It plays an important role in immune system, metabolism functions and even carcinognesis of certain cancers, hence microbiota can be used to predict various disease with emerging sequencing technologies [1,2,3,4,5,6]. However, there are three major challenges to realize the predictions in practice [7]. First, the low number of samples, together with the large number of features, leads to the curse of dimensionality. Second, there is a research gap in using strain-level profiles to classify samples into patient and healthy control groups across different diseases. Third, a rigorous validation framework is essential. Prior research has shown that tuning hyperparameters on the test set without a separate validation set may lead to an overestimation of model performance [8,9,10].\n","\n","The DeepMicro paper proposes the deployment of autoencoders to learn low-dimensional representations from microbiota data and then predict disease by another classification model based on the learned representations, with both trainings having thorough validation schemes [7]. The authors hypothesize that these innovations can contribute to the followings:\n","\n","1. The appropriate autoencoders can effectively solve the curse of dimensionality.\n","\n","2. They also reduce latency compared with alternative models without representation learing, in the mean time maintaining favorable training metrics.\n","\n","In this project, our primary objective revolves around the implementation and evaluation of various autoencoder architectures. We aim to validate the disease prediction capabilities as outlined in the paper. Furthermore, we discuss the ablations inherent in the innovation of autoencoder. The rationale for choosing this paper stems from our pursuit of knowledge in deep representation learning.\n"]},{"cell_type":"markdown","metadata":{"id":"2bwDxCqqbwhb"},"source":["# 2. Github Repository\n","\n","Our project is published on [CS598-DLH-SP24](https://github.com/ZKingQ/CS598-DLH-SP24/tree/main)."]},{"cell_type":"markdown","metadata":{"id":"uygL9tTPSVHB"},"source":["# 3. Scope of Reproducibility\n","\n","<!-- List hypotheses from the paper you will test and the corresponding experiments you will run.\n","\n","\n","1.   Hypothesis 1: xxxxxxx\n","2.   Hypothesis 2: xxxxxxx\n","\n","You can insert images in this notebook text, [see this link](https://stackoverflow.com/questions/50670920/how-to-insert-an-inline-image-in-google-colaboratory-from-google-drive) and example below:\n","\n","![sample_image.png](https://drive.google.com/uc?export=view&id=1g2efvsRJDxTxKz-OY3loMhihrEUdBxbc)\n","\n","\n","You can also use code to display images, see the code below.\n","\n","The images must be saved in Google Drive first.\n","\n","-->\n","\n","Our team achieved successful replication of the model by leveraging its open source codebase, resulting in metrics that are comparable to those showcased in the DeepMicro paper. In our endeavor, we thoroughly examined and addressed the claims below put forth in the original paper.\n","\n","1. **Dimensionality reduction engineering with traditional statistical techniques including Principal Component Analysis (PCA) and Gaussian Random Projection (GRP)**\n","\n","- Principal Component Analysis (PCA) aims to capture the most significant patterns and variations in a dataset by identifying orthogonal axes, known as principal components, that maximize the data variance.\n","- Gaussian Random Projection (GRP) seeks to preserve pairwise distances between data points by projecting them onto a lower-dimensional space using random projections drawn from a Gaussian distribution.\n","\n","2. **Innovated representation learning employing with four different autoencoders including Shallow Autoencoder (SAE), Deep Autoencoder (DAE), Variational Autoencoder (VAE), and Convolutional Autoencoder (CAE)**\n","\n","- Shallow Autoencoder (SAE): This is the simplest form of an autoencoder, consisting of a fully\n","connected encoder layer and a decoder layer. The latent representation is obtained from the encoder\n","layer, which is a lower-dimensional space compared to the original input.\n","- Deep Autoencoder (DAE): In addition to the encoder and decoder layers, DAE introduces hidden\n","layers between the input and latent layers and between the latent and output layers. Rectified Linear\n","Unit (ReLU) activation functions are used in the hidden layers.\n","- Variational Autoencoder (VAE): VAE learns probabilistic representations by approximating the\n","true posterior distribution of latent embeddings. It assumes that the posterior distribution follows\n","a Gaussian distribution. VAE uses an encoder network to encode the means and variances of the\n","Gaussian distribution and samples the latent representation from this distribution. The decoder\n","network then reconstructs the input based on the sampled latent representation.\n","- Convolutional Autoencoder (CAE): Instead of fully connected layers, CAE incorporates convo-\n","lutional layers, where each unit is connected to local regions of the previous layer. Convolutional layers use filters (kernels) to perform convolution operations. CAE employs convolutional transpose layers (deconvolutional layers) to make the decoder symmetric to the encoder. No pooling layers\n","are used in CAE.\n","\n","3. **Classification learning including including Support Vector Machine (SVM), Random Forest (RF), and Multi-Layer Perceptron (MLP)**\n","- Support Vector Machine (SVM) is a supervised learning algorithm that aims to find an optimal hyperplane to classify data by maximizing the margin between different classes.\n","- Random Forest (RF) is an ensemble learning method that constructs a multitude of decision trees and combines their predictions to make classifications.\n","- Multi-Layer Perceptron (MLP) is a type of neural network that consists of multiple layers of interconnected nodes, enabling it to learn non-linear relationships and perform classification tasks.\n"]},{"cell_type":"markdown","metadata":{"id":"xWAHJ_1CdtaA"},"source":["# 4. Methodology\n","\n","<!-- This methodology is the core of your project. It consists of run-able codes with necessary annotations to show the expeiment you executed for testing the hypotheses.\n","\n","The methodology at least contains two subsections **data** and **model** in your experiment. -->"]},{"cell_type":"markdown","metadata":{"id":"zPov0WCKWbPh"},"source":["## 4.1. Environment"]},{"cell_type":"markdown","metadata":{"id":"0dxJ28Zaccme"},"source":["### 4.1.1. Python Version\n","\n","We use colab default python environment which is `python==3.10.12`."]},{"cell_type":"code","execution_count":1,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":355,"status":"ok","timestamp":1715020005091,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"oTXyVLh8YFl7","outputId":"e437c9a0-18fe-42b9-fa68-78ac4f820336"},"outputs":[{"name":"stdout","output_type":"stream","text":["Python 3.10.12\n"]}],"source":["# Python version\n","!python --version"]},{"cell_type":"markdown","metadata":{"id":"-IMSOXGXcYM9"},"source":["### 4.1.2. Packages\n","\n","We utilize PyTorch to build our models so this project primarily depends on `sklearn` and `pytorch`."]},{"cell_type":"code","execution_count":2,"metadata":{"ExecuteTime":{"end_time":"2024-04-29T11:48:38.502933Z","start_time":"2024-04-29T11:48:33.190976Z"},"executionInfo":{"elapsed":2242,"status":"ok","timestamp":1715020007658,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"Lyf5Jfp7sGWw"},"outputs":[],"source":["# import packages needed in this project\n","import numpy as np\n","import pandas as pd\n","import time\n","import datetime\n","import matplotlib\n","matplotlib.use('agg')\n","%matplotlib inline\n","import matplotlib.pyplot as plt\n","\n","from sklearn.neural_network import MLPClassifier\n","from sklearn.model_selection import train_test_split\n","from sklearn.decomposition import PCA\n","from sklearn.random_projection import GaussianRandomProjection\n","from sklearn.model_selection import GridSearchCV\n","from sklearn.svm import SVC\n","from sklearn.ensemble import RandomForestClassifier\n","from sklearn.model_selection import StratifiedKFold\n","from sklearn.metrics import roc_auc_score\n","from sklearn.metrics import accuracy_score\n","from sklearn.metrics import precision_score\n","from sklearn.metrics import recall_score\n","from sklearn.metrics import f1_score\n","\n","# importing PyTorch\n","import torch\n","import torch.nn as nn\n","import torch.nn.functional as F\n","from torch.utils.data import DataLoader, TensorDataset"]},{"cell_type":"markdown","metadata":{"id":"2NbPHUTMbkD3"},"source":["##  4.2. Data\n","<!-- Data includes raw data (MIMIC III tables), descriptive statistics (our homework questions), and data processing (feature engineering).\n","  * Source of the data: where the data is collected from; if data is synthetic or self-generated, explain how. If possible, please provide a link to the raw datasets.\n","  * Statistics: include basic descriptive statistics of the dataset like size, cross validation split, label distribution, etc.\n","  * Data process: how do you munipulate the data, e.g., change the class labels, split the dataset to train/valid/test, refining the dataset.\n","  * Illustration: printing results, plotting figures for illustration.\n","  * You can upload your raw dataset to Google Drive and mount this Colab to the same directory. If your raw dataset is too large, you can upload the processed dataset and have a code to load the processed dataset. -->"]},{"cell_type":"markdown","metadata":{"id":"SRRY_dsgYL2Q"},"source":["### 4.2.1 Data Download Instruction\n","\n","We store two datasets, abundance and marker from the [DeepMicro codebase](https://github.com/minoh0201/DeepMicro/tree/master/data) in the path `raw_data_dir` of the [colab project](https://drive.google.com/file/d/1c6ZPVp346e1wQH6opg4YwlCRjRgfTXCk/view?usp=drive_link).\n","\n","When we run this project locally, we use the function `fetch_data_from_remote` to download the datasets from the google drive to current working directory."]},{"cell_type":"code","execution_count":3,"metadata":{"ExecuteTime":{"end_time":"2024-04-29T11:48:47.979130Z","start_time":"2024-04-29T11:48:47.975250Z"},"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":270,"status":"ok","timestamp":1715020007920,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"yGCbNWqfcRcs","outputId":"8797c9a9-6a05-428b-c742-9bf58ff5a7bc"},"outputs":[{"name":"stdout","output_type":"stream","text":["Data already exists locally\n","abundance_Cirrhosis.txt   abundance_IBD.txt\t abundance_T2D.txt\n","abundance_Colorectal.txt  abundance_Obesity.txt  abundance_WT2D.txt\n","marker_Cirrhosis.txt   marker_IBD.txt\t   marker_T2D.txt\n","marker_Colorectal.txt  marker_Obesity.txt  marker_WT2D.txt\n"]}],"source":["# data dir\n","raw_data_dir = 'data/'\n","\n","# Load data\n","import os\n","import requests\n","from zipfile import ZipFile\n","from io import BytesIO\n","\n","def fetch_data_from_remote():\n","    # cloud location where the .zip file of the data used by this project is stored\n","    url = 'https://drive.google.com/file/d/1c6ZPVp346e1wQH6opg4YwlCRjRgfTXCk/view?usp=drive_link'\n","\n","    file_id = url.split('/')[-2]\n","    dwn_url = 'https://drive.google.com/uc?export=download&id=' + file_id\n","\n","    # unzip the data into the local working directory\n","    response = requests.get(dwn_url)\n","    zip_content = BytesIO(response.content)\n","    with ZipFile(zip_content, 'r') as main_zip:\n","        main_zip.extractall()\n","        with main_zip.open('data/abundance.zip') as nested_zip_abundance:\n","            with ZipFile(nested_zip_abundance) as zip_ref_abundance:\n","                zip_ref_abundance.extractall('data/abundance')\n","\n","        with main_zip.open('data/marker.zip') as nested_zip_marker:\n","            with ZipFile(nested_zip_marker) as zip_ref_marker:\n","                zip_ref_marker.extractall('data/marker')\n","\n","if os.path.exists('data'):\n","    print(\"Data already exists locally\")\n","else:\n","    fetch_data_from_remote()\n","    print(\"Data fetched and unzipped successfully\")\n","\n","# show that the data is available locally now\n","!ls data/abundance/\n","!ls data/marker/"]},{"cell_type":"markdown","metadata":{"id":"FJNbhZZ7cTjS"},"source":["### 4.2.2. Data Description\n","\n","Our reproductivity utilizes the same datasets as the original paper, which include six disease (Table 1). They are inflammatory bowel disease (IBD), type 2 diabetes in European women (EW-T2D), type 2 diabetes in Chinese (C-T2D), obesity (Obesity), liver cirrhosis (Cirrhosis), and colorectal cancer (Colorectal).\n","\n","![](https://github.com/ZKingQ/CS598-DLH-SP24/blob/main/assets/Data_Table1.png?raw=1)\n","\n","In each dataset, marker profile and abundance profile of microbiome are used to train our models (Table 2).\n","\n","![](https://github.com/ZKingQ/CS598-DLH-SP24/blob/main/assets/Data_Table2.png?raw=1)\n","\n","All the data are stored in txt format and the data path structure is as following.\n","\n","```\n",".\n","├── ...\n","├── data                              # Data folder\n","│   ├── marker                        # Marker profile data\n","│       ├── marker_IBD.txt            # Inflammatory bowel disease (IBD)\n","│       ├── marker_WT2D.txt           # Type 2 diabetes in European women (EW-T2D)\n","│       ├── marker_T2D.txt            # Type 2 diabetes in Chinese (C-T2D)\n","│       ├── marker_Obesity.txt        # Obesity (Obesity)\n","│       ├── marker_Cirrhosis.txt      # Liver cirrhosis (Cirrhosis)\n","│       ├── marker_Colorectal.txt     # Colorectal cancer (Colorectal)\n","│   ├── abundance                     # Abundance profile data\n","│       ├── abundance_IBD.txt         # Inflammatory bowel disease (IBD)\n","│       ├── abundance_WT2D.txt        # Type 2 diabetes in European women (EW-T2D)\n","│       ├── abundance_T2D.txt         # Type 2 diabetes in Chinese (C-T2D)\n","│       ├── abundance_Obesity.txt     # Obesity (Obesity)\n","│       ├── abundance_Cirrhosis.txt   # Liver cirrhosis (Cirrhosis)\n","│       ├── abundance_Colorectal.txt  # Colorectal cancer (Colorectal)\n","└── ...\n","```\n","In each txt file, it has different number of features and data points as shown in Table 1 and Table 2 above. In the sample demonstration and visualizations below. The colorectal cancer of microbiome abundance profile has 503 features (exluding those dummy labels) and 121 entries.\n","\n"]},{"cell_type":"code","execution_count":4,"metadata":{"ExecuteTime":{"end_time":"2024-04-29T11:51:20.979319Z","start_time":"2024-04-29T11:51:20.828998Z"},"colab":{"base_uri":"https://localhost:8080/","height":585},"executionInfo":{"elapsed":8,"status":"ok","timestamp":1715020007920,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"BYGSgWXzjdeX","outputId":"7e70741b-fd94-44b2-d924-032363e17cd0"},"outputs":[{"data":{"application/vnd.google.colaboratory.intrinsic+json":{"type":"dataframe","variable_name":"sample"},"text/html":["\n","  <div id=\"df-4fa532fe-4b46-4e85-a6b9-00dec30d3426\" class=\"colab-df-container\">\n","    <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>dataset_name</th>\n","      <th>sampleID</th>\n","      <th>subjectID</th>\n","      <th>bodysite</th>\n","      <th>disease</th>\n","      <th>age</th>\n","      <th>gender</th>\n","      <th>country</th>\n","      <th>sequencing_technology</th>\n","      <th>pubmedid</th>\n","      <th>...</th>\n","      <th>k__Eukaryota|p__Ascomycota|c__Saccharomycetes|o__Saccharomycetales|f__Saccharomycetaceae|g__Eremothecium|s__Eremothecium_unclassified</th>\n","      <th>k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactobacillales|f__Lactobacillaceae|g__Lactobacillus|s__Lactobacillus_antri</th>\n","      <th>k__Bacteria|p__Firmicutes|c__Bacilli|o__Bacillales|f__Bacillaceae|g__Lysinibacillus|s__Lysinibacillus_fusiformis</th>\n","      <th>k__Archaea|p__Euryarchaeota|c__Methanobacteria|o__Methanobacteriales|f__Methanobacteriaceae|g__Methanobacterium|s__Methanobacterium_unclassified</th>\n","      <th>k__Bacteria|p__Firmicutes|c__Bacilli|o__Bacillales|f__Bacillaceae|g__Lysinibacillus|s__Lysinibacillus_boronitolerans</th>\n","      <th>k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactobacillales|f__Enterococcaceae|g__Bavariicoccus|s__Bavariicoccus_seileri</th>\n","      <th>k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactobacillales|f__Enterococcaceae|g__Enterococcus|s__Enterococcus_gilvus</th>\n","      <th>k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactobacillales|f__Lactobacillaceae|g__Lactobacillus|s__Lactobacillus_otakiensis</th>\n","      <th>k__Bacteria|p__Firmicutes|c__Clostridia|o__Clostridiales|f__Peptococcaceae|g__Desulfotomaculum|s__Desulfotomaculum_ruminis</th>\n","      <th>k__Bacteria|p__Firmicutes|c__Negativicutes|o__Selenomonadales|f__Veillonellaceae|g__Megasphaera|s__Megasphaera_sp_BV3C16_1</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>1</th>\n","      <td>Zeller_fecal_colorectal_cancer</td>\n","      <td>CCIS00146684ST-4-0</td>\n","      <td>fr-726</td>\n","      <td>stool</td>\n","      <td>n</td>\n","      <td>72</td>\n","      <td>female</td>\n","      <td>france</td>\n","      <td>Illumina</td>\n","      <td>25432777</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>Zeller_fecal_colorectal_cancer</td>\n","      <td>CCIS00281083ST-3-0</td>\n","      <td>fr-060</td>\n","      <td>stool</td>\n","      <td>n</td>\n","      <td>53</td>\n","      <td>male</td>\n","      <td>france</td>\n","      <td>Illumina</td>\n","      <td>25432777</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Zeller_fecal_colorectal_cancer</td>\n","      <td>CCIS02124300ST-4-0</td>\n","      <td>fr-568</td>\n","      <td>stool</td>\n","      <td>n</td>\n","      <td>35</td>\n","      <td>male</td>\n","      <td>france</td>\n","      <td>Illumina</td>\n","      <td>25432777</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>Zeller_fecal_colorectal_cancer</td>\n","      <td>CCIS02379307ST-4-0</td>\n","      <td>fr-828</td>\n","      <td>stool</td>\n","      <td>cancer</td>\n","      <td>67</td>\n","      <td>male</td>\n","      <td>france</td>\n","      <td>Illumina</td>\n","      <td>25432777</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>Zeller_fecal_colorectal_cancer</td>\n","      <td>CCIS03473770ST-4-0</td>\n","      <td>fr-192</td>\n","      <td>stool</td>\n","      <td>n</td>\n","      <td>29</td>\n","      <td>male</td>\n","      <td>france</td>\n","      <td>Illumina</td>\n","      <td>25432777</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0.03756</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>Zeller_fecal_colorectal_cancer</td>\n","      <td>CCIS06260551ST-3-0</td>\n","      <td>fr-200</td>\n","      <td>stool</td>\n","      <td>cancer</td>\n","      <td>58</td>\n","      <td>male</td>\n","      <td>france</td>\n","      <td>Illumina</td>\n","      <td>25432777</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0.31121</td>\n","      <td>0</td>\n","      <td>0.03562</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>Zeller_fecal_colorectal_cancer</td>\n","      <td>CCIS07539127ST-4-0</td>\n","      <td>fr-460</td>\n","      <td>stool</td>\n","      <td>n</td>\n","      <td>77</td>\n","      <td>female</td>\n","      <td>france</td>\n","      <td>Illumina</td>\n","      <td>25432777</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>Zeller_fecal_colorectal_cancer</td>\n","      <td>CCIS07648107ST-4-0</td>\n","      <td>fr-053</td>\n","      <td>stool</td>\n","      <td>n</td>\n","      <td>62</td>\n","      <td>female</td>\n","      <td>france</td>\n","      <td>Illumina</td>\n","      <td>25432777</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>9</th>\n","      <td>Zeller_fecal_colorectal_cancer</td>\n","      <td>CCIS08668806ST-3-0</td>\n","      <td>fr-214</td>\n","      <td>stool</td>\n","      <td>small_adenoma</td>\n","      <td>63</td>\n","      <td>male</td>\n","      <td>france</td>\n","      <td>Illumina</td>\n","      <td>25432777</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>10</th>\n","      <td>Zeller_fecal_colorectal_cancer</td>\n","      <td>CCIS09568613ST-4-0</td>\n","      <td>fr-400</td>\n","      <td>stool</td>\n","      <td>n</td>\n","      <td>67</td>\n","      <td>male</td>\n","      <td>france</td>\n","      <td>Illumina</td>\n","      <td>25432777</td>\n","      <td>...</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>10 rows × 714 columns</p>\n","</div>\n","    <div class=\"colab-df-buttons\">\n","\n","  <div class=\"colab-df-container\">\n","    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4fa532fe-4b46-4e85-a6b9-00dec30d3426')\"\n","            title=\"Convert this dataframe to an interactive table.\"\n","            style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n","    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n","  </svg>\n","    </button>\n","\n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    .colab-df-buttons div {\n","      margin-bottom: 4px;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","    <script>\n","      const buttonEl =\n","        document.querySelector('#df-4fa532fe-4b46-4e85-a6b9-00dec30d3426 button.colab-df-convert');\n","      buttonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","      async function convertToInteractive(key) {\n","        const element = document.querySelector('#df-4fa532fe-4b46-4e85-a6b9-00dec30d3426');\n","        const dataTable =\n","          await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                    [key], {});\n","        if (!dataTable) return;\n","\n","        const docLinkHtml = 'Like what you see? Visit the ' +\n","          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","          + ' to learn more about interactive tables.';\n","        element.innerHTML = '';\n","        dataTable['output_type'] = 'display_data';\n","        await google.colab.output.renderOutput(dataTable, element);\n","        const docLink = document.createElement('div');\n","        docLink.innerHTML = docLinkHtml;\n","        element.appendChild(docLink);\n","      }\n","    </script>\n","  </div>\n","\n","\n","<div id=\"df-1f9c8bed-c489-46d0-9242-bfe899ac27ac\">\n","  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-1f9c8bed-c489-46d0-9242-bfe899ac27ac')\"\n","            title=\"Suggest charts\"\n","            style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","  </button>\n","\n","<style>\n","  .colab-df-quickchart {\n","      --bg-color: #E8F0FE;\n","      --fill-color: #1967D2;\n","      --hover-bg-color: #E2EBFA;\n","      --hover-fill-color: #174EA6;\n","      --disabled-fill-color: #AAA;\n","      --disabled-bg-color: #DDD;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","      --bg-color: #3B4455;\n","      --fill-color: #D2E3FC;\n","      --hover-bg-color: #434B5C;\n","      --hover-fill-color: #FFFFFF;\n","      --disabled-bg-color: #3B4455;\n","      --disabled-fill-color: #666;\n","  }\n","\n","  .colab-df-quickchart {\n","    background-color: var(--bg-color);\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: var(--fill-color);\n","    height: 32px;\n","    padding: 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: var(--hover-bg-color);\n","    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: var(--button-hover-fill-color);\n","  }\n","\n","  .colab-df-quickchart-complete:disabled,\n","  .colab-df-quickchart-complete:disabled:hover {\n","    background-color: var(--disabled-bg-color);\n","    fill: var(--disabled-fill-color);\n","    box-shadow: none;\n","  }\n","\n","  .colab-df-spinner {\n","    border: 2px solid var(--fill-color);\n","    border-color: transparent;\n","    border-bottom-color: var(--fill-color);\n","    animation:\n","      spin 1s steps(1) infinite;\n","  }\n","\n","  @keyframes spin {\n","    0% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","      border-left-color: var(--fill-color);\n","    }\n","    20% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    30% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","      border-right-color: var(--fill-color);\n","    }\n","    40% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    60% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","    }\n","    80% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-bottom-color: var(--fill-color);\n","    }\n","    90% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","    }\n","  }\n","</style>\n","\n","  <script>\n","    async function quickchart(key) {\n","      const quickchartButtonEl =\n","        document.querySelector('#' + key + ' button');\n","      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n","      quickchartButtonEl.classList.add('colab-df-spinner');\n","      try {\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      } catch (error) {\n","        console.error('Error during call to suggestCharts:', error);\n","      }\n","      quickchartButtonEl.classList.remove('colab-df-spinner');\n","      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n","    }\n","    (() => {\n","      let quickchartButtonEl =\n","        document.querySelector('#df-1f9c8bed-c489-46d0-9242-bfe899ac27ac button');\n","      quickchartButtonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","    })();\n","  </script>\n","</div>\n","\n","    </div>\n","  </div>\n"],"text/plain":["0                     dataset_name            sampleID subjectID bodysite  \\\n","1   Zeller_fecal_colorectal_cancer  CCIS00146684ST-4-0    fr-726    stool   \n","2   Zeller_fecal_colorectal_cancer  CCIS00281083ST-3-0    fr-060    stool   \n","3   Zeller_fecal_colorectal_cancer  CCIS02124300ST-4-0    fr-568    stool   \n","4   Zeller_fecal_colorectal_cancer  CCIS02379307ST-4-0    fr-828    stool   \n","5   Zeller_fecal_colorectal_cancer  CCIS03473770ST-4-0    fr-192    stool   \n","6   Zeller_fecal_colorectal_cancer  CCIS06260551ST-3-0    fr-200    stool   \n","7   Zeller_fecal_colorectal_cancer  CCIS07539127ST-4-0    fr-460    stool   \n","8   Zeller_fecal_colorectal_cancer  CCIS07648107ST-4-0    fr-053    stool   \n","9   Zeller_fecal_colorectal_cancer  CCIS08668806ST-3-0    fr-214    stool   \n","10  Zeller_fecal_colorectal_cancer  CCIS09568613ST-4-0    fr-400    stool   \n","\n","0         disease age  gender country sequencing_technology  pubmedid  ...  \\\n","1               n  72  female  france              Illumina  25432777  ...   \n","2               n  53    male  france              Illumina  25432777  ...   \n","3               n  35    male  france              Illumina  25432777  ...   \n","4          cancer  67    male  france              Illumina  25432777  ...   \n","5               n  29    male  france              Illumina  25432777  ...   \n","6          cancer  58    male  france              Illumina  25432777  ...   \n","7               n  77  female  france              Illumina  25432777  ...   \n","8               n  62  female  france              Illumina  25432777  ...   \n","9   small_adenoma  63    male  france              Illumina  25432777  ...   \n","10              n  67    male  france              Illumina  25432777  ...   \n","\n","0  k__Eukaryota|p__Ascomycota|c__Saccharomycetes|o__Saccharomycetales|f__Saccharomycetaceae|g__Eremothecium|s__Eremothecium_unclassified  \\\n","1                                                   0                                                                                      \n","2                                                   0                                                                                      \n","3                                                   0                                                                                      \n","4                                                   0                                                                                      \n","5                                                   0                                                                                      \n","6                                                   0                                                                                      \n","7                                                   0                                                                                      \n","8                                                   0                                                                                      \n","9                                                   0                                                                                      \n","10                                                  0                                                                                      \n","\n","0  k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactobacillales|f__Lactobacillaceae|g__Lactobacillus|s__Lactobacillus_antri  \\\n","1                                                   0                                                                    \n","2                                                   0                                                                    \n","3                                                   0                                                                    \n","4                                                   0                                                                    \n","5                                                   0                                                                    \n","6                                                   0                                                                    \n","7                                                   0                                                                    \n","8                                                   0                                                                    \n","9                                                   0                                                                    \n","10                                                  0                                                                    \n","\n","0  k__Bacteria|p__Firmicutes|c__Bacilli|o__Bacillales|f__Bacillaceae|g__Lysinibacillus|s__Lysinibacillus_fusiformis  \\\n","1                                                   0                                                                 \n","2                                                   0                                                                 \n","3                                                   0                                                                 \n","4                                                   0                                                                 \n","5                                                   0                                                                 \n","6                                             0.31121                                                                 \n","7                                                   0                                                                 \n","8                                                   0                                                                 \n","9                                                   0                                                                 \n","10                                                  0                                                                 \n","\n","0  k__Archaea|p__Euryarchaeota|c__Methanobacteria|o__Methanobacteriales|f__Methanobacteriaceae|g__Methanobacterium|s__Methanobacterium_unclassified  \\\n","1                                                   0                                                                                                 \n","2                                                   0                                                                                                 \n","3                                                   0                                                                                                 \n","4                                                   0                                                                                                 \n","5                                                   0                                                                                                 \n","6                                                   0                                                                                                 \n","7                                                   0                                                                                                 \n","8                                                   0                                                                                                 \n","9                                                   0                                                                                                 \n","10                                                  0                                                                                                 \n","\n","0  k__Bacteria|p__Firmicutes|c__Bacilli|o__Bacillales|f__Bacillaceae|g__Lysinibacillus|s__Lysinibacillus_boronitolerans  \\\n","1                                                   0                                                                     \n","2                                                   0                                                                     \n","3                                                   0                                                                     \n","4                                                   0                                                                     \n","5                                                   0                                                                     \n","6                                             0.03562                                                                     \n","7                                                   0                                                                     \n","8                                                   0                                                                     \n","9                                                   0                                                                     \n","10                                                  0                                                                     \n","\n","0  k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactobacillales|f__Enterococcaceae|g__Bavariicoccus|s__Bavariicoccus_seileri  \\\n","1                                                   0                                                                     \n","2                                                   0                                                                     \n","3                                                   0                                                                     \n","4                                                   0                                                                     \n","5                                                   0                                                                     \n","6                                                   0                                                                     \n","7                                                   0                                                                     \n","8                                                   0                                                                     \n","9                                                   0                                                                     \n","10                                                  0                                                                     \n","\n","0  k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactobacillales|f__Enterococcaceae|g__Enterococcus|s__Enterococcus_gilvus  \\\n","1                                                   0                                                                  \n","2                                                   0                                                                  \n","3                                                   0                                                                  \n","4                                                   0                                                                  \n","5                                             0.03756                                                                  \n","6                                                   0                                                                  \n","7                                                   0                                                                  \n","8                                                   0                                                                  \n","9                                                   0                                                                  \n","10                                                  0                                                                  \n","\n","0  k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactobacillales|f__Lactobacillaceae|g__Lactobacillus|s__Lactobacillus_otakiensis  \\\n","1                                                   0                                                                         \n","2                                                   0                                                                         \n","3                                                   0                                                                         \n","4                                                   0                                                                         \n","5                                                   0                                                                         \n","6                                                   0                                                                         \n","7                                                   0                                                                         \n","8                                                   0                                                                         \n","9                                                   0                                                                         \n","10                                                  0                                                                         \n","\n","0  k__Bacteria|p__Firmicutes|c__Clostridia|o__Clostridiales|f__Peptococcaceae|g__Desulfotomaculum|s__Desulfotomaculum_ruminis  \\\n","1                                                   0                                                                           \n","2                                                   0                                                                           \n","3                                                   0                                                                           \n","4                                                   0                                                                           \n","5                                                   0                                                                           \n","6                                                   0                                                                           \n","7                                                   0                                                                           \n","8                                                   0                                                                           \n","9                                                   0                                                                           \n","10                                                  0                                                                           \n","\n","0  k__Bacteria|p__Firmicutes|c__Negativicutes|o__Selenomonadales|f__Veillonellaceae|g__Megasphaera|s__Megasphaera_sp_BV3C16_1  \n","1                                                   0                                                                          \n","2                                                   0                                                                          \n","3                                                   0                                                                          \n","4                                                   0                                                                          \n","5                                                   0                                                                          \n","6                                                   0                                                                          \n","7                                                   0                                                                          \n","8                                                   0                                                                          \n","9                                                   0                                                                          \n","10                                                  0                                                                          \n","\n","[10 rows x 714 columns]"]},"execution_count":4,"metadata":{},"output_type":"execute_result"}],"source":["# Extract a sample data of colorectal cancer of microbiome abundance profile\n","sample = pd.read_csv(os.path.join(raw_data_dir, \"abundance\", \"abundance_Colorectal.txt\"), sep='\\t', index_col=0, header=None)\n","sample = sample.T\n","sample.head(10)"]},{"cell_type":"code","execution_count":5,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":6,"status":"ok","timestamp":1715020007920,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"5EjqDkrWm0lc","outputId":"f637e820-d563-4e73-8193-495d99308c8d"},"outputs":[{"data":{"text/plain":["503"]},"execution_count":5,"metadata":{},"output_type":"execute_result"}],"source":["# Count the features excluding patients information\n","len([ _ for _ in sample.columns if '|' in _ ])"]},{"cell_type":"code","execution_count":6,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":5,"status":"ok","timestamp":1715020007921,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"i93VorpFlPnl","outputId":"762d3db0-e9f0-4654-8a67-7999921f8b9c"},"outputs":[{"data":{"text/plain":["(121, 714)"]},"execution_count":6,"metadata":{},"output_type":"execute_result"}],"source":["# Calculate the dimensions of the sameple dataset\n","sample.shape"]},{"cell_type":"code","execution_count":7,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":455},"executionInfo":{"elapsed":971,"status":"ok","timestamp":1715020008886,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"PPFi9QXolIoa","outputId":"9021913f-4e92-4268-f670-43afe0234ceb"},"outputs":[{"data":{"application/vnd.google.colaboratory.intrinsic+json":{"summary":"{\n  \"name\": \"sample\",\n  \"rows\": 714,\n  \"fields\": [\n    {\n      \"column\": \"count\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"121\",\n        \"max\": \"121\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"121\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"unique\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 1,\n        \"max\": 121,\n        \"num_unique_values\": 103,\n        \"samples\": [\n          23\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"top\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 21,\n        \"samples\": [\n          \"Zeller_fecal_colorectal_cancer\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"freq\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"1\",\n        \"max\": \"121\",\n        \"num_unique_values\": 105,\n        \"samples\": [\n          \"77\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}","type":"dataframe"},"text/html":["\n","  <div id=\"df-c2cdb80d-a471-402d-96da-51d5a05b457c\" class=\"colab-df-container\">\n","    <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>count</th>\n","      <th>unique</th>\n","      <th>top</th>\n","      <th>freq</th>\n","    </tr>\n","    <tr>\n","      <th>0</th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>dataset_name</th>\n","      <td>121</td>\n","      <td>1</td>\n","      <td>Zeller_fecal_colorectal_cancer</td>\n","      <td>121</td>\n","    </tr>\n","    <tr>\n","      <th>sampleID</th>\n","      <td>121</td>\n","      <td>121</td>\n","      <td>CCIS98832363ST-4-0</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>subjectID</th>\n","      <td>121</td>\n","      <td>121</td>\n","      <td>fr-552</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>bodysite</th>\n","      <td>121</td>\n","      <td>1</td>\n","      <td>stool</td>\n","      <td>121</td>\n","    </tr>\n","    <tr>\n","      <th>disease</th>\n","      <td>121</td>\n","      <td>3</td>\n","      <td>cancer</td>\n","      <td>48</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactobacillales|f__Enterococcaceae|g__Bavariicoccus|s__Bavariicoccus_seileri</th>\n","      <td>121</td>\n","      <td>2</td>\n","      <td>0</td>\n","      <td>120</td>\n","    </tr>\n","    <tr>\n","      <th>k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactobacillales|f__Enterococcaceae|g__Enterococcus|s__Enterococcus_gilvus</th>\n","      <td>121</td>\n","      <td>5</td>\n","      <td>0</td>\n","      <td>117</td>\n","    </tr>\n","    <tr>\n","      <th>k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactobacillales|f__Lactobacillaceae|g__Lactobacillus|s__Lactobacillus_otakiensis</th>\n","      <td>121</td>\n","      <td>2</td>\n","      <td>0</td>\n","      <td>120</td>\n","    </tr>\n","    <tr>\n","      <th>k__Bacteria|p__Firmicutes|c__Clostridia|o__Clostridiales|f__Peptococcaceae|g__Desulfotomaculum|s__Desulfotomaculum_ruminis</th>\n","      <td>121</td>\n","      <td>2</td>\n","      <td>0</td>\n","      <td>120</td>\n","    </tr>\n","    <tr>\n","      <th>k__Bacteria|p__Firmicutes|c__Negativicutes|o__Selenomonadales|f__Veillonellaceae|g__Megasphaera|s__Megasphaera_sp_BV3C16_1</th>\n","      <td>121</td>\n","      <td>2</td>\n","      <td>0</td>\n","      <td>120</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>714 rows × 4 columns</p>\n","</div>\n","    <div class=\"colab-df-buttons\">\n","\n","  <div class=\"colab-df-container\">\n","    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c2cdb80d-a471-402d-96da-51d5a05b457c')\"\n","            title=\"Convert this dataframe to an interactive table.\"\n","            style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n","    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n","  </svg>\n","    </button>\n","\n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    .colab-df-buttons div {\n","      margin-bottom: 4px;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","    <script>\n","      const buttonEl =\n","        document.querySelector('#df-c2cdb80d-a471-402d-96da-51d5a05b457c button.colab-df-convert');\n","      buttonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","      async function convertToInteractive(key) {\n","        const element = document.querySelector('#df-c2cdb80d-a471-402d-96da-51d5a05b457c');\n","        const dataTable =\n","          await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                    [key], {});\n","        if (!dataTable) return;\n","\n","        const docLinkHtml = 'Like what you see? Visit the ' +\n","          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","          + ' to learn more about interactive tables.';\n","        element.innerHTML = '';\n","        dataTable['output_type'] = 'display_data';\n","        await google.colab.output.renderOutput(dataTable, element);\n","        const docLink = document.createElement('div');\n","        docLink.innerHTML = docLinkHtml;\n","        element.appendChild(docLink);\n","      }\n","    </script>\n","  </div>\n","\n","\n","<div id=\"df-eee25e10-061b-46c5-b25f-6ce345ec4a83\">\n","  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-eee25e10-061b-46c5-b25f-6ce345ec4a83')\"\n","            title=\"Suggest charts\"\n","            style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","  </button>\n","\n","<style>\n","  .colab-df-quickchart {\n","      --bg-color: #E8F0FE;\n","      --fill-color: #1967D2;\n","      --hover-bg-color: #E2EBFA;\n","      --hover-fill-color: #174EA6;\n","      --disabled-fill-color: #AAA;\n","      --disabled-bg-color: #DDD;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","      --bg-color: #3B4455;\n","      --fill-color: #D2E3FC;\n","      --hover-bg-color: #434B5C;\n","      --hover-fill-color: #FFFFFF;\n","      --disabled-bg-color: #3B4455;\n","      --disabled-fill-color: #666;\n","  }\n","\n","  .colab-df-quickchart {\n","    background-color: var(--bg-color);\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: var(--fill-color);\n","    height: 32px;\n","    padding: 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: var(--hover-bg-color);\n","    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: var(--button-hover-fill-color);\n","  }\n","\n","  .colab-df-quickchart-complete:disabled,\n","  .colab-df-quickchart-complete:disabled:hover {\n","    background-color: var(--disabled-bg-color);\n","    fill: var(--disabled-fill-color);\n","    box-shadow: none;\n","  }\n","\n","  .colab-df-spinner {\n","    border: 2px solid var(--fill-color);\n","    border-color: transparent;\n","    border-bottom-color: var(--fill-color);\n","    animation:\n","      spin 1s steps(1) infinite;\n","  }\n","\n","  @keyframes spin {\n","    0% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","      border-left-color: var(--fill-color);\n","    }\n","    20% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    30% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","      border-right-color: var(--fill-color);\n","    }\n","    40% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    60% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","    }\n","    80% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-bottom-color: var(--fill-color);\n","    }\n","    90% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","    }\n","  }\n","</style>\n","\n","  <script>\n","    async function quickchart(key) {\n","      const quickchartButtonEl =\n","        document.querySelector('#' + key + ' button');\n","      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n","      quickchartButtonEl.classList.add('colab-df-spinner');\n","      try {\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      } catch (error) {\n","        console.error('Error during call to suggestCharts:', error);\n","      }\n","      quickchartButtonEl.classList.remove('colab-df-spinner');\n","      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n","    }\n","    (() => {\n","      let quickchartButtonEl =\n","        document.querySelector('#df-eee25e10-061b-46c5-b25f-6ce345ec4a83 button');\n","      quickchartButtonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","    })();\n","  </script>\n","</div>\n","\n","    </div>\n","  </div>\n"],"text/plain":["                                                   count unique  \\\n","0                                                                 \n","dataset_name                                         121      1   \n","sampleID                                             121    121   \n","subjectID                                            121    121   \n","bodysite                                             121      1   \n","disease                                              121      3   \n","...                                                  ...    ...   \n","k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactoba...   121      2   \n","k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactoba...   121      5   \n","k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactoba...   121      2   \n","k__Bacteria|p__Firmicutes|c__Clostridia|o__Clos...   121      2   \n","k__Bacteria|p__Firmicutes|c__Negativicutes|o__S...   121      2   \n","\n","                                                                               top  \\\n","0                                                                                    \n","dataset_name                                        Zeller_fecal_colorectal_cancer   \n","sampleID                                                        CCIS98832363ST-4-0   \n","subjectID                                                                   fr-552   \n","bodysite                                                                     stool   \n","disease                                                                     cancer   \n","...                                                                            ...   \n","k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactoba...                               0   \n","k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactoba...                               0   \n","k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactoba...                               0   \n","k__Bacteria|p__Firmicutes|c__Clostridia|o__Clos...                               0   \n","k__Bacteria|p__Firmicutes|c__Negativicutes|o__S...                               0   \n","\n","                                                   freq  \n","0                                                        \n","dataset_name                                        121  \n","sampleID                                              1  \n","subjectID                                             1  \n","bodysite                                            121  \n","disease                                              48  \n","...                                                 ...  \n","k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactoba...  120  \n","k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactoba...  117  \n","k__Bacteria|p__Firmicutes|c__Bacilli|o__Lactoba...  120  \n","k__Bacteria|p__Firmicutes|c__Clostridia|o__Clos...  120  \n","k__Bacteria|p__Firmicutes|c__Negativicutes|o__S...  120  \n","\n","[714 rows x 4 columns]"]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["# Have a generate descrption of the sample dataset\n","sample.describe().T"]},{"cell_type":"markdown","metadata":{"id":"QTkZz-HrofL1"},"source":["### 4.2.3. Data Preprocess\n","\n","According to the original paper, the original data provided in the codebase have been preprocessed and cleaned properly. However, we still need to identify the features columns and labels to extract `X` and `Y`, respectively.\n","\n","In abundance data, we extradct features by `k__`. In marker data, the features are chracterized with `gi|`.\n","\n","For labels of different deseases, we created an dictionary `label_dict` to make the labels into binary numbers (i.e. either 1 or 0)."]},{"cell_type":"code","execution_count":8,"metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1715020008887,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"acW06e9MbqgM"},"outputs":[],"source":["# define random seed\n","seed = 42\n","\n","label_dict = {\n","    # Controls\n","    'n': 0,\n","    # Chirrhosis\n","    'cirrhosis': 1,\n","    # Colorectal Cancer\n","    'cancer': 1, 'small_adenoma': 0,\n","    # IBD\n","    'ibd_ulcerative_colitis': 1, 'ibd_crohn_disease': 1,\n","    # T2D and WT2D\n","    't2d': 1,\n","    # Obesity\n","    'leaness': 0, 'obesity': 1,\n","}\n","\n","\n","def load_data(data_dir, filename, dtype=None):\n","    feature_string = ''\n","    if filename.split('_')[0] == 'abundance':\n","        feature_string = \"k__\"\n","    if filename.split('_')[0] == 'marker':\n","        feature_string = \"gi|\"\n","    # read file\n","    filename = data_dir + filename\n","    if not os.path.isfile(filename):\n","        print(\"FileNotFoundError: File {} does not exist\".format(filename))\n","        exit()\n","    raw = pd.read_csv(filename, sep='\\t', index_col=0, header=None)\n","\n","    # select rows having feature index identifier string\n","    X = raw.loc[raw.index.str.contains(feature_string, regex=False)].T\n","\n","    # get class labels\n","    Y = raw.loc['disease']\n","    Y = Y.replace(label_dict)\n","\n","    # train and test split\n","    X_train, X_test, y_train, y_test = train_test_split(\n","        X.values.astype(dtype), Y.values.astype('int'), test_size=0.2,\n","        random_state=seed, stratify=Y.values\n","        )\n","    print(\"Train data shape: \", X_train.shape)\n","    print(\"Test data shape: \", X_test.shape)\n","    return X_train, X_test, y_train, y_test"]},{"cell_type":"markdown","metadata":{"id":"GXSpN9sMbqgN"},"source":["##   4.3. Model\n","\n","### 4.3.1. Citation to the Original Paper\n","\n","Oh, M., & Zhang, L. (2020). DeepMicro: deep representation learning for disease prediction based on microbiome data. Scientific reports, 10(1), 6026.\n","\n","### 4.3.2. Link to the Original Paper's Repo\n","[DeepMicro Github Repository](https://github.com/minoh0201/DeepMicro)\n","\n","### 4.3.3. Model Descriptions\n","\n","An autoencoder represents a type of neural network designed for the purpose of reconstructing its input data, denoted as $x$. In its fundamental structure, it comprises an encoder function, denoted as $f_\\phi(⋅)$, and a decoder function, denoted as $f_\\theta′(⋅)$, with $\\phi$ and $\\theta$ serving as the parameters associated with the encoder and decoder functions, respectively.\n","The training objective of an autoencoder is to minimize the disparity between the original input $x$ and its reconstructed counterpart $x′$. This discrepancy, typically quantified using a reconstruction loss metric such as squared error, can be mathematically expressed as $L(x, x')=||x-x'||^2=||x-f_\\theta'(f_\\phi(x))||^2$.\n","\n","In this project, we focus on the utilization of a trained autoencoder to obtain a lower-dimensional latent representation $z = f_\\phi(x)$ of the input. There are four autoencoders that we incorporate as below.\n","\n","1. Shallow Autoencoder (SAE): a fully connected encoder connecting the input layer to the latent layer, and a decoder producing the reconstructed input $x′$ by combining the outputs of the latent layer using weighted sums, with both the latent and output layers utilizing a linear activation function.\n","\n","2. Deep Autoencoder (DAE): enhanced SAE model by inserting hidden layers with Rectified Linear Unit (ReLu) activation function and Glorot uniform initializer between the input and latent layers, maintaining an equal number of hidden layers (either one or two layers) in both the encoder and decoder sections.\n","\n","3. Variational autoencoder (VAE): it learns probabilistic representations $z$ by approximating the true posterior distribution with $q_\\phi(z|x)$ assuming a Gaussian distribution. The encoder encodes the means and variances of the Gaussian distribution, allowing sampling of latent representation $z$. This sampled representation is then fed into the decoder network to generate the reconstructed input $x′ \\sim g_\\theta(x|z)$.\n","\n","4. convolutional autoencoder (CAE): equipped with convolutional layers where each unit is connected locally to the previous layer. The layers consist of multiple filters with weights for convolution operations. We employed ReLu activation, Glorot uniform initializer, and avoided pooling layers to prevent excessive generalization. The $n$-dimensional input vector was reshaped into a squared image of size $d \\times d \\times 1$, where $d = ⌊ \\sqrt{n} ⌋ + 1$.\n","\n","After getting the low-dimentional representations of profiles, we build classification models.\n","\n","1. Support Vector Machine (SVM): a grid search using SVM is conducted to explore the hyper-parameter space. We considered both radial basis function (RBF) and linear kernels, adjusting penalty parameter C and kernel coefficient gamma for RBF.\n","\n","2. Random Forest (RF): We examine two criteria, Gini impurity and information gain, for selecting features to split a node in a decision tree. The maximum number of features considered for the best split at each node was determined using the square root of the sample size and the logarithm to base 2 of the sample size.\n","\n","3. Multi-Layer Perceptron (MLP): ReLu activations were used for the hidden layers, while the output layer employed sigmoid activation with a single unit. The number of units in the hidden layers was set to half of the preceding layer, excluding the first hidden layer.\n","\n","The overall workflow is demonstrated in the Figure 1 below.\n","\n","![](https://github.com/ZKingQ/CS598-DLH-SP24/blob/main/assets/Model_Figure1.png?raw=1)\n","\n","### 4.3.4. Implementation Code\n","\n","Please find the implementations of autoencoders in the section [Autoencoders Definitions](#autoencoders-definitions) below."]},{"cell_type":"markdown","metadata":{"id":"c3XEF8AmtwSI"},"source":["#### Autoencoders Definitions\n","\n"]},{"cell_type":"code","execution_count":9,"metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1715020008887,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"pWkNlD2ORZoG"},"outputs":[],"source":["class EarlyStopper:\n","    def __init__(self, patience=5, delta=0.0):\n","        self.patience = patience\n","        self.delta = delta\n","        self.counter = 0\n","        self.best_loss = None\n","        self.early_stop = False\n","\n","    def __call__(self, loss):\n","        if self.best_loss is None:\n","            self.best_loss = loss\n","        elif loss > self.best_loss + self.delta:\n","            self.counter += 1\n","            if self.counter >= self.patience:\n","                self.early_stop = True\n","        else:\n","            self.best_loss = loss\n","            self.counter = 0\n","\n","        return self.early_stop\n","\n","\n","class ShallowAutoEncoder(nn.Module):\n","    def __init__(self, input_dim=28 * 28):\n","        super(ShallowAutoEncoder, self).__init__()\n","        self.input_dim = input_dim\n","        self.encoder1 = nn.Linear(input_dim, 32)\n","        self.decoder1 = nn.Linear(32, input_dim)\n","\n","    def encoder(self, x):\n","        return torch.tanh(self.encoder1(x))\n","\n","    def decoder(self, z):\n","        return torch.sigmoid(self.decoder1(z))\n","\n","    def forward(self, x):\n","        z = self.encoder(x.view(-1, self.input_dim))\n","        return self.decoder(z)\n","\n","    @staticmethod\n","    def loss_func(x_hat, x):\n","        return F.mse_loss(x_hat, x, reduction='sum')\n","\n","\n","class AutoEncoder(nn.Module):\n","    def __init__(self, input_dim=28*28):\n","        super(AutoEncoder, self).__init__()\n","        self.input_dim = input_dim\n","\n","        # DNN as encoder\n","        # 512-256-128\n","        self.encoder1 = nn.Linear(input_dim, 512)\n","        self.encoder2 = nn.Linear(512, 256)\n","        self.encoder3 = nn.Linear(256, 128)\n","\n","        # DNN as decoder\n","        self.decoder1 = nn.Linear(128, 256)\n","        self.decoder2 = nn.Linear(256, 512)\n","        self.decoder3 = nn.Linear(512, input_dim)\n","\n","    def encoder(self, x):\n","        h = torch.tanh(self.encoder1(x))\n","        h = torch.tanh(self.encoder2(h))\n","        return torch.tanh(self.encoder3(h))\n","\n","    def decoder(self, z):\n","        h = torch.tanh(self.decoder1(z))\n","        h = torch.tanh(self.decoder2(h))\n","        return torch.sigmoid(self.decoder3(h))\n","\n","    def forward(self, x):\n","        z = self.encoder(x.view(-1, self.input_dim))\n","        return self.decoder(z)\n","\n","    @staticmethod\n","    def loss_func(x_hat, x):\n","        return F.mse_loss(x_hat, x)\n","\n","\n","class CAE(nn.Module):\n","    def __init__(self):\n","        super(CAE, self).__init__()\n","\n","        # CNN as encoder\n","        self.encoder = nn.Sequential(\n","            nn.Conv2d(1, 16, 3, stride=2, padding=1),\n","            nn.ReLU(),\n","            nn.Conv2d(16, 32, 3, stride=2, padding=1),\n","            nn.ReLU(),\n","            nn.Conv2d(32, 64, 6)\n","        )\n","\n","        # CNN as decoder\n","        self.decoder = nn.Sequential(\n","            nn.ConvTranspose2d(64, 32, 6),\n","            nn.ReLU(),\n","            nn.ConvTranspose2d(32, 16, 3, stride=2, padding=1, output_padding=1),\n","            nn.ReLU(),\n","            nn.ConvTranspose2d(16, 1, 3, stride=2, padding=1, output_padding=1),\n","            nn.Sigmoid()\n","        )\n","\n","    def forward(self, x):\n","        z = self.encoder(x)\n","        return self.decoder(z)\n","\n","    @staticmethod\n","    def loss_func(x_hat, x):\n","        return F.mse_loss(x_hat, x)\n","\n","\n","class VAE(nn.Module):\n","    def __init__(self, input_dim=28 * 28, latent_dim=8):\n","        super(VAE, self).__init__()\n","        self.input_dim = input_dim\n","\n","        # DNN as encoder\n","        self.encoder1 = nn.Linear(input_dim, 128)\n","        self.mu = nn.Linear(128, 8)\n","        self.log_sigma2 = nn.Linear(128, 8)\n","\n","        # DNN as decoder\n","        self.decoder1 = nn.Linear(8, 128)\n","        self.decoder2 = nn.Linear(128, input_dim)\n","\n","    def encoder(self, x):\n","        h = torch.tanh(self.encoder1(x))\n","        return self.mu(h), torch.sqrt(torch.exp(self.log_sigma2(h)))\n","\n","    @staticmethod\n","    def sampling(mu, std):  # Re parameterization trick\n","        eps = torch.randn_like(std)\n","        return mu + eps * std\n","\n","    def decoder(self, z):\n","        h = torch.tanh(self.decoder1(z))\n","        return torch.sigmoid(self.decoder2(h))\n","\n","    def forward(self, x):\n","        mu, std = self.encoder(x.view(-1, self.input_dim))\n","        z = self.sampling(mu, std)\n","        return self.decoder(z), mu, std\n","\n","    @staticmethod\n","    def loss_func(x_hat, x, mu, std):\n","        # define the reconstruction loss\n","        ERR = F.binary_cross_entropy(x_hat, x, reduction='sum')\n","        # define the KL divergence loss\n","        KLD = -0.5 * torch.sum(1 + torch.log(std ** 2) - mu ** 2 - std ** 2)\n","        return ERR + KLD, ERR, KLD"]},{"cell_type":"markdown","metadata":{"id":"f349y3iuE-Cz"},"source":["#### Classification Models"]},{"cell_type":"code","execution_count":10,"metadata":{"executionInfo":{"elapsed":4,"status":"ok","timestamp":1715020008887,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"s2hbJUcCjiEY"},"outputs":[],"source":["global_data_identifier = '' # will be set later\n","cache_trained_model = False # will be set later\n","train_only = False # will be set later\n","enable_grid_search = False # will be set later\n","\n","def get_metrics(clf, X_test_encoded, y_test):\n","    y_true, y_pred = y_test, clf.predict(X_test_encoded)\n","    y_prob = clf.predict_proba(X_test_encoded)[:, 1]\n","    # Performance Metrics: AUC, ACC, Recall, Precision, F1_score\n","    metrics = {\n","        'AUC': roc_auc_score(y_true, y_prob),\n","        'ACC': accuracy_score(y_true, y_pred),\n","        'Recall': recall_score(y_true, y_pred),\n","        'Precision': precision_score(y_true, y_pred),\n","        'F1_score': f1_score(y_true, y_pred),\n","    }\n","    return metrics\n","\n","\n","def classification_with_pytorch_model(model, X_train, y_train, X_test, y_test, device, is_vae=False):\n","    model = model.to(device)\n","    # check if X_train is tensor already\n","    if not torch.is_tensor(X_train):\n","        X_train = torch.tensor(X_train).to(device)\n","        X_test = torch.tensor(X_test).to(device)\n","\n","    if is_vae:\n","        X_train_encoded = model.encoder(X_train)[0].cpu().detach().numpy()\n","        X_test_encoded = model.encoder(X_test)[0].cpu().detach().numpy()\n","    else:\n","        X_train_encoded = model.encoder(X_train).cpu().detach().numpy()\n","        X_test_encoded = model.encoder(X_test).cpu().detach().numpy()\n","    X_train_encoded = np.reshape(X_train_encoded, (X_train_encoded.shape[0], -1))\n","    X_test_encoded = np.reshape(X_test_encoded, (X_test_encoded.shape[0], -1))\n","    return classification(X_train_encoded, y_train, X_test_encoded, y_test)\n","\n","\n","def run_sub_classification(method, X_train_encoded, y_train, X_test_encoded, y_test):\n","    clf = \"\"\n","    hyper_parameters = {}\n","    # hyper-parameter grids for classifiers\n","    rf_hyper_parameters = [{'n_estimators': [s for s in range(100, 1001, 200)],\n","                            'max_features': ['sqrt', 'log2'],\n","                            'min_samples_leaf': [1, 2, 3, 4, 5],\n","                            'criterion': ['gini', 'entropy']\n","                            }, ]\n","    svm_hyper_parameters = [{'C': [2 ** s for s in range(-5, 6, 2)], 'kernel': ['linear']},\n","                            {'C': [2 ** s for s in range(-5, 6, 2)], 'gamma': [2 ** s for s in range(3, -15, -2)],\n","                             'kernel': ['rbf']}]\n","    mlp_hyper_parameters = [{'hidden_layer_sizes': [(64, 64), (128, 128), (256, 256), (512, 512)],\n","                             'alpha': [2 ** s for s in range(-15, 1, 2)]}]\n","\n","    if method == 'SVM':\n","        clf = SVC(kernel='linear', probability=True)\n","        hyper_parameters = svm_hyper_parameters\n","    elif method == 'Random Forest':\n","        clf = RandomForestClassifier(n_estimators=100)\n","        hyper_parameters = rf_hyper_parameters\n","    elif method == 'MLP':\n","        clf = MLPClassifier(hidden_layer_sizes=(64, 64), max_iter=1000)\n","        hyper_parameters = mlp_hyper_parameters\n","    if enable_grid_search:\n","        clf = GridSearchCV(clf, param_grid=hyper_parameters,\n","                                cv=StratifiedKFold(n_splits=5, shuffle=True), scoring='roc_auc',\n","                                verbose=1, n_jobs=2)\n","    clf.fit(X_train_encoded, y_train)\n","    metrics = get_metrics(clf, X_test_encoded, y_test)\n","    metrics['classification_method'] = method\n","    print(metrics)\n","    if enable_grid_search:\n","        print(\"Best hyper-parameters: \", clf.best_params_)\n","        metrics['hyper_parameters'] = clf.best_params_\n","    return metrics\n","\n","\n","# Classification by using the transformed data\n","def classification(X_train_encoded, y_train, X_test_encoded, y_test):\n","    methods = ['MLP', 'SVM', 'Random Forest']\n","    metrics_list = []\n","    for method in methods:\n","        classification_start_time = time.time()\n","        metrics = run_sub_classification(method, X_train_encoded.copy(), y_train.copy(), X_test_encoded.copy(),\n","                                         y_test.copy())\n","        classification_end_time = time.time()\n","        print(method, \"Classification time: \", classification_end_time - classification_start_time)\n","        metrics['classification_time'] = classification_end_time - classification_start_time\n","        metrics_list.append(metrics)\n","    return metrics_list"]},{"cell_type":"markdown","metadata":{"id":"LS3jIzt9uj9B"},"source":["## 4.4. Training\n","\n","### 4.4.1. Training Objectives\n","- Split each dataset into a training set, validation set, and test set (64% training, 16% validation, and 20% test).\n","- Exclude the test set from model training.\n","- Implement early-stopping strategy: train models on the training set, compute reconstruction loss for the validation set after each epoch, stop training if no improvement in validation loss is observed for 20 epochs.\n","- Select the model with the lowest validation loss as the best model.\n","Utilize mean squared error as the reconstruction loss metric.\n","- Apply adaptive moment estimation (Adam) optimizer with default parameters (learning rate: 0.001, epsilon: 1e-07) as specified in the original paper.\n","- Utilize the encoder part of the best model to generate low-dimensional representations of microbiome data for subsequent disease prediction tasks.\n","\n","### 4.4.2. Hyperparams\n","\n","In the base case of svm classification, we have hyperparameter number of folds equal to 5. The epoch is set at max 200-2000, batch set is 32 for each autoencoder algorithm.\n","\n","The original implementation use command line arguments to facilitate the execution of experiments with varying configurations. In order to enhance usability within a Colab notebook environment, we modified this approach by setting the hyperparameters in our model directly.\n","\n","\n","### 4.4.3. Computation Requirements\n","\n","According to the original paper, an equipment with an Intel Core i9-9820X CPU (10 cores), 64 GB Memory, and a GPU of NVIDIA GTX 1080 Ti, running on Ubuntu 18.04 have the latency in time as in Table 3 below.\n","\n","![Table 3](https://github.com/ZKingQ/CS598-DLH-SP24/blob/main/assets/Computation_Table3.png?raw=1)\n","\n","In our experiments, we utilized the following computational resources\n","\n","1. Hardware:\n","  - iMac with Apple M1 chip / 16GB RAM / 8-core CPU / 8-core GPU (for initial code verification and testing)\n","  - Google Colab Pro with GPU and High-RAM (for AE models training)\n","  - Google Colab Pro with CPU (for sklearn classification models)\n","2. Average runtime for each epoch:\n","  - PCA: 0.6s\n","  - RP: 0.6s\n","  - Shallow AE: 0.3s\n","  - AE: 2.6s\n","  - CAE: 1.5s\n","  - VAE: 1.0s\n","3. GPU hours used: around 1 hour for all AE models training, excluding the time for data download, preprocessing and classification models\n","4. CPU hours used: since the sklearn classification models are run on CPU, the time is around 6 hour for all classification models training for each dataset.\n"]},{"cell_type":"markdown","metadata":{"id":"qGdp9R-IFwAZ"},"source":["#### Training Code"]},{"cell_type":"code","execution_count":11,"metadata":{"executionInfo":{"elapsed":4,"status":"ok","timestamp":1715020008887,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"xhLeFpd1Fobs"},"outputs":[],"source":["max_epoch_num = 2000\n","\n","def train_vae(model, train_loader, test_loader, device):\n","    model = model.to(device)\n","    optimizer = torch.optim.AdamW(model.parameters(), lr=0.001)\n","    rec_loss, kl_loss = [], []\n","    early_stopper = EarlyStopper(patience=5, delta=0.0)\n","    for epoch in range(1, max_epoch_num + 1):\n","\n","        \"\"\" model training \"\"\"\n","        model.train()\n","        cur_rec_loss, cur_kl_loss = [], []\n","        for batch_idx, (data, _) in enumerate(train_loader):\n","            data = data.to(device)\n","            rec, mu, std = model(data)\n","            loss, err, kl = model.loss_func(rec, data.reshape(-1, model.input_dim), mu, std)\n","\n","            optimizer.zero_grad()\n","            loss.backward()\n","            optimizer.step()\n","\n","            cur_rec_loss.append(err.item())\n","            cur_kl_loss.append(kl.item())\n","\n","        rec_loss.append(np.mean(cur_rec_loss))\n","        kl_loss.append(np.mean(cur_kl_loss))\n","\n","        \"\"\" model evaluation \"\"\"\n","        with torch.no_grad():\n","            test_loss = []\n","            for batch_idx, (data, _) in enumerate(test_loader):\n","                data = data.to(device)\n","                rec, mu, std = model(data)\n","                _, mse, _ = model.loss_func(rec, data.reshape(data.shape[0], -1), mu, std)\n","                test_loss.append(mse.item())\n","            if early_stopper(np.mean(test_loss)):\n","                print(f\"Early stopping at epoch {epoch}\")\n","                break\n","\n","        if epoch == 1 or epoch % 10 == 0:\n","            print(\n","                f\"-- epoch {epoch} --, train MSE: {np.mean(cur_rec_loss)}, train KL: {np.mean(cur_kl_loss)}, test MSE: {np.mean(test_loss)}\")\n","    return model\n","\n","\n","def train_ae(model, train_loader, test_loader, device):\n","    optimizer = torch.optim.AdamW(model.parameters(), lr=0.002)\n","\n","    losses = []\n","    early_stopper = EarlyStopper(patience=5, delta=0.0)\n","    for epoch in range(1, max_epoch_num + 1):\n","\n","        \"\"\" model training \"\"\"\n","        model.train()\n","        cur_rec_loss = []\n","        for batch_idx, (data, _) in enumerate(train_loader):\n","            data = data.to(device)\n","            rec = model(data)\n","            loss = model.loss_func(rec, data)\n","\n","            optimizer.zero_grad()\n","            loss.backward()\n","            optimizer.step()\n","\n","            cur_rec_loss.append(loss.item())\n","        losses.append(np.mean(cur_rec_loss))\n","\n","        \"\"\" model evaluation \"\"\"\n","        with torch.no_grad():\n","            test_loss = []\n","            for batch_idx, (data, _) in enumerate(test_loader):\n","                data = data.to(device)\n","                rec = model(data)\n","                loss = model.loss_func(rec, data)\n","                test_loss.append(loss.item())\n","            if early_stopper(np.mean(test_loss)):\n","                print(f\"Early stopping at epoch {epoch}\")\n","                break\n","\n","        if epoch == 1 or epoch % 10 == 0:\n","            print(f\"-- epoch {epoch} --, train MSE: {np.mean(cur_rec_loss)}, test MSE: {np.mean(test_loss)}\")\n","\n","    return model\n","\n","def run_model_training_and_classification(model_type, model, X_train, X_test, y_train, y_test, train_loader, test_loader, device):\n","    path = './cached_model/' + global_data_identifier + '_' + model_type + '_model.pth'\n","    print(\"Checking model path: \", path)\n","    if cache_trained_model and os.path.exists(path):\n","        model = torch.load(path, map_location=device)\n","    else:\n","        if model_type == 'vae':\n","            model = train_vae(model, train_loader, test_loader, device)\n","        else:\n","            model = train_ae(model, train_loader, test_loader, device)\n","        if cache_trained_model:\n","            torch.save(model, path)\n","    if train_only:\n","        return []\n","    return classification_with_pytorch_model(model, X_train, y_train, X_test, y_test, device, is_vae=(model_type == 'vae'))\n"]},{"cell_type":"code","execution_count":12,"metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1715020008888,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"a5AIqsThG-Sk"},"outputs":[],"source":["def run_shallow_ae_experiment(X_train, X_test, y_train, y_test, train_loader, test_loader, device):\n","    model = ShallowAutoEncoder(input_dim=X_train.shape[1]).to(device)\n","    return run_model_training_and_classification('shallow_ae', model, X_train, X_test, y_train, y_test, train_loader, test_loader, device)\n","\n","\n","def run_ae_experiment(X_train, X_test, y_train, y_test, train_loader, test_loader, device):\n","    model = AutoEncoder(input_dim=X_train.shape[1]).to(device)\n","    return run_model_training_and_classification('ae', model, X_train, X_test, y_train, y_test, train_loader, test_loader, device)\n","\n","\n","def run_cae_experiment(X_train, X_test, y_train, y_test, device):\n","    # one_side_dim = int(math.sqrt(X_train.shape[1])) + 1\n","    one_side_dim = 348\n","    enlarged_dim = one_side_dim ** 2\n","    X_train = np.column_stack((X_train, np.zeros((X_train.shape[0], enlarged_dim - X_train.shape[1]))))\n","    X_test = np.column_stack((X_test, np.zeros((X_test.shape[0], enlarged_dim - X_test.shape[1]))))\n","    # reshape\n","    X_train = np.reshape(X_train, (len(X_train), one_side_dim, one_side_dim, 1))\n","    X_test = np.reshape(X_test, (len(X_test), one_side_dim, one_side_dim, 1))\n","    # to float32\n","    X_train = X_train.astype('float32')\n","    X_test = X_test.astype('float32')\n","    X_train = torch.tensor(X_train).to(device)\n","    X_test = torch.tensor(X_test).to(device)\n","    X_train = X_train.transpose(1, 3)\n","    X_test = X_test.transpose(1, 3)\n","    train_loader = DataLoader(TensorDataset(torch.tensor(X_train), torch.tensor(y_train)), batch_size=32, shuffle=True)\n","    test_loader = DataLoader(TensorDataset(torch.tensor(X_test), torch.tensor(y_test)), batch_size=32, shuffle=False)\n","    model = CAE().to(device)\n","    return run_model_training_and_classification('cae', model, X_train, X_test, y_train, y_test, train_loader, test_loader, device)\n","\n","\n","def run_vae_experiment(X_train, X_test, y_train, y_test, train_loader, test_loader, device):\n","    model = VAE(input_dim=X_train.shape[1]).to(device)\n","    return run_model_training_and_classification('vae', model, X_train, X_test, y_train, y_test, train_loader, test_loader, device)\n","\n","\n","def run_pca_experiment(X_train, X_test, y_train, y_test):\n","    if train_only:\n","        return []\n","    pca = PCA(n_components=10)\n","    pca.fit(X_train)\n","    X_train_pca = pca.transform(X_train)\n","    X_test_pca = pca.transform(X_test)\n","    return classification(X_train_pca, y_train, X_test_pca, y_test)\n","\n","\n","def run_rp_experiment(X_train, X_test, y_train, y_test):\n","    if train_only:\n","        return []\n","    rp = GaussianRandomProjection(eps=0.5)\n","    rp.fit(X_train)\n","    X_train_rp = rp.transform(X_train)\n","    X_test_rp = rp.transform(X_test)\n","    return classification(X_train_rp, y_train, X_test_rp, y_test)\n","\n","\n","def run_plain_experiment(X_train, X_test, y_train, y_test):\n","    if train_only:\n","        return []\n","    return classification(X_train, y_train, X_test, y_test)"]},{"cell_type":"markdown","metadata":{"id":"Nbdu_gV5FetZ"},"source":["## 4.5. Evaluation\n","\n","### 4.5.1. Metrics Descriptions\n","\n","- Conduct 5-fold cross-validation on the reduced training set. This involves dividing the training set into five subsets, using four subsets for training and one for validation in each fold. This is to vary hyper-parameters and explore different combinations to find the best performing configuration.\n","- Evaluate the performance of the models using the area under the receiver operating characteristics curve (AUC). This metric assesses the model's ability to distinguish between different classes and is commonly used for classification tasks.\n","- Train a final classification model using the entire training set and the best hyper-parameter combination identified during cross-validation. This model aims to achieve optimal performance based on the selected configuration. Then test the final classification model on the separate test set, which was not used during training. This evaluation provides an unbiased assessment of the model's performance on unseen data.\n","- Repeat the entire procedure five times, each time using a different random partition seed to create new training, validation, and test sets. This helps account for potential variations in performance due to the specific data splits. Average the resulting AUC scores obtained from the five repetitions. This average serves as a summary metric to compare the performance of different models or approaches. It provides a more robust assessment by considering multiple iterations of the evaluation process.\n","\n","### 4.5.2. Evaluation Code\n","\n","The detailed cross-validation codes are in the `run_sub_classification` function of the [Classification Models](#classification-models) section."]},{"cell_type":"code","execution_count":13,"metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1715020008888,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"5WDva6WosGE6"},"outputs":[],"source":["def run_sub_experiment_on_data(model_type, X_train, X_test, y_train, y_test, train_loader, test_loader, device):\n","    metrics_list = []\n","    if model_type == 'shallow_ae':\n","        metrics_list = run_shallow_ae_experiment(X_train, X_test, y_train, y_test, train_loader, test_loader, device)\n","    elif model_type == 'ae':\n","        metrics_list = run_ae_experiment(X_train, X_test, y_train, y_test, train_loader, test_loader, device)\n","    elif model_type == 'cae':\n","        metrics_list = run_cae_experiment(X_train, X_test, y_train, y_test, device)\n","    elif model_type == 'vae':\n","        metrics_list = run_vae_experiment(X_train, X_test, y_train, y_test, train_loader, test_loader, device)\n","    elif model_type == 'pca':\n","        metrics_list = run_pca_experiment(X_train, X_test, y_train, y_test)\n","    elif model_type == 'rp':\n","        metrics_list = run_rp_experiment(X_train, X_test, y_train, y_test)\n","    elif model_type == 'plain':\n","        metrics_list = run_plain_experiment(X_train, X_test, y_train, y_test)\n","    return metrics_list\n","\n","\n","def run_experiment_on_data(data_dir, data_string):\n","    X_train, X_test, y_train, y_test = load_data(data_dir, data_string, dtype='float32')\n","    train_loader = DataLoader(\n","        TensorDataset(torch.tensor(X_train), torch.tensor(y_train)),\n","        batch_size=32, shuffle=True)\n","    test_loader = DataLoader(TensorDataset(torch.tensor(X_test), torch.tensor(y_test)),\n","                             batch_size=32, shuffle=False)\n","    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n","    if torch.backends.mps.is_available():\n","        device = torch.device(\"mps\")\n","\n","    all_metrics_df_on_current_data = pd.DataFrame()\n","    model_types = ['cae', 'shallow_ae', 'ae', 'vae', 'pca', 'rp', 'plain']\n","    for model_type in model_types:\n","        print(\"Running experiment on model: \", model_type)\n","        experiment_start_time = time.time()\n","        metrics_list = run_sub_experiment_on_data(model_type, X_train, X_test, y_train, y_test, train_loader,\n","                                                  test_loader, device)\n","        experiment_end_time = time.time()\n","        print(\"Experiment time: \", experiment_end_time - experiment_start_time)\n","        if train_only:\n","            continue\n","\n","        for metrics in metrics_list:\n","            metrics['experiment_time'] = experiment_end_time - experiment_start_time\n","            metrics['data'] = data_string\n","            metrics['model'] = model_type\n","            metrics['seed'] = seed\n","            metrics['max_epoch_num'] = max_epoch_num\n","            metrics['device'] = device\n","            print(metrics)\n","            # save metrics\n","        metrics_df = pd.DataFrame(metrics_list)\n","        all_metrics_df_on_current_data = pd.concat([all_metrics_df_on_current_data, metrics_df], ignore_index=True)\n","    return all_metrics_df_on_current_data\n","\n","\n","def run_experiment():\n","    data_dir = './data/marker/'\n","    data_list = os.listdir(data_dir)\n","    for data_string in data_list:\n","        global global_data_identifier\n","        global_data_identifier = data_string.split('.')[0]\n","        main_metrics_df = pd.DataFrame()\n","        print(\"Running experiment on data: \", data_string)\n","        all_metrics_df_on_current_data = run_experiment_on_data(data_dir, data_string)\n","        if train_only:\n","            continue\n","        main_metrics_df = pd.concat([main_metrics_df, all_metrics_df_on_current_data], ignore_index=True)\n","        csv_file_name = datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\") + '_metrics.csv'\n","        main_metrics_df.to_csv(csv_file_name, index=False)\n","        print(\"Metrics saved to file: \", csv_file_name)"]},{"cell_type":"markdown","metadata":{"id":"hY7oRbSRldrn"},"source":["#### Run Experiments"]},{"cell_type":"code","execution_count":14,"metadata":{"executionInfo":{"elapsed":5,"status":"ok","timestamp":1715020008888,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"eTKS0Y5CG-Sm"},"outputs":[],"source":["# run_experiment()"]},{"cell_type":"markdown","metadata":{"id":"ck2YsLNfG-Sn"},"source":["Due to run_experiment() takes a long time to run, we will run the following code to test the functions"]},{"cell_type":"code","execution_count":15,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"elapsed":45469,"status":"ok","timestamp":1715020054353,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"LLlAcNQoG-Sn","outputId":"b525523c-b38f-4200-ddc5-b9fb8471bea0"},"outputs":[{"name":"stderr","output_type":"stream","text":["<ipython-input-8-f62ecde9ce71>:31: DtypeWarning: Columns (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232) have mixed types. Specify dtype option on import or set low_memory=False.\n","  raw = pd.read_csv(filename, sep='\\t', index_col=0, header=None)\n"]},{"name":"stdout","output_type":"stream","text":["Train data shape:  (185, 120553)\n","Test data shape:  (47, 120553)\n","Running experiment on model:  cae\n"]},{"name":"stderr","output_type":"stream","text":["<ipython-input-12-0b552378b430>:27: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n","  train_loader = DataLoader(TensorDataset(torch.tensor(X_train), torch.tensor(y_train)), batch_size=32, shuffle=True)\n","<ipython-input-12-0b552378b430>:28: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n","  test_loader = DataLoader(TensorDataset(torch.tensor(X_test), torch.tensor(y_test)), batch_size=32, shuffle=False)\n"]},{"name":"stdout","output_type":"stream","text":["Checking model path:  ./cached_model/_cae_model.pth\n","-- epoch 1 --, train MSE: 0.17658995712796846, test MSE: 0.12242444977164268\n","{'AUC': 0.5, 'ACC': 0.5106382978723404, 'Recall': 1.0, 'Precision': 0.5106382978723404, 'F1_score': 0.676056338028169, 'classification_method': 'MLP'}\n","MLP Classification time:  7.871400594711304\n","{'AUC': 0.9601449275362319, 'ACC': 0.8723404255319149, 'Recall': 0.8333333333333334, 'Precision': 0.9090909090909091, 'F1_score': 0.8695652173913043, 'classification_method': 'SVM'}\n","SVM Classification time:  11.516283750534058\n","{'AUC': 0.9347826086956521, 'ACC': 0.851063829787234, 'Recall': 0.75, 'Precision': 0.9473684210526315, 'F1_score': 0.8372093023255814, 'classification_method': 'Random Forest'}\n","Random Forest Classification time:  3.333646059036255\n","Experiment time:  25.416157007217407\n","{'AUC': 0.5, 'ACC': 0.5106382978723404, 'Recall': 1.0, 'Precision': 0.5106382978723404, 'F1_score': 0.676056338028169, 'classification_method': 'MLP', 'classification_time': 7.871400594711304, 'experiment_time': 25.416157007217407, 'data': 'marker_Cirrhosis.txt', 'model': 'cae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.9601449275362319, 'ACC': 0.8723404255319149, 'Recall': 0.8333333333333334, 'Precision': 0.9090909090909091, 'F1_score': 0.8695652173913043, 'classification_method': 'SVM', 'classification_time': 11.516283750534058, 'experiment_time': 25.416157007217407, 'data': 'marker_Cirrhosis.txt', 'model': 'cae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.9347826086956521, 'ACC': 0.851063829787234, 'Recall': 0.75, 'Precision': 0.9473684210526315, 'F1_score': 0.8372093023255814, 'classification_method': 'Random Forest', 'classification_time': 3.333646059036255, 'experiment_time': 25.416157007217407, 'data': 'marker_Cirrhosis.txt', 'model': 'cae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","Running experiment on model:  shallow_ae\n","Checking model path:  ./cached_model/_shallow_ae_model.pth\n","-- epoch 1 --, train MSE: 963287.21875, test MSE: 650516.984375\n","{'AUC': 0.875, 'ACC': 0.7021276595744681, 'Recall': 1.0, 'Precision': 0.631578947368421, 'F1_score': 0.7741935483870968, 'classification_method': 'MLP'}\n","MLP Classification time:  0.1101541519165039\n","{'AUC': 0.858695652173913, 'ACC': 0.6808510638297872, 'Recall': 1.0, 'Precision': 0.6153846153846154, 'F1_score': 0.761904761904762, 'classification_method': 'SVM'}\n","SVM Classification time:  0.017426729202270508\n","{'AUC': 0.8695652173913044, 'ACC': 0.7872340425531915, 'Recall': 1.0, 'Precision': 0.7058823529411765, 'F1_score': 0.8275862068965517, 'classification_method': 'Random Forest'}\n","Random Forest Classification time:  0.18676280975341797\n","Experiment time:  0.5060217380523682\n","{'AUC': 0.875, 'ACC': 0.7021276595744681, 'Recall': 1.0, 'Precision': 0.631578947368421, 'F1_score': 0.7741935483870968, 'classification_method': 'MLP', 'classification_time': 0.1101541519165039, 'experiment_time': 0.5060217380523682, 'data': 'marker_Cirrhosis.txt', 'model': 'shallow_ae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.858695652173913, 'ACC': 0.6808510638297872, 'Recall': 1.0, 'Precision': 0.6153846153846154, 'F1_score': 0.761904761904762, 'classification_method': 'SVM', 'classification_time': 0.017426729202270508, 'experiment_time': 0.5060217380523682, 'data': 'marker_Cirrhosis.txt', 'model': 'shallow_ae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.8695652173913044, 'ACC': 0.7872340425531915, 'Recall': 1.0, 'Precision': 0.7058823529411765, 'F1_score': 0.8275862068965517, 'classification_method': 'Random Forest', 'classification_time': 0.18676280975341797, 'experiment_time': 0.5060217380523682, 'data': 'marker_Cirrhosis.txt', 'model': 'shallow_ae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","Running experiment on model:  ae\n","Checking model path:  ./cached_model/_ae_model.pth\n","-- epoch 1 --, train MSE: 0.17200066521763802, test MSE: 0.0746481604874134\n","{'AUC': 0.14945652173913043, 'ACC': 0.5106382978723404, 'Recall': 1.0, 'Precision': 0.5106382978723404, 'F1_score': 0.676056338028169, 'classification_method': 'MLP'}\n","MLP Classification time:  0.034488677978515625\n","{'AUC': 0.10144927536231883, 'ACC': 0.5106382978723404, 'Recall': 1.0, 'Precision': 0.5106382978723404, 'F1_score': 0.676056338028169, 'classification_method': 'SVM'}\n","SVM Classification time:  0.02523040771484375\n","{'AUC': 0.7590579710144928, 'ACC': 0.8297872340425532, 'Recall': 0.6666666666666666, 'Precision': 1.0, 'F1_score': 0.8, 'classification_method': 'Random Forest'}\n","Random Forest Classification time:  0.20232725143432617\n","Experiment time:  1.7575924396514893\n","{'AUC': 0.14945652173913043, 'ACC': 0.5106382978723404, 'Recall': 1.0, 'Precision': 0.5106382978723404, 'F1_score': 0.676056338028169, 'classification_method': 'MLP', 'classification_time': 0.034488677978515625, 'experiment_time': 1.7575924396514893, 'data': 'marker_Cirrhosis.txt', 'model': 'ae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.10144927536231883, 'ACC': 0.5106382978723404, 'Recall': 1.0, 'Precision': 0.5106382978723404, 'F1_score': 0.676056338028169, 'classification_method': 'SVM', 'classification_time': 0.02523040771484375, 'experiment_time': 1.7575924396514893, 'data': 'marker_Cirrhosis.txt', 'model': 'ae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.7590579710144928, 'ACC': 0.8297872340425532, 'Recall': 0.6666666666666666, 'Precision': 1.0, 'F1_score': 0.8, 'classification_method': 'Random Forest', 'classification_time': 0.20232725143432617, 'experiment_time': 1.7575924396514893, 'data': 'marker_Cirrhosis.txt', 'model': 'ae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","Running experiment on model:  vae\n","Checking model path:  ./cached_model/_vae_model.pth\n","-- epoch 1 --, train MSE: 2528070.25, train KL: 176.29419195652008, test MSE: 1772184.625\n","{'AUC': 0.8740942028985508, 'ACC': 0.8297872340425532, 'Recall': 0.9583333333333334, 'Precision': 0.7666666666666667, 'F1_score': 0.8518518518518519, 'classification_method': 'MLP'}\n","MLP Classification time:  0.6301736831665039\n","{'AUC': 0.8759057971014493, 'ACC': 0.7446808510638298, 'Recall': 0.9583333333333334, 'Precision': 0.6764705882352942, 'F1_score': 0.7931034482758621, 'classification_method': 'SVM'}\n","SVM Classification time:  0.01723194122314453\n","{'AUC': 0.8233695652173914, 'ACC': 0.723404255319149, 'Recall': 0.6666666666666666, 'Precision': 0.7619047619047619, 'F1_score': 0.7111111111111111, 'classification_method': 'Random Forest'}\n","Random Forest Classification time:  0.19273972511291504\n","Experiment time:  1.4009058475494385\n","{'AUC': 0.8740942028985508, 'ACC': 0.8297872340425532, 'Recall': 0.9583333333333334, 'Precision': 0.7666666666666667, 'F1_score': 0.8518518518518519, 'classification_method': 'MLP', 'classification_time': 0.6301736831665039, 'experiment_time': 1.4009058475494385, 'data': 'marker_Cirrhosis.txt', 'model': 'vae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.8759057971014493, 'ACC': 0.7446808510638298, 'Recall': 0.9583333333333334, 'Precision': 0.6764705882352942, 'F1_score': 0.7931034482758621, 'classification_method': 'SVM', 'classification_time': 0.01723194122314453, 'experiment_time': 1.4009058475494385, 'data': 'marker_Cirrhosis.txt', 'model': 'vae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.8233695652173914, 'ACC': 0.723404255319149, 'Recall': 0.6666666666666666, 'Precision': 0.7619047619047619, 'F1_score': 0.7111111111111111, 'classification_method': 'Random Forest', 'classification_time': 0.19273972511291504, 'experiment_time': 1.4009058475494385, 'data': 'marker_Cirrhosis.txt', 'model': 'vae', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","Running experiment on model:  pca\n","{'AUC': 0.9293478260869565, 'ACC': 0.8085106382978723, 'Recall': 0.75, 'Precision': 0.8571428571428571, 'F1_score': 0.7999999999999999, 'classification_method': 'MLP'}\n","MLP Classification time:  0.4155449867248535\n","{'AUC': 0.9547101449275361, 'ACC': 0.8723404255319149, 'Recall': 0.8333333333333334, 'Precision': 0.9090909090909091, 'F1_score': 0.8695652173913043, 'classification_method': 'SVM'}\n","SVM Classification time:  0.407397985458374\n","{'AUC': 0.9030797101449276, 'ACC': 0.8936170212765957, 'Recall': 0.8333333333333334, 'Precision': 0.9523809523809523, 'F1_score': 0.888888888888889, 'classification_method': 'Random Forest'}\n","Random Forest Classification time:  0.17244529724121094\n","Experiment time:  1.7776365280151367\n","{'AUC': 0.9293478260869565, 'ACC': 0.8085106382978723, 'Recall': 0.75, 'Precision': 0.8571428571428571, 'F1_score': 0.7999999999999999, 'classification_method': 'MLP', 'classification_time': 0.4155449867248535, 'experiment_time': 1.7776365280151367, 'data': 'marker_Cirrhosis.txt', 'model': 'pca', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.9547101449275361, 'ACC': 0.8723404255319149, 'Recall': 0.8333333333333334, 'Precision': 0.9090909090909091, 'F1_score': 0.8695652173913043, 'classification_method': 'SVM', 'classification_time': 0.407397985458374, 'experiment_time': 1.7776365280151367, 'data': 'marker_Cirrhosis.txt', 'model': 'pca', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.9030797101449276, 'ACC': 0.8936170212765957, 'Recall': 0.8333333333333334, 'Precision': 0.9523809523809523, 'F1_score': 0.888888888888889, 'classification_method': 'Random Forest', 'classification_time': 0.17244529724121094, 'experiment_time': 1.7776365280151367, 'data': 'marker_Cirrhosis.txt', 'model': 'pca', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","Running experiment on model:  rp\n","{'AUC': 0.9094202898550725, 'ACC': 0.8297872340425532, 'Recall': 0.9166666666666666, 'Precision': 0.7857142857142857, 'F1_score': 0.8461538461538461, 'classification_method': 'MLP'}\n","MLP Classification time:  0.21117329597473145\n","{'AUC': 0.9257246376811594, 'ACC': 0.8936170212765957, 'Recall': 0.9166666666666666, 'Precision': 0.88, 'F1_score': 0.8979591836734694, 'classification_method': 'SVM'}\n","SVM Classification time:  0.03352499008178711\n","{'AUC': 0.9239130434782609, 'ACC': 0.7872340425531915, 'Recall': 0.75, 'Precision': 0.8181818181818182, 'F1_score': 0.7826086956521738, 'classification_method': 'Random Forest'}\n","Random Forest Classification time:  0.2612457275390625\n","Experiment time:  1.5291056632995605\n","{'AUC': 0.9094202898550725, 'ACC': 0.8297872340425532, 'Recall': 0.9166666666666666, 'Precision': 0.7857142857142857, 'F1_score': 0.8461538461538461, 'classification_method': 'MLP', 'classification_time': 0.21117329597473145, 'experiment_time': 1.5291056632995605, 'data': 'marker_Cirrhosis.txt', 'model': 'rp', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.9257246376811594, 'ACC': 0.8936170212765957, 'Recall': 0.9166666666666666, 'Precision': 0.88, 'F1_score': 0.8979591836734694, 'classification_method': 'SVM', 'classification_time': 0.03352499008178711, 'experiment_time': 1.5291056632995605, 'data': 'marker_Cirrhosis.txt', 'model': 'rp', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.9239130434782609, 'ACC': 0.7872340425531915, 'Recall': 0.75, 'Precision': 0.8181818181818182, 'F1_score': 0.7826086956521738, 'classification_method': 'Random Forest', 'classification_time': 0.2612457275390625, 'experiment_time': 1.5291056632995605, 'data': 'marker_Cirrhosis.txt', 'model': 'rp', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","Running experiment on model:  plain\n","{'AUC': 0.9945652173913043, 'ACC': 0.9148936170212766, 'Recall': 1.0, 'Precision': 0.8571428571428571, 'F1_score': 0.923076923076923, 'classification_method': 'MLP'}\n","MLP Classification time:  5.112205982208252\n","{'AUC': 0.9836956521739131, 'ACC': 0.9148936170212766, 'Recall': 0.875, 'Precision': 0.9545454545454546, 'F1_score': 0.9130434782608695, 'classification_method': 'SVM'}\n","SVM Classification time:  3.034834623336792\n","{'AUC': 0.9855072463768115, 'ACC': 0.9574468085106383, 'Recall': 0.9166666666666666, 'Precision': 1.0, 'F1_score': 0.9565217391304348, 'classification_method': 'Random Forest'}\n","Random Forest Classification time:  0.5710349082946777\n","Experiment time:  8.719501256942749\n","{'AUC': 0.9945652173913043, 'ACC': 0.9148936170212766, 'Recall': 1.0, 'Precision': 0.8571428571428571, 'F1_score': 0.923076923076923, 'classification_method': 'MLP', 'classification_time': 5.112205982208252, 'experiment_time': 8.719501256942749, 'data': 'marker_Cirrhosis.txt', 'model': 'plain', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.9836956521739131, 'ACC': 0.9148936170212766, 'Recall': 0.875, 'Precision': 0.9545454545454546, 'F1_score': 0.9130434782608695, 'classification_method': 'SVM', 'classification_time': 3.034834623336792, 'experiment_time': 8.719501256942749, 'data': 'marker_Cirrhosis.txt', 'model': 'plain', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n","{'AUC': 0.9855072463768115, 'ACC': 0.9574468085106383, 'Recall': 0.9166666666666666, 'Precision': 1.0, 'F1_score': 0.9565217391304348, 'classification_method': 'Random Forest', 'classification_time': 0.5710349082946777, 'experiment_time': 8.719501256942749, 'data': 'marker_Cirrhosis.txt', 'model': 'plain', 'seed': 42, 'max_epoch_num': 1, 'device': device(type='cuda')}\n"]},{"data":{"application/vnd.google.colaboratory.intrinsic+json":{"summary":"{\n  \"name\": \"run_experiment_on_data(data_dir, data_string)\",\n  \"rows\": 21,\n  \"fields\": [\n    {\n      \"column\": \"AUC\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.251869848309385,\n        \"min\": 0.10144927536231883,\n        \"max\": 0.9945652173913043,\n        \"num_unique_values\": 21,\n        \"samples\": [\n          0.5,\n          0.9239130434782609,\n          0.9094202898550725\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ACC\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.13454898621834344,\n        \"min\": 0.5106382978723404,\n        \"max\": 0.9574468085106383,\n        \"num_unique_values\": 13,\n        \"samples\": [\n          0.9148936170212766,\n          0.8085106382978723,\n          0.5106382978723404\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Recall\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.11497498002498646,\n        \"min\": 0.6666666666666666,\n        \"max\": 1.0,\n        \"num_unique_values\": 7,\n        \"samples\": [\n          1.0,\n          0.8333333333333334,\n          0.9166666666666666\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Precision\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.1618139959950731,\n        \"min\": 0.5106382978723404,\n        \"max\": 1.0,\n        \"num_unique_values\": 16,\n        \"samples\": [\n          0.5106382978723404,\n          0.9090909090909091,\n          0.7058823529411765\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"F1_score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.08299265073312334,\n        \"min\": 0.676056338028169,\n        \"max\": 0.9565217391304348,\n        \"num_unique_values\": 18,\n        \"samples\": [\n          0.676056338028169,\n          0.8695652173913043,\n          0.7931034482758621\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"classification_method\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"MLP\",\n          \"SVM\",\n          \"Random Forest\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"classification_time\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3.050936773561446,\n        \"min\": 0.01723194122314453,\n        \"max\": 11.516283750534058,\n        \"num_unique_values\": 21,\n        \"samples\": [\n          7.871400594711304,\n          0.2612457275390625,\n          0.21117329597473145\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"experiment_time\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 8.585624327678431,\n        \"min\": 0.5060217380523682,\n        \"max\": 25.416157007217407,\n        \"num_unique_values\": 7,\n        \"samples\": [\n          25.416157007217407,\n          0.5060217380523682,\n          1.5291056632995605\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"data\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"marker_Cirrhosis.txt\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"model\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 7,\n        \"samples\": [\n          \"cae\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"seed\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 42,\n        \"max\": 42,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          42\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"max_epoch_num\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"device\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"cuda\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}","type":"dataframe"},"text/html":["\n","  <div id=\"df-c3627949-391a-438e-9c5e-846d0a020152\" class=\"colab-df-container\">\n","    <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>AUC</th>\n","      <th>ACC</th>\n","      <th>Recall</th>\n","      <th>Precision</th>\n","      <th>F1_score</th>\n","      <th>classification_method</th>\n","      <th>classification_time</th>\n","      <th>experiment_time</th>\n","      <th>data</th>\n","      <th>model</th>\n","      <th>seed</th>\n","      <th>max_epoch_num</th>\n","      <th>device</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.500000</td>\n","      <td>0.510638</td>\n","      <td>1.000000</td>\n","      <td>0.510638</td>\n","      <td>0.676056</td>\n","      <td>MLP</td>\n","      <td>7.871401</td>\n","      <td>25.416157</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>0.960145</td>\n","      <td>0.872340</td>\n","      <td>0.833333</td>\n","      <td>0.909091</td>\n","      <td>0.869565</td>\n","      <td>SVM</td>\n","      <td>11.516284</td>\n","      <td>25.416157</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0.934783</td>\n","      <td>0.851064</td>\n","      <td>0.750000</td>\n","      <td>0.947368</td>\n","      <td>0.837209</td>\n","      <td>Random Forest</td>\n","      <td>3.333646</td>\n","      <td>25.416157</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>0.875000</td>\n","      <td>0.702128</td>\n","      <td>1.000000</td>\n","      <td>0.631579</td>\n","      <td>0.774194</td>\n","      <td>MLP</td>\n","      <td>0.110154</td>\n","      <td>0.506022</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0.858696</td>\n","      <td>0.680851</td>\n","      <td>1.000000</td>\n","      <td>0.615385</td>\n","      <td>0.761905</td>\n","      <td>SVM</td>\n","      <td>0.017427</td>\n","      <td>0.506022</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>0.869565</td>\n","      <td>0.787234</td>\n","      <td>1.000000</td>\n","      <td>0.705882</td>\n","      <td>0.827586</td>\n","      <td>Random Forest</td>\n","      <td>0.186763</td>\n","      <td>0.506022</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>0.149457</td>\n","      <td>0.510638</td>\n","      <td>1.000000</td>\n","      <td>0.510638</td>\n","      <td>0.676056</td>\n","      <td>MLP</td>\n","      <td>0.034489</td>\n","      <td>1.757592</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>0.101449</td>\n","      <td>0.510638</td>\n","      <td>1.000000</td>\n","      <td>0.510638</td>\n","      <td>0.676056</td>\n","      <td>SVM</td>\n","      <td>0.025230</td>\n","      <td>1.757592</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>0.759058</td>\n","      <td>0.829787</td>\n","      <td>0.666667</td>\n","      <td>1.000000</td>\n","      <td>0.800000</td>\n","      <td>Random Forest</td>\n","      <td>0.202327</td>\n","      <td>1.757592</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>9</th>\n","      <td>0.874094</td>\n","      <td>0.829787</td>\n","      <td>0.958333</td>\n","      <td>0.766667</td>\n","      <td>0.851852</td>\n","      <td>MLP</td>\n","      <td>0.630174</td>\n","      <td>1.400906</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>10</th>\n","      <td>0.875906</td>\n","      <td>0.744681</td>\n","      <td>0.958333</td>\n","      <td>0.676471</td>\n","      <td>0.793103</td>\n","      <td>SVM</td>\n","      <td>0.017232</td>\n","      <td>1.400906</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>11</th>\n","      <td>0.823370</td>\n","      <td>0.723404</td>\n","      <td>0.666667</td>\n","      <td>0.761905</td>\n","      <td>0.711111</td>\n","      <td>Random Forest</td>\n","      <td>0.192740</td>\n","      <td>1.400906</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>12</th>\n","      <td>0.929348</td>\n","      <td>0.808511</td>\n","      <td>0.750000</td>\n","      <td>0.857143</td>\n","      <td>0.800000</td>\n","      <td>MLP</td>\n","      <td>0.415545</td>\n","      <td>1.777637</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>13</th>\n","      <td>0.954710</td>\n","      <td>0.872340</td>\n","      <td>0.833333</td>\n","      <td>0.909091</td>\n","      <td>0.869565</td>\n","      <td>SVM</td>\n","      <td>0.407398</td>\n","      <td>1.777637</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>14</th>\n","      <td>0.903080</td>\n","      <td>0.893617</td>\n","      <td>0.833333</td>\n","      <td>0.952381</td>\n","      <td>0.888889</td>\n","      <td>Random Forest</td>\n","      <td>0.172445</td>\n","      <td>1.777637</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>15</th>\n","      <td>0.909420</td>\n","      <td>0.829787</td>\n","      <td>0.916667</td>\n","      <td>0.785714</td>\n","      <td>0.846154</td>\n","      <td>MLP</td>\n","      <td>0.211173</td>\n","      <td>1.529106</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>16</th>\n","      <td>0.925725</td>\n","      <td>0.893617</td>\n","      <td>0.916667</td>\n","      <td>0.880000</td>\n","      <td>0.897959</td>\n","      <td>SVM</td>\n","      <td>0.033525</td>\n","      <td>1.529106</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>17</th>\n","      <td>0.923913</td>\n","      <td>0.787234</td>\n","      <td>0.750000</td>\n","      <td>0.818182</td>\n","      <td>0.782609</td>\n","      <td>Random Forest</td>\n","      <td>0.261246</td>\n","      <td>1.529106</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>18</th>\n","      <td>0.994565</td>\n","      <td>0.914894</td>\n","      <td>1.000000</td>\n","      <td>0.857143</td>\n","      <td>0.923077</td>\n","      <td>MLP</td>\n","      <td>5.112206</td>\n","      <td>8.719501</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>19</th>\n","      <td>0.983696</td>\n","      <td>0.914894</td>\n","      <td>0.875000</td>\n","      <td>0.954545</td>\n","      <td>0.913043</td>\n","      <td>SVM</td>\n","      <td>3.034835</td>\n","      <td>8.719501</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","    <tr>\n","      <th>20</th>\n","      <td>0.985507</td>\n","      <td>0.957447</td>\n","      <td>0.916667</td>\n","      <td>1.000000</td>\n","      <td>0.956522</td>\n","      <td>Random Forest</td>\n","      <td>0.571035</td>\n","      <td>8.719501</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>1</td>\n","      <td>cuda</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","    <div class=\"colab-df-buttons\">\n","\n","  <div class=\"colab-df-container\">\n","    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c3627949-391a-438e-9c5e-846d0a020152')\"\n","            title=\"Convert this dataframe to an interactive table.\"\n","            style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n","    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n","  </svg>\n","    </button>\n","\n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    .colab-df-buttons div {\n","      margin-bottom: 4px;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","    <script>\n","      const buttonEl =\n","        document.querySelector('#df-c3627949-391a-438e-9c5e-846d0a020152 button.colab-df-convert');\n","      buttonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","      async function convertToInteractive(key) {\n","        const element = document.querySelector('#df-c3627949-391a-438e-9c5e-846d0a020152');\n","        const dataTable =\n","          await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                    [key], {});\n","        if (!dataTable) return;\n","\n","        const docLinkHtml = 'Like what you see? Visit the ' +\n","          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","          + ' to learn more about interactive tables.';\n","        element.innerHTML = '';\n","        dataTable['output_type'] = 'display_data';\n","        await google.colab.output.renderOutput(dataTable, element);\n","        const docLink = document.createElement('div');\n","        docLink.innerHTML = docLinkHtml;\n","        element.appendChild(docLink);\n","      }\n","    </script>\n","  </div>\n","\n","\n","<div id=\"df-b048feb4-5618-49bb-a0f9-f93f49008575\">\n","  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-b048feb4-5618-49bb-a0f9-f93f49008575')\"\n","            title=\"Suggest charts\"\n","            style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","  </button>\n","\n","<style>\n","  .colab-df-quickchart {\n","      --bg-color: #E8F0FE;\n","      --fill-color: #1967D2;\n","      --hover-bg-color: #E2EBFA;\n","      --hover-fill-color: #174EA6;\n","      --disabled-fill-color: #AAA;\n","      --disabled-bg-color: #DDD;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","      --bg-color: #3B4455;\n","      --fill-color: #D2E3FC;\n","      --hover-bg-color: #434B5C;\n","      --hover-fill-color: #FFFFFF;\n","      --disabled-bg-color: #3B4455;\n","      --disabled-fill-color: #666;\n","  }\n","\n","  .colab-df-quickchart {\n","    background-color: var(--bg-color);\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: var(--fill-color);\n","    height: 32px;\n","    padding: 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: var(--hover-bg-color);\n","    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: var(--button-hover-fill-color);\n","  }\n","\n","  .colab-df-quickchart-complete:disabled,\n","  .colab-df-quickchart-complete:disabled:hover {\n","    background-color: var(--disabled-bg-color);\n","    fill: var(--disabled-fill-color);\n","    box-shadow: none;\n","  }\n","\n","  .colab-df-spinner {\n","    border: 2px solid var(--fill-color);\n","    border-color: transparent;\n","    border-bottom-color: var(--fill-color);\n","    animation:\n","      spin 1s steps(1) infinite;\n","  }\n","\n","  @keyframes spin {\n","    0% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","      border-left-color: var(--fill-color);\n","    }\n","    20% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    30% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","      border-right-color: var(--fill-color);\n","    }\n","    40% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    60% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","    }\n","    80% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-bottom-color: var(--fill-color);\n","    }\n","    90% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","    }\n","  }\n","</style>\n","\n","  <script>\n","    async function quickchart(key) {\n","      const quickchartButtonEl =\n","        document.querySelector('#' + key + ' button');\n","      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n","      quickchartButtonEl.classList.add('colab-df-spinner');\n","      try {\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      } catch (error) {\n","        console.error('Error during call to suggestCharts:', error);\n","      }\n","      quickchartButtonEl.classList.remove('colab-df-spinner');\n","      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n","    }\n","    (() => {\n","      let quickchartButtonEl =\n","        document.querySelector('#df-b048feb4-5618-49bb-a0f9-f93f49008575 button');\n","      quickchartButtonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","    })();\n","  </script>\n","</div>\n","\n","    </div>\n","  </div>\n"],"text/plain":["         AUC       ACC    Recall  Precision  F1_score classification_method  \\\n","0   0.500000  0.510638  1.000000   0.510638  0.676056                   MLP   \n","1   0.960145  0.872340  0.833333   0.909091  0.869565                   SVM   \n","2   0.934783  0.851064  0.750000   0.947368  0.837209         Random Forest   \n","3   0.875000  0.702128  1.000000   0.631579  0.774194                   MLP   \n","4   0.858696  0.680851  1.000000   0.615385  0.761905                   SVM   \n","5   0.869565  0.787234  1.000000   0.705882  0.827586         Random Forest   \n","6   0.149457  0.510638  1.000000   0.510638  0.676056                   MLP   \n","7   0.101449  0.510638  1.000000   0.510638  0.676056                   SVM   \n","8   0.759058  0.829787  0.666667   1.000000  0.800000         Random Forest   \n","9   0.874094  0.829787  0.958333   0.766667  0.851852                   MLP   \n","10  0.875906  0.744681  0.958333   0.676471  0.793103                   SVM   \n","11  0.823370  0.723404  0.666667   0.761905  0.711111         Random Forest   \n","12  0.929348  0.808511  0.750000   0.857143  0.800000                   MLP   \n","13  0.954710  0.872340  0.833333   0.909091  0.869565                   SVM   \n","14  0.903080  0.893617  0.833333   0.952381  0.888889         Random Forest   \n","15  0.909420  0.829787  0.916667   0.785714  0.846154                   MLP   \n","16  0.925725  0.893617  0.916667   0.880000  0.897959                   SVM   \n","17  0.923913  0.787234  0.750000   0.818182  0.782609         Random Forest   \n","18  0.994565  0.914894  1.000000   0.857143  0.923077                   MLP   \n","19  0.983696  0.914894  0.875000   0.954545  0.913043                   SVM   \n","20  0.985507  0.957447  0.916667   1.000000  0.956522         Random Forest   \n","\n","    classification_time  experiment_time                  data       model  \\\n","0              7.871401        25.416157  marker_Cirrhosis.txt         cae   \n","1             11.516284        25.416157  marker_Cirrhosis.txt         cae   \n","2              3.333646        25.416157  marker_Cirrhosis.txt         cae   \n","3              0.110154         0.506022  marker_Cirrhosis.txt  shallow_ae   \n","4              0.017427         0.506022  marker_Cirrhosis.txt  shallow_ae   \n","5              0.186763         0.506022  marker_Cirrhosis.txt  shallow_ae   \n","6              0.034489         1.757592  marker_Cirrhosis.txt          ae   \n","7              0.025230         1.757592  marker_Cirrhosis.txt          ae   \n","8              0.202327         1.757592  marker_Cirrhosis.txt          ae   \n","9              0.630174         1.400906  marker_Cirrhosis.txt         vae   \n","10             0.017232         1.400906  marker_Cirrhosis.txt         vae   \n","11             0.192740         1.400906  marker_Cirrhosis.txt         vae   \n","12             0.415545         1.777637  marker_Cirrhosis.txt         pca   \n","13             0.407398         1.777637  marker_Cirrhosis.txt         pca   \n","14             0.172445         1.777637  marker_Cirrhosis.txt         pca   \n","15             0.211173         1.529106  marker_Cirrhosis.txt          rp   \n","16             0.033525         1.529106  marker_Cirrhosis.txt          rp   \n","17             0.261246         1.529106  marker_Cirrhosis.txt          rp   \n","18             5.112206         8.719501  marker_Cirrhosis.txt       plain   \n","19             3.034835         8.719501  marker_Cirrhosis.txt       plain   \n","20             0.571035         8.719501  marker_Cirrhosis.txt       plain   \n","\n","    seed  max_epoch_num device  \n","0     42              1   cuda  \n","1     42              1   cuda  \n","2     42              1   cuda  \n","3     42              1   cuda  \n","4     42              1   cuda  \n","5     42              1   cuda  \n","6     42              1   cuda  \n","7     42              1   cuda  \n","8     42              1   cuda  \n","9     42              1   cuda  \n","10    42              1   cuda  \n","11    42              1   cuda  \n","12    42              1   cuda  \n","13    42              1   cuda  \n","14    42              1   cuda  \n","15    42              1   cuda  \n","16    42              1   cuda  \n","17    42              1   cuda  \n","18    42              1   cuda  \n","19    42              1   cuda  \n","20    42              1   cuda  "]},"execution_count":15,"metadata":{},"output_type":"execute_result"}],"source":["data_dir = './data/marker/'\n","data_string = 'marker_Cirrhosis.txt'\n","max_epoch_num = 1\n","enable_grid_search = False\n","run_experiment_on_data(data_dir, data_string)"]},{"cell_type":"markdown","metadata":{"id":"gX6bCcZNuxmz"},"source":["# Results\n","We recorded the best AUC scores for each model, classification algorithm, and dataset. The results are stored in the following csv files."]},{"cell_type":"code","execution_count":16,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"elapsed":278,"status":"ok","timestamp":1715020054613,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"Y4WgUnso82d1","outputId":"18d5387c-77fd-47ff-cd61-4152c00b17c0"},"outputs":[{"data":{"application/vnd.google.colaboratory.intrinsic+json":{"summary":"{\n  \"name\": \"df\",\n  \"rows\": 126,\n  \"fields\": [\n    {\n      \"column\": \"AUC\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.17947797573183263,\n        \"min\": 0.2373737373737373,\n        \"max\": 0.9927536231884058,\n        \"num_unique_values\": 104,\n        \"samples\": [\n          0.5016835016835017,\n          0.7933333333333334,\n          0.8466666666666667\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ACC\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.12687554285852,\n        \"min\": 0.3529411764705882,\n        \"max\": 0.9361702127659576,\n        \"num_unique_values\": 39,\n        \"samples\": [\n          0.6363636363636364,\n          0.45,\n          0.8085106382978723\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Recall\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.375955354420995,\n        \"min\": 0.0,\n        \"max\": 1.0,\n        \"num_unique_values\": 37,\n        \"samples\": [\n          0.8235294117647058,\n          0.7272727272727273,\n          0.6666666666666666\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Precision\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.32601476135639157,\n        \"min\": 0.0,\n        \"max\": 1.0,\n        \"num_unique_values\": 53,\n        \"samples\": [\n          0.7058823529411765,\n          0.8333333333333334,\n          0.5384615384615384\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"F1_score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.3218139527022976,\n        \"min\": 0.0,\n        \"max\": 0.9333333333333332,\n        \"num_unique_values\": 61,\n        \"samples\": [\n          0.8260869565217391,\n          0.8,\n          0.3076923076923076\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"classification_method\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"MLP\",\n          \"SVM\",\n          \"Random Forest\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"hyper_parameters\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 77,\n        \"samples\": [\n          \"{'C': 2, 'kernel': 'linear'}\",\n          \"{'criterion': 'gini', 'max_features': 'log2', 'min_samples_leaf': 2, 'n_estimators': 500}\",\n          \"{'criterion': 'entropy', 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100}\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"classification_time\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1566.621191691799,\n        \"min\": 0.2289099693298339,\n        \"max\": 10209.280279874802,\n        \"num_unique_values\": 126,\n        \"samples\": [\n          0.3991360664367676,\n          2277.429843902588,\n          80.39075970649719\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"experiment_time\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3513.292497526278,\n        \"min\": 87.20031785964966,\n        \"max\": 14115.473663568497,\n        \"num_unique_values\": 42,\n        \"samples\": [\n          102.14193177223206,\n          191.006667137146,\n          4146.935734510422\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"data\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 6,\n        \"samples\": [\n          \"marker_Cirrhosis.txt\",\n          \"marker_Obesity.txt\",\n          \"marker_WT2D.txt\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"model\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 7,\n        \"samples\": [\n          \"shallow_ae\",\n          \"ae\",\n          \"rp\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"seed\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 42,\n        \"max\": 42,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          42\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"max_epoch_num\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 903,\n        \"min\": 200,\n        \"max\": 2000,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          2000\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"device\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"cpu\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}","type":"dataframe","variable_name":"df"},"text/html":["\n","  <div id=\"df-d3fa703a-ef5a-4d98-93cd-ab61bc2a3daf\" class=\"colab-df-container\">\n","    <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>AUC</th>\n","      <th>ACC</th>\n","      <th>Recall</th>\n","      <th>Precision</th>\n","      <th>F1_score</th>\n","      <th>classification_method</th>\n","      <th>hyper_parameters</th>\n","      <th>classification_time</th>\n","      <th>experiment_time</th>\n","      <th>data</th>\n","      <th>model</th>\n","      <th>seed</th>\n","      <th>max_epoch_num</th>\n","      <th>device</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>0.947464</td>\n","      <td>0.829787</td>\n","      <td>0.791667</td>\n","      <td>0.863636</td>\n","      <td>0.826087</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.0001220703125, 'hidden_layer_sizes...</td>\n","      <td>66.259109</td>\n","      <td>238.745647</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>0.916667</td>\n","      <td>0.872340</td>\n","      <td>0.875000</td>\n","      <td>0.875000</td>\n","      <td>0.875000</td>\n","      <td>SVM</td>\n","      <td>{'C': 32, 'gamma': 0.0078125, 'kernel': 'rbf'}</td>\n","      <td>1.120851</td>\n","      <td>238.745647</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>0.943841</td>\n","      <td>0.829787</td>\n","      <td>0.791667</td>\n","      <td>0.863636</td>\n","      <td>0.826087</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'sqrt...</td>\n","      <td>111.178497</td>\n","      <td>238.745647</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>0.926630</td>\n","      <td>0.829787</td>\n","      <td>0.750000</td>\n","      <td>0.900000</td>\n","      <td>0.818182</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>64.426869</td>\n","      <td>730.488681</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>0.926630</td>\n","      <td>0.851064</td>\n","      <td>0.750000</td>\n","      <td>0.947368</td>\n","      <td>0.837209</td>\n","      <td>SVM</td>\n","      <td>{'C': 2, 'kernel': 'linear'}</td>\n","      <td>1.863737</td>\n","      <td>730.488681</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>0.902174</td>\n","      <td>0.829787</td>\n","      <td>0.916667</td>\n","      <td>0.785714</td>\n","      <td>0.846154</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>119.985000</td>\n","      <td>730.488681</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>0.927536</td>\n","      <td>0.829787</td>\n","      <td>0.666667</td>\n","      <td>1.000000</td>\n","      <td>0.800000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.0001220703125, 'hidden_layer_sizes...</td>\n","      <td>2337.573239</td>\n","      <td>13683.493345</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>0.981884</td>\n","      <td>0.893617</td>\n","      <td>0.875000</td>\n","      <td>0.913043</td>\n","      <td>0.893617</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>8910.858763</td>\n","      <td>13683.493345</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>0.956522</td>\n","      <td>0.851064</td>\n","      <td>0.750000</td>\n","      <td>0.947368</td>\n","      <td>0.837209</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'sqrt...</td>\n","      <td>2144.996647</td>\n","      <td>13683.493345</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>9</th>\n","      <td>0.934783</td>\n","      <td>0.808511</td>\n","      <td>0.750000</td>\n","      <td>0.857143</td>\n","      <td>0.800000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.0078125, 'hidden_layer_sizes': (25...</td>\n","      <td>111.703098</td>\n","      <td>433.025449</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>10</th>\n","      <td>0.951087</td>\n","      <td>0.829787</td>\n","      <td>0.708333</td>\n","      <td>0.944444</td>\n","      <td>0.809524</td>\n","      <td>SVM</td>\n","      <td>{'C': 2, 'gamma': 0.0001220703125, 'kernel': '...</td>\n","      <td>1.306651</td>\n","      <td>433.025449</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>11</th>\n","      <td>0.936594</td>\n","      <td>0.851064</td>\n","      <td>0.791667</td>\n","      <td>0.904762</td>\n","      <td>0.844444</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'sqrt...</td>\n","      <td>124.495656</td>\n","      <td>433.025449</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>12</th>\n","      <td>0.922101</td>\n","      <td>0.829787</td>\n","      <td>0.708333</td>\n","      <td>0.944444</td>\n","      <td>0.809524</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>81.842762</td>\n","      <td>243.937610</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>13</th>\n","      <td>0.971014</td>\n","      <td>0.936170</td>\n","      <td>0.875000</td>\n","      <td>1.000000</td>\n","      <td>0.933333</td>\n","      <td>SVM</td>\n","      <td>{'C': 8, 'gamma': 0.0001220703125, 'kernel': '...</td>\n","      <td>30.035022</td>\n","      <td>243.937610</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>14</th>\n","      <td>0.934783</td>\n","      <td>0.872340</td>\n","      <td>0.791667</td>\n","      <td>0.950000</td>\n","      <td>0.863636</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>131.445379</td>\n","      <td>243.937610</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>15</th>\n","      <td>0.980072</td>\n","      <td>0.872340</td>\n","      <td>0.875000</td>\n","      <td>0.875000</td>\n","      <td>0.875000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.5, 'hidden_layer_sizes': (256, 256)}</td>\n","      <td>123.187775</td>\n","      <td>315.226107</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>16</th>\n","      <td>0.974638</td>\n","      <td>0.914894</td>\n","      <td>0.833333</td>\n","      <td>1.000000</td>\n","      <td>0.909091</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.5, 'gamma': 0.0001220703125, 'kernel':...</td>\n","      <td>4.674169</td>\n","      <td>315.226107</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>17</th>\n","      <td>0.965580</td>\n","      <td>0.914894</td>\n","      <td>0.875000</td>\n","      <td>0.954545</td>\n","      <td>0.913043</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>186.715222</td>\n","      <td>315.226107</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>18</th>\n","      <td>0.989130</td>\n","      <td>0.829787</td>\n","      <td>1.000000</td>\n","      <td>0.750000</td>\n","      <td>0.857143</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.0001220703125, 'hidden_layer_sizes...</td>\n","      <td>5058.035164</td>\n","      <td>7645.472544</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>19</th>\n","      <td>0.983696</td>\n","      <td>0.914894</td>\n","      <td>0.875000</td>\n","      <td>0.954545</td>\n","      <td>0.913043</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>2277.429844</td>\n","      <td>7645.472544</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>20</th>\n","      <td>0.992754</td>\n","      <td>0.914894</td>\n","      <td>0.833333</td>\n","      <td>1.000000</td>\n","      <td>0.909091</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>310.007509</td>\n","      <td>7645.472544</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>21</th>\n","      <td>0.500000</td>\n","      <td>0.352941</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>1486.891025</td>\n","      <td>9414.390030</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>22</th>\n","      <td>0.592593</td>\n","      <td>0.607843</td>\n","      <td>0.757576</td>\n","      <td>0.675676</td>\n","      <td>0.714286</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>7005.718553</td>\n","      <td>9414.390030</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>23</th>\n","      <td>0.640572</td>\n","      <td>0.627451</td>\n","      <td>0.969697</td>\n","      <td>0.640000</td>\n","      <td>0.771084</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'sqrt...</td>\n","      <td>918.833811</td>\n","      <td>9414.390030</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>24</th>\n","      <td>0.560606</td>\n","      <td>0.450980</td>\n","      <td>0.424242</td>\n","      <td>0.608696</td>\n","      <td>0.500000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.03125, 'hidden_layer_sizes': (128,...</td>\n","      <td>2762.416396</td>\n","      <td>4146.935735</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>25</th>\n","      <td>0.681818</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.125, 'gamma': 0.0001220703125, 'kernel...</td>\n","      <td>1190.073292</td>\n","      <td>4146.935735</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>26</th>\n","      <td>0.739057</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>194.442154</td>\n","      <td>4146.935735</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>27</th>\n","      <td>0.500000</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>16.176070</td>\n","      <td>100.520304</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>28</th>\n","      <td>0.500000</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.125, 'gamma': 0.125, 'kernel': 'rbf'}</td>\n","      <td>0.634451</td>\n","      <td>100.520304</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>29</th>\n","      <td>0.500000</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>83.237592</td>\n","      <td>100.520304</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>30</th>\n","      <td>0.439394</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.00048828125, 'hidden_layer_sizes':...</td>\n","      <td>23.258092</td>\n","      <td>118.358175</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>31</th>\n","      <td>0.524411</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>0.865882</td>\n","      <td>118.358175</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>32</th>\n","      <td>0.524411</td>\n","      <td>0.627451</td>\n","      <td>0.969697</td>\n","      <td>0.640000</td>\n","      <td>0.771084</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>89.596979</td>\n","      <td>118.358175</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>33</th>\n","      <td>0.370370</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.001953125, 'hidden_layer_sizes': (...</td>\n","      <td>25.970036</td>\n","      <td>90.790034</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>34</th>\n","      <td>0.526936</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.125, 'kernel': 'linear'}</td>\n","      <td>3.912843</td>\n","      <td>90.790034</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>35</th>\n","      <td>0.501684</td>\n","      <td>0.607843</td>\n","      <td>0.878788</td>\n","      <td>0.644444</td>\n","      <td>0.743590</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'sqrt...</td>\n","      <td>60.502643</td>\n","      <td>90.790034</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>36</th>\n","      <td>0.673401</td>\n","      <td>0.627451</td>\n","      <td>0.727273</td>\n","      <td>0.705882</td>\n","      <td>0.716418</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.03125, 'hidden_layer_sizes': (64, ...</td>\n","      <td>45.460609</td>\n","      <td>412.816045</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>37</th>\n","      <td>0.654882</td>\n","      <td>0.647059</td>\n","      <td>0.969697</td>\n","      <td>0.653061</td>\n","      <td>0.780488</td>\n","      <td>SVM</td>\n","      <td>{'C': 32, 'kernel': 'linear'}</td>\n","      <td>292.242113</td>\n","      <td>412.816045</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>38</th>\n","      <td>0.675084</td>\n","      <td>0.686275</td>\n","      <td>0.969697</td>\n","      <td>0.680851</td>\n","      <td>0.800000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>74.022311</td>\n","      <td>412.816045</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>39</th>\n","      <td>0.646465</td>\n","      <td>0.686275</td>\n","      <td>0.909091</td>\n","      <td>0.697674</td>\n","      <td>0.789474</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.5, 'hidden_layer_sizes': (64, 64)}</td>\n","      <td>62.771949</td>\n","      <td>191.006667</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>40</th>\n","      <td>0.326599</td>\n","      <td>0.607843</td>\n","      <td>0.666667</td>\n","      <td>0.709677</td>\n","      <td>0.687500</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>5.681792</td>\n","      <td>191.006667</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>41</th>\n","      <td>0.685185</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>121.447971</td>\n","      <td>191.006667</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>42</th>\n","      <td>0.500000</td>\n","      <td>0.507246</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.0001220703125, 'hidden_layer_sizes...</td>\n","      <td>2492.601276</td>\n","      <td>14115.473664</td>\n","      <td>marker_T2D.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>43</th>\n","      <td>0.801681</td>\n","      <td>0.753623</td>\n","      <td>0.794118</td>\n","      <td>0.729730</td>\n","      <td>0.760563</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>10209.280280</td>\n","      <td>14115.473664</td>\n","      <td>marker_T2D.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>44</th>\n","      <td>0.755462</td>\n","      <td>0.710145</td>\n","      <td>0.794118</td>\n","      <td>0.675000</td>\n","      <td>0.729730</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'sqrt...</td>\n","      <td>1409.363473</td>\n","      <td>14115.473664</td>\n","      <td>marker_T2D.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>45</th>\n","      <td>0.807563</td>\n","      <td>0.739130</td>\n","      <td>0.764706</td>\n","      <td>0.722222</td>\n","      <td>0.742857</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.00048828125, 'hidden_layer_sizes':...</td>\n","      <td>5410.766285</td>\n","      <td>8130.763001</td>\n","      <td>marker_T2D.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>46</th>\n","      <td>0.771429</td>\n","      <td>0.753623</td>\n","      <td>0.823529</td>\n","      <td>0.717949</td>\n","      <td>0.767123</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>2465.565559</td>\n","      <td>8130.763001</td>\n","      <td>marker_T2D.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>47</th>\n","      <td>0.727731</td>\n","      <td>0.623188</td>\n","      <td>0.617647</td>\n","      <td>0.617647</td>\n","      <td>0.617647</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>254.427690</td>\n","      <td>8130.763001</td>\n","      <td>marker_T2D.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>48</th>\n","      <td>0.546639</td>\n","      <td>0.507246</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.0078125, 'hidden_layer_sizes': (64...</td>\n","      <td>18.924324</td>\n","      <td>100.779192</td>\n","      <td>marker_T2D.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>49</th>\n","      <td>0.485294</td>\n","      <td>0.507246</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'gamma': 0.001953125, 'kernel':...</td>\n","      <td>1.156454</td>\n","      <td>100.779192</td>\n","      <td>marker_T2D.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>50</th>\n","      <td>0.500000</td>\n","      <td>0.507246</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>80.019774</td>\n","      <td>100.779192</td>\n","      <td>marker_T2D.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>51</th>\n","      <td>0.452521</td>\n","      <td>0.492754</td>\n","      <td>1.000000</td>\n","      <td>0.492754</td>\n","      <td>0.660194</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>28.255768</td>\n","      <td>132.101581</td>\n","      <td>marker_T2D.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>52</th>\n","      <td>0.397479</td>\n","      <td>0.507246</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>1.663867</td>\n","      <td>132.101581</td>\n","      <td>marker_T2D.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>53</th>\n","      <td>0.697899</td>\n","      <td>0.608696</td>\n","      <td>0.264706</td>\n","      <td>0.818182</td>\n","      <td>0.400000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>96.230592</td>\n","      <td>132.101581</td>\n","      <td>marker_T2D.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>54</th>\n","      <td>0.502521</td>\n","      <td>0.507246</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>17.820294</td>\n","      <td>100.809295</td>\n","      <td>marker_T2D.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>55</th>\n","      <td>0.500840</td>\n","      <td>0.507246</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>SVM</td>\n","      <td>{'C': 8, 'gamma': 8, 'kernel': 'rbf'}</td>\n","      <td>1.023353</td>\n","      <td>100.809295</td>\n","      <td>marker_T2D.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>56</th>\n","      <td>0.500840</td>\n","      <td>0.507246</td>\n","      <td>0.029412</td>\n","      <td>0.500000</td>\n","      <td>0.055556</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>80.574382</td>\n","      <td>100.809295</td>\n","      <td>marker_T2D.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>57</th>\n","      <td>0.608403</td>\n","      <td>0.579710</td>\n","      <td>0.617647</td>\n","      <td>0.567568</td>\n","      <td>0.591549</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.00048828125, 'hidden_layer_sizes':...</td>\n","      <td>61.699997</td>\n","      <td>229.635357</td>\n","      <td>marker_T2D.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>58</th>\n","      <td>0.622689</td>\n","      <td>0.579710</td>\n","      <td>0.588235</td>\n","      <td>0.571429</td>\n","      <td>0.579710</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.5, 'gamma': 0.001953125, 'kernel': 'rbf'}</td>\n","      <td>71.348738</td>\n","      <td>229.635357</td>\n","      <td>marker_T2D.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>59</th>\n","      <td>0.625630</td>\n","      <td>0.594203</td>\n","      <td>0.558824</td>\n","      <td>0.593750</td>\n","      <td>0.575758</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'log2...</td>\n","      <td>94.713534</td>\n","      <td>229.635357</td>\n","      <td>marker_T2D.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>60</th>\n","      <td>0.567227</td>\n","      <td>0.536232</td>\n","      <td>0.470588</td>\n","      <td>0.533333</td>\n","      <td>0.500000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.0001220703125, 'hidden_layer_sizes...</td>\n","      <td>80.663704</td>\n","      <td>219.272567</td>\n","      <td>marker_T2D.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>61</th>\n","      <td>0.610084</td>\n","      <td>0.579710</td>\n","      <td>0.529412</td>\n","      <td>0.580645</td>\n","      <td>0.553846</td>\n","      <td>SVM</td>\n","      <td>{'C': 2, 'gamma': 0.0001220703125, 'kernel': '...</td>\n","      <td>3.265329</td>\n","      <td>219.272567</td>\n","      <td>marker_T2D.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>62</th>\n","      <td>0.640336</td>\n","      <td>0.623188</td>\n","      <td>0.500000</td>\n","      <td>0.653846</td>\n","      <td>0.566667</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>134.022436</td>\n","      <td>219.272567</td>\n","      <td>marker_T2D.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>63</th>\n","      <td>0.516667</td>\n","      <td>0.600000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.0078125, 'hidden_layer_sizes': (25...</td>\n","      <td>22.348105</td>\n","      <td>102.883997</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>64</th>\n","      <td>0.550000</td>\n","      <td>0.600000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>SVM</td>\n","      <td>{'C': 32, 'gamma': 0.0001220703125, 'kernel': ...</td>\n","      <td>0.305911</td>\n","      <td>102.883997</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>65</th>\n","      <td>0.583333</td>\n","      <td>0.600000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>54.457502</td>\n","      <td>102.883997</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>66</th>\n","      <td>0.500000</td>\n","      <td>0.600000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.001953125, 'hidden_layer_sizes': (...</td>\n","      <td>21.643693</td>\n","      <td>316.989440</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>67</th>\n","      <td>0.500000</td>\n","      <td>0.600000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'gamma': 0.00048828125, 'kernel...</td>\n","      <td>0.401798</td>\n","      <td>316.989440</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>68</th>\n","      <td>0.500000</td>\n","      <td>0.600000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>55.509926</td>\n","      <td>316.989440</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>69</th>\n","      <td>0.740000</td>\n","      <td>0.800000</td>\n","      <td>0.500000</td>\n","      <td>1.000000</td>\n","      <td>0.666667</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.0001220703125, 'hidden_layer_sizes...</td>\n","      <td>1359.514105</td>\n","      <td>3378.567152</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>70</th>\n","      <td>0.773333</td>\n","      <td>0.720000</td>\n","      <td>0.500000</td>\n","      <td>0.714286</td>\n","      <td>0.588235</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>1354.391862</td>\n","      <td>3378.567152</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>71</th>\n","      <td>0.753333</td>\n","      <td>0.680000</td>\n","      <td>0.400000</td>\n","      <td>0.666667</td>\n","      <td>0.500000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>521.247247</td>\n","      <td>3378.567152</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>72</th>\n","      <td>0.640000</td>\n","      <td>0.640000</td>\n","      <td>0.500000</td>\n","      <td>0.555556</td>\n","      <td>0.526316</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.5, 'hidden_layer_sizes': (256, 256)}</td>\n","      <td>40.422624</td>\n","      <td>189.599260</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>73</th>\n","      <td>0.653333</td>\n","      <td>0.640000</td>\n","      <td>0.100000</td>\n","      <td>1.000000</td>\n","      <td>0.181818</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.5, 'gamma': 0.03125, 'kernel': 'rbf'}</td>\n","      <td>0.399136</td>\n","      <td>189.599260</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>74</th>\n","      <td>0.540000</td>\n","      <td>0.640000</td>\n","      <td>0.100000</td>\n","      <td>1.000000</td>\n","      <td>0.181818</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'sqrt...</td>\n","      <td>59.198460</td>\n","      <td>189.599260</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>75</th>\n","      <td>0.846667</td>\n","      <td>0.760000</td>\n","      <td>0.800000</td>\n","      <td>0.666667</td>\n","      <td>0.727273</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>30.598888</td>\n","      <td>102.141932</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>76</th>\n","      <td>0.793333</td>\n","      <td>0.760000</td>\n","      <td>0.500000</td>\n","      <td>0.833333</td>\n","      <td>0.625000</td>\n","      <td>SVM</td>\n","      <td>{'C': 32, 'gamma': 0.00048828125, 'kernel': 'r...</td>\n","      <td>11.931073</td>\n","      <td>102.141932</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>77</th>\n","      <td>0.766667</td>\n","      <td>0.600000</td>\n","      <td>0.300000</td>\n","      <td>0.500000</td>\n","      <td>0.375000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'sqrt...</td>\n","      <td>59.363168</td>\n","      <td>102.141932</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>78</th>\n","      <td>0.793333</td>\n","      <td>0.720000</td>\n","      <td>0.400000</td>\n","      <td>0.800000</td>\n","      <td>0.533333</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.125, 'hidden_layer_sizes': (512, 5...</td>\n","      <td>49.870998</td>\n","      <td>120.945065</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>79</th>\n","      <td>0.793333</td>\n","      <td>0.600000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.125, 'gamma': 0.0001220703125, 'kernel...</td>\n","      <td>0.744758</td>\n","      <td>120.945065</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>80</th>\n","      <td>0.733333</td>\n","      <td>0.640000</td>\n","      <td>0.200000</td>\n","      <td>0.666667</td>\n","      <td>0.307692</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'log2...</td>\n","      <td>69.876427</td>\n","      <td>120.945065</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>81</th>\n","      <td>0.806667</td>\n","      <td>0.720000</td>\n","      <td>0.400000</td>\n","      <td>0.800000</td>\n","      <td>0.533333</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>1673.659658</td>\n","      <td>2080.805042</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>82</th>\n","      <td>0.780000</td>\n","      <td>0.680000</td>\n","      <td>0.400000</td>\n","      <td>0.666667</td>\n","      <td>0.500000</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>288.527356</td>\n","      <td>2080.805042</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>83</th>\n","      <td>0.820000</td>\n","      <td>0.840000</td>\n","      <td>0.600000</td>\n","      <td>1.000000</td>\n","      <td>0.750000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>118.617998</td>\n","      <td>2080.805042</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>84</th>\n","      <td>0.535294</td>\n","      <td>0.772727</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>25.468925</td>\n","      <td>103.051002</td>\n","      <td>marker_IBD.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>85</th>\n","      <td>0.523529</td>\n","      <td>0.772727</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>SVM</td>\n","      <td>{'C': 2, 'gamma': 0.5, 'kernel': 'rbf'}</td>\n","      <td>0.228910</td>\n","      <td>103.051002</td>\n","      <td>marker_IBD.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>86</th>\n","      <td>0.488235</td>\n","      <td>0.772727</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>55.099191</td>\n","      <td>103.051002</td>\n","      <td>marker_IBD.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>87</th>\n","      <td>0.558824</td>\n","      <td>0.772727</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.0078125, 'hidden_layer_sizes': (25...</td>\n","      <td>24.641263</td>\n","      <td>285.467308</td>\n","      <td>marker_IBD.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>88</th>\n","      <td>0.588235</td>\n","      <td>0.772727</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>SVM</td>\n","      <td>{'C': 32, 'gamma': 2, 'kernel': 'rbf'}</td>\n","      <td>0.297684</td>\n","      <td>285.467308</td>\n","      <td>marker_IBD.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>89</th>\n","      <td>0.511765</td>\n","      <td>0.727273</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'sqrt...</td>\n","      <td>58.754943</td>\n","      <td>285.467308</td>\n","      <td>marker_IBD.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>90</th>\n","      <td>0.547059</td>\n","      <td>0.727273</td>\n","      <td>0.200000</td>\n","      <td>0.333333</td>\n","      <td>0.250000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>1456.672904</td>\n","      <td>3058.399564</td>\n","      <td>marker_IBD.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>91</th>\n","      <td>0.800000</td>\n","      <td>0.727273</td>\n","      <td>0.200000</td>\n","      <td>0.333333</td>\n","      <td>0.250000</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>1137.848792</td>\n","      <td>3058.399564</td>\n","      <td>marker_IBD.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>92</th>\n","      <td>0.929412</td>\n","      <td>0.727273</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'log2...</td>\n","      <td>331.307379</td>\n","      <td>3058.399564</td>\n","      <td>marker_IBD.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>93</th>\n","      <td>0.858824</td>\n","      <td>0.818182</td>\n","      <td>0.400000</td>\n","      <td>0.666667</td>\n","      <td>0.500000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.00048828125, 'hidden_layer_sizes':...</td>\n","      <td>40.435831</td>\n","      <td>183.918113</td>\n","      <td>marker_IBD.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>94</th>\n","      <td>0.764706</td>\n","      <td>0.772727</td>\n","      <td>0.200000</td>\n","      <td>0.500000</td>\n","      <td>0.285714</td>\n","      <td>SVM</td>\n","      <td>{'C': 32, 'gamma': 0.03125, 'kernel': 'rbf'}</td>\n","      <td>0.346800</td>\n","      <td>183.918113</td>\n","      <td>marker_IBD.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>95</th>\n","      <td>0.847059</td>\n","      <td>0.772727</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>58.136487</td>\n","      <td>183.918113</td>\n","      <td>marker_IBD.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>96</th>\n","      <td>0.800000</td>\n","      <td>0.772727</td>\n","      <td>0.600000</td>\n","      <td>0.500000</td>\n","      <td>0.545455</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.5, 'hidden_layer_sizes': (128, 128)}</td>\n","      <td>27.264308</td>\n","      <td>87.200318</td>\n","      <td>marker_IBD.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>97</th>\n","      <td>0.764706</td>\n","      <td>0.772727</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'gamma': 0.001953125, 'kernel':...</td>\n","      <td>1.467895</td>\n","      <td>87.200318</td>\n","      <td>marker_IBD.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>98</th>\n","      <td>0.811765</td>\n","      <td>0.772727</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'sqrt...</td>\n","      <td>58.264915</td>\n","      <td>87.200318</td>\n","      <td>marker_IBD.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>99</th>\n","      <td>0.705882</td>\n","      <td>0.636364</td>\n","      <td>0.200000</td>\n","      <td>0.200000</td>\n","      <td>0.200000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.00048828125, 'hidden_layer_sizes':...</td>\n","      <td>45.109154</td>\n","      <td>112.673155</td>\n","      <td>marker_IBD.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>100</th>\n","      <td>0.788235</td>\n","      <td>0.727273</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>SVM</td>\n","      <td>{'C': 2, 'gamma': 0.0001220703125, 'kernel': '...</td>\n","      <td>0.618138</td>\n","      <td>112.673155</td>\n","      <td>marker_IBD.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>101</th>\n","      <td>0.823529</td>\n","      <td>0.772727</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'log2...</td>\n","      <td>66.569057</td>\n","      <td>112.673155</td>\n","      <td>marker_IBD.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>102</th>\n","      <td>0.776471</td>\n","      <td>0.727273</td>\n","      <td>0.600000</td>\n","      <td>0.428571</td>\n","      <td>0.500000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.5, 'hidden_layer_sizes': (64, 64)}</td>\n","      <td>2195.381789</td>\n","      <td>2497.345249</td>\n","      <td>marker_IBD.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>103</th>\n","      <td>0.811765</td>\n","      <td>0.727273</td>\n","      <td>0.200000</td>\n","      <td>0.333333</td>\n","      <td>0.250000</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>207.017738</td>\n","      <td>2497.345249</td>\n","      <td>marker_IBD.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>104</th>\n","      <td>0.823529</td>\n","      <td>0.772727</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>94.945693</td>\n","      <td>2497.345249</td>\n","      <td>marker_IBD.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>105</th>\n","      <td>0.500000</td>\n","      <td>0.550000</td>\n","      <td>1.000000</td>\n","      <td>0.550000</td>\n","      <td>0.709677</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.0001220703125, 'hidden_layer_sizes...</td>\n","      <td>1405.755020</td>\n","      <td>2495.791879</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>106</th>\n","      <td>0.595960</td>\n","      <td>0.600000</td>\n","      <td>0.545455</td>\n","      <td>0.666667</td>\n","      <td>0.600000</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>834.236857</td>\n","      <td>2495.791879</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>107</th>\n","      <td>0.626263</td>\n","      <td>0.500000</td>\n","      <td>0.636364</td>\n","      <td>0.538462</td>\n","      <td>0.583333</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'log2...</td>\n","      <td>254.800342</td>\n","      <td>2495.791879</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>108</th>\n","      <td>0.666667</td>\n","      <td>0.600000</td>\n","      <td>0.363636</td>\n","      <td>0.800000</td>\n","      <td>0.500000</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.001953125, 'hidden_layer_sizes': (...</td>\n","      <td>2478.668654</td>\n","      <td>2730.213370</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>109</th>\n","      <td>0.636364</td>\n","      <td>0.500000</td>\n","      <td>0.545455</td>\n","      <td>0.545455</td>\n","      <td>0.545455</td>\n","      <td>SVM</td>\n","      <td>{'C': 2, 'gamma': 0.0001220703125, 'kernel': '...</td>\n","      <td>137.955329</td>\n","      <td>2730.213370</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>110</th>\n","      <td>0.525253</td>\n","      <td>0.450000</td>\n","      <td>0.636364</td>\n","      <td>0.500000</td>\n","      <td>0.560000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'log2...</td>\n","      <td>113.584896</td>\n","      <td>2730.213370</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>111</th>\n","      <td>0.500000</td>\n","      <td>0.550000</td>\n","      <td>1.000000</td>\n","      <td>0.550000</td>\n","      <td>0.709677</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.001953125, 'hidden_layer_sizes': (...</td>\n","      <td>13.699150</td>\n","      <td>92.950624</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>112</th>\n","      <td>0.500000</td>\n","      <td>0.550000</td>\n","      <td>1.000000</td>\n","      <td>0.550000</td>\n","      <td>0.709677</td>\n","      <td>SVM</td>\n","      <td>{'C': 8, 'gamma': 0.5, 'kernel': 'rbf'}</td>\n","      <td>0.413084</td>\n","      <td>92.950624</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>113</th>\n","      <td>0.500000</td>\n","      <td>0.550000</td>\n","      <td>1.000000</td>\n","      <td>0.550000</td>\n","      <td>0.709677</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>78.685357</td>\n","      <td>92.950624</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>shallow_ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>114</th>\n","      <td>0.515152</td>\n","      <td>0.550000</td>\n","      <td>1.000000</td>\n","      <td>0.550000</td>\n","      <td>0.709677</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.125, 'hidden_layer_sizes': (64, 64)}</td>\n","      <td>22.683855</td>\n","      <td>107.277794</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>115</th>\n","      <td>0.717172</td>\n","      <td>0.550000</td>\n","      <td>1.000000</td>\n","      <td>0.550000</td>\n","      <td>0.709677</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'gamma': 0.001953125, 'kernel':...</td>\n","      <td>0.406561</td>\n","      <td>107.277794</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>116</th>\n","      <td>0.393939</td>\n","      <td>0.500000</td>\n","      <td>0.818182</td>\n","      <td>0.529412</td>\n","      <td>0.642857</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'log2...</td>\n","      <td>80.390760</td>\n","      <td>107.277794</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>ae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>117</th>\n","      <td>0.237374</td>\n","      <td>0.550000</td>\n","      <td>1.000000</td>\n","      <td>0.550000</td>\n","      <td>0.709677</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.03125, 'hidden_layer_sizes': (64, ...</td>\n","      <td>16.322586</td>\n","      <td>95.779552</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>118</th>\n","      <td>0.500000</td>\n","      <td>0.550000</td>\n","      <td>1.000000</td>\n","      <td>0.550000</td>\n","      <td>0.709677</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'gamma': 0.00048828125, 'kernel...</td>\n","      <td>0.387627</td>\n","      <td>95.779552</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>119</th>\n","      <td>0.500000</td>\n","      <td>0.550000</td>\n","      <td>1.000000</td>\n","      <td>0.550000</td>\n","      <td>0.709677</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>78.629580</td>\n","      <td>95.779552</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>vae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>120</th>\n","      <td>0.626263</td>\n","      <td>0.600000</td>\n","      <td>0.636364</td>\n","      <td>0.636364</td>\n","      <td>0.636364</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>25.624079</td>\n","      <td>120.230908</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>121</th>\n","      <td>0.545455</td>\n","      <td>0.550000</td>\n","      <td>0.454545</td>\n","      <td>0.625000</td>\n","      <td>0.526316</td>\n","      <td>SVM</td>\n","      <td>{'C': 8, 'gamma': 0.0001220703125, 'kernel': '...</td>\n","      <td>12.671258</td>\n","      <td>120.230908</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>122</th>\n","      <td>0.479798</td>\n","      <td>0.400000</td>\n","      <td>0.363636</td>\n","      <td>0.444444</td>\n","      <td>0.400000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'sqrt...</td>\n","      <td>81.428668</td>\n","      <td>120.230908</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>123</th>\n","      <td>0.474747</td>\n","      <td>0.450000</td>\n","      <td>0.454545</td>\n","      <td>0.500000</td>\n","      <td>0.476190</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.00048828125, 'hidden_layer_sizes':...</td>\n","      <td>38.448539</td>\n","      <td>126.877143</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>124</th>\n","      <td>0.303030</td>\n","      <td>0.550000</td>\n","      <td>1.000000</td>\n","      <td>0.550000</td>\n","      <td>0.709677</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'gamma': 0.00048828125, 'kernel...</td>\n","      <td>0.625493</td>\n","      <td>126.877143</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>125</th>\n","      <td>0.737374</td>\n","      <td>0.650000</td>\n","      <td>0.909091</td>\n","      <td>0.625000</td>\n","      <td>0.740741</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>86.934503</td>\n","      <td>126.877143</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","    <div class=\"colab-df-buttons\">\n","\n","  <div class=\"colab-df-container\">\n","    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d3fa703a-ef5a-4d98-93cd-ab61bc2a3daf')\"\n","            title=\"Convert this dataframe to an interactive table.\"\n","            style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n","    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n","  </svg>\n","    </button>\n","\n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    .colab-df-buttons div {\n","      margin-bottom: 4px;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","    <script>\n","      const buttonEl =\n","        document.querySelector('#df-d3fa703a-ef5a-4d98-93cd-ab61bc2a3daf button.colab-df-convert');\n","      buttonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","      async function convertToInteractive(key) {\n","        const element = document.querySelector('#df-d3fa703a-ef5a-4d98-93cd-ab61bc2a3daf');\n","        const dataTable =\n","          await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                    [key], {});\n","        if (!dataTable) return;\n","\n","        const docLinkHtml = 'Like what you see? Visit the ' +\n","          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","          + ' to learn more about interactive tables.';\n","        element.innerHTML = '';\n","        dataTable['output_type'] = 'display_data';\n","        await google.colab.output.renderOutput(dataTable, element);\n","        const docLink = document.createElement('div');\n","        docLink.innerHTML = docLinkHtml;\n","        element.appendChild(docLink);\n","      }\n","    </script>\n","  </div>\n","\n","\n","<div id=\"df-4e028611-350e-4b61-b766-9b70aa3c3ab2\">\n","  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-4e028611-350e-4b61-b766-9b70aa3c3ab2')\"\n","            title=\"Suggest charts\"\n","            style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","  </button>\n","\n","<style>\n","  .colab-df-quickchart {\n","      --bg-color: #E8F0FE;\n","      --fill-color: #1967D2;\n","      --hover-bg-color: #E2EBFA;\n","      --hover-fill-color: #174EA6;\n","      --disabled-fill-color: #AAA;\n","      --disabled-bg-color: #DDD;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","      --bg-color: #3B4455;\n","      --fill-color: #D2E3FC;\n","      --hover-bg-color: #434B5C;\n","      --hover-fill-color: #FFFFFF;\n","      --disabled-bg-color: #3B4455;\n","      --disabled-fill-color: #666;\n","  }\n","\n","  .colab-df-quickchart {\n","    background-color: var(--bg-color);\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: var(--fill-color);\n","    height: 32px;\n","    padding: 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: var(--hover-bg-color);\n","    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: var(--button-hover-fill-color);\n","  }\n","\n","  .colab-df-quickchart-complete:disabled,\n","  .colab-df-quickchart-complete:disabled:hover {\n","    background-color: var(--disabled-bg-color);\n","    fill: var(--disabled-fill-color);\n","    box-shadow: none;\n","  }\n","\n","  .colab-df-spinner {\n","    border: 2px solid var(--fill-color);\n","    border-color: transparent;\n","    border-bottom-color: var(--fill-color);\n","    animation:\n","      spin 1s steps(1) infinite;\n","  }\n","\n","  @keyframes spin {\n","    0% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","      border-left-color: var(--fill-color);\n","    }\n","    20% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    30% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","      border-right-color: var(--fill-color);\n","    }\n","    40% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    60% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","    }\n","    80% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-bottom-color: var(--fill-color);\n","    }\n","    90% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","    }\n","  }\n","</style>\n","\n","  <script>\n","    async function quickchart(key) {\n","      const quickchartButtonEl =\n","        document.querySelector('#' + key + ' button');\n","      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n","      quickchartButtonEl.classList.add('colab-df-spinner');\n","      try {\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      } catch (error) {\n","        console.error('Error during call to suggestCharts:', error);\n","      }\n","      quickchartButtonEl.classList.remove('colab-df-spinner');\n","      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n","    }\n","    (() => {\n","      let quickchartButtonEl =\n","        document.querySelector('#df-4e028611-350e-4b61-b766-9b70aa3c3ab2 button');\n","      quickchartButtonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","    })();\n","  </script>\n","</div>\n","\n","  <div id=\"id_f6a3def7-2c01-42ba-a51a-6d8a1f5b56cd\">\n","    <style>\n","      .colab-df-generate {\n","        background-color: #E8F0FE;\n","        border: none;\n","        border-radius: 50%;\n","        cursor: pointer;\n","        display: none;\n","        fill: #1967D2;\n","        height: 32px;\n","        padding: 0 0 0 0;\n","        width: 32px;\n","      }\n","\n","      .colab-df-generate:hover {\n","        background-color: #E2EBFA;\n","        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","        fill: #174EA6;\n","      }\n","\n","      [theme=dark] .colab-df-generate {\n","        background-color: #3B4455;\n","        fill: #D2E3FC;\n","      }\n","\n","      [theme=dark] .colab-df-generate:hover {\n","        background-color: #434B5C;\n","        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","        fill: #FFFFFF;\n","      }\n","    </style>\n","    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n","            title=\"Generate code using this dataframe.\"\n","            style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n","  </svg>\n","    </button>\n","    <script>\n","      (() => {\n","      const buttonEl =\n","        document.querySelector('#id_f6a3def7-2c01-42ba-a51a-6d8a1f5b56cd button.colab-df-generate');\n","      buttonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","      buttonEl.onclick = () => {\n","        google.colab.notebook.generateWithVariable('df');\n","      }\n","      })();\n","    </script>\n","  </div>\n","\n","    </div>\n","  </div>\n"],"text/plain":["          AUC       ACC    Recall  Precision  F1_score classification_method  \\\n","0    0.947464  0.829787  0.791667   0.863636  0.826087                   MLP   \n","1    0.916667  0.872340  0.875000   0.875000  0.875000                   SVM   \n","2    0.943841  0.829787  0.791667   0.863636  0.826087         Random Forest   \n","3    0.926630  0.829787  0.750000   0.900000  0.818182                   MLP   \n","4    0.926630  0.851064  0.750000   0.947368  0.837209                   SVM   \n","5    0.902174  0.829787  0.916667   0.785714  0.846154         Random Forest   \n","6    0.927536  0.829787  0.666667   1.000000  0.800000                   MLP   \n","7    0.981884  0.893617  0.875000   0.913043  0.893617                   SVM   \n","8    0.956522  0.851064  0.750000   0.947368  0.837209         Random Forest   \n","9    0.934783  0.808511  0.750000   0.857143  0.800000                   MLP   \n","10   0.951087  0.829787  0.708333   0.944444  0.809524                   SVM   \n","11   0.936594  0.851064  0.791667   0.904762  0.844444         Random Forest   \n","12   0.922101  0.829787  0.708333   0.944444  0.809524                   MLP   \n","13   0.971014  0.936170  0.875000   1.000000  0.933333                   SVM   \n","14   0.934783  0.872340  0.791667   0.950000  0.863636         Random Forest   \n","15   0.980072  0.872340  0.875000   0.875000  0.875000                   MLP   \n","16   0.974638  0.914894  0.833333   1.000000  0.909091                   SVM   \n","17   0.965580  0.914894  0.875000   0.954545  0.913043         Random Forest   \n","18   0.989130  0.829787  1.000000   0.750000  0.857143                   MLP   \n","19   0.983696  0.914894  0.875000   0.954545  0.913043                   SVM   \n","20   0.992754  0.914894  0.833333   1.000000  0.909091         Random Forest   \n","21   0.500000  0.352941  0.000000   0.000000  0.000000                   MLP   \n","22   0.592593  0.607843  0.757576   0.675676  0.714286                   SVM   \n","23   0.640572  0.627451  0.969697   0.640000  0.771084         Random Forest   \n","24   0.560606  0.450980  0.424242   0.608696  0.500000                   MLP   \n","25   0.681818  0.647059  1.000000   0.647059  0.785714                   SVM   \n","26   0.739057  0.647059  1.000000   0.647059  0.785714         Random Forest   \n","27   0.500000  0.647059  1.000000   0.647059  0.785714                   MLP   \n","28   0.500000  0.647059  1.000000   0.647059  0.785714                   SVM   \n","29   0.500000  0.647059  1.000000   0.647059  0.785714         Random Forest   \n","30   0.439394  0.647059  1.000000   0.647059  0.785714                   MLP   \n","31   0.524411  0.647059  1.000000   0.647059  0.785714                   SVM   \n","32   0.524411  0.627451  0.969697   0.640000  0.771084         Random Forest   \n","33   0.370370  0.647059  1.000000   0.647059  0.785714                   MLP   \n","34   0.526936  0.647059  1.000000   0.647059  0.785714                   SVM   \n","35   0.501684  0.607843  0.878788   0.644444  0.743590         Random Forest   \n","36   0.673401  0.627451  0.727273   0.705882  0.716418                   MLP   \n","37   0.654882  0.647059  0.969697   0.653061  0.780488                   SVM   \n","38   0.675084  0.686275  0.969697   0.680851  0.800000         Random Forest   \n","39   0.646465  0.686275  0.909091   0.697674  0.789474                   MLP   \n","40   0.326599  0.607843  0.666667   0.709677  0.687500                   SVM   \n","41   0.685185  0.647059  1.000000   0.647059  0.785714         Random Forest   \n","42   0.500000  0.507246  0.000000   0.000000  0.000000                   MLP   \n","43   0.801681  0.753623  0.794118   0.729730  0.760563                   SVM   \n","44   0.755462  0.710145  0.794118   0.675000  0.729730         Random Forest   \n","45   0.807563  0.739130  0.764706   0.722222  0.742857                   MLP   \n","46   0.771429  0.753623  0.823529   0.717949  0.767123                   SVM   \n","47   0.727731  0.623188  0.617647   0.617647  0.617647         Random Forest   \n","48   0.546639  0.507246  0.000000   0.000000  0.000000                   MLP   \n","49   0.485294  0.507246  0.000000   0.000000  0.000000                   SVM   \n","50   0.500000  0.507246  0.000000   0.000000  0.000000         Random Forest   \n","51   0.452521  0.492754  1.000000   0.492754  0.660194                   MLP   \n","52   0.397479  0.507246  0.000000   0.000000  0.000000                   SVM   \n","53   0.697899  0.608696  0.264706   0.818182  0.400000         Random Forest   \n","54   0.502521  0.507246  0.000000   0.000000  0.000000                   MLP   \n","55   0.500840  0.507246  0.000000   0.000000  0.000000                   SVM   \n","56   0.500840  0.507246  0.029412   0.500000  0.055556         Random Forest   \n","57   0.608403  0.579710  0.617647   0.567568  0.591549                   MLP   \n","58   0.622689  0.579710  0.588235   0.571429  0.579710                   SVM   \n","59   0.625630  0.594203  0.558824   0.593750  0.575758         Random Forest   \n","60   0.567227  0.536232  0.470588   0.533333  0.500000                   MLP   \n","61   0.610084  0.579710  0.529412   0.580645  0.553846                   SVM   \n","62   0.640336  0.623188  0.500000   0.653846  0.566667         Random Forest   \n","63   0.516667  0.600000  0.000000   0.000000  0.000000                   MLP   \n","64   0.550000  0.600000  0.000000   0.000000  0.000000                   SVM   \n","65   0.583333  0.600000  0.000000   0.000000  0.000000         Random Forest   \n","66   0.500000  0.600000  0.000000   0.000000  0.000000                   MLP   \n","67   0.500000  0.600000  0.000000   0.000000  0.000000                   SVM   \n","68   0.500000  0.600000  0.000000   0.000000  0.000000         Random Forest   \n","69   0.740000  0.800000  0.500000   1.000000  0.666667                   MLP   \n","70   0.773333  0.720000  0.500000   0.714286  0.588235                   SVM   \n","71   0.753333  0.680000  0.400000   0.666667  0.500000         Random Forest   \n","72   0.640000  0.640000  0.500000   0.555556  0.526316                   MLP   \n","73   0.653333  0.640000  0.100000   1.000000  0.181818                   SVM   \n","74   0.540000  0.640000  0.100000   1.000000  0.181818         Random Forest   \n","75   0.846667  0.760000  0.800000   0.666667  0.727273                   MLP   \n","76   0.793333  0.760000  0.500000   0.833333  0.625000                   SVM   \n","77   0.766667  0.600000  0.300000   0.500000  0.375000         Random Forest   \n","78   0.793333  0.720000  0.400000   0.800000  0.533333                   MLP   \n","79   0.793333  0.600000  0.000000   0.000000  0.000000                   SVM   \n","80   0.733333  0.640000  0.200000   0.666667  0.307692         Random Forest   \n","81   0.806667  0.720000  0.400000   0.800000  0.533333                   MLP   \n","82   0.780000  0.680000  0.400000   0.666667  0.500000                   SVM   \n","83   0.820000  0.840000  0.600000   1.000000  0.750000         Random Forest   \n","84   0.535294  0.772727  0.000000   0.000000  0.000000                   MLP   \n","85   0.523529  0.772727  0.000000   0.000000  0.000000                   SVM   \n","86   0.488235  0.772727  0.000000   0.000000  0.000000         Random Forest   \n","87   0.558824  0.772727  0.000000   0.000000  0.000000                   MLP   \n","88   0.588235  0.772727  0.000000   0.000000  0.000000                   SVM   \n","89   0.511765  0.727273  0.000000   0.000000  0.000000         Random Forest   \n","90   0.547059  0.727273  0.200000   0.333333  0.250000                   MLP   \n","91   0.800000  0.727273  0.200000   0.333333  0.250000                   SVM   \n","92   0.929412  0.727273  0.000000   0.000000  0.000000         Random Forest   \n","93   0.858824  0.818182  0.400000   0.666667  0.500000                   MLP   \n","94   0.764706  0.772727  0.200000   0.500000  0.285714                   SVM   \n","95   0.847059  0.772727  0.000000   0.000000  0.000000         Random Forest   \n","96   0.800000  0.772727  0.600000   0.500000  0.545455                   MLP   \n","97   0.764706  0.772727  0.000000   0.000000  0.000000                   SVM   \n","98   0.811765  0.772727  0.000000   0.000000  0.000000         Random Forest   \n","99   0.705882  0.636364  0.200000   0.200000  0.200000                   MLP   \n","100  0.788235  0.727273  0.000000   0.000000  0.000000                   SVM   \n","101  0.823529  0.772727  0.000000   0.000000  0.000000         Random Forest   \n","102  0.776471  0.727273  0.600000   0.428571  0.500000                   MLP   \n","103  0.811765  0.727273  0.200000   0.333333  0.250000                   SVM   \n","104  0.823529  0.772727  0.000000   0.000000  0.000000         Random Forest   \n","105  0.500000  0.550000  1.000000   0.550000  0.709677                   MLP   \n","106  0.595960  0.600000  0.545455   0.666667  0.600000                   SVM   \n","107  0.626263  0.500000  0.636364   0.538462  0.583333         Random Forest   \n","108  0.666667  0.600000  0.363636   0.800000  0.500000                   MLP   \n","109  0.636364  0.500000  0.545455   0.545455  0.545455                   SVM   \n","110  0.525253  0.450000  0.636364   0.500000  0.560000         Random Forest   \n","111  0.500000  0.550000  1.000000   0.550000  0.709677                   MLP   \n","112  0.500000  0.550000  1.000000   0.550000  0.709677                   SVM   \n","113  0.500000  0.550000  1.000000   0.550000  0.709677         Random Forest   \n","114  0.515152  0.550000  1.000000   0.550000  0.709677                   MLP   \n","115  0.717172  0.550000  1.000000   0.550000  0.709677                   SVM   \n","116  0.393939  0.500000  0.818182   0.529412  0.642857         Random Forest   \n","117  0.237374  0.550000  1.000000   0.550000  0.709677                   MLP   \n","118  0.500000  0.550000  1.000000   0.550000  0.709677                   SVM   \n","119  0.500000  0.550000  1.000000   0.550000  0.709677         Random Forest   \n","120  0.626263  0.600000  0.636364   0.636364  0.636364                   MLP   \n","121  0.545455  0.550000  0.454545   0.625000  0.526316                   SVM   \n","122  0.479798  0.400000  0.363636   0.444444  0.400000         Random Forest   \n","123  0.474747  0.450000  0.454545   0.500000  0.476190                   MLP   \n","124  0.303030  0.550000  1.000000   0.550000  0.709677                   SVM   \n","125  0.737374  0.650000  0.909091   0.625000  0.740741         Random Forest   \n","\n","                                      hyper_parameters  classification_time  \\\n","0    {'alpha': 0.0001220703125, 'hidden_layer_sizes...            66.259109   \n","1       {'C': 32, 'gamma': 0.0078125, 'kernel': 'rbf'}             1.120851   \n","2    {'criterion': 'entropy', 'max_features': 'sqrt...           111.178497   \n","3    {'alpha': 3.0517578125e-05, 'hidden_layer_size...            64.426869   \n","4                         {'C': 2, 'kernel': 'linear'}             1.863737   \n","5    {'criterion': 'gini', 'max_features': 'log2', ...           119.985000   \n","6    {'alpha': 0.0001220703125, 'hidden_layer_sizes...          2337.573239   \n","7                   {'C': 0.03125, 'kernel': 'linear'}          8910.858763   \n","8    {'criterion': 'entropy', 'max_features': 'sqrt...          2144.996647   \n","9    {'alpha': 0.0078125, 'hidden_layer_sizes': (25...           111.703098   \n","10   {'C': 2, 'gamma': 0.0001220703125, 'kernel': '...             1.306651   \n","11   {'criterion': 'entropy', 'max_features': 'sqrt...           124.495656   \n","12   {'alpha': 3.0517578125e-05, 'hidden_layer_size...            81.842762   \n","13   {'C': 8, 'gamma': 0.0001220703125, 'kernel': '...            30.035022   \n","14   {'criterion': 'gini', 'max_features': 'log2', ...           131.445379   \n","15    {'alpha': 0.5, 'hidden_layer_sizes': (256, 256)}           123.187775   \n","16   {'C': 0.5, 'gamma': 0.0001220703125, 'kernel':...             4.674169   \n","17   {'criterion': 'gini', 'max_features': 'log2', ...           186.715222   \n","18   {'alpha': 0.0001220703125, 'hidden_layer_sizes...          5058.035164   \n","19                  {'C': 0.03125, 'kernel': 'linear'}          2277.429844   \n","20   {'criterion': 'gini', 'max_features': 'log2', ...           310.007509   \n","21   {'alpha': 3.0517578125e-05, 'hidden_layer_size...          1486.891025   \n","22                  {'C': 0.03125, 'kernel': 'linear'}          7005.718553   \n","23   {'criterion': 'entropy', 'max_features': 'sqrt...           918.833811   \n","24   {'alpha': 0.03125, 'hidden_layer_sizes': (128,...          2762.416396   \n","25   {'C': 0.125, 'gamma': 0.0001220703125, 'kernel...          1190.073292   \n","26   {'criterion': 'gini', 'max_features': 'log2', ...           194.442154   \n","27   {'alpha': 3.0517578125e-05, 'hidden_layer_size...            16.176070   \n","28       {'C': 0.125, 'gamma': 0.125, 'kernel': 'rbf'}             0.634451   \n","29   {'criterion': 'gini', 'max_features': 'sqrt', ...            83.237592   \n","30   {'alpha': 0.00048828125, 'hidden_layer_sizes':...            23.258092   \n","31                  {'C': 0.03125, 'kernel': 'linear'}             0.865882   \n","32   {'criterion': 'gini', 'max_features': 'sqrt', ...            89.596979   \n","33   {'alpha': 0.001953125, 'hidden_layer_sizes': (...            25.970036   \n","34                    {'C': 0.125, 'kernel': 'linear'}             3.912843   \n","35   {'criterion': 'entropy', 'max_features': 'sqrt...            60.502643   \n","36   {'alpha': 0.03125, 'hidden_layer_sizes': (64, ...            45.460609   \n","37                       {'C': 32, 'kernel': 'linear'}           292.242113   \n","38   {'criterion': 'gini', 'max_features': 'log2', ...            74.022311   \n","39      {'alpha': 0.5, 'hidden_layer_sizes': (64, 64)}            62.771949   \n","40                  {'C': 0.03125, 'kernel': 'linear'}             5.681792   \n","41   {'criterion': 'gini', 'max_features': 'sqrt', ...           121.447971   \n","42   {'alpha': 0.0001220703125, 'hidden_layer_sizes...          2492.601276   \n","43                  {'C': 0.03125, 'kernel': 'linear'}         10209.280280   \n","44   {'criterion': 'entropy', 'max_features': 'sqrt...          1409.363473   \n","45   {'alpha': 0.00048828125, 'hidden_layer_sizes':...          5410.766285   \n","46                  {'C': 0.03125, 'kernel': 'linear'}          2465.565559   \n","47   {'criterion': 'gini', 'max_features': 'log2', ...           254.427690   \n","48   {'alpha': 0.0078125, 'hidden_layer_sizes': (64...            18.924324   \n","49   {'C': 0.03125, 'gamma': 0.001953125, 'kernel':...             1.156454   \n","50   {'criterion': 'gini', 'max_features': 'sqrt', ...            80.019774   \n","51   {'alpha': 3.0517578125e-05, 'hidden_layer_size...            28.255768   \n","52                  {'C': 0.03125, 'kernel': 'linear'}             1.663867   \n","53   {'criterion': 'gini', 'max_features': 'sqrt', ...            96.230592   \n","54   {'alpha': 3.0517578125e-05, 'hidden_layer_size...            17.820294   \n","55               {'C': 8, 'gamma': 8, 'kernel': 'rbf'}             1.023353   \n","56   {'criterion': 'gini', 'max_features': 'sqrt', ...            80.574382   \n","57   {'alpha': 0.00048828125, 'hidden_layer_sizes':...            61.699997   \n","58   {'C': 0.5, 'gamma': 0.001953125, 'kernel': 'rbf'}            71.348738   \n","59   {'criterion': 'entropy', 'max_features': 'log2...            94.713534   \n","60   {'alpha': 0.0001220703125, 'hidden_layer_sizes...            80.663704   \n","61   {'C': 2, 'gamma': 0.0001220703125, 'kernel': '...             3.265329   \n","62   {'criterion': 'gini', 'max_features': 'log2', ...           134.022436   \n","63   {'alpha': 0.0078125, 'hidden_layer_sizes': (25...            22.348105   \n","64   {'C': 32, 'gamma': 0.0001220703125, 'kernel': ...             0.305911   \n","65   {'criterion': 'gini', 'max_features': 'sqrt', ...            54.457502   \n","66   {'alpha': 0.001953125, 'hidden_layer_sizes': (...            21.643693   \n","67   {'C': 0.03125, 'gamma': 0.00048828125, 'kernel...             0.401798   \n","68   {'criterion': 'gini', 'max_features': 'sqrt', ...            55.509926   \n","69   {'alpha': 0.0001220703125, 'hidden_layer_sizes...          1359.514105   \n","70                  {'C': 0.03125, 'kernel': 'linear'}          1354.391862   \n","71   {'criterion': 'gini', 'max_features': 'sqrt', ...           521.247247   \n","72    {'alpha': 0.5, 'hidden_layer_sizes': (256, 256)}            40.422624   \n","73       {'C': 0.5, 'gamma': 0.03125, 'kernel': 'rbf'}             0.399136   \n","74   {'criterion': 'entropy', 'max_features': 'sqrt...            59.198460   \n","75   {'alpha': 3.0517578125e-05, 'hidden_layer_size...            30.598888   \n","76   {'C': 32, 'gamma': 0.00048828125, 'kernel': 'r...            11.931073   \n","77   {'criterion': 'entropy', 'max_features': 'sqrt...            59.363168   \n","78   {'alpha': 0.125, 'hidden_layer_sizes': (512, 5...            49.870998   \n","79   {'C': 0.125, 'gamma': 0.0001220703125, 'kernel...             0.744758   \n","80   {'criterion': 'entropy', 'max_features': 'log2...            69.876427   \n","81   {'alpha': 3.0517578125e-05, 'hidden_layer_size...          1673.659658   \n","82                  {'C': 0.03125, 'kernel': 'linear'}           288.527356   \n","83   {'criterion': 'gini', 'max_features': 'sqrt', ...           118.617998   \n","84   {'alpha': 3.0517578125e-05, 'hidden_layer_size...            25.468925   \n","85             {'C': 2, 'gamma': 0.5, 'kernel': 'rbf'}             0.228910   \n","86   {'criterion': 'gini', 'max_features': 'log2', ...            55.099191   \n","87   {'alpha': 0.0078125, 'hidden_layer_sizes': (25...            24.641263   \n","88              {'C': 32, 'gamma': 2, 'kernel': 'rbf'}             0.297684   \n","89   {'criterion': 'entropy', 'max_features': 'sqrt...            58.754943   \n","90   {'alpha': 3.0517578125e-05, 'hidden_layer_size...          1456.672904   \n","91                  {'C': 0.03125, 'kernel': 'linear'}          1137.848792   \n","92   {'criterion': 'entropy', 'max_features': 'log2...           331.307379   \n","93   {'alpha': 0.00048828125, 'hidden_layer_sizes':...            40.435831   \n","94        {'C': 32, 'gamma': 0.03125, 'kernel': 'rbf'}             0.346800   \n","95   {'criterion': 'gini', 'max_features': 'sqrt', ...            58.136487   \n","96    {'alpha': 0.5, 'hidden_layer_sizes': (128, 128)}            27.264308   \n","97   {'C': 0.03125, 'gamma': 0.001953125, 'kernel':...             1.467895   \n","98   {'criterion': 'entropy', 'max_features': 'sqrt...            58.264915   \n","99   {'alpha': 0.00048828125, 'hidden_layer_sizes':...            45.109154   \n","100  {'C': 2, 'gamma': 0.0001220703125, 'kernel': '...             0.618138   \n","101  {'criterion': 'entropy', 'max_features': 'log2...            66.569057   \n","102     {'alpha': 0.5, 'hidden_layer_sizes': (64, 64)}          2195.381789   \n","103                 {'C': 0.03125, 'kernel': 'linear'}           207.017738   \n","104  {'criterion': 'gini', 'max_features': 'log2', ...            94.945693   \n","105  {'alpha': 0.0001220703125, 'hidden_layer_sizes...          1405.755020   \n","106                 {'C': 0.03125, 'kernel': 'linear'}           834.236857   \n","107  {'criterion': 'entropy', 'max_features': 'log2...           254.800342   \n","108  {'alpha': 0.001953125, 'hidden_layer_sizes': (...          2478.668654   \n","109  {'C': 2, 'gamma': 0.0001220703125, 'kernel': '...           137.955329   \n","110  {'criterion': 'entropy', 'max_features': 'log2...           113.584896   \n","111  {'alpha': 0.001953125, 'hidden_layer_sizes': (...            13.699150   \n","112            {'C': 8, 'gamma': 0.5, 'kernel': 'rbf'}             0.413084   \n","113  {'criterion': 'gini', 'max_features': 'sqrt', ...            78.685357   \n","114   {'alpha': 0.125, 'hidden_layer_sizes': (64, 64)}            22.683855   \n","115  {'C': 0.03125, 'gamma': 0.001953125, 'kernel':...             0.406561   \n","116  {'criterion': 'entropy', 'max_features': 'log2...            80.390760   \n","117  {'alpha': 0.03125, 'hidden_layer_sizes': (64, ...            16.322586   \n","118  {'C': 0.03125, 'gamma': 0.00048828125, 'kernel...             0.387627   \n","119  {'criterion': 'gini', 'max_features': 'sqrt', ...            78.629580   \n","120  {'alpha': 3.0517578125e-05, 'hidden_layer_size...            25.624079   \n","121  {'C': 8, 'gamma': 0.0001220703125, 'kernel': '...            12.671258   \n","122  {'criterion': 'entropy', 'max_features': 'sqrt...            81.428668   \n","123  {'alpha': 0.00048828125, 'hidden_layer_sizes':...            38.448539   \n","124  {'C': 0.03125, 'gamma': 0.00048828125, 'kernel...             0.625493   \n","125  {'criterion': 'gini', 'max_features': 'log2', ...            86.934503   \n","\n","     experiment_time                   data       model  seed  max_epoch_num  \\\n","0         238.745647   marker_Cirrhosis.txt  shallow_ae    42            200   \n","1         238.745647   marker_Cirrhosis.txt  shallow_ae    42            200   \n","2         238.745647   marker_Cirrhosis.txt  shallow_ae    42            200   \n","3         730.488681   marker_Cirrhosis.txt          ae    42            200   \n","4         730.488681   marker_Cirrhosis.txt          ae    42            200   \n","5         730.488681   marker_Cirrhosis.txt          ae    42            200   \n","6       13683.493345   marker_Cirrhosis.txt         cae    42            200   \n","7       13683.493345   marker_Cirrhosis.txt         cae    42            200   \n","8       13683.493345   marker_Cirrhosis.txt         cae    42            200   \n","9         433.025449   marker_Cirrhosis.txt         vae    42            200   \n","10        433.025449   marker_Cirrhosis.txt         vae    42            200   \n","11        433.025449   marker_Cirrhosis.txt         vae    42            200   \n","12        243.937610   marker_Cirrhosis.txt         pca    42            200   \n","13        243.937610   marker_Cirrhosis.txt         pca    42            200   \n","14        243.937610   marker_Cirrhosis.txt         pca    42            200   \n","15        315.226107   marker_Cirrhosis.txt          rp    42            200   \n","16        315.226107   marker_Cirrhosis.txt          rp    42            200   \n","17        315.226107   marker_Cirrhosis.txt          rp    42            200   \n","18       7645.472544   marker_Cirrhosis.txt       plain    42            200   \n","19       7645.472544   marker_Cirrhosis.txt       plain    42            200   \n","20       7645.472544   marker_Cirrhosis.txt       plain    42            200   \n","21       9414.390030     marker_Obesity.txt         cae    42           2000   \n","22       9414.390030     marker_Obesity.txt         cae    42           2000   \n","23       9414.390030     marker_Obesity.txt         cae    42           2000   \n","24       4146.935735     marker_Obesity.txt       plain    42           2000   \n","25       4146.935735     marker_Obesity.txt       plain    42           2000   \n","26       4146.935735     marker_Obesity.txt       plain    42           2000   \n","27        100.520304     marker_Obesity.txt  shallow_ae    42           2000   \n","28        100.520304     marker_Obesity.txt  shallow_ae    42           2000   \n","29        100.520304     marker_Obesity.txt  shallow_ae    42           2000   \n","30        118.358175     marker_Obesity.txt          ae    42           2000   \n","31        118.358175     marker_Obesity.txt          ae    42           2000   \n","32        118.358175     marker_Obesity.txt          ae    42           2000   \n","33         90.790034     marker_Obesity.txt         vae    42           2000   \n","34         90.790034     marker_Obesity.txt         vae    42           2000   \n","35         90.790034     marker_Obesity.txt         vae    42           2000   \n","36        412.816045     marker_Obesity.txt         pca    42           2000   \n","37        412.816045     marker_Obesity.txt         pca    42           2000   \n","38        412.816045     marker_Obesity.txt         pca    42           2000   \n","39        191.006667     marker_Obesity.txt          rp    42           2000   \n","40        191.006667     marker_Obesity.txt          rp    42           2000   \n","41        191.006667     marker_Obesity.txt          rp    42           2000   \n","42      14115.473664         marker_T2D.txt         cae    42           2000   \n","43      14115.473664         marker_T2D.txt         cae    42           2000   \n","44      14115.473664         marker_T2D.txt         cae    42           2000   \n","45       8130.763001         marker_T2D.txt       plain    42           2000   \n","46       8130.763001         marker_T2D.txt       plain    42           2000   \n","47       8130.763001         marker_T2D.txt       plain    42           2000   \n","48        100.779192         marker_T2D.txt  shallow_ae    42           2000   \n","49        100.779192         marker_T2D.txt  shallow_ae    42           2000   \n","50        100.779192         marker_T2D.txt  shallow_ae    42           2000   \n","51        132.101581         marker_T2D.txt          ae    42           2000   \n","52        132.101581         marker_T2D.txt          ae    42           2000   \n","53        132.101581         marker_T2D.txt          ae    42           2000   \n","54        100.809295         marker_T2D.txt         vae    42           2000   \n","55        100.809295         marker_T2D.txt         vae    42           2000   \n","56        100.809295         marker_T2D.txt         vae    42           2000   \n","57        229.635357         marker_T2D.txt         pca    42           2000   \n","58        229.635357         marker_T2D.txt         pca    42           2000   \n","59        229.635357         marker_T2D.txt         pca    42           2000   \n","60        219.272567         marker_T2D.txt          rp    42           2000   \n","61        219.272567         marker_T2D.txt          rp    42           2000   \n","62        219.272567         marker_T2D.txt          rp    42           2000   \n","63        102.883997  marker_Colorectal.txt  shallow_ae    42            200   \n","64        102.883997  marker_Colorectal.txt  shallow_ae    42            200   \n","65        102.883997  marker_Colorectal.txt  shallow_ae    42            200   \n","66        316.989440  marker_Colorectal.txt          ae    42            200   \n","67        316.989440  marker_Colorectal.txt          ae    42            200   \n","68        316.989440  marker_Colorectal.txt          ae    42            200   \n","69       3378.567152  marker_Colorectal.txt         cae    42            200   \n","70       3378.567152  marker_Colorectal.txt         cae    42            200   \n","71       3378.567152  marker_Colorectal.txt         cae    42            200   \n","72        189.599260  marker_Colorectal.txt         vae    42            200   \n","73        189.599260  marker_Colorectal.txt         vae    42            200   \n","74        189.599260  marker_Colorectal.txt         vae    42            200   \n","75        102.141932  marker_Colorectal.txt         pca    42            200   \n","76        102.141932  marker_Colorectal.txt         pca    42            200   \n","77        102.141932  marker_Colorectal.txt         pca    42            200   \n","78        120.945065  marker_Colorectal.txt          rp    42            200   \n","79        120.945065  marker_Colorectal.txt          rp    42            200   \n","80        120.945065  marker_Colorectal.txt          rp    42            200   \n","81       2080.805042  marker_Colorectal.txt       plain    42            200   \n","82       2080.805042  marker_Colorectal.txt       plain    42            200   \n","83       2080.805042  marker_Colorectal.txt       plain    42            200   \n","84        103.051002         marker_IBD.txt  shallow_ae    42            200   \n","85        103.051002         marker_IBD.txt  shallow_ae    42            200   \n","86        103.051002         marker_IBD.txt  shallow_ae    42            200   \n","87        285.467308         marker_IBD.txt          ae    42            200   \n","88        285.467308         marker_IBD.txt          ae    42            200   \n","89        285.467308         marker_IBD.txt          ae    42            200   \n","90       3058.399564         marker_IBD.txt         cae    42            200   \n","91       3058.399564         marker_IBD.txt         cae    42            200   \n","92       3058.399564         marker_IBD.txt         cae    42            200   \n","93        183.918113         marker_IBD.txt         vae    42            200   \n","94        183.918113         marker_IBD.txt         vae    42            200   \n","95        183.918113         marker_IBD.txt         vae    42            200   \n","96         87.200318         marker_IBD.txt         pca    42            200   \n","97         87.200318         marker_IBD.txt         pca    42            200   \n","98         87.200318         marker_IBD.txt         pca    42            200   \n","99        112.673155         marker_IBD.txt          rp    42            200   \n","100       112.673155         marker_IBD.txt          rp    42            200   \n","101       112.673155         marker_IBD.txt          rp    42            200   \n","102      2497.345249         marker_IBD.txt       plain    42            200   \n","103      2497.345249         marker_IBD.txt       plain    42            200   \n","104      2497.345249         marker_IBD.txt       plain    42            200   \n","105      2495.791879        marker_WT2D.txt         cae    42           2000   \n","106      2495.791879        marker_WT2D.txt         cae    42           2000   \n","107      2495.791879        marker_WT2D.txt         cae    42           2000   \n","108      2730.213370        marker_WT2D.txt       plain    42           2000   \n","109      2730.213370        marker_WT2D.txt       plain    42           2000   \n","110      2730.213370        marker_WT2D.txt       plain    42           2000   \n","111        92.950624        marker_WT2D.txt  shallow_ae    42           2000   \n","112        92.950624        marker_WT2D.txt  shallow_ae    42           2000   \n","113        92.950624        marker_WT2D.txt  shallow_ae    42           2000   \n","114       107.277794        marker_WT2D.txt          ae    42           2000   \n","115       107.277794        marker_WT2D.txt          ae    42           2000   \n","116       107.277794        marker_WT2D.txt          ae    42           2000   \n","117        95.779552        marker_WT2D.txt         vae    42           2000   \n","118        95.779552        marker_WT2D.txt         vae    42           2000   \n","119        95.779552        marker_WT2D.txt         vae    42           2000   \n","120       120.230908        marker_WT2D.txt         pca    42           2000   \n","121       120.230908        marker_WT2D.txt         pca    42           2000   \n","122       120.230908        marker_WT2D.txt         pca    42           2000   \n","123       126.877143        marker_WT2D.txt          rp    42           2000   \n","124       126.877143        marker_WT2D.txt          rp    42           2000   \n","125       126.877143        marker_WT2D.txt          rp    42           2000   \n","\n","    device  \n","0      mps  \n","1      mps  \n","2      mps  \n","3      mps  \n","4      mps  \n","5      mps  \n","6      mps  \n","7      mps  \n","8      mps  \n","9      mps  \n","10     mps  \n","11     mps  \n","12     mps  \n","13     mps  \n","14     mps  \n","15     mps  \n","16     mps  \n","17     mps  \n","18     mps  \n","19     mps  \n","20     mps  \n","21     cpu  \n","22     cpu  \n","23     cpu  \n","24     cpu  \n","25     cpu  \n","26     cpu  \n","27     cpu  \n","28     cpu  \n","29     cpu  \n","30     cpu  \n","31     cpu  \n","32     cpu  \n","33     cpu  \n","34     cpu  \n","35     cpu  \n","36     cpu  \n","37     cpu  \n","38     cpu  \n","39     cpu  \n","40     cpu  \n","41     cpu  \n","42     cpu  \n","43     cpu  \n","44     cpu  \n","45     cpu  \n","46     cpu  \n","47     cpu  \n","48     cpu  \n","49     cpu  \n","50     cpu  \n","51     cpu  \n","52     cpu  \n","53     cpu  \n","54     cpu  \n","55     cpu  \n","56     cpu  \n","57     cpu  \n","58     cpu  \n","59     cpu  \n","60     cpu  \n","61     cpu  \n","62     cpu  \n","63     mps  \n","64     mps  \n","65     mps  \n","66     mps  \n","67     mps  \n","68     mps  \n","69     mps  \n","70     mps  \n","71     mps  \n","72     mps  \n","73     mps  \n","74     mps  \n","75     mps  \n","76     mps  \n","77     mps  \n","78     mps  \n","79     mps  \n","80     mps  \n","81     mps  \n","82     mps  \n","83     mps  \n","84     mps  \n","85     mps  \n","86     mps  \n","87     mps  \n","88     mps  \n","89     mps  \n","90     mps  \n","91     mps  \n","92     mps  \n","93     mps  \n","94     mps  \n","95     mps  \n","96     mps  \n","97     mps  \n","98     mps  \n","99     mps  \n","100    mps  \n","101    mps  \n","102    mps  \n","103    mps  \n","104    mps  \n","105    cpu  \n","106    cpu  \n","107    cpu  \n","108    cpu  \n","109    cpu  \n","110    cpu  \n","111    cpu  \n","112    cpu  \n","113    cpu  \n","114    cpu  \n","115    cpu  \n","116    cpu  \n","117    cpu  \n","118    cpu  \n","119    cpu  \n","120    cpu  \n","121    cpu  \n","122    cpu  \n","123    cpu  \n","124    cpu  \n","125    cpu  "]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["url = 'https://raw.githubusercontent.com/ZKingQ/CS598-DLH-SP24/main/result/marker_merged_metrics.csv'\n","df = pd.read_csv(url)\n","# show all rows\n","pd.set_option('display.max_rows', None)\n","df"]},{"cell_type":"markdown","metadata":{"id":"8EAWAy_LwHlV"},"source":["## Model comparison"]},{"cell_type":"code","execution_count":17,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":497},"executionInfo":{"elapsed":10,"status":"ok","timestamp":1715020054613,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"kz2DU4ZTK5tE","outputId":"bc52d113-f0e8-4e4a-bc88-27d5c22e0b8f"},"outputs":[{"data":{"application/vnd.google.colaboratory.intrinsic+json":{"repr_error":"cannot insert data, already exists","type":"dataframe"},"text/html":["\n","  <div id=\"df-ed1a573b-d1bc-4288-a596-042047d65637\" class=\"colab-df-container\">\n","    <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>AUC</th>\n","      <th>ACC</th>\n","      <th>Recall</th>\n","      <th>Precision</th>\n","      <th>F1_score</th>\n","      <th>classification_method</th>\n","      <th>hyper_parameters</th>\n","      <th>classification_time</th>\n","      <th>experiment_time</th>\n","      <th>data</th>\n","      <th>model</th>\n","      <th>seed</th>\n","      <th>max_epoch_num</th>\n","      <th>device</th>\n","    </tr>\n","    <tr>\n","      <th>data</th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>marker_Cirrhosis.txt</th>\n","      <td>0.992754</td>\n","      <td>0.914894</td>\n","      <td>0.833333</td>\n","      <td>1.000000</td>\n","      <td>0.909091</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>310.007509</td>\n","      <td>7645.472544</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>marker_Colorectal.txt</th>\n","      <td>0.846667</td>\n","      <td>0.760000</td>\n","      <td>0.800000</td>\n","      <td>0.666667</td>\n","      <td>0.727273</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>30.598888</td>\n","      <td>102.141932</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>marker_IBD.txt</th>\n","      <td>0.929412</td>\n","      <td>0.727273</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'log2...</td>\n","      <td>331.307379</td>\n","      <td>3058.399564</td>\n","      <td>marker_IBD.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>marker_Obesity.txt</th>\n","      <td>0.739057</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>194.442154</td>\n","      <td>4146.935735</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>marker_T2D.txt</th>\n","      <td>0.807563</td>\n","      <td>0.739130</td>\n","      <td>0.764706</td>\n","      <td>0.722222</td>\n","      <td>0.742857</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 0.00048828125, 'hidden_layer_sizes':...</td>\n","      <td>5410.766285</td>\n","      <td>8130.763001</td>\n","      <td>marker_T2D.txt</td>\n","      <td>plain</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>marker_WT2D.txt</th>\n","      <td>0.737374</td>\n","      <td>0.650000</td>\n","      <td>0.909091</td>\n","      <td>0.625000</td>\n","      <td>0.740741</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>86.934503</td>\n","      <td>126.877143</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","    <div class=\"colab-df-buttons\">\n","\n","  <div class=\"colab-df-container\">\n","    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ed1a573b-d1bc-4288-a596-042047d65637')\"\n","            title=\"Convert this dataframe to an interactive table.\"\n","            style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n","    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n","  </svg>\n","    </button>\n","\n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    .colab-df-buttons div {\n","      margin-bottom: 4px;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","    <script>\n","      const buttonEl =\n","        document.querySelector('#df-ed1a573b-d1bc-4288-a596-042047d65637 button.colab-df-convert');\n","      buttonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","      async function convertToInteractive(key) {\n","        const element = document.querySelector('#df-ed1a573b-d1bc-4288-a596-042047d65637');\n","        const dataTable =\n","          await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                    [key], {});\n","        if (!dataTable) return;\n","\n","        const docLinkHtml = 'Like what you see? Visit the ' +\n","          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","          + ' to learn more about interactive tables.';\n","        element.innerHTML = '';\n","        dataTable['output_type'] = 'display_data';\n","        await google.colab.output.renderOutput(dataTable, element);\n","        const docLink = document.createElement('div');\n","        docLink.innerHTML = docLinkHtml;\n","        element.appendChild(docLink);\n","      }\n","    </script>\n","  </div>\n","\n","\n","<div id=\"df-b00f4231-d9ca-4daf-87c4-569a440fabd5\">\n","  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-b00f4231-d9ca-4daf-87c4-569a440fabd5')\"\n","            title=\"Suggest charts\"\n","            style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","  </button>\n","\n","<style>\n","  .colab-df-quickchart {\n","      --bg-color: #E8F0FE;\n","      --fill-color: #1967D2;\n","      --hover-bg-color: #E2EBFA;\n","      --hover-fill-color: #174EA6;\n","      --disabled-fill-color: #AAA;\n","      --disabled-bg-color: #DDD;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","      --bg-color: #3B4455;\n","      --fill-color: #D2E3FC;\n","      --hover-bg-color: #434B5C;\n","      --hover-fill-color: #FFFFFF;\n","      --disabled-bg-color: #3B4455;\n","      --disabled-fill-color: #666;\n","  }\n","\n","  .colab-df-quickchart {\n","    background-color: var(--bg-color);\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: var(--fill-color);\n","    height: 32px;\n","    padding: 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: var(--hover-bg-color);\n","    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: var(--button-hover-fill-color);\n","  }\n","\n","  .colab-df-quickchart-complete:disabled,\n","  .colab-df-quickchart-complete:disabled:hover {\n","    background-color: var(--disabled-bg-color);\n","    fill: var(--disabled-fill-color);\n","    box-shadow: none;\n","  }\n","\n","  .colab-df-spinner {\n","    border: 2px solid var(--fill-color);\n","    border-color: transparent;\n","    border-bottom-color: var(--fill-color);\n","    animation:\n","      spin 1s steps(1) infinite;\n","  }\n","\n","  @keyframes spin {\n","    0% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","      border-left-color: var(--fill-color);\n","    }\n","    20% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    30% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","      border-right-color: var(--fill-color);\n","    }\n","    40% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    60% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","    }\n","    80% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-bottom-color: var(--fill-color);\n","    }\n","    90% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","    }\n","  }\n","</style>\n","\n","  <script>\n","    async function quickchart(key) {\n","      const quickchartButtonEl =\n","        document.querySelector('#' + key + ' button');\n","      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n","      quickchartButtonEl.classList.add('colab-df-spinner');\n","      try {\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      } catch (error) {\n","        console.error('Error during call to suggestCharts:', error);\n","      }\n","      quickchartButtonEl.classList.remove('colab-df-spinner');\n","      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n","    }\n","    (() => {\n","      let quickchartButtonEl =\n","        document.querySelector('#df-b00f4231-d9ca-4daf-87c4-569a440fabd5 button');\n","      quickchartButtonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","    })();\n","  </script>\n","</div>\n","\n","    </div>\n","  </div>\n"],"text/plain":["                            AUC       ACC    Recall  Precision  F1_score  \\\n","data                                                                       \n","marker_Cirrhosis.txt   0.992754  0.914894  0.833333   1.000000  0.909091   \n","marker_Colorectal.txt  0.846667  0.760000  0.800000   0.666667  0.727273   \n","marker_IBD.txt         0.929412  0.727273  0.000000   0.000000  0.000000   \n","marker_Obesity.txt     0.739057  0.647059  1.000000   0.647059  0.785714   \n","marker_T2D.txt         0.807563  0.739130  0.764706   0.722222  0.742857   \n","marker_WT2D.txt        0.737374  0.650000  0.909091   0.625000  0.740741   \n","\n","                      classification_method  \\\n","data                                          \n","marker_Cirrhosis.txt          Random Forest   \n","marker_Colorectal.txt                   MLP   \n","marker_IBD.txt                Random Forest   \n","marker_Obesity.txt            Random Forest   \n","marker_T2D.txt                          MLP   \n","marker_WT2D.txt               Random Forest   \n","\n","                                                        hyper_parameters  \\\n","data                                                                       \n","marker_Cirrhosis.txt   {'criterion': 'gini', 'max_features': 'log2', ...   \n","marker_Colorectal.txt  {'alpha': 3.0517578125e-05, 'hidden_layer_size...   \n","marker_IBD.txt         {'criterion': 'entropy', 'max_features': 'log2...   \n","marker_Obesity.txt     {'criterion': 'gini', 'max_features': 'log2', ...   \n","marker_T2D.txt         {'alpha': 0.00048828125, 'hidden_layer_sizes':...   \n","marker_WT2D.txt        {'criterion': 'gini', 'max_features': 'log2', ...   \n","\n","                       classification_time  experiment_time  \\\n","data                                                          \n","marker_Cirrhosis.txt            310.007509      7645.472544   \n","marker_Colorectal.txt            30.598888       102.141932   \n","marker_IBD.txt                  331.307379      3058.399564   \n","marker_Obesity.txt              194.442154      4146.935735   \n","marker_T2D.txt                 5410.766285      8130.763001   \n","marker_WT2D.txt                  86.934503       126.877143   \n","\n","                                        data  model  seed  max_epoch_num  \\\n","data                                                                       \n","marker_Cirrhosis.txt    marker_Cirrhosis.txt  plain    42            200   \n","marker_Colorectal.txt  marker_Colorectal.txt    pca    42            200   \n","marker_IBD.txt                marker_IBD.txt    cae    42            200   \n","marker_Obesity.txt        marker_Obesity.txt  plain    42           2000   \n","marker_T2D.txt                marker_T2D.txt  plain    42           2000   \n","marker_WT2D.txt              marker_WT2D.txt     rp    42           2000   \n","\n","                      device  \n","data                          \n","marker_Cirrhosis.txt     mps  \n","marker_Colorectal.txt    mps  \n","marker_IBD.txt           mps  \n","marker_Obesity.txt       cpu  \n","marker_T2D.txt           cpu  \n","marker_WT2D.txt          cpu  "]},"execution_count":17,"metadata":{},"output_type":"execute_result"}],"source":["# show best auc for each data set and its param config\n","df.groupby(['data']).apply(lambda x: x.loc[x['AUC'].idxmax()])"]},{"cell_type":"code","execution_count":18,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":462},"executionInfo":{"elapsed":9,"status":"ok","timestamp":1715020054613,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"taWIFOiiMJYS","outputId":"bf70fb3b-1c3e-429e-8e94-4608d2dadf4f"},"outputs":[{"data":{"application/vnd.google.colaboratory.intrinsic+json":{"repr_error":"cannot insert data, already exists","type":"dataframe"},"text/html":["\n","  <div id=\"df-422dacd0-4d5b-4175-ab0d-85b570b17fa8\" class=\"colab-df-container\">\n","    <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>AUC</th>\n","      <th>ACC</th>\n","      <th>Recall</th>\n","      <th>Precision</th>\n","      <th>F1_score</th>\n","      <th>classification_method</th>\n","      <th>hyper_parameters</th>\n","      <th>classification_time</th>\n","      <th>experiment_time</th>\n","      <th>data</th>\n","      <th>model</th>\n","      <th>seed</th>\n","      <th>max_epoch_num</th>\n","      <th>device</th>\n","    </tr>\n","    <tr>\n","      <th>data</th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>marker_Cirrhosis.txt</th>\n","      <td>0.981884</td>\n","      <td>0.893617</td>\n","      <td>0.875000</td>\n","      <td>0.913043</td>\n","      <td>0.893617</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>8910.858763</td>\n","      <td>13683.493345</td>\n","      <td>marker_Cirrhosis.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>marker_Colorectal.txt</th>\n","      <td>0.846667</td>\n","      <td>0.760000</td>\n","      <td>0.800000</td>\n","      <td>0.666667</td>\n","      <td>0.727273</td>\n","      <td>MLP</td>\n","      <td>{'alpha': 3.0517578125e-05, 'hidden_layer_size...</td>\n","      <td>30.598888</td>\n","      <td>102.141932</td>\n","      <td>marker_Colorectal.txt</td>\n","      <td>pca</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>marker_IBD.txt</th>\n","      <td>0.929412</td>\n","      <td>0.727273</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>0.000000</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'entropy', 'max_features': 'log2...</td>\n","      <td>331.307379</td>\n","      <td>3058.399564</td>\n","      <td>marker_IBD.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>200</td>\n","      <td>mps</td>\n","    </tr>\n","    <tr>\n","      <th>marker_Obesity.txt</th>\n","      <td>0.685185</td>\n","      <td>0.647059</td>\n","      <td>1.000000</td>\n","      <td>0.647059</td>\n","      <td>0.785714</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'sqrt', ...</td>\n","      <td>121.447971</td>\n","      <td>191.006667</td>\n","      <td>marker_Obesity.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>marker_T2D.txt</th>\n","      <td>0.801681</td>\n","      <td>0.753623</td>\n","      <td>0.794118</td>\n","      <td>0.729730</td>\n","      <td>0.760563</td>\n","      <td>SVM</td>\n","      <td>{'C': 0.03125, 'kernel': 'linear'}</td>\n","      <td>10209.280280</td>\n","      <td>14115.473664</td>\n","      <td>marker_T2D.txt</td>\n","      <td>cae</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","    <tr>\n","      <th>marker_WT2D.txt</th>\n","      <td>0.737374</td>\n","      <td>0.650000</td>\n","      <td>0.909091</td>\n","      <td>0.625000</td>\n","      <td>0.740741</td>\n","      <td>Random Forest</td>\n","      <td>{'criterion': 'gini', 'max_features': 'log2', ...</td>\n","      <td>86.934503</td>\n","      <td>126.877143</td>\n","      <td>marker_WT2D.txt</td>\n","      <td>rp</td>\n","      <td>42</td>\n","      <td>2000</td>\n","      <td>cpu</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","    <div class=\"colab-df-buttons\">\n","\n","  <div class=\"colab-df-container\">\n","    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-422dacd0-4d5b-4175-ab0d-85b570b17fa8')\"\n","            title=\"Convert this dataframe to an interactive table.\"\n","            style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n","    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n","  </svg>\n","    </button>\n","\n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    .colab-df-buttons div {\n","      margin-bottom: 4px;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","    <script>\n","      const buttonEl =\n","        document.querySelector('#df-422dacd0-4d5b-4175-ab0d-85b570b17fa8 button.colab-df-convert');\n","      buttonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","      async function convertToInteractive(key) {\n","        const element = document.querySelector('#df-422dacd0-4d5b-4175-ab0d-85b570b17fa8');\n","        const dataTable =\n","          await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                    [key], {});\n","        if (!dataTable) return;\n","\n","        const docLinkHtml = 'Like what you see? Visit the ' +\n","          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","          + ' to learn more about interactive tables.';\n","        element.innerHTML = '';\n","        dataTable['output_type'] = 'display_data';\n","        await google.colab.output.renderOutput(dataTable, element);\n","        const docLink = document.createElement('div');\n","        docLink.innerHTML = docLinkHtml;\n","        element.appendChild(docLink);\n","      }\n","    </script>\n","  </div>\n","\n","\n","<div id=\"df-b8d790c2-9311-4ded-b0f9-7079fd725d41\">\n","  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-b8d790c2-9311-4ded-b0f9-7079fd725d41')\"\n","            title=\"Suggest charts\"\n","            style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","  </button>\n","\n","<style>\n","  .colab-df-quickchart {\n","      --bg-color: #E8F0FE;\n","      --fill-color: #1967D2;\n","      --hover-bg-color: #E2EBFA;\n","      --hover-fill-color: #174EA6;\n","      --disabled-fill-color: #AAA;\n","      --disabled-bg-color: #DDD;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","      --bg-color: #3B4455;\n","      --fill-color: #D2E3FC;\n","      --hover-bg-color: #434B5C;\n","      --hover-fill-color: #FFFFFF;\n","      --disabled-bg-color: #3B4455;\n","      --disabled-fill-color: #666;\n","  }\n","\n","  .colab-df-quickchart {\n","    background-color: var(--bg-color);\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: var(--fill-color);\n","    height: 32px;\n","    padding: 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: var(--hover-bg-color);\n","    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: var(--button-hover-fill-color);\n","  }\n","\n","  .colab-df-quickchart-complete:disabled,\n","  .colab-df-quickchart-complete:disabled:hover {\n","    background-color: var(--disabled-bg-color);\n","    fill: var(--disabled-fill-color);\n","    box-shadow: none;\n","  }\n","\n","  .colab-df-spinner {\n","    border: 2px solid var(--fill-color);\n","    border-color: transparent;\n","    border-bottom-color: var(--fill-color);\n","    animation:\n","      spin 1s steps(1) infinite;\n","  }\n","\n","  @keyframes spin {\n","    0% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","      border-left-color: var(--fill-color);\n","    }\n","    20% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    30% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","      border-right-color: var(--fill-color);\n","    }\n","    40% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    60% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","    }\n","    80% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-bottom-color: var(--fill-color);\n","    }\n","    90% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","    }\n","  }\n","</style>\n","\n","  <script>\n","    async function quickchart(key) {\n","      const quickchartButtonEl =\n","        document.querySelector('#' + key + ' button');\n","      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n","      quickchartButtonEl.classList.add('colab-df-spinner');\n","      try {\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      } catch (error) {\n","        console.error('Error during call to suggestCharts:', error);\n","      }\n","      quickchartButtonEl.classList.remove('colab-df-spinner');\n","      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n","    }\n","    (() => {\n","      let quickchartButtonEl =\n","        document.querySelector('#df-b8d790c2-9311-4ded-b0f9-7079fd725d41 button');\n","      quickchartButtonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","    })();\n","  </script>\n","</div>\n","\n","    </div>\n","  </div>\n"],"text/plain":["                            AUC       ACC    Recall  Precision  F1_score  \\\n","data                                                                       \n","marker_Cirrhosis.txt   0.981884  0.893617  0.875000   0.913043  0.893617   \n","marker_Colorectal.txt  0.846667  0.760000  0.800000   0.666667  0.727273   \n","marker_IBD.txt         0.929412  0.727273  0.000000   0.000000  0.000000   \n","marker_Obesity.txt     0.685185  0.647059  1.000000   0.647059  0.785714   \n","marker_T2D.txt         0.801681  0.753623  0.794118   0.729730  0.760563   \n","marker_WT2D.txt        0.737374  0.650000  0.909091   0.625000  0.740741   \n","\n","                      classification_method  \\\n","data                                          \n","marker_Cirrhosis.txt                    SVM   \n","marker_Colorectal.txt                   MLP   \n","marker_IBD.txt                Random Forest   \n","marker_Obesity.txt            Random Forest   \n","marker_T2D.txt                          SVM   \n","marker_WT2D.txt               Random Forest   \n","\n","                                                        hyper_parameters  \\\n","data                                                                       \n","marker_Cirrhosis.txt                  {'C': 0.03125, 'kernel': 'linear'}   \n","marker_Colorectal.txt  {'alpha': 3.0517578125e-05, 'hidden_layer_size...   \n","marker_IBD.txt         {'criterion': 'entropy', 'max_features': 'log2...   \n","marker_Obesity.txt     {'criterion': 'gini', 'max_features': 'sqrt', ...   \n","marker_T2D.txt                        {'C': 0.03125, 'kernel': 'linear'}   \n","marker_WT2D.txt        {'criterion': 'gini', 'max_features': 'log2', ...   \n","\n","                       classification_time  experiment_time  \\\n","data                                                          \n","marker_Cirrhosis.txt           8910.858763     13683.493345   \n","marker_Colorectal.txt            30.598888       102.141932   \n","marker_IBD.txt                  331.307379      3058.399564   \n","marker_Obesity.txt              121.447971       191.006667   \n","marker_T2D.txt                10209.280280     14115.473664   \n","marker_WT2D.txt                  86.934503       126.877143   \n","\n","                                        data model  seed  max_epoch_num device  \n","data                                                                            \n","marker_Cirrhosis.txt    marker_Cirrhosis.txt   cae    42            200    mps  \n","marker_Colorectal.txt  marker_Colorectal.txt   pca    42            200    mps  \n","marker_IBD.txt                marker_IBD.txt   cae    42            200    mps  \n","marker_Obesity.txt        marker_Obesity.txt    rp    42           2000    cpu  \n","marker_T2D.txt                marker_T2D.txt   cae    42           2000    cpu  \n","marker_WT2D.txt              marker_WT2D.txt    rp    42           2000    cpu  "]},"execution_count":18,"metadata":{},"output_type":"execute_result"}],"source":["# show best auc for each data set and its config, when model is not plain\n","df[df['model'] != 'plain'].groupby(['data']).apply(lambda x: x.loc[x['AUC'].idxmax()])"]},{"cell_type":"code","execution_count":19,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":534},"executionInfo":{"elapsed":236,"status":"ok","timestamp":1715020054842,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"B-DgTDuvMc4T","outputId":"336f6730-952c-4c6a-f959-693aad8a97db"},"outputs":[{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAuMAAAIFCAYAAAB4YPM7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB+aklEQVR4nO3deVxO6f8/8Ndd2utu00oqspSybzHIMkIYZN+SpUEhO2M3xjaD7DsZwxi7GY21ZEzIvo7tQ2QpIZVCpfv8/vDrfLtVVNSJ83o+Hvfj0X2d6z7nfc5939d5d93XuY5CEAQBRERERERU5DSkDoCIiIiISK6YjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRRJiMF1PTpk2DQqHIU12FQoFp06YVbkASyDwGz549K/RtSCU8PBwKhQLh4eFq5Zs2bUKlSpWgpaUFExMTAICHhwc8PDyKPMbg4GAoFArcu3evyLedF7kdQym9ffsWY8eOhZ2dHTQ0NNC+fXupQ/rsMj8XZ8+elTqUAvmUz01x/E4U5/OAh4cHXF1dC30bUrSPmXI6l+TWDkj1XvXt2xcODg5Fvt3iIPP7vmPHjkLfVkGOc4GT8eXLl0OhUKBu3boFXQUR5eDGjRvo27cvypUrhzVr1mD16tVFst1Zs2Zhz549RbKtgli+fDmCg4OlDiNP1q9fj59//hmdOnXCxo0bMWLECKlDIqIiJkU78PjxY0ybNg0XL14s9G0VR1u2bEFQUJDUYeRbiYK+cPPmzXBwcMDp06fxv//9D05OTp8zLtmbNGkSxo8fL3UYVMgaNWqE169fQ1tbWywLDw+HSqXCokWL1L5Xhw4dKtRYZs2ahU6dOmXrxe3duze6desGHR2dQt3+xyxfvhwlS5ZE37591cpzOoZSCwsLQ6lSpbBw4UKpQyGiIpDTOTu3duD169coUaLA6dcHPX78GNOnT4eDgwOqVaumtmzNmjVQqVSFst3iYsuWLbh69SoCAwOlDiVfCtQzHhUVhRMnTmDBggWwsLDA5s2bP3dcH6VSqfDmzZsi325hS0lJAQCUKFECurq6Ekfz+b19+xZpaWlSh1FsaGhoQFdXFxoa//dVjIuLAwBxeEombW1tSRJOTU1N6OrqSjqc50NyOoZSi4uLy/b+fYqvtb0j+lrkdM7OrR3Q1dUttGT8Q7S0tCTvVKGcFejstXnzZpiamsLLywudOnVSS8bT09NhZmYGX1/fbK9LSkqCrq4uRo8eLZalpqZi6tSpcHJygo6ODuzs7DB27FikpqaqvVahUCAgIACbN29G5cqVoaOjgwMHDgAAfvnlF9SvXx/m5ubQ09NDzZo1cxwX9Pr1awwbNgwlS5aEkZER2rVrh0ePHuU4fuvRo0fo168frKysoKOjg8qVK2P9+vV5Oj553U7mGLP//vsPPXr0gKmpKb755hu1ZVmlpqZixIgRsLCwENf78OHDPMUEAEuWLEHlypWhr68PU1NT1KpVC1u2bMn3fqelpWHKlCmoWbMmjI2NYWBggIYNG+Lo0aNq9e7duweFQoFffvkFQUFBKFeuHHR0dPDff/8BeDcco0uXLrCwsICenh4qVqyIiRMnZos7ISEBffv2hYmJCYyNjeHr64tXr17laZ8jIyPRunVrmJqawsDAAFWqVMGiRYs++JoNGzagadOmsLS0hI6ODlxcXLBixYps9c6ePQtPT0+ULFkSenp6cHR0RL9+/dTqbN26FTVr1oSRkRGUSiXc3NzUtv/+uFUHBwdMnToVAGBhYaH2mclpTOSbN28wbdo0VKhQAbq6urCxsUHHjh1x584dsU5evh8KhQIpKSnYuHEjFAoFFAqF2AOd2/jY5cuXi99FW1tb+Pv7IyEhQa1O5ljR//77D02aNIG+vj5KlSqFefPmfegtyMbBwQHXrl3DsWPHxPgyj0VOY38zt3v58mU0btwY+vr6cHJyEvf72LFjqFu3rvi5O3LkSLZtFqQNyPzMHz16FNeuXRNjzYwtJSUFo0aNgp2dHXR0dFCxYkX88ssvEARBbT0fau9ys3//fjRs2BAGBgYwMjKCl5cXrl27plbn8uXL6Nu3L8qWLQtdXV1YW1ujX79+eP78eY77379/f9ja2kJHRweOjo4YPHhwtn+mU1NTMXLkSFhYWMDAwAAdOnTA06dPPxgr8G5cpaGhIaKjo9GmTRsYGhqiVKlSWLZsGQDgypUraNq0KQwMDGBvb5+trQKAu3fvonPnzjAzM4O+vj7q1auHkJCQbPUePnyI9u3bw8DAAJaWlhgxYkS2c0ymyMhItGzZEsbGxtDX10fjxo0RERHx0f3JSV6Pd2Z7/7///e+jbd2nngfyes7NazsIvPvsNW7cWGznateuneP79SntwG+//YY6deqI569GjRp98NfCvJ6ngI+30+np6Zg+fTrKly8PXV1dmJub45tvvsHhw4fFOlnP2R9rB3LLOT70fYuPj8fo0aPh5uYGQ0NDKJVKtGrVCpcuXRLXER4ejtq1awMAfH19xe1mDu/LaSxzftukPXv2wNXVVWwTP9YuZcalUCiwbds2TJ8+HaVKlYKRkRE6deqExMREpKamIjAwEJaWljA0NISvr2+O38/ffvsNNWvWhJ6eHszMzNCtWzc8ePBAXO7h4YGQkBDcv39f3Pf391elUuGnn35C6dKloauri2bNmuF///tftm1t375d3FbJkiXRq1cvPHr0KFu9zOOhq6sLV1dX7N69+6PHIycF+tds8+bN6NixI7S1tdG9e3esWLECZ86cQe3ataGlpYUOHTpg165dWLVqlVpP3p49e5Camopu3boBeHdQ2rVrh3///Rd+fn5wdnbGlStXsHDhQty6dSvb+NWwsDBs27YNAQEBKFmypHiQFy1ahHbt2qFnz55IS0vD1q1b0blzZ+zbtw9eXl7i6/v27Ytt27ahd+/eqFevHo4dO6a2PNOTJ09Qr1498cNnYWGB/fv3o3///khKSvrozx953U6mzp07o3z58pg1a1a2L0BWAwYMwG+//YYePXqgfv36CAsL++B6s1qzZg2GDRuGTp06Yfjw4Xjz5g0uX76MyMhI9OjRI1/7nZSUhLVr16J79+4YOHAgXr58iXXr1sHT0xOnT5/O9tPYhg0b8ObNG/j5+UFHRwdmZma4fPkyGjZsCC0tLfj5+cHBwQF37tzBX3/9hZ9++knt9V26dIGjoyNmz56N8+fPY+3atbC0tMTcuXM/uM+HDx9GmzZtYGNjg+HDh8Pa2hrXr1/Hvn37MHz48Fxft2LFClSuXBnt2rVDiRIl8Ndff2HIkCFQqVTw9/cH8K7Ho0WLFrCwsMD48eNhYmKCe/fuYdeuXWrb7969O5o1aybGev36dUREROS6/aCgIPz666/YvXs3VqxYAUNDQ1SpUiXHuhkZGWjTpg1CQ0PRrVs3DB8+HC9fvsThw4dx9epVlCtXDkDevh+bNm3CgAEDUKdOHfj5+QGA+PqcTJs2DdOnT0fz5s0xePBg3Lx5U2wHIiIioKWlJdZ98eIFWrZsiY4dO6JLly7YsWMHxo0bBzc3N7Rq1SrXbbx/XIYOHQpDQ0PxHzYrK6sPvubFixdo06YNunXrhs6dO2PFihXo1q0bNm/ejMDAQAwaNAg9evQQx3Q+ePAARkZGAAreBlhYWGDTpk346aefkJycjNmzZwMAnJ2dIQgC2rVrh6NHj6J///6oVq0aDh48iDFjxuDRo0fZfsrOrb3LyaZNm+Dj4wNPT0/MnTsXr169wooVK/DNN9/gwoUL4msPHz6Mu3fvwtfXF9bW1rh27RpWr16Na9eu4dSpU2Iy8fjxY9SpUwcJCQnw8/NDpUqV8OjRI+zYsQOvXr1Sa9eHDh0KU1NTTJ06Fffu3UNQUBACAgLwxx9/fPD9Ad59hlu1aoVGjRph3rx52Lx5MwICAmBgYICJEyeiZ8+e6NixI1auXIk+ffrA3d0djo6O4ntUv359vHr1CsOGDYO5uTk2btyIdu3aYceOHejQoQOAd50jzZo1Q3R0NIYNGwZbW1ts2rQJYWFh2eIJCwtDq1atULNmTUydOhUaGhpiUnr8+HHUqVPno/uUVV6Pd6a8tHWfch7Izzk3L+0g8O6f9X79+qFy5cqYMGECTExMcOHCBRw4cEA8twCf1g5Mnz4d06ZNQ/369TFjxgxoa2sjMjISYWFhaNGiRY6vyet5Ki/t9LRp0zB79myxjUxKSsLZs2dx/vx5fPvtt9m2/aF2ICd5+b7dvXsXe/bsQefOneHo6IgnT55g1apVaNy4Mf777z/Y2trC2dkZM2bMwJQpU+Dn54eGDRsCAOrXr5/jdvPbJv3777/YtWsXhgwZAiMjIyxevBje3t6Ijo6Gubn5B99DAJg9ezb09PQwfvx4/O9//8OSJUugpaUFDQ0NvHjxAtOmTcOpU6cQHBwMR0dHTJkyRXztTz/9hMmTJ6NLly4YMGAAnj59iiVLlqBRo0a4cOECTExMMHHiRCQmJuLhw4di7IaGhmoxzJkzBxoaGhg9ejQSExMxb9489OzZE5GRkWKd4OBg+Pr6onbt2pg9ezaePHmCRYsWISIiQtwW8G7oqLe3N1xcXDB79mw8f/4cvr6+KF269EePRU5vRr6cPXtWACAcPnxYEARBUKlUQunSpYXhw4eLdQ4ePCgAEP766y+117Zu3VooW7as+HzTpk2ChoaGcPz4cbV6K1euFAAIERERYhkAQUNDQ7h27Vq2mF69eqX2PC0tTXB1dRWaNm0qlp07d04AIAQGBqrV7du3rwBAmDp1qljWv39/wcbGRnj27Jla3W7dugnGxsbZtpdVfrYzdepUAYDQvXv3bOvJXJbp4sWLAgBhyJAhavV69OiRbb05+e6774TKlSt/sE5e9/vt27dCamqqWp0XL14IVlZWQr9+/cSyqKgoAYCgVCqFuLg4tfqNGjUSjIyMhPv376uVq1Qq8e/MY5B1nYIgCB06dBDMzc0/uC9v374VHB0dBXt7e+HFixcf3UZWOb2/np6eap/d3bt3CwCEM2fO5BrD8OHDBaVSKbx9+zbXOkePHhUACEePHs0W09OnT9XqNm7cWGjcuLH4fP369QIAYcGCBdnWm3Uf8/L9EARBMDAwEHx8fLKta8OGDQIAISoqShAEQYiLixO0tbWFFi1aCBkZGWK9pUuXCgCE9evXq8UMQPj111/FstTUVMHa2lrw9vbOfkA+oHLlymr7nymnY5i53S1btohlN27cENuRU6dOieWZ7dWGDRvEsk9pAzK3//73bc+ePQIAYebMmWrlnTp1EhQKhfC///1PLPtQe/e+ly9fCiYmJsLAgQPVymNjYwVjY2O18pzi/v333wUAwj///COW9enTR9DQ0Mjx85352cr8XDRv3lzt8zZixAhBU1NTSEhI+GDcPj4+AgBh1qxZYtmLFy8EPT09QaFQCFu3bhXLM9+7rO1cYGCgAEDt/PHy5UvB0dFRcHBwED+bQUFBAgBh27ZtYr2UlBTByclJ7XOjUqmE8uXLC56entm+P46OjsK3334rlr3/nchNXo93Xtu6Tz0P5Oecm5d2MCEhQTAyMhLq1q0rvH79Wq1u1mP4Ke3A7du3BQ0NDaFDhw5q7U1O28jaPuT1PJWXdrpq1aqCl5fXB+PM6VySUzsgCEK29yov37c3b95k2/+oqChBR0dHmDFjhlh25syZbO1ZJh8fH8He3l58nt82SVtbW63s0qVLAgBhyZIl2baVVWYb7erqKqSlpYnl3bt3FxQKhdCqVSu1+u7u7mpx3rt3T9DU1BR++ukntXpXrlwRSpQooVbu5eWl9tr3Y3B2dlb7XCxatEgAIFy5ckUQhHfnR0tLS8HV1VXtM71v3z4BgDBlyhSxrFq1aoKNjY1aW3fo0CEBQI4xfEi+h6ls3rwZVlZWaNKkCYB3P1107doVW7duRUZGBgCgadOmKFmypFrPyIsXL3D48GF07dpVLNu+fTucnZ1RqVIlPHv2THw0bdoUALL9nNS4cWO4uLhki0lPT09tO4mJiWjYsCHOnz8vlmf+lDJkyBC11w4dOlTtuSAI2LlzJ9q2bQtBENTi8vT0RGJiotp635fX7WQ1aNCgXJdl+vvvvwEAw4YNUyvP60UKJiYmePjwIc6cOZPj8vzst6amptgzplKpEB8fj7dv36JWrVo5Hhtvb29YWFiIz58+fYp//vkH/fr1Q5kyZdTq5jQu+f3j07BhQzx//hxJSUm57u+FCxcQFRWFwMDAbGP2Pjb2OevnKTExEc+ePUPjxo1x9+5dJCYmAvi/8dz79u1Denp6jusxMTFBSkqK2k+Zn9POnTtRsmTJHD9bWfcxL9+P/Dhy5AjS0tIQGBioNk574MCBUCqV2YYJGBoaolevXuJzbW1t1KlTB3fv3i3Q9vPK0NBQ/BUOACpWrAgTExM4OzurzQKV+XdmPJ/aBuTm77//hqamZrbv8KhRoyAIAvbv369Wnlt7977Dhw8jISEB3bt3V4tVU1MTdevWVWtHs34W3rx5g2fPnqFevXoAIO6TSqXCnj170LZtW9SqVSvb9t7//vj5+amVNWzYEBkZGbh///5HYwfe9fRmMjExQcWKFWFgYIAuXbqI5ZnvXdbPzN9//406deqIQ/uAd++5n58f7t27Jw6H+/vvv2FjY4NOnTqJ9fT19cVfgDJdvHgRt2/fRo8ePfD8+XPxOKakpKBZs2b4559/8n3xW16Od1Yfa+s+9TyQn3NuXtrBw4cP4+XLlxg/fny28dLvf04K2g7s2bMHKpUKU6ZMyXZdyIfa8ryep/LSTpuYmODatWu4ffv2B2MtiLx+33R0dMT9z8jIwPPnz2FoaIiKFSsWuC3Pb5vUvHlztV9Mq1SpAqVSmee2vE+fPmq/mtatWxeCIGQb3lm3bl08ePAAb9++BQDs2rULKpUKXbp0UfvcWltbo3z58jkOPcqNr6+v2i97mb8eZO7D2bNnERcXhyFDhqh9pr28vFCpUiXx/BYTE4OLFy/Cx8cHxsbGYr1vv/02T+32+/I1TCUjIwNbt25FkyZNEBUVJZbXrVsX8+fPR2hoKFq0aIESJUrA29sbW7ZsQWpqKnR0dLBr1y6kp6erJeO3b9/G9evX1RK1rDIvZMuU+fPk+/bt24eZM2fi4sWLauOMsn5R79+/Dw0NjWzreH8WmKdPnyIhIQGrV6/OdUq59+PKKq/bySq3/cppve8PHahYseJHXwsA48aNw5EjR1CnTh04OTmhRYsW6NGjBxo0aAAg//u9ceNGzJ8/Hzdu3FBLRnPal/fLMj/0eZ139v2E3dTUFMC7xFKpVOb4mswx0wWZ2zYiIgJTp07FyZMns43XTExMhLGxMRo3bgxvb29Mnz4dCxcuhIeHB9q3b48ePXqIF8gMGTIE27ZtQ6tWrVCqVCm0aNECXbp0QcuWLfMdU07u3LmDihUrfvRCoLx8P/IjM8l6/7Onra2NsmXLZkvCSpcunW1bpqamuHz5coG2n1c5bdfY2Bh2dnbZyoB3nyfg09uA3Ny/fx+2trbiUJhMmT9dv3/c8tIuABAThMyE6n1ZvyPx8fGYPn06tm7dmm0fMhOsp0+fIikp6bN8Pz9GV1c3W/tvbGyc63uXdZ3379/PcWrdrMfT1dUV9+/fh5OTU7b1vf/5zTyOPj4+ucabmJgo7l9e5OV4Z/Wxtu5TzwP5OefmpR3MTztb0Hbgzp070NDQKFCCk5fzVF7a6RkzZuC7775DhQoV4OrqipYtW6J37965DiHMj7x+3zJn2Fq+fDmioqLEzk8AeRoikpP8tknvfz6Bd+9hXr7rOb0+s+3NqU1WqVRITEyEubk5bt++DUEQUL58+RzXmzXBz28M77dXuZ3fAKBSpUr4999/1erlFFNB/kHKVzIeFhaGmJgYbN26FVu3bs22fPPmzeL4rW7dumHVqlXYv38/2rdvj23btqFSpUqoWrWqWF+lUsHNzQ0LFizIcXvvv0FZ/1PPdPz4cbRr1w6NGjXC8uXLYWNjAy0tLWzYsCHHC0g+JrPno1evXrk2yp/jC5hVTvv1uTk7O+PmzZvYt28fDhw4gJ07d2L58uWYMmUKpk+fnq/9/u2339C3b1+0b98eY8aMgaWlJTQ1NTF79my1Cwczfer+aWpq5lgufGB8fUHduXMHzZo1Q6VKlbBgwQLY2dlBW1sbf//9NxYuXCgep8ybB5w6dQp//fUXDh48iH79+mH+/Pk4deoUDA0NYWlpiYsXL+LgwYPYv38/9u/fjw0bNqBPnz7YuHHjZ489J5/7+1EQRfn+5WW7H4tHijYgJ3n93mTGu2nTJlhbW2dbnvWftS5duuDEiRMYM2YMqlWrBkNDQ6hUKrRs2bLAU559yvtb0PeoMGTu/88//5ztupdM748//Zj8Hu/C3u+8nnPz2g7mR1G/p3k9T+WlnW7UqBHu3LmDvXv34tChQ1i7di0WLlyIlStXqv2yU5hmzZqFyZMno1+/fvjxxx9hZmYGDQ0NBAYGFtl0hZ/6Hn5Km6xQKLB///4c6+bneynV+ehj8pWMb968GZaWluLV7lnt2rULu3fvxsqVK6Gnp4dGjRrBxsYGf/zxB7755huEhYVlmymjXLlyuHTpEpo1a1bgXrqdO3dCV1cXBw8eVJuyZ8OGDWr17O3toVKpEBUVpfafzPtX0WZeoZ6RkYHmzZvnO568bqeg683sDc108+bNPK/DwMAAXbt2RdeuXZGWloaOHTvip59+woQJE/K13zt27EDZsmWxa9cutfctcxaQjylbtiwA4OrVq3mOPb8ye46uXr2ar/fxr7/+QmpqKv7880+1/6Bz+xmsXr16qFevHn766Sds2bIFPXv2xNatW8UGWltbG23btkXbtm2hUqkwZMgQrFq1CpMnT/7kufnLlSuHyMhIpKen59ozkNfvB5D3nnJ7e3sA7z57me8l8G72gqioqAJ9b/KiqKZW/NQ2IDf29vY4cuQIXr58qdYTdePGDXF5QWR+1i0tLT8Y74sXLxAaGorp06erXRj1/k/vFhYWUCqVhfr9/Bzs7e1zbP/eP5729va4evUqBEFQ+wy9/9rM46hUKj/L+57X450fn3oeyOs5N6/tYNZ2trDuNVKuXDmoVCr8999/uf6TlJP8nKfy0k5nzhLn6+uL5ORkNGrUCNOmTfvkZDyv37cdO3agSZMmWLdunVp5QkICSpYsKT7PTztZWG3S51auXDkIggBHR0dUqFDhg3U/9TyR9fz2/q+NN2/eVGtXgJy/z/nJyzLlecz469evsWvXLrRp0wadOnXK9ggICMDLly/x559/vluxhgY6deqEv/76C5s2bcLbt2/VhqgA73oNHj16hDVr1uS4vcw5tz9EU1MTCoVC7Sebe/fuZZuJxdPTE8C76diyWrJkSbb1eXt7Y+fOnTl+OT42ZVdet5NfmVebL168WK08r3eaen8qLW1tbbi4uEAQBKSnp+drvzP/s8z6n2RkZCROnjyZp1gsLCzQqFEjrF+/HtHR0WrLPtd/pzVq1ICjoyOCgoKyTbf3oW3ktG+JiYnZktcXL15kW0/miSJzKMj7x1xDQ0PsUc1tWrX88Pb2xrNnz7B06dJsyzJjy+v3A3j3z9r7xyonzZs3h7a2NhYvXqx2DNatW4fExMQ8z+yQX3mN71N9ahuQm9atWyMjIyPb+7Vw4UIoFIo8zyzzPk9PTyiVSsyaNSvH6xcy483psw1kb0Myb9v9119/5Xire6l7kDK1bt0ap0+fVmt3UlJSsHr1ajg4OIjDGlq3bo3Hjx+rTef56tWrbEOQatasiXLlyuGXX35BcnJytu3l933P6/HOj089D+T1nJvXdrBFixYwMjLC7Nmzs82D/7k+J+3bt4eGhgZmzJiRrQc4v215TuepvLTT79cxNDSEk5PTZ2nH8/p909TUzLa/27dvzzbdnoGBAQDkqa0srDbpc+vYsSM0NTUxffr0bMdAEAS198fAwCDHIWB5VatWLVhaWmLlypVq7+/+/ftx/fp18fxmY2ODatWqYePGjWrbO3z4sHi9Sn7kuWf8zz//xMuXL9GuXbscl9erV0+8AVBm0t21a1csWbIEU6dOhZubW7ZpfXr37o1t27Zh0KBBOHr0KBo0aICMjAzcuHED27Ztw8GDB3O8oCErLy8vLFiwAC1btkSPHj0QFxeHZcuWwcnJSW0sWs2aNeHt7Y2goCA8f/5cnHLw1q1bANT/m5ozZw6OHj2KunXrYuDAgXBxcUF8fDzOnz+PI0eOID4+Ptd48rOd/KhWrRq6d++O5cuXIzExEfXr10doaGiee9xbtGgBa2trNGjQAFZWVrh+/TqWLl0KLy8v8T/ivO53mzZtsGvXLnTo0AFeXl6IiorCypUr4eLikuNJLCeLFy/GN998gxo1asDPzw+Ojo64d+8eQkJCPsttfDU0NLBixQq0bdsW1apVg6+vL2xsbHDjxg1cu3YNBw8ezPU4ZfaSfP/990hOTsaaNWtgaWmJmJgYsd7GjRuxfPlydOjQAeXKlcPLly+xZs0aKJVKtG7dGsC7C9Pi4+PRtGlTlC5dGvfv38eSJUtQrVq1XKe4yo8+ffrg119/xciRI3H69Gk0bNgQKSkpOHLkCIYMGYLvvvsuz98P4N1n98iRI1iwYAFsbW3h6OiY45hcCwsLTJgwAdOnT0fLli3Rrl073Lx5E8uXL0ft2rXVLtL6nGrWrIkVK1Zg5syZcHJygqWlZa7jpD/Vp7QBuWnbti2aNGmCiRMn4t69e6hatSoOHTqEvXv3IjAw8INTSX6IUqnEihUr0Lt3b9SoUQPdunWDhYUFoqOjERISggYNGmDp0qVQKpXiFILp6ekoVaoUDh06pHb9T6ZZs2bh0KFDaNy4sTgFXkxMDLZv345///33s97QqKDGjx+P33//Ha1atcKwYcNgZmaGjRs3IioqCjt37hQvdhs4cCCWLl2KPn364Ny5c7CxscGmTZugr6+vtj4NDQ2sXbsWrVq1QuXKleHr64tSpUrh0aNHOHr0KJRKJf766688x5ef451Xn3oeyOs5N6/toFKpxMKFCzFgwADUrl1bvF/GpUuX8OrVq88yHM/JyQkTJ07Ejz/+iIYNG6Jjx47Q0dHBmTNnYGtrK04b+L68nqfy0k67uLjAw8MDNWvWhJmZGc6ePYsdO3YgICDgk/cPyNv3rU2bNpgxYwZ8fX1Rv359XLlyBZs3b1b7dRJ414tsYmKClStXwsjICAYGBqhbt26O16AUVpv0uZUrVw4zZ87EhAkTcO/ePbRv3x5GRkaIiorC7t274efnJ96/pmbNmvjjjz8wcuRI1K5dG4aGhmjbtm2et6WlpYW5c+fC19cXjRs3Rvfu3cWpDR0cHDBixAix7uzZs+Hl5YVvvvkG/fr1Q3x8vHg/l7zmQqK8TrvStm1bQVdXV0hJScm1Tt++fQUtLS1xOjCVSiXY2dnlOHVOprS0NGHu3LlC5cqVBR0dHcHU1FSoWbOmMH36dCExMVGsB0Dw9/fPcR3r1q0TypcvL+jo6AiVKlUSNmzYkOM0QykpKYK/v79gZmYmGBoaCu3btxdu3rwpABDmzJmjVvfJkyeCv7+/YGdnJ2hpaQnW1tZCs2bNhNWrV3/0WOV1O7lNYZd1WVavX78Whg0bJpibmwsGBgZC27ZthQcPHuRpSqtVq1YJjRo1EszNzQUdHR2hXLlywpgxY9SOcV73W6VSCbNmzRLs7e0FHR0doXr16sK+ffuyTZuUObXhzz//nGNMV69eFTp06CCYmJgIurq6QsWKFYXJkyd/9PjkdVoxQRCEf//9V/j2228FIyMjwcDAQKhSpYraNEw5Hec///xTqFKliqCrqys4ODgIc+fOFacRzNzm+fPnhe7duwtlypQRdHR0BEtLS6FNmzbC2bNnxfXs2LFDaNGihWBpaSloa2sLZcqUEb7//nshJiZGrPMpUxsKwrvpxyZOnCg4OjqK71enTp2EO3fuiHXy+v24ceOG0KhRI0FPT08AIE5zmNvxXrp0qVCpUiVBS0tLsLKyEgYPHpxtGsncpvZ6/7OSF7GxsYKXl5dgZGQkABCPRW5TG+a0XXt7+xynKMupffmUNiC37b98+VIYMWKEYGtrK2hpaQnly5cXfv75Z7Up2nKL52OOHj0qeHp6CsbGxoKurq5Qrlw5oW/fvmqfyYcPH4rfOWNjY6Fz587C48ePc2xD7t+/L/Tp00ewsLAQdHR0hLJlywr+/v7itGCZn4v3p2PL6f3IiY+Pj2BgYJCtPD/v3Z07d4ROnTqJbUidOnWEffv2ZXvt/fv3hXbt2gn6+vpCyZIlheHDhwsHDhzIMc4LFy4IHTt2FNtKe3t7oUuXLkJoaKhYJ69tUF6Pd37auk85DwhC3s+5eWkHs9atX7++oKenJyiVSqFOnTrC77//Li7/HO3A+vXrherVq4sxN27cWJxiOXMbWdvHvJ6n8tJOz5w5U6hTp45gYmIi6OnpCZUqVRJ++ukntWn6PmVqQ0H4+PftzZs3wqhRowQbGxtBT09PaNCggXDy5Mkczwt79+4VXFxchBIlSqhNc5jT8f7UNsne3j7HKXGzymwTtm/frlaeWxuS2/dh586dwjfffCMYGBgIBgYGQqVKlQR/f3/h5s2bYp3k5GShR48egomJidoUg7nFkJmnvD8V5B9//CF+3szMzISePXsKDx8+zLZvO3fuFJydnQUdHR3BxcVF2LVrV4HObwpBKCa/OUrk4sWLqF69On777Tf07Nnzi98OEREREX058j3P+Jfs9evX2cqCgoKgoaGBRo0afXHbISIiIqIvW75mU/nSzZs3D+fOnUOTJk1QokQJcRojPz+/bNMofgnbIfoaPH36VO0C0/dpa2vDzMysCCMiIiIqOrIapnL48GFMnz4d//33H5KTk1GmTBn07t0bEydO/OiNU4rjdoi+Bg4ODh+8W2Pjxo0RHh5edAEREREVIVkl40RU/EREROQ4tCuTqakpatasWYQRERERFR0m40REREREEpHVBZxERERERMUJBzDLgEqlwuPHj2FkZFRktxQnIiKiTyMIAl6+fAlbW1vxRlb09WEyLgOPHz/mLC5ERERfqAcPHqB06dJSh0GFhMm4DGTe7v7BgwdQKpUSR0NERER5kZSUBDs7O/E8Tl8nJuMykDk0RalUMhknIiL6wnCI6deNA5CIiIiIiCTCZJyIiIiISCJMxomIiIiIJMIx40REJHsZGRlIT0+XOgySGS0tLWhqakodBkmMyXgBTZs2DdOnT1crq1ixIm7cuAEAePPmDUaNGoWtW7ciNTUVnp6eWL58OaysrMT60dHRGDx4MI4ePQpDQ0P4+Phg9uzZKFHi/96W8PBwjBw5EteuXYOdnR0mTZqEvn37Fsk+EhF97QRBQGxsLBISEqQOhWTKxMQE1tbWvEhTxpiMf4LKlSvjyJEj4vOsSfSIESMQEhKC7du3w9jYGAEBAejYsSMiIiIAvOuF8fLygrW1NU6cOIGYmBj06dMHWlpamDVrFgAgKioKXl5eGDRoEDZv3ozQ0FAMGDAANjY28PT0LNqdJSL6CmUm4paWltDX12dCREVGEAS8evUKcXFxAAAbGxuJIyKpKARBEKQO4ks0bdo07NmzBxcvXsy2LDExERYWFtiyZQs6deoEALhx4wacnZ1x8uRJ1KtXD/v370ebNm3w+PFjsbd85cqVGDduHJ4+fQptbW2MGzcOISEhuHr1qrjubt26ISEhAQcOHMhzrElJSTA2NkZiYiKnNiQi+v8yMjJw69YtWFpawtzcXOpwSKaeP3+OuLg4VKhQIduQFZ6/5YEXcH6C27dvw9bWFmXLlkXPnj0RHR0NADh37hzS09PRvHlzsW6lSpVQpkwZnDx5EgBw8uRJuLm5qQ1b8fT0RFJSEq5duybWybqOzDqZ68hNamoqkpKS1B5ERKQuc4y4vr6+xJGQnGV+/njNgnwxGS+gunXrIjg4GAcOHMCKFSsQFRWFhg0b4uXLl4iNjYW2tjZMTEzUXmNlZYXY2FgA734azZqIZy7PXPahOklJSXj9+nWusc2ePRvGxsbiw87O7lN3l4joq8WhKSQlfv6IY8YLqFWrVuLfVapUQd26dWFvb49t27ZBT09PwsiACRMmYOTIkeLzzNvpEhEREVHxwp7xz8TExAQVKlTA//73P1hbWyMtLS3b1flPnjyBtbU1AMDa2hpPnjzJtjxz2YfqKJXKDyb8Ojo6UCqVag8iIvo87t27B4VCkeM1Q59bcHBwtl9ZV69eDTs7O2hoaCAoKAjTpk1DtWrVCj0WBwcHBAUFFfp2pBAeHg6FQlEos+ooFArs2bPns6+Xvh5Mxj+T5ORk3LlzBzY2NqhZsya0tLQQGhoqLr958yaio6Ph7u4OAHB3d8eVK1fEq6gB4PDhw1AqlXBxcRHrZF1HZp3MdRAR0deta9euuHXrlvg8KSkJAQEBGDduHB49egQ/Pz+MHj0627niU+T0DwAAnDlzBn5+fp9tO1Lx8PBAYGCg1GEQiThMpYBGjx6Ntm3bwt7eHo8fP8bUqVOhqamJ7t27w9jYGP3798fIkSNhZmYGpVKJoUOHwt3dHfXq1QMAtGjRAi4uLujduzfmzZuH2NhYTJo0Cf7+/tDR0QEADBo0CEuXLsXYsWPRr18/hIWFYdu2bQgJCZFy14mIqIjo6emp/RIaHR2N9PR0eHl5qU2FZ2hoWOixWFhYFPo2iOSIPeMF9PDhQ3Tv3h0VK1ZEly5dYG5ujlOnTomN1cKFC9GmTRt4e3ujUaNGsLa2xq5du8TXa2pqYt++fdDU1IS7uzt69eqFPn36YMaMGWIdR0dHhISE4PDhw6hatSrmz5+PtWvXco5xIqIioFKpMG/ePDg5OUFHRwdlypTBTz/9lK1eRkYG+vfvD0dHR+jp6aFixYpYtGiRWp3w8HDUqVMHBgYGMDExQYMGDXD//n0AwKVLl9CkSRMYGRlBqVSiZs2aOHv2LAD1Xurg4GC4ubkBAMqWLQuFQoF79+7lOExl/fr1qFy5MnR0dGBjY4OAgABx2YIFC+Dm5gYDAwPY2dlhyJAhSE5OFuP09fVFYmIiFAoFFAoFpk2bBiD7MJXo6Gh89913MDQ0hFKpRJcuXdSGVmbGtWnTJjg4OMDY2BjdunXDy5cv83T8PTw8MHToUAQGBsLU1BRWVlZYs2YNUlJS4OvrCyMjIzg5OWH//v1qr7t69SpatWoFQ0NDWFlZoXfv3nj27BkAoG/fvjh27BgWLVok7t+9e/fE1547dw61atWCvr4+6tevj5s3b6qte8WKFShXrhy0tbVRsWJFbNq0SW357du30ahRI+jq6sLFxQWHDx/O076SvLFnvIC2bt36weW6urpYtmwZli1blmsde3t7/P333x9cj4eHBy5cuFCgGOnr4bbRLV/1r/hcKaRIiORjwoQJWLNmDRYuXIhvvvkGMTEx4l2Ws1KpVChdujS2b98Oc3NznDhxAn5+frCxsUGXLl3w9u1btG/fHgMHDsTvv/+OtLQ0nD59WpxFo2fPnqhevTpWrFgBTU1NXLx4EVpaWtm207VrV9jZ2aF58+Y4ffo07OzscuytXrFiBUaOHIk5c+agVatWSExMFG84BwAaGhpYvHgxHB0dcffuXQwZMgRjx47F8uXLUb9+fQQFBWHKlCliIppTr7tKpRIT8WPHjuHt27fw9/dH165dER4eLta7c+cO9uzZg3379uHFixfo0qUL5syZk+M/NTnZuHEjxo4di9OnT+OPP/7A4MGDsXv3bnTo0AE//PADFi5ciN69eyM6Ohr6+vpISEhA06ZNMWDAACxcuBCvX7/GuHHj0KVLF4SFhWHRokW4desWXF1dxc4vCwsLMSGfOHEi5s+fDwsLCwwaNAj9+vUTj93u3bsxfPhwBAUFoXnz5ti3bx98fX1RunRpNGnSBCqVCh07doSVlRUiIyORmJjI4TCUNwJ99RITEwUAQmJiotShUAG5Brvm60FEH/f69Wvhv//+E16/fp1tWVJSkqCjoyOsWbMm27KoqCgBgHDhwoVc1+3v7y94e3sLgiAIz58/FwAI4eHhOdY1MjISgoODc1y2YcMGwdjYWHx+4cIFAYAQFRUllk2dOlWoWrWq+NzW1laYOHFirrG9b/v27YK5uXmu28xkb28vLFy4UBAEQTh06JCgqakpREdHi8uvXbsmABBOnz4txqWvry8kJSWJdcaMGSPUrVs3T3E1btxY+Oabb8Tnb9++FQwMDITevXuLZTExMQIA4eTJk4IgCMKPP/4otGjRQm09Dx48EAAIN2/eFNc7fPhwtTpHjx4VAAhHjhwRy0JCQgQA4uejfv36wsCBA9Ve17lzZ6F169aCIAjCwYMHhRIlSgiPHj0Sl+/fv18AIOzevTvX/fzQ55Dnb3ngMBUiIqL3XL9+HampqWjWrFme6i9btgw1a9aEhYUFDA0NsXr1avFGcGZmZujbty88PT3Rtm1bLFq0CDExMeJrR44ciQEDBqB58+aYM2cO7ty5U+C44+Li8Pjx4w/GfeTIETRr1gylSpWCkZERevfujefPn+PVq1d53s7169dhZ2enNm2ui4sLTExMcP36dbHMwcEBRkZG4nMbGxu1iQs+pkqVKuLfmpqaMDc3F4fqAP93f47MdV66dAlHjx6FoaGh+KhUqRIA5Om4Zt1e5pj8zHVfv34dDRo0UKvfoEEDcX8zj4mtra24nBMuUF4wGSciInpPfu4XsXXrVowePRr9+/fHoUOHcPHiRfj6+iItLU2ss2HDBpw8eRL169fHH3/8gQoVKuDUqVMA3o2tvnbtGry8vBAWFgYXFxfs3r27UOK+d+8e2rRpgypVqmDnzp04d+6cOJwya7yfy/vDbRQKBVQq1Se9PmtZ5lCfzHUmJyejbdu2uHjxotojcyx3frb3/rqJCguTcSIioveUL18eenp6eZoyMCIiAvXr18eQIUNQvXp1ODk55dgLW716dUyYMAEnTpyAq6srtmzZIi6rUKECRowYgUOHDqFjx47YsGFDgeI2MjKCg4NDrnGfO3cOKpUK8+fPR7169VChQgU8fvxYrY62tjYyMjI+uB1nZ2c8ePAADx48EMv+++8/JCQkiNPzSqFGjRq4du0aHBwc4OTkpPYwMDAAkLf9y4mzs7Pa2Hvg3Xufub+ZxyTrrx6Z/3ARfQiTcSIiovfo6upi3LhxGDt2LH799VfcuXMHp06dwrp167LVLV++PM6ePYuDBw/i1q1bmDx5Ms6cOSMuj4qKwoQJE3Dy5Encv38fhw4dwu3bt+Hs7IzXr18jICAA4eHhuH//PiIiInDmzBk4OzsXOPZp06Zh/vz5WLx4MW7fvo3z589jyZIlAAAnJyekp6djyZIluHv3LjZt2oSVK1eqvd7BwQHJyckIDQ3Fs2fPchy+0rx5c7i5uaFnz544f/48Tp8+jT59+qBx48aoVatWgWP/VP7+/oiPj0f37t1x5swZ3LlzBwcPHoSvr6+YgDs4OCAyMhL37t3Ds2fP8tzzPWbMGAQHB2PFihW4ffs2FixYgF27dmH06NEA3h2TChUqwMfHB5cuXcLx48cxceLEQttX+nowGSciIsrB5MmTMWrUKEyZMgXOzs7o2rVrjuOdv//+e3Ts2BFdu3ZF3bp18fz5cwwZMkRcrq+vjxs3bsDb2xsVKlSAn58f/P398f3330NTUxPPnz9Hnz59UKFCBXTp0gWtWrXC9OnTCxy3j48PgoKCsHz5clSuXBlt2rTB7du3AQBVq1bFggULMHfuXLi6umLz5s2YPXu22uvr16+PQYMGoWvXrrCwsMC8efOybUOhUGDv3r0wNTVFo0aN0Lx5c5QtWxZ//PFHgeP+HGxtbREREYGMjAy0aNECbm5uCAwMhImJCTQ03qU8o0ePhqamJlxcXGBhYSGO7f+Y9u3bY9GiRfjll19QuXJlrFq1Chs2bICHhweAd7PU7N69G69fv0adOnUwYMCAPM8aQ/KmEARBkDoIKlxJSUkwNjZGYmIilEql1OFQAXBqQ6LP782bN4iKioKjoyN0dXWlDodk6kOfQ56/5YE940REREREEmEyTkREREUqOjpabfrB9x95HTpC9DXgHTiJiIioSNna2uLixYsfXE4kF0zGiYiIqEiVKFECTk5OUodBVCxwmAoRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEUxsSERF9gMP4kCLd3r05Xvmq37dvX2zcuBHff/89Vq5cqbbM398fy5cvh4+PD4KDg9G3b18kJCRgz549Oa7LwcEB9+/fBwDo6+ujYsWKmDBhAjp37lygfSGij2PPOBER0RfOzs4OW7duxevXr8WyN2/eYMuWLShTpky+1jVjxgzExMTgwoULqF27Nrp27YoTJ0587pCJ6P9jMk5ERPSFq1GjBuzs7LBr1y6xbNeuXShTpgyqV6+er3UZGRnB2toaFSpUwLJly6Cnp4e//vrrc4dMRP8fk3EiIqKvQL9+/bBhwwbx+fr16+Hr6/tJ6yxRogS0tLSQlpb2qeERUS6YjBMREX0FevXqhX///Rf379/H/fv3ERERgV69ehV4fWlpaZg9ezYSExPRtGnTzxgpEWXFCziJiIi+AhYWFvDy8kJwcDAEQYCXlxdKliyZ7/WMGzcOkyZNwps3b2BoaIg5c+bAyyt/F5USUd4xGSciIvpK9OvXDwEBAQCAZcuWFWgdY8aMQd++fWFoaAgrKysoFIrPGSIRvYfJOBER0VeiZcuWSEtLg0KhgKenZ4HWUbJkSTg5OX3myIgoN0zGiYiIvhKampq4fv26+HdOEhMTcfHiRbUyc3Nz2NnZFXZ4RJQDJuNEREQfkN+b8EhNqVR+cHl4eHi26Q779++PtWvXFmZYRJQLJuNERERfsODg4A8uz3q3zeDg4A/Wv3fv3meJiYjyjlMbEhERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgREREVmEKhULvLJxHlTwmpAyAiIirWphkX8fYS81W9b9++2LhxIwCgRIkSKF26NDp37owZM2ZAV1e3MCIsFrLud1a3b9+Gk5OTBBG9iykhIYH/nFC+MBknIiL6wrVs2RIbNmxAeno6zp07Bx8fHygUCsydO1fq0ApV5n5nZWFhUaB1paWlQVtb+3OERZQvHKZCRET0hdPR0YG1tTXs7OzQvn17NG/eHIcPHxaXP3/+HN27d0epUqWgr68PNzc3/P7772rr8PDwwLBhwzB27FiYmZnB2toa06ZNU6tz+/ZtNGrUCLq6unBxcVHbRqYrV66gadOm0NPTg7m5Ofz8/JCcnCwu79u3L9q3b49Zs2bBysoKJiYmmDFjBt6+fYsxY8bAzMwMpUuXzpZkf2i/sz40NTUBAMeOHUOdOnWgo6MDGxsbjB8/Hm/fvlXb34CAAAQGBqJkyZLw9PQEAFy9ehWtWrWCoaEhrKys0Lt3bzx79kx83Y4dO+Dm5ibuX/PmzZGSkoJp06Zh48aN2Lt3LxQKBRQKBcLDwz+6D0RMxomIiL4iV69exYkTJ9R6ed+8eYOaNWsiJCQEV69ehZ+fH3r37o3Tp0+rvXbjxo0wMDBAZGQk5s2bhxkzZogJt0qlQseOHaGtrY3IyEisXLkS48aNU3t9SkoKPD09YWpqijNnzmD79u04cuQIAgIC1OqFhYXh8ePH+Oeff7BgwQJMnToVbdq0gampKSIjIzFo0CB8//33ePjwYYGOwaNHj9C6dWvUrl0bly5dwooVK7Bu3TrMnDkz2/5qa2sjIiICK1euREJCApo2bYrq1avj7NmzOHDgAJ48eYIuXboAAGJiYtC9e3f069cP169fR3h4ODp27AhBEDB69Gh06dIFLVu2RExMDGJiYlC/fv0CxU/ywmEqREREX7h9+/bB0NAQb9++RWpqKjQ0NLB06VJxealSpTB69Gjx+dChQ3Hw4EFs27YNderUEcurVKmCqVOnAgDKly+PpUuXIjQ0FN9++y2OHDmCGzdu4ODBg7C1tQUAzJo1C61atRJfv2XLFrx58wa//vorDAwMAABLly5F27ZtMXfuXFhZWQEAzMzMsHjxYmhoaKBixYqYN28eXr16hR9++AEAMGHCBMyZMwf//vsvunXr9tH9ztSqVSts374dy5cvh52dHZYuXQqFQoFKlSrh8ePHGDduHKZMmQINDQ1xH+fNmye+fubMmahevTpmzZollq1fvx52dna4desWkpOT8fbtW3Ts2BH29vYAADc3N7Gunp4eUlNTYW1t/eE3jCgLJuNERERfuCZNmmDFihVISUnBwoULUaJECXh7e4vLMzIyMGvWLGzbtg2PHj1CWloaUlNToa+vr7aeKlWqqD23sbFBXFwcAOD69euws7MTE3EAcHd3V6t//fp1VK1aVUzEAaBBgwZQqVS4efOmmIxXrlxZTIgBwMrKCq6uruJzTU1NmJubi9v+2H5nytzu9evX4e7uDoVCoRZHcnIyHj58iDJlygAAatasqba+S5cu4ejRo2oJfqY7d+6gRYsWaNasGdzc3ODp6YkWLVqgU6dOMDU1/WCcRB/CZJyIiOgLZ2BgIM4gsn79elStWhXr1q1D//79AQA///wzFi1ahKCgILi5ucHAwACBgYFIS0tTW4+Wlpbac4VCAZVK9dnjzWk7Bdl21v0uiKz/NABAcnKy2Iv/PhsbG2hqauLw4cM4ceIEDh06hCVLlmDixImIjIyEo6NjgeMgeeOYcSIioq+IhoYGfvjhB0yaNAmvX78GAEREROC7775Dr169ULVqVZQtWxa3bt3K13qdnZ3x4MEDxMTEiGWnTp3KVufSpUtISUkRyyIiIsThKEXF2dkZJ0+ehCAIanEYGRmhdOnSub6uRo0auHbtGhwcHODk5KT2yEzcFQoFGjRogOnTp+PChQvQ1tbG7t27AQDa2trIyMgo3J2jrw6TcSIioq9M586doampiWXLlgF4NzY6s0f3+vXr+P777/HkyZN8rbN58+aoUKECfHx8cOnSJRw/fhwTJ05Uq9OzZ0/o6urCx8cHV69exdGjRzF06FD07t1bHKJSFIYMGYIHDx5g6NChuHHjBvbu3YupU6di5MiRasNj3ufv74/4+Hh0794dZ86cwZ07d3Dw4EH4+voiIyMDkZGRmDVrFs6ePYvo6Gjs2rULT58+hbOzMwDAwcEBly9fxs2bN/Hs2TOkp6cX1S7TF4zDVIiIiD4knzfhKQ5KlCiBgIAAzJs3D4MHD8akSZNw9+5deHp6Ql9fH35+fmjfvj0SE/O+bxoaGti9ezf69++POnXqwMHBAYsXL0bLli3FOvr6+jh48CCGDx+O2rVrQ19fH97e3liwYEFh7GauSpUqhb///htjxoxB1apVYWZmhv79+2PSpEkffJ2trS0iIiIwbtw4tGjRAqmpqbC3t0fLli2hoaEBpVKJf/75B0FBQUhKSoK9vT3mz58vXsQ6cOBAhIeHo1atWkhOTsbRo0fh4eFRBHtMXzKFkPU3HPoqJSUlwdjYGImJiVAqlVKHQwXgttHt45WyuOJzpZAiIfp6vHnzBlFRUXB0dPyq71RJxduHPoc8f8sDh6kQEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRREpIHQAREVFx5rbRrUi3d8XnSr5f8/TpU0yZMgUhISF48uQJTE1NUbVqVfzwww/w9vbG6NGjMX78+Gyv+/HHH7F06VI8fPgQmzdvhq+vLypVqoTr16+r1du+fTu6dOkCe3t73Lt3r6C7RkQ5YM84ERHRF87b2xsXLlzAxo0bcevWLfz555/w8PBAYmIievXqhQ0bNmR7jSAICA4ORp8+faClpQUAMDAwQFxcHE6ePKlWd926dShTpkyR7AuR3DAZ/0zmzJkDhUKBwMBAsezNmzfw9/eHubk5DA0N4e3tjSdPnqi9Ljo6Gl5eXtDX14elpSXGjBmDt2/fqtUJDw9HjRo1oKOjAycnJwQHBxfBHhER0ZcgISEBx48fx9y5c9GkSRPY29ujTp06mDBhAtq1a4f+/fvj1q1b+Pfff9Ved+zYMdy9exf9+/cXy0qUKIEePXpg/fr1YtnDhw8RHh6OHj16FNk+EckJk/HP4MyZM1i1ahWqVKmiVj5ixAj89ddf2L59O44dO4bHjx+jY8eO4vKMjAx4eXkhLS0NJ06cwMaNGxEcHIwpU6aIdaKiouDl5YUmTZrg4sWLCAwMxIABA3Dw4MEi2z8iIiq+DA0NYWhoiD179iA1NTXbcjc3N9SuXVstwQaADRs2oH79+qhUqZJaeb9+/bBt2za8evUKABAcHIyWLVvCysqq8HaCSMaYjH+i5ORk9OzZE2vWrIGpqalYnpiYiHXr1mHBggVo2rQpatasiQ0bNuDEiRM4deoUAODQoUP477//8Ntvv6FatWpo1aoVfvzxRyxbtgxpaWkAgJUrV8LR0RHz58+Hs7MzAgIC0KlTJyxcuFCS/SUiouKlRIkSCA4OxsaNG2FiYoIGDRrghx9+wOXLl8U6/fv3x/bt25GcnAwAePnyJXbs2IF+/fplW1/16tVRtmxZ7NixQxzKklM9Ivo8mIx/In9/f3h5eaF58+Zq5efOnUN6erpaeaVKlVCmTBlxLN7Jkyfh5uam1tvg6emJpKQkXLt2Tazz/ro9PT2zjefLKjU1FUlJSWoPIiL6enl7e+Px48f4888/0bJlS3F4Y+awxu7duyMjIwPbtm0DAPzxxx/Q0NBA165dc1xfv379sGHDBhw7dgwpKSlo3bp1Ue0KkewwGf8EW7duxfnz5zF79uxsy2JjY6GtrQ0TExO1cisrK8TGxop13v/ZL/P5x+okJSXh9evXOcY1e/ZsGBsbiw87O7sC7R8REX05dHV18e2332Ly5Mk4ceIE+vbti6lTpwIAlEolOnXqJF7IuWHDBnTp0gWGhoY5rqtnz544deoUpk2bht69e6NECU6+RlRYmIwX0IMHDzB8+HBs3rwZurq6UoejZsKECUhMTBQfDx48kDokIiIqYi4uLkhJSRGf9+/fH//++y/27duHEydOqF24+T4zMzO0a9cOx44d4xAVokLGZLyAzp07h7i4ONSoUQMlSpRAiRIlcOzYMSxevBglSpSAlZUV0tLSkJCQoPa6J0+ewNraGgBgbW2dbXaVzOcfq6NUKqGnp5djbDo6OlAqlWoPIiL6Oj1//hxNmzbFb7/9hsuXLyMqKgrbt2/HvHnz8N1334n1GjVqBCcnJ/Tp0weVKlVC/fr1P7je4OBgPHv2LNsFnkT0efF3pwJq1qwZrlxRvzFD5s0Sxo0bBzs7O2hpaSE0NBTe3t4AgJs3byI6Ohru7u4AAHd3d/z000+Ii4uDpaUlAODw4cNQKpVwcXER6/z9999q2zl8+LC4DiIiKlwFuQlPUTI0NETdunWxcOFC3LlzB+np6bCzs8PAgQPxww8/iPUUCgX69euHH374ARMmTPjoevX09HLt9CGiz0chCIIgdRBfCw8PD1SrVg1BQUEAgMGDB+Pvv/9GcHAwlEolhg4dCgA4ceIEgHdTG1arVg22traYN28eYmNj0bt3bwwYMACzZs0C8G5qQ1dXV/j7+6Nfv34ICwvDsGHDEBISAk9PzzzFlZSUBGNjYyQmJrKX/AuV3zsAFvfkgag4ePPmDaKiouDo6FjshhuSfHzoc8jztzywZ7wQLVy4EBoaGvD29kZqaio8PT2xfPlycbmmpib27duHwYMHw93dHQYGBvDx8cGMGTPEOo6OjggJCcGIESOwaNEilC5dGmvXrs1zIk5ERERExRd7xmWA/1l/+dgzTvT5sWecigP2jBMv4CQiIiIikgiTcSIiIiIiiTAZJyIiWeNoTZISP3/EZJyIiGRJS0sLAPDq1SuJIyE5y/z8ZX4eSX44mwoREcmSpqYmTExMEBcXBwDQ19eHQqGQOCqSC0EQ8OrVK8TFxcHExASamppSh0QSYTJORESylXm348yEnKiomZiYiJ9Dkicm40REJFsKhQI2NjawtLREenq61OGQzGhpabFHnJiMExERaWpqMikiIknwAk4iIiIiIokwGSciIiIikgiTcSIiIiIiiTAZJyIiIiKSCJNxIiIiIiKJMBknIiIiIpIIk3EiIiIiIokwGSciIiIikgiTcSIiIiIiiTAZJyIiIiKSCJNxIiIiIiKJMBknIiIiIpIIk3EiIiIiIokwGSciIiIikgiTcSIiIiIiiTAZJyIiIiKSCJNxIiIiIiKJMBknIiIiIpIIk3EiIiIiIokwGSciIiIikgiTcSIiIiIiiTAZJyIiIiKSCJNxIiIiIiKJMBknIiIiIpIIk3EiIiIiIonIMhm/c+cOJk2ahO7duyMuLg4AsH//fly7dk3iyIiIiIhITmSXjB87dgxubm6IjIzErl27kJycDAC4dOkSpk6dKnF0RERERCQnskvGx48fj5kzZ+Lw4cPQ1tYWy5s2bYpTp05JGBkRERERyY3skvErV66gQ4cO2cotLS3x7NkzCSIiIiIiIrmSXTJuYmKCmJiYbOUXLlxAqVKlJIiIiIiIiORKdsl4t27dMG7cOMTGxkKhUEClUiEiIgKjR49Gnz59pA6PiIiIiGREdsn4rFmzUKlSJdjZ2SE5ORkuLi5o1KgR6tevj0mTJkkdHhERERHJSAmpAyhq2traWLNmDSZPnoyrV68iOTkZ1atXR/ny5aUOjYiIiIhkRnbJeKYyZcqgTJkyUodBRERERDImu2RcEATs2LEDR48eRVxcHFQqldryXbt2SRQZEREREcmN7JLxwMBArFq1Ck2aNIGVlRUUCoXUIRERERGRTMkuGd+0aRN27dqF1q1bSx0KEREREcmc7JJxY2NjlC1bVuowiIjgMD4kX/XvzfEqpEiIiEgqspvacNq0aZg+fTpev34tdShEREREJHOy6xnv0qULfv/9d1haWsLBwQFaWlpqy8+fPy9RZEREREQkN7JLxn18fHDu3Dn06tWLF3ASERERkaRkl4yHhITg4MGD+Oabb6QOhYiIiIhkTnZjxu3s7KBUKqUOg4iIiIhIfsn4/PnzMXbsWNy7d0/qUIiIiIhI5mQ3TKVXr1549eoVypUrB319/WwXcMbHx0sUGRERERHJjeyS8aCgIKlDICIiIiICIMNk3MfHR+oQiIiIiIgAyCQZT0pKEi/aTEpK+mBdXtxJREREREVFFsm4qakpYmJiYGlpCRMTkxznFhcEAQqFAhkZGRJESERERERyJItkPCwsDGZmZgCAo0ePShwNEREREdE7skjGGzduLP7t6OgIOzu7bL3jgiDgwYMHRR0aEREREcmY7OYZd3R0xNOnT7OVx8fHw9HRUYKIiIiIiEiuZJeMZ44Nf19ycjJ0dXXzvJ4VK1agSpUqUCqVUCqVcHd3x/79+8Xlb968gb+/P8zNzWFoaAhvb288efJEbR3R0dHw8vKCvr4+LC0tMWbMGLx9+1atTnh4OGrUqAEdHR04OTkhODg4fztMRERERMWWLIapAMDIkSMBAAqFApMnT4a+vr64LCMjA5GRkahWrVqe11e6dGnMmTMH5cuXhyAI2LhxI7777jtcuHABlStXxogRIxASEoLt27fD2NgYAQEB6NixIyIiIsRtenl5wdraGidOnEBMTAz69OkDLS0tzJo1CwAQFRUFLy8vDBo0CJs3b0ZoaCgGDBgAGxsbeHp6fr6DQ0RERESSUAiCIEgdRFFo0qQJAODYsWNwd3eHtra2uExbWxsODg4YPXo0ypcvX+BtmJmZ4eeff0anTp1gYWGBLVu2oFOnTgCAGzduwNnZGSdPnkS9evWwf/9+tGnTBo8fP4aVlRUAYOXKlRg3bhyePn0KbW1tjBs3DiEhIbh69aq4jW7duiEhIQEHDhzIc1xJSUkwNjZGYmIip278QrltdMtX/Ss+VwopEvqcHMaH5Kv+vTlehRQJERVHPH/Lg2x6xjNnUfH19cWiRYs++qF++PAhbG1toaHx8ZE8GRkZ2L59O1JSUuDu7o5z584hPT0dzZs3F+tUqlQJZcqUEZPxkydPws3NTUzEAcDT0xODBw/GtWvXUL16dZw8eVJtHZl1AgMDPxhPamoqUlNTxecfm1udiIiIiKQhuzHjGzZsyNN/ly4uLrh3794H61y5cgWGhobQ0dHBoEGDsHv3bri4uCA2Nhba2towMTFRq29lZYXY2FgAQGxsrFoinrk8c9mH6iQlJeH169e5xjV79mwYGxuLDzs7u4/uLxEREREVPdkl43mVl9E7FStWxMWLFxEZGYnBgwfDx8cH//33XxFE92ETJkxAYmKi+OCUjURERETFk2yGqRQGbW1tODk5AQBq1qyJM2fOYNGiRejatSvS0tKQkJCg1jv+5MkTWFtbAwCsra1x+vRptfVlzraStc77M7A8efIESqUSenp6ucalo6MDHR2dT94/IiIiIipc7Bn/jFQqFVJTU1GzZk1oaWkhNDRUXHbz5k1ER0fD3d0dAODu7o4rV64gLi5OrHP48GEolUq4uLiIdbKuI7NO5jqIiIiI6MvGnvECmjBhAlq1aoUyZcrg5cuX2LJlC8LDw3Hw4EEYGxujf//+GDlyJMzMzKBUKjF06FC4u7ujXr16AIAWLVrAxcUFvXv3xrx58xAbG4tJkybB399f7NUeNGgQli5dirFjx6Jfv34ICwvDtm3bEBKSvxkYiIiIiKh4YjKei5xuDJRVXFwc+vTpg5iYGBgbG6NKlSo4ePAgvv32WwDAwoULoaGhAW9vb6SmpsLT0xPLly8XX6+pqYl9+/Zh8ODBcHd3h4GBAXx8fDBjxgyxjqOjI0JCQjBixAgsWrQIpUuXxtq1aznHOBEREdFXQjbzjOeXkZERLl26hLJly0odyifjPKVfPs4z/nXiPONE9CE8f8sDe8Zz8d9//8HW1lbqMIiIiIjoKya7ZDwlJQVz5sxBaGgo4uLioFKp1JbfvXsXADg3NxEREREVOtkl4wMGDMCxY8fQu3dv2NjYfHRsOBERERFRYZFdMr5//36EhISgQYMGUodCRERERDInu3nGTU1NYWZmJnUYRERERETyS8Z//PFHTJkyBa9evZI6FCIiIiKSOdkNU5k/fz7u3LkDKysrODg4QEtLS235+fPnJYqMiIiIiORGdsl4+/btpQ6BiIiIiAiADJPxqVOnSh0CEREREREAGSbjmc6dO4fr168DACpXrozq1atLHBERERERyY3skvG4uDh069YN4eHhMDExAQAkJCSgSZMm2Lp1KywsLKQNkIiIiIhkQ3azqQwdOhQvX77EtWvXEB8fj/j4eFy9ehVJSUkYNmyY1OERERERkYzIrmf8wIEDOHLkCJydncUyFxcXLFu2DC1atJAwMiIiIiKSG9n1jKtUqmzTGQKAlpYWVCqVBBERERERkVzJLhlv2rQphg8fjsePH4tljx49wogRI9CsWTMJIyMiIiIiuZFdMr506VIkJSXBwcEB5cqVQ7ly5eDo6IikpCQsWbJE6vCIiIiISEZkN2bczs4O58+fx5EjR3Djxg0AgLOzM5o3by5xZEREREQkN7JLxgFAoVDg22+/xbfffit1KEREREQkY7JIxhcvXgw/Pz/o6upi8eLFH6zL6Q2JiIiIqKjIIhlfuHAhevbsCV1dXSxcuDDXegqFgsk4ERERERUZWSTjUVFROf5NRERERCQl2c2mMmPGDLx69Spb+evXrzFjxgwJIiIiIiIiuZJdMj59+nQkJydnK3/16hWmT58uQUREREREJFeyS8YFQYBCochWfunSJZiZmUkQERERERHJlSzGjAOAqakpFAoFFAoFKlSooJaQZ2RkIDk5GYMGDZIwQiIiIiKSG9kk40FBQRAEAf369cP06dNhbGwsLtPW1oaDgwPc3d0ljJCIiIiI5EY2ybiPjw8AwNHREfXr14eWlpbEERERERGR3MkmGc/UuHFj8e83b94gLS1NbblSqSzqkIiIiIhIpmR3AeerV68QEBAAS0tLGBgYwNTUVO1BRERERFRUZJeMjxkzBmFhYVixYgV0dHSwdu1aTJ8+Hba2tvj111+lDo+IiIiIZER2w1T++usv/Prrr/Dw8ICvry8aNmwIJycn2NvbY/PmzejZs6fUIRIRERGRTMiuZzw+Ph5ly5YF8G58eHx8PADgm2++wT///CNlaEREREQkM7JLxsuWLYuoqCgAQKVKlbBt2zYA73rMTUxMJIyMiIiIiORGdsm4r68vLl26BAAYP348li1bBl1dXYwYMQJjxoyRODoiIiIikhPZjRkfMWKE+Hfz5s1x48YNnDt3Dk5OTqhSpYqEkRERERGR3MguGX+fvb097O3tpQ6DiIiIiGRIdsNUhg0bhsWLF2crX7p0KQIDA4s+ICIiIiKSLdkl4zt37kSDBg2yldevXx87duyQICIiIiIikivZJePPnz+HsbFxtnKlUolnz55JEBERERERyZXsknEnJyccOHAgW/n+/fvF+ceJiIiIiIqC7C7gHDlyJAICAvD06VM0bdoUABAaGor58+cjKChI2uCIiIiISFZkl4z369cPqamp+Omnn/Djjz8CABwcHLBixQr06dNH4uiIiIiISE5kl4wDwODBgzF48GA8ffoUenp6MDQ0lDokIiIiIpIhWSbjmSwsLKQOgYiIiIhkTBbJeI0aNRAaGgpTU1NUr14dCoUi17rnz58vwsiIiIiISM5kkYx/99130NHRAQC0b99e2mCIiIiIiP4/WSTjpqam0NB4N4ujr68vSpcuLT4nIiIiIpKKLDLSkSNHIikpCQDg6OjIm/sQERERUbEgi55xW1tb7Ny5E61bt4YgCHj48CHevHmTY90yZcoUcXREREREJFeySMYnTZqEoUOHIiAgAAqFArVr185WRxAEKBQKZGRkSBAhEREREcmRLJJxPz8/dO/eHffv30eVKlVw5MgRmJubSx0WEREREcmcLJJxADAyMoKrqys2bNiABg0aiLOrEBERERFJRTbJeCYfHx+pQyAiIiIiAiCTZNzMzAy3bt1CyZIlYWpq+sGb/sTHxxdhZEREREQkZ7JIxhcuXAgjIyPx7w8l40RERERERUUWyXjWoSl9+/aVLhAiIiIioixkcdOfrM6fP48rV66Iz/fu3Yv27dvjhx9+QFpamoSREREREZHcyC4Z//7773Hr1i0AwN27d9G1a1fo6+tj+/btGDt2rMTREREREZGcyC4Zv3XrFqpVqwYA2L59Oxo3bowtW7YgODgYO3fulDY4IiIiIpIV2SXjgiBApVIBAI4cOYLWrVsDAOzs7PDs2TMpQyMiIiIimZFdMl6rVi3MnDkTmzZtwrFjx+Dl5QUAiIqKgpWVlcTREREREZGcyC4ZDwoKwvnz5xEQEICJEyfCyckJALBjxw7Ur18/z+uZPXs2ateuDSMjI1haWqJ9+/a4efOmWp03b97A398f5ubmMDQ0hLe3N548eaJWJzo6Gl5eXtDX14elpSXGjBmDt2/fqtUJDw9HjRo1oKOjAycnJwQHBxds54mIiIioWJHF1IZZValSRW02lUw///wzNDU187yeY8eOwd/fH7Vr18bbt2/xww8/oEWLFvjvv/9gYGAAABgxYgRCQkKwfft2GBsbIyAgAB07dkRERAQAICMjA15eXrC2tsaJEycQExODPn36QEtLC7NmzQLwrsfey8sLgwYNwubNmxEaGooBAwbAxsYGnp6en+GIEBEREZFUFIIgCFIHUZQePHgAhUKB0qVLAwBOnz6NLVu2wMXFBX5+fgVe79OnT2FpaYljx46hUaNGSExMhIWFBbZs2YJOnToBAG7cuAFnZ2ecPHkS9erVw/79+9GmTRs8fvxYHCKzcuVKjBs3Dk+fPoW2tjbGjRuHkJAQXL16VdxWt27dkJCQgAMHDuQptqSkJBgbGyMxMRFKpbLA+0jScdvolq/6V3yy/8NJxY/D+JB81b83x6uQIiGi4ojnb3mQ3TCVHj164OjRowCA2NhYfPvttzh9+jQmTpyIGTNmFHi9iYmJAAAzMzMAwLlz55Ceno7mzZuLdSpVqoQyZcrg5MmTAICTJ0/Czc1Nbay6p6cnkpKScO3aNbFO1nVk1slcR05SU1ORlJSk9iAiIiKi4kd2yfjVq1dRp04dAMC2bdvg6uqKEydOYPPmzQUei61SqRAYGIgGDRrA1dUVwLtEX1tbGyYmJmp1raysEBsbK9Z5/6LRzOcfq5OUlITXr1/nGM/s2bNhbGwsPuzs7Aq0X0RERERUuGSXjKenp0NHRwfAu6kN27VrB+Bdr3VMTEyB1unv74+rV69i69atny3OTzFhwgQkJiaKjwcPHkgdEhERERHlQHbJeOXKlbFy5UocP34chw8fRsuWLQEAjx8/hrm5eb7XFxAQgH379uHo0aPiOHQAsLa2RlpaGhISEtTqP3nyBNbW1mKd92dXyXz+sTpKpRJ6eno5xqSjowOlUqn2ICIiIqLiR3bJ+Ny5c7Fq1Sp4eHige/fuqFq1KgDgzz//FIev5IUgCAgICMDu3bsRFhYGR0dHteU1a9aElpYWQkNDxbKbN28iOjoa7u7uAAB3d3dcuXIFcXFxYp3Dhw9DqVTCxcVFrJN1HZl1MtdBRERERF8u2U1t6OHhgWfPniEpKQmmpqZiuZ+fH/T19fO8Hn9/f2zZsgV79+6FkZGROMbb2NgYenp6MDY2Rv/+/TFy5EiYmZlBqVRi6NChcHd3R7169QAALVq0gIuLC3r37o158+YhNjYWkyZNgr+/vziUZtCgQVi6dCnGjh2Lfv36ISwsDNu2bUNISP5mYSAiIiKi4kd2yTgAaGpqqiXiAODg4JCvdaxYsQLAu+Q+qw0bNqBv374AgIULF0JDQwPe3t5ITU2Fp6cnli9frhbHvn37MHjwYLi7u8PAwAA+Pj5qs7o4OjoiJCQEI0aMwKJFi1C6dGmsXbuWc4wTERERfQVkN8848O5um9u2bUN0dDTS0tLUlp0/f16iqAoP5yn98nGe8a8T5xknog/h+VseZDdmfPHixfD19YWVlRUuXLiAOnXqwNzcHHfv3kWrVq2kDo+IiIiIZER2yfjy5cuxevVqLFmyBNra2hg7diwOHz6MYcOGiTfuISIiIiIqCrJLxqOjo1G/fn0AgJ6eHl6+fAkA6N27N37//XcpQyMiIiIimZFdMm5tbY34+HgAQJkyZXDq1CkAQFRUFGQ4fJ6IiIiIJCS7ZLxp06b4888/AQC+vr4YMWIEvv32W3Tt2hUdOnSQODoiIiIikhPZTW24evVqqFQqAO/mCjc3N8eJEyfQrl07fP/99xJHR0RERERyIrtkXENDAxoa//eDQLdu3dCtWzcJIyIiIiIiuZJFMn758uU8161SpUohRkJERERE9H9kkYxXq1YNCoXioxdoKhQKZGRkFFFURERERCR3skjGo6KipA6BiIiIiCgbWSTj9vb24t+zZ8+GlZUV+vXrp1Zn/fr1ePr0KcaNG1fU4RERERGRTMkiGc9q1apV2LJlS7byypUro1u3bkzGqUAcxofkq/69OV6FFAkRERF9SWQ3z3hsbCxsbGyylVtYWCAmJkaCiIiIiIhIrmSXjNvZ2SEiIiJbeUREBGxtbSWIiIiIiIjkSnbDVAYOHIjAwECkp6ejadOmAIDQ0FCMHTsWo0aNkjg6IiIiIpIT2SXjY8aMwfPnzzFkyBCkpaUBAHR1dTFu3DhMmDBB4uiIiIiISE5kl4wrFArMnTsXkydPxvXr16Gnp4fy5ctDR0dH6tCIiIiISGZkl4xnMjQ0RO3ataUOg4iIiIhkTHYXcBIRERERFRdMxomIiIiIJMJknIiIiIhIIkzGiYiIiIgkwmSciIiIiEgiTMaJiIiIiCTCZJyIiIiISCJMxomIiIiIJMJknIiIiIhIIkzGiYiIiIgkwmSciIiIiEgiTMaJiIiIiCTCZJyIiIiISCJMxomIiIiIJMJknIiIiIhIIkzGiYiIiIgkwmSciIiIiEgiTMaJiIiIiCTCZJyIiIiISCJMxomIiIiIJMJknIiIiIhIIkzGiYiIiIgkwmSciIiIiEgiTMaJiIiIiCTCZJyIiIiISCJMxomIiIiIJMJknIiIiIhIIkzGiYiIiIgkwmSciIiIiEgiTMaJiIiIiCTCZJyIiIiISCJMxomIiIiIJMJknIiIiIhIIkzGiYiIiIgkwmSciIiIiEgiTMaJiIiIiCTCZJyIiIiISCJMxomIiIiIJMJknIiIiIhIIkzGiYiIiIgkUkLqAIiIqHC4bXTLc90rPlcKMRIiIsoNe8aJiIiIiCTCZJyIiIiISCJMxgvon3/+Qdu2bWFrawuFQoE9e/aoLRcEAVOmTIGNjQ309PTQvHlz3L59W61OfHw8evbsCaVSCRMTE/Tv3x/JyclqdS5fvoyGDRtCV1cXdnZ2mDdvXmHvGhEREREVESbjBZSSkoKqVati2bJlOS6fN28eFi9ejJUrVyIyMhIGBgbw9PTEmzdvxDo9e/bEtWvXcPjwYezbtw///PMP/Pz8xOVJSUlo0aIF7O3tce7cOfz888+YNm0aVq9eXej7R0RERESFjxdwFlCrVq3QqlWrHJcJgoCgoCBMmjQJ3333HQDg119/hZWVFfbs2YNu3brh+vXrOHDgAM6cOYNatWoBAJYsWYLWrVvjl19+ga2tLTZv3oy0tDSsX78e2traqFy5Mi5evIgFCxaoJe1ERERE9GViz3ghiIqKQmxsLJo3by6WGRsbo27dujh58iQA4OTJkzAxMRETcQBo3rw5NDQ0EBkZKdZp1KgRtLW1xTqenp64efMmXrx4kev2U1NTkZSUpPYgIiIiouKHyXghiI2NBQBYWVmplVtZWYnLYmNjYWlpqba8RIkSMDMzU6uT0zqybiMns2fPhrGxsfiws7P7tB0iIiIiokLBZPwrNGHCBCQmJoqPBw8eSB0SEREREeWAyXghsLa2BgA8efJErfzJkyfiMmtra8TFxaktf/v2LeLj49Xq5LSOrNvIiY6ODpRKpdqDiIiIiIofJuOFwNHREdbW1ggNDRXLkpKSEBkZCXd3dwCAu7s7EhIScO7cObFOWFgYVCoV6tatK9b5559/kJ6eLtY5fPgwKlasCFNT0yLaGyIiIiIqLEzGCyg5ORkXL17ExYsXAby7aPPixYuIjo6GQqFAYGAgZs6ciT///BNXrlxBnz59YGtri/bt2wMAnJ2d0bJlSwwcOBCnT59GREQEAgIC0K1bN9ja2gIAevToAW1tbfTv3x/Xrl3DH3/8gUWLFmHkyJES7TURERERfU6c2rCAzp49iyZNmojPMxNkHx8fBAcHY+zYsUhJSYGfnx8SEhLwzTff4MCBA9DV1RVfs3nzZgQEBKBZs2bQ0NCAt7c3Fi9eLC43NjbGoUOH4O/vj5o1a6JkyZKYMmUKpzX8Gkwzzl99xzKFEwcRERFJisl4AXl4eEAQhFyXKxQKzJgxAzNmzMi1jpmZGbZs2fLB7VSpUgXHjx8vcJxEREREVHxxmAoRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRREpIHQAREeXRNOP81XcsUzhxEBHRZ8OecSIiIiIiiTAZJyIiIiKSCJNxIiIiIiKJMBknIiIiIpIIk3EiIiIiIolwNhUiIvry5WemmWmJhRcHEVE+MRmnwpPfadh4giQiIiKZYTJORMVLPv+Jc8vnXNpXfK7kqz7RF42dIrnjsaFigsn4F2LZsmX4+eefERsbi6pVq2LJkiWoU6eO1GERERUKh/Eh+ap/T7eQAiEiKmRMxr8Af/zxB0aOHImVK1eibt26CAoKgqenJ27evAlLS0upwyP6ICZVRFRU8tPesK2h4oLJ+BdgwYIFGDhwIHx9fQEAK1euREhICNavX4/x48cXWRxMqoiIiIg+LybjxVxaWhrOnTuHCRMmiGUaGhpo3rw5Tp48meNrUlNTkZqaKj5PTHw3zi0pKemTYlGlvspX/SSFkL8NfGJ8UirsY5PxOiN/6y9Gx5LHJnfF6dgUp+MCFPKxKWb7ml+uUw/mue5VXfm0w0D+Pjdfwjkq83spCPmMlb4oCoHvcLH2+PFjlCpVCidOnIC7u7tYPnbsWBw7dgyRkZHZXjNt2jRMnz69KMMkIiKiQvLgwQOULl1a6jCokLBn/Cs0YcIEjBw5UnyuUqkQHx8Pc3NzKBQKCSN791++nZ0dHjx4AKVSKWksxQ2PTe54bHLHY5M7Hpuc8bjkrrgdG0EQ8PLlS9ja2kodChUiJuPFXMmSJaGpqYknT56olT958gTW1tY5vkZHRwc6OjpqZSYmJoUVYoEolcpi0dAVRzw2ueOxyR2PTe54bHLG45K74nRsjI3zOQUjfXE0pA6APkxbWxs1a9ZEaGioWKZSqRAaGqo2bIWIiIiIvjzsGf8CjBw5Ej4+PqhVqxbq1KmDoKAgpKSkiLOrEBEREdGXicn4F6Br1654+vQppkyZgtjYWFSrVg0HDhyAlZWV1KHlm46ODqZOnZptGA3x2HwIj03ueGxyx2OTMx6X3PHYkBQ4mwoRERERkUQ4ZpyIiIiISCJMxomIiIiIJMJknIiIiIhIIkzGiYiIiIgkwmSciIiIiEgiTMaJiok3b95IHQIREREVMSbjVCTevn2LI0eOYNWqVXj58iUA4PHjx0hOTpY4MmmpVCr8+OOPKFWqFAwNDXH37l0AwOTJk7Fu3TqJoyOir0lCQgLWrl2LCRMmID4+HgBw/vx5PHr0SOLIpHfnzh1MmjQJ3bt3R1xcHABg//79uHbtmsSRkRwwGadCd//+fbi5ueG7776Dv78/nj59CgCYO3cuRo8eLXF00po5cyaCg4Mxb948aGtri+Wurq5Yu3athJERfblu3ryJgIAANGvWDM2aNUNAQABu3rwpdViSunz5MipUqIC5c+fil19+QUJCAgBg165dmDBhgrTBSezYsWNwc3NDZGQkdu3aJXYSXbp0CVOnTpU4OpIDJuNU6IYPH45atWrhxYsX0NPTE8s7dOiA0NBQCSOT3q+//orVq1ejZ8+e0NTUFMurVq2KGzduSBhZ8bBjxw506dIF9erVQ40aNdQecpaSkoLJkyejfv36cHJyQtmyZdUecrZz5064urri3LlzqFq1KqpWrYrz58/D1dUVO3fulDo8yYwcORJ9+/bF7du3oaurK5a3bt0a//zzj4SRSW/8+PGYOXMmDh8+rNYp0rRpU5w6dUrCyEguSkgdAH39jh8/jhMnTqg1cgDg4OAg+59HHz16BCcnp2zlKpUK6enpEkRUfCxevBgTJ05E3759sXfvXvj6+uLOnTs4c+YM/P39pQ5PUgMGDMCxY8fQu3dv2NjYQKFQSB1SsTF27FhMmDABM2bMUCufOnUqxo4dC29vb4kik9aZM2ewatWqbOWlSpVCbGysBBEVH1euXMGWLVuylVtaWuLZs2cSRERyw2ScCp1KpUJGRka28ocPH8LIyEiCiIoPFxcXHD9+HPb29mrlO3bsQPXq1SWKqnhYvnw5Vq9eje7duyM4OBhjx45F2bJlMWXKFHG8q1zt378fISEhaNCggdShFDsxMTHo06dPtvJevXrh559/liCi4kFHRwdJSUnZym/dugULCwsJIio+TExMEBMTA0dHR7XyCxcuoFSpUhJFRXLCYSpU6Fq0aIGgoCDxuUKhQHJyMqZOnYrWrVtLF1gxMGXKFAQEBGDu3LlQqVTYtWsXBg4ciJ9++glTpkyROjxJRUdHo379+gAAPT098cLf3r174/fff5cyNMmZmprCzMxM6jCKJQ8PDxw/fjxb+b///ouGDRtKEFHx0K5dO8yYMUP8xU2hUCA6Ohrjxo2T7a8Fmbp164Zx48YhNjYWCoUCKpUKERERGD16dI7/2BF9bgpBEASpg6Cv28OHD+Hp6QlBEHD79m3UqlULt2/fhrm5OY4fPw5LS0upQ5TU8ePHMWPGDFy6dAnJycmoUaMGpkyZghYtWkgdmqTKli2LnTt3onr16qhVqxYGDhyI77//HocOHUK3bt1k3Tv+22+/Ye/evdi4cSP09fWlDqdYWblyJaZMmSJeawAAp06dwvbt2zF9+nTY2tqKddu1aydVmEUuMTERnTp1wtmzZ/Hy5UvY2toiNjYW7u7u+Pvvv2FgYCB1iJJJS0uDv78/goODkZGRgRIlSiAjIwM9evRAcHCw2vU8RIWByTgVibdv3+KPP/5QSzh79uypdkEnUVYDBgyAnZ0dpk6dimXLlmHMmDFo0KABzp49i44dO8p66sfq1avjzp07EAQBDg4O0NLSUlt+/vx5iSKTnoZG3n7wVSgUOQ6f+9r9+++/uHz5stgON2/eXOqQio0HDx7gypUrSE5ORvXq1VG+fHmpQyKZYDJORea///5DdHQ00tLS1Mrl1DtFeadSqaBSqVCixLtLW7Zu3YoTJ06gfPny+P7777NdECwn06dP/+ByuU7Hlp6eDk9PT6xcuRIVKlSQOhwiojxhMk6F7u7du+jQoQOuXLkChUIBQRDUZn+QY+9UpoyMDCxcuBDbtm3L8R8VOQ/FICoICwsLnDx5MsdZiuQuJSUFx44dy7GtGTZsmERRFQ8PHz7En3/+meOxWbBggURRkVxwNhUqdMOHD4ejoyNCQ0Ph6OiIyMhIxMfHY9SoUfjll1+kDk9S06dPx9q1azFq1ChMmjQJEydOxL1797Bnzx7ZX8C5YcMGGBoaonPnzmrl27dvx6tXr+Dj4yNRZFSc9erVC2vXrsWcOXOkDqVYuXDhAlq3bo1Xr14hJSUFZmZmePbsGfT19WFpaSnrZDw0NBTt2rVD2bJlcePGDbi6uuLevXsQBEH29zSgosGecSp0JUuWRFhYGKpUqQJjY2OcPn0aFStWRFhYGEaNGoULFy5IHaJkypUrh8WLF8PLywtGRka4ePGiWHbq1Kkc576ViwoVKmDVqlVo0qSJWvmxY8fg5+cnuzsqmpmZ4datWyhZsiRMTU0/OLe4nH9RGTp0KH799VeUL18eNWvWzHZholx7OT08PFChQgWsXLkSxsbGuHTpErS0tNCrVy8MHz4cHTt2lDpEydSpUwetWrXC9OnTYWRkhEuXLsHS0hI9e/ZEy5YtMXjwYKlDpK8ce8ap0GVkZIjziZcsWRKPHz9GxYoVYW9vL7uE6n2xsbFwc3MDABgaGiIxMREA0KZNG0yePFnK0CQXHR2dbd5fALC3t0d0dLQEEUlr4cKF4vco61ShpO7q1atib+atW7fUlsn55kgXL17EqlWroKGhAU1NTaSmpqJs2bKYN28efHx8ZJ2MX79+XZwutUSJEnj9+jUMDQ0xY8YMfPfdd0zGqdAxGadC5+rqikuXLsHR0RF169bFvHnzoK2tjdWrV8v+1t2lS5dGTEwMypQpg3LlyuHQoUOoUaMGzpw5Ax0dHanDk5SlpSUuX74MBwcHtfJLly7B3NxcmqAklHVYDofo5O7o0aNSh1AsaWlpiTPNWFpaIjo6Gs7OzjA2NsaDBw8kjk5aBgYG4jhxGxsb3LlzB5UrVwYA3oGTigSTcSp0kyZNQkpKCgBgxowZaNOmDRo2bAhzc3P88ccfEkcnrQ4dOiA0NBR169bF0KFD0atXL6xbtw7R0dEYMWKE1OFJqnv37hg2bBiMjIzQqFEjAO+GqAwfPhzdunWTOLri482bN9kuOFMqlRJFQ8VV9erVcebMGZQvXx6NGzfGlClT8OzZM2zatAmurq5ShyepevXq4d9//4WzszNat26NUaNG4cqVK9i1a5c4Vz1RYeKYcZJEfHz8R8e9ytGpU6fE6fvatm0rdTiSSktLQ+/evbF9+3ZxesOMjAz4+Phg5cqVsp7aMCUlBePGjcO2bdvw/PnzbMvlPEMR5SzzZj9NmjRBXFwc+vTpI7Y169evR9WqVaUOUTJ3795FcnIyqlSpgpSUFIwaNUo8NgsWLIC9vb3UIdJXjsk4kYRmz54NKysr9OvXT618/fr1ePr0KcaNGydRZMXH7du3ceHCBejp6aFKlSo8MQLw9/fH0aNH8eOPP6J3795YtmwZHj16hFWrVmHOnDno2bOn1CHSFyoiIgK1atWS1TC5AQMGoFevXvDw8JA6FJKpvN2qjIgKxapVq1CpUqVs5ZUrV8bKlSsliKh4WbduHTp06IDevXujU6dO8PLywtq1a6UOS3J//fUXli9fDm9vb5QoUQINGzbEpEmTMGvWLGzevFnq8OgL1qpVKzx69EjqMIrU06dP0bJlS9jZ2WHMmDG4dOmS1CGRzDAZJ5JQbGwsbGxsspVbWFggJiZGgoiKjylTpmD48OFo27Yttm/fju3bt6Nt27YYMWKE7Odgj4+PFy9+ViqV4lSG33zzDf755x8pQ6MvnBx/LN+7dy9iYmIwefJknDlzBjVq1EDlypUxa9Ys3Lt3T+rwSAaYjBNJyM7ODhEREdnKIyIiYGtrK0FExceKFSuwZs0azJ49G+3atUO7du0we/ZsrF69GsuXL5c6PEmVLVsWUVFRAIBKlSph27ZtAN71mJuYmEgYGdGXydTUFH5+fggPD8f9+/fRt29fbNq0iXdypSLB2VSIJDRw4EAEBgYiPT0dTZs2BfDubnBjx47FqFGjJI5OWunp6ahVq1a28po1a+Lt27cSRFR8+Pr64tKlS2jcuDHGjx+Ptm3bYunSpUhPT5ftTW2IPof09HScPXsWkZGRuHfvHqysrKQOiWSAF3ASSUgQBIwfPx6LFy8Wp6fT1dXFuHHjZD8UY+jQodDS0sqWXI4ePRqvX7/GsmXLJIqs+Ll//z7OnTsHJycnVKlSRepw6AuWeQdKud0D4ujRo9iyZQt27twJlUqFjh07omfPnmjatCln/aJCx2ScqBhITk7G9evXoaenh/Lly8tqJoPcZN7W3M7OTpzrNzIyEtHR0ejTpw+0tLTEuuwNJvo8lEolLl68KKtkvFSpUoiPj0fLli3Rs2dPtG3blm0wFSkm40RULDVp0iRP9RQKBcLCwgo5GuktXrw4z3WHDRtWiJHQ10yOPeNr1qxB586deb0FSYbJOBHRF8DR0TFP9RQKBe7evVvI0dCXZv369WjSpEmeP0dEVHSYjBMRfcEym3COa6UPKV++PO7evYtSpUqhcePGaNy4MTw8PDhbCFExwKkNiYi+QOvWrYOrqyt0dXWhq6sLV1dX3hCJcnX79m1ER0dj9uzZ0NfXxy+//IKKFSuidOnS6NWrl9ThEckae8aJiL4wU6ZMwYIFCzB06FC4u7sDAE6ePImlS5dixIgRmDFjhsQRUnH26tUrHD9+HL///js2b94MQRBkP10okZSYjBMRfWEsLCywePFidO/eXa38999/x9ChQ/Hs2TOJIqPi6tChQwgPD0d4eDguXLgAZ2dncahKo0aNYGpqKnWIRLLFm/4QEX1heEMkyq+WLVvCwsICo0aNwt9//82ZQ4iKEfaMExF9YXhDJMqvoKAg/PPPP/jnn3+go6Mj9op7eHigQoUKUodHJGtMxomIvjC8IRJ9iitXruDYsWMICwvDvn37YGlpiYcPH0odFpFscZgKEdEX5urVq6hRowYA4M6dOwCAkiVLomTJkrh69apYj9MdUlaCIODChQsIDw/H0aNH8e+//0KlUsHCwkLq0IhkjT3jREREX7m2bdsiIiICSUlJqFq1Kjw8PNC4cWM0atSI48eJJMaecSIioq9cpUqV8P3336Nhw4YwNjaWOhwiyoI940REREREEuEdOImIiGTg2LFjaNu2LZycnODk5IR27drh+PHjUodFJHtMxomIiL5yv/32G5o3bw59fX0MGzYMw4YNg56eHpo1a4YtW7ZIHR6RrHGYChER0VfO2dkZfn5+GDFihFr5ggULsGbNGly/fl2iyIiIyTgREdFXTkdHB9euXYOTk5Na+f/+9z+4urrizZs3EkVGRBymQkRE9JWzs7NDaGhotvIjR47Azs5OgoiIKBOnNiQiIvrKjRo1CsOGDcPFixdRv359AEBERASCg4OxaNEiiaMjkjcOUyEiIpKB3bt3Y/78+eL4cGdnZ4wZMwbfffedxJERyRuTcSIiIiIiiXDMOBERERGRRDhmnIiI6CtkamoKhUKRp7rx8fGFHA0R5YbJOBER0VcoKChI6hCIKA84ZpyIiIiISCLsGSciIvoKJSUl5bmuUqksxEiI6EPYM05ERPQV0tDQ+OiYcUEQoFAokJGRUURREdH72DNORET0FTp69KjUIRBRHrBnnIiIiIhIIuwZJyIikolXr14hOjoaaWlpauVVqlSRKCIiYjJORET0lXv69Cl8fX2xf//+HJdzzDiRdHgHTiIioq9cYGAgEhISEBkZCT09PRw4cAAbN25E+fLl8eeff0odHpGssWeciIjoKxcWFoa9e/eiVq1a0NDQgL29Pb799lsolUrMnj0bXl5eUodIJFvsGSciIvrKpaSkwNLSEgBgamqKp0+fAgDc3Nxw/vx5KUMjkj0m40RERF+5ihUr4ubNmwCAqlWrYtWqVXj06BFWrlwJGxsbiaMjkjdObUhERPSV++233/D27Vv07dsX586dQ8uWLREfHw9tbW0EBweja9euUodIJFtMxomIiGTm1atXuHHjBsqUKYOSJUtKHQ6RrDEZJyIiIiKSCGdTISIi+splZGQgODgYoaGhiIuLg0qlUlseFhYmUWRExGSciIjoKzd8+HAEBwfDy8sLrq6uUCgUUodERP8fh6kQERF95UqWLIlff/0VrVu3ljoUInoPpzYkIiL6ymlra8PJyUnqMIgoB0zGiYiIvnKjRo3CokWLwB/DiYofDlMhIiL6CnXs2FHteVhYGMzMzFC5cmVoaWmpLdu1a1dRhkZEWfACTiIioq+QsbGx2vMOHTpIFAkRfQh7xomIiL5yr1+/hkqlgoGBAQDg3r172LNnD5ydneHp6SlxdETyxjHjREREX7nvvvsOmzZtAgAkJCSgXr16mD9/Ptq3b48VK1ZIHB2RvDEZJyIi+sqdP38eDRs2BADs2LEDVlZWuH//Pn799VcsXrxY4uiI5I3JOBER0Vfu1atXMDIyAgAcOnQIHTt2hIaGBurVq4f79+9LHB2RvDEZJyIi+so5OTlhz549ePDgAQ4ePIgWLVoAAOLi4qBUKiWOjkjemIwTERF95aZMmYLRo0fDwcEBdevWhbu7O4B3veTVq1eXODoieeNsKkRERDIQGxuLmJgYVK1aFRoa7/riTp8+DaVSiUqVKkkcHZF8MRknIiIiIpIIh6kQEREREUmEyTgRERERkUSYjBMRERERSYTJOBHRV8jDwwOBgYF5rh8cHAwTE5NCi4eIiHLGZJyIiIiISCJMxomIiIiIJMJknIioCHl4eGDo0KEIDAyEqakprKyssGbNGqSkpMDX1xdGRkZwcnLC/v37xdccO3YMderUgY6ODmxsbDB+/Hi8fftWXJ6SkoI+ffrA0NAQNjY2mD9/frbtpqamYvTo0ShVqhQMDAxQt25dhIeHF8UuExHRBzAZJyIqYhs3bkTJkiVx+vRpDB06FIMHD0bnzp1Rv359nD9/Hi1atEDv3r3x6tUrPHr0CK1bt0bt2rVx6dIlrFixAuvWrcPMmTPF9Y0ZMwbHjh3D3r17cejQIYSHh+P8+fNq2wwICMDJkyexdetWXL58GZ07d0bLli1x+/btot59IiLKgjf9ISIqQh4eHsjIyMDx48cBABkZGTA2NkbHjh3x66+/Anh3p0QbGxucPHkSf/31F3bu3Inr169DoVAAAJYvX45x48YhMTERr169grm5OX777Td07twZABAfH4/SpUvDz88PQUFBiI6ORtmyZREdHQ1bW1sxlubNm6NOnTqYNWsWgoODERgYiISEhKI9IEREMldC6gCIiOSmSpUq4t+ampowNzeHm5ubWGZlZQUAiIuLw/Xr1+Hu7i4m4gDQoEEDJCcn4+HDh3jx4gXS0tJQt25dcbmZmRkqVqwoPr9y5QoyMjJQoUIFtThSU1Nhbm7+2fePiIjyjsk4EVER09LSUnuuUCjUyjITb5VK9Vm2l5ycDE1NTZw7dw6amppqywwNDT/LNoiIqGCYjBMRFWPOzs7YuXMnBEEQk/SIiAgYGRmhdOnSMDMzg5aWFiIjI1GmTBkAwIsXL3Dr1i00btwYAFC9enVkZGQgLi4ODRs2lGxfiIgoO17ASURUjA0ZMgQPHjzA0KFDcePGDezduxdTp07FyJEjoaGhAUNDQ/Tv3x9jxoxBWFgYrl69ir59+0JD4/+a9woVKqBnz57o06cPdu3ahaioKJw+fRqzZ89GSEiIhHtHRETsGSciKsZKlSqFv//+G2PGjEHVqlVhZmaG/v37Y9KkSWKdn3/+GcnJyWjbti2MjIwwatQoJCYmqq1nw4YNmDlzJkaNGoVHjx6hZMmSqFevHtq0aVPUu0RERFlwNhUiIiIiIolwmAoRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUmEyTgRERERkUSYjBMRERERSYTJOBERERGRRJiMExERERFJhMk4EREREZFEmIwTEREREUnk/wEM7NWeZF+uGAAAAABJRU5ErkJggg==","text/plain":["<Figure size 640x480 with 1 Axes>"]},"metadata":{},"output_type":"display_data"}],"source":["df.groupby(['model', 'classification_method'])['classification_time'].mean().unstack().plot(kind='bar')\n","plt.ylabel('classification_time')\n","plt.title('Average grid search classification_time for each model and each classification method')\n","plt.show()"]},{"cell_type":"code","execution_count":20,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":608},"executionInfo":{"elapsed":297,"status":"ok","timestamp":1715020055135,"user":{"displayName":"Matthew Trueblood","userId":"14999859395916803982"},"user_tz":420},"id":"xEdCFIOtO7bU","outputId":"aba44ed2-a83c-46f1-823c-547a9643a048"},"outputs":[{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAjcAAAJPCAYAAACAWfUCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB7qklEQVR4nO3deXxM1/8/8NckssoqKxGJJSS22Iktggoqttp9JWIvqqTWKqG0iD0otQa1a2snNBL7voZaIkTUFoJsSJi5vz/8Mp9OM4mEydyZm9fz8ZhHO+feO/O+10zyyr3nniMTBEEAERERkUQYiF0AERERkSYx3BAREZGkMNwQERGRpDDcEBERkaQw3BAREZGkMNwQERGRpDDcEBERkaQw3BAREZGkMNwQERGRpDDcEGnQ+/fvMXbsWLi6usLAwAAdO3YUuySNi4iIgEwmw/nz58Uu5ZPExMRAJpMhJiamwNtm73tCQoLG6/pUMpkMU6ZMEbsMtZo1a4aqVatq/HV1eZ9JNzDckPIH9r8fjo6O8PPzw/79+wvtfV+/fo0pU6Z80i+Zffv2QSaToVSpUlAoFGrXkclkGD58uNpl27dvz/UXXExMDDp37gxnZ2cYGxvD0dERAQEB+OOPPz5a1+rVqzF79mx06dIFa9euxahRowq0X0RUeP7++29MmTJFp8Lpf508eRJTpkzBq1evxC5FrzHckNKPP/6I9evXY926dRg7diyePXuGtm3bYs+ePYXyfq9fv8bUqVM/Kdxs2LAB7u7uePz4MQ4fPqyxmkJDQ+Hn54dr165h8ODBWLZsGcaMGYP09HR89dVX2LhxY57bHz58GC4uLpg/fz769OkDX19fjdVGRJ/n77//xtSpU3U+3EydOpXh5jMVE7sA0h1t2rRBnTp1lM/79+8PJycnbNq0Ce3atROxMlUZGRnYuXMnZsyYgTVr1mDDhg1o2bLlZ7/u9u3b8eOPP6JLly7YuHEjjIyMlMvGjBmDyMhIvHv3Ls/XSEpKgo2NzWfXkk2hUCArKwumpqYae00iIqnjmRvKlY2NDczMzFCsmGoGVigUWLBgAapUqQJTU1M4OTlh8ODBePnypcp658+fh7+/P+zt7WFmZoayZcuiX79+AICEhAQ4ODgAAKZOnaq8HJaf6+h//vkn3rx5g65du6JHjx74448/8Pbt28/e30mTJqFEiRJYvXq1SrDJ5u/vn2vIS0hIgEwmQ3R0NK5fv67cn+yzUhkZGfjuu+/g6uoKExMTVKpUCXPmzIEgCCqvk30pbcOGDahSpQpMTExw4MCBPOvev38/mjRpguLFi8PS0hJffvklrl+/rrLO1atX0bdvX5QrVw6mpqZwdnZGv379kJycnOP1Hj58iP79+6NUqVIwMTFB2bJl8fXXXyMrK0tlvczMTISEhMDBwQHFixdHp06d8OzZszxrBYC+ffvCwsICiYmJaNeuHSwsLODi4oIlS5YAAGJjY9G8eXMUL14cbm5uas+W3b17F127dkWJEiVgbm6OBg0aYO/evTnW++eff9CxY0cUL14cjo6OGDVqFDIzM9XWdebMGbRu3RrW1tYwNzeHr68vTpw48dH9USe/x3vKlCmQyWS4c+cO+vbtCxsbG1hbWyM4OBivX79WWTczMxOjRo2Cg4MDLC0t0b59e/zzzz/5rikzMxOhoaGoUKECTExM4OrqirFjx+Y4HmvWrEHz5s3h6OgIExMTVK5cGUuXLlX7mvv374evry8sLS1hZWWFunXrqv33+vvvv+Hn5wdzc3O4uLggLCws3zXnZ5/v37+PoUOHolKlSjAzM4OdnR26du2qcoYmIiICXbt2BQD4+fnl+I7u3LkTX375pfJzX758eUybNg1yufyjdaalpWHkyJFwd3eHiYkJHB0d8cUXX+DixYsq633sMzZlyhSMGTMGAFC2bFlljbp8pklX8cwNKaWkpOD58+cQBAFJSUlYtGgR0tPT8X//938q6w0ePBgREREIDg7GiBEjcO/ePSxevBiXLl3CiRMnYGRkhKSkJLRq1QoODg4YP348bGxskJCQoOy34uDggKVLl+Lrr79Gp06d0LlzZwBA9erVP1rnhg0b4OfnB2dnZ/To0QPjx4/H7t27lT+4PkVcXBxu3ryJfv36wdLSssDbOzg4YP369fjpp5+Qnp6OGTNmAAC8vLwgCALat2+P6Oho9O/fHzVq1EBkZCTGjBmDhw8fYv78+SqvdfjwYWzduhXDhw+Hvb093N3dc33f9evXIygoCP7+/pg1axZev36NpUuXonHjxrh06ZJy20OHDuHu3bsIDg6Gs7Mzrl+/juXLl+P69es4ffo0ZDIZAODRo0eoV68eXr16hUGDBsHT0xMPHz7E9u3b8fr1axgbGyvf+5tvvoGtrS1CQ0ORkJCABQsWYPjw4diyZctHj5dcLkebNm3QtGlThIWFYcOGDRg+fDiKFy+OiRMnonfv3ujcuTOWLVuGwMBA+Pj4oGzZsgCAp0+fomHDhnj9+jVGjBgBOzs7rF27Fu3bt8f27dvRqVMnAMCbN2/QokULJCYmYsSIEShVqhTWr1+v9jLm4cOH0aZNG9SuXRuhoaEwMDBQ/pI/duwY6tWr99F9+rf8Hu9s3bp1Q9myZTFjxgxcvHgRK1euhKOjI2bNmqVcZ8CAAfjtt9/Qq1cvNGzYEIcPH8aXX36Zr3oUCgXat2+P48ePY9CgQfDy8kJsbCzmz5+P27dvY8eOHcp1ly5diipVqqB9+/YoVqwYdu/ejaFDh0KhUGDYsGHK9SIiItCvXz9UqVIFEyZMgI2NDS5duoQDBw6gV69eyvVevnyJ1q1bo3PnzujWrRu2b9+OcePGoVq1amjTpk2eded3n8+dO4eTJ0+iR48eKF26NBISErB06VI0a9YMf//9N8zNzdG0aVOMGDEC4eHh+P777+Hl5QUAyv9GRETAwsICISEhsLCwwOHDhzF58mSkpqZi9uzZedY5ZMgQbN++HcOHD0flypWRnJyM48eP48aNG6hVqxaA/H3GOnfujNu3b2PTpk2YP38+7O3tAUD5hyAVgEBF3po1awQAOR4mJiZCRESEyrrHjh0TAAgbNmxQaT9w4IBK+59//ikAEM6dO5fr+z579kwAIISGhua71qdPnwrFihUTVqxYoWxr2LCh0KFDhxzrAhCGDRum9nW2bdsmABCio6MFQRCEnTt3CgCE+fPn57sWdXx9fYUqVaqotO3YsUMAIEyfPl2lvUuXLoJMJhPu3LmjUrOBgYFw/fr1j75XWlqaYGNjIwwcOFCl/cmTJ4K1tbVK++vXr3Nsv2nTJgGAcPToUWVbYGCgYGBgoPbfTaFQCILwv89Ly5YtlW2CIAijRo0SDA0NhVevXuVZd1BQkABA+Pnnn5VtL1++FMzMzASZTCZs3rxZ2X7z5s0cn5GRI0cKAIRjx46pHIuyZcsK7u7uglwuFwRBEBYsWCAAELZu3apcLyMjQ6hQoYLKv71CoRA8PDwEf39/lf15/fq1ULZsWeGLL75QtmXv+7179/Lcx/we79DQUAGA0K9fP5V1O3XqJNjZ2SmfX758WQAgDB06VGW9Xr165es7tH79esHAwEDlmAmCICxbtkwAIJw4cSLP2v39/YVy5copn7969UqwtLQU6tevL7x580Zl3X8fQ19fXwGAsG7dOmVbZmam4OzsLHz11Vd51lyQfVZX86lTp3K893+/9/+m7jUGDx4smJubC2/fvs2zVmtr61x/1ghCwT5js2fPztdnjPLGy1KktGTJEhw6dAiHDh3Cb7/9Bj8/PwwYMEDlLqFt27bB2toaX3zxBZ4/f6581K5dGxYWFoiOjgYAZb+TPXv2fLSfSkFs3rwZBgYG+Oqrr5RtPXv2xP79+3NcFiuI1NRUAPikszYfs2/fPhgaGmLEiBEq7d999x0EQchxR5qvry8qV6780dc9dOgQXr16hZ49e6r8WxgaGqJ+/frKfwsAMDMzU/7/27dv8fz5czRo0AAAlKfOFQoFduzYgYCAAJW+V9n+e7Zh0KBBKm1NmjSBXC7H/fv3P1o78OGv8mw2NjaoVKkSihcvjm7duinbK1WqBBsbG9y9e1fZtm/fPtSrVw+NGzdWtllYWGDQoEFISEjA33//rVyvZMmS6NKli3I9c3NzDBo0SKWOy5cvIy4uDr169UJycrLyOGZkZKBFixY4evRornfk5SY/x/vfhgwZovK8SZMmSE5OVn4u9+3bBwA5PkMjR47MVz3btm2Dl5cXPD09VT4rzZs3B4BcPyvZZ3N9fX1x9+5dpKSkAPjw2UtLS8P48eNz9Af77+fEwsJC5eyvsbEx6tWrp/Jvqk5B9vnfNb979w7JycmoUKECbGxs1B5vdf79GmlpaXj+/DmaNGmC169f4+bNm3lua2NjgzNnzuDRo0dqlxfGZ4zyxstSpFSvXj2VX2o9e/ZEzZo1MXz4cLRr1w7GxsaIi4tDSkoKHB0d1b5GUlISgA+/oL/66itMnToV8+fPR7NmzdCxY0f06tULJiYmn1zjb7/9hnr16iE5OVnZf6FmzZrIysrCtm3bcvzi+pjsH8RWVlYAPvxQ07T79++jVKlSOYJT9unw/4aB7MsvHxMXFwcAyl9Q/5W9TwDw4sULTJ06FZs3b1b+G2XL/oX17NkzpKam5ntckjJlyqg8t7W1BYB8hUxTU9Mcp9qtra1RunTpHL8cra2tVV7z/v37qF+/fo7X/PfxrFq1Ku7fv48KFSrkeL1KlSqpPM8+jkFBQbnWm5KSoty//MjP8f63vI6llZUV7t+/DwMDA5QvXz7PfclNXFwcbty4kevljX/XeOLECYSGhuLUqVM5+v2kpKTA2toa8fHxAJCvz4q6f1NbW1tcvXo1z+0Kss9v3rxR3mDw8OFDlb5s6o63OtevX8cPP/yAw4cPK0Nlfl8jLCwMQUFBcHV1Re3atdG2bVsEBgaiXLlyAArnM0Z5Y7ihXBkYGMDPzw8LFy5EXFwcqlSpAoVCAUdHR2zYsEHtNtk/PGUyGbZv347Tp09j9+7diIyMRL9+/TB37lycPn0aFhYWBa4nLi4O586dAwB4eHjkWL5hwwaVcGNiYoI3b96ofa3sH9rZf3V6enoC+NCZVWz//gsyL9l/6a1fvx7Ozs45lv+7I3i3bt1w8uRJjBkzBjVq1ICFhQUUCgVat279yX8xGhoaqm0X/tNJuiDbfs5rfqrs/Z89ezZq1Kihdp2Cfl4LerwLe78VCgWqVauGefPmqV3u6uoKAIiPj0eLFi3g6emJefPmwdXVFcbGxti3bx/mz5//SZ8VbfybfvPNN1izZg1GjhwJHx8fWFtbQyaToUePHvmq+dWrV/D19YWVlRV+/PFHlC9fHqamprh48SLGjRv30dfo1q0bmjRpgj///BMHDx7E7NmzMWvWLPzxxx9o06ZNoXzGKG8MN5Sn9+/fAwDS09MBAOXLl8dff/2FRo0a5euXcIMGDdCgQQP89NNP2LhxI3r37o3NmzdjwIABOf6a+5gNGzbAyMgI69evz/ED8/jx4wgPD0diYqLyr2A3NzfcunVL7Wtlt7u5uQEAKlasiEqVKmHnzp1YuHChRn/QuLm54a+//kJaWprK2ZvsU93ZNRRU9l+0jo6Oed4K//LlS0RFRWHq1KmYPHmysj37r8lsDg4OsLKywrVr1z6pHm3J7d/1v8fTzc0N165dgyAIKp+1/26bfRytrKw0MqRAfo93Qbi5uUGhUCA+Pl7lzEVun+//Kl++PK5cuYIWLVrk+b3bvXs3MjMzsWvXLpWzSf++bJX9egBw7do1VKhQoSC7km8F2eft27cjKCgIc+fOVba9ffs2x1gxue17TEwMkpOT8ccff6Bp06bK9nv37uW73pIlS2Lo0KEYOnQokpKSUKtWLfz0009o06ZNgT5jBf25SOqxzw3l6t27dzh48CCMjY2Vp/y7desGuVyOadOm5Vj//fv3yh8mL1++zPGXWfZfLNm3npqbmwNAvger2rBhA5o0aYLu3bujS5cuKo/s2yc3bdqkXL9t27Y4ffo0Lly4oPI6r169woYNG1CjRg2VMx5Tp05FcnIyBgwYoAx1/3bw4MFPGtCwbdu2kMvlWLx4sUr7/PnzIZPJPnrHSG78/f1hZWWFn3/+WW2/puzbsrOD4H//PRYsWKDyPHu6iN27d6udWqEwz54URNu2bXH27FmcOnVK2ZaRkYHly5fD3d1d2V+pbdu2ePToEbZv365c7/Xr11i+fLnK69WuXRvly5fHnDlzlCH+3/Jze/u/5fd4F0T2ZyQ8PPyTXrNbt254+PAhVqxYkWPZmzdvkJGRAUB97SkpKVizZo3KNq1atYKlpSVmzJiRYxgGTX1OCrLPhoaGOd530aJFOW7jLl68OICcP3PU7XdWVhZ++eWXj9Ypl8tzXLZydHREqVKllD/rCvIZy61GKhieuSGl/fv3K//6TUpKwsaNGxEXF4fx48cr+2/4+vpi8ODBmDFjBi5fvoxWrVrByMgIcXFx2LZtGxYuXKiceuCXX35Bp06dUL58eaSlpWHFihWwsrJC27ZtAXy4/FK5cmVs2bIFFStWRIkSJVC1alW11/HPnDmDO3fu5DqdgouLC2rVqoUNGzZg3LhxAIDx48dj27ZtaNq0KQYPHgxPT088evQIERERePz4cY4f2N27d0dsbCx++uknXLp0CT179oSbmxuSk5Nx4MABREVFfXSEYnUCAgLg5+eHiRMnIiEhAd7e3jh48CB27tyJkSNH5uhTkF9WVlZYunQp+vTpg1q1aqFHjx5wcHBAYmIi9u7di0aNGmHx4sWwsrJS3nL97t07uLi44ODBg2r/Kv35559x8OBB+Pr6Km8Zfvz4MbZt24bjx49rdIDCTzV+/Hhs2rQJbdq0wYgRI1CiRAmsXbsW9+7dw++//w4Dgw9/sw0cOBCLFy9GYGAgLly4gJIlS2L9+vXKUJ3NwMAAK1euRJs2bVClShUEBwfDxcUFDx8+RHR0NKysrLB79+5811eQ451fNWrUQM+ePfHLL78gJSUFDRs2RFRUFO7cuZOv7fv06YOtW7diyJAhiI6ORqNGjSCXy3Hz5k1s3boVkZGRqFOnDlq1agVjY2MEBARg8ODBSE9Px4oVK+Do6IjHjx+r7OP8+fMxYMAA1K1bF7169YKtrS2uXLmC169fY+3atZ+8r5+yz+3atcP69ethbW2NypUr49SpU/jrr79gZ2eX4zUNDQ0xa9YspKSkwMTEBM2bN0fDhg1ha2uLoKAgjBgxAjKZDOvXr89XUEtLS0Pp0qXRpUsXeHt7w8LCAn/99RfOnTunPJNUkM9Y7dq1AQATJ05Ejx49YGRkhICAAGXooXzS/g1apGvU3Qpuamoq1KhRQ1i6dKnKrYvZli9fLtSuXVswMzMTLC0thWrVqgljx44VHj16JAiCIFy8eFHo2bOnUKZMGcHExERwdHQU2rVrJ5w/f17ldU6ePCnUrl1bMDY2zvOW1m+++UYAIMTHx+e6H1OmTBEACFeuXFG2/fPPP8KAAQMEFxcXoVixYkKJEiWEdu3aCadPn871daKiooQOHToIjo6OQrFixQQHBwchICBA2LlzZ16HURAE9beCC8KHW5VHjRollCpVSjAyMhI8PDyE2bNn5zi2yOP29dxER0cL/v7+grW1tWBqaiqUL19e6Nu3r8qx/ueff4ROnToJNjY2grW1tdC1a1fh0aNHao/5/fv3hcDAQMHBwUEwMTERypUrJwwbNkzIzMwUBOF/n5f/3i4eHR2d6222/xYUFCQUL148R3tux87NzU348ssvVdri4+OFLl26CDY2NoKpqalQr149Yc+ePTm2vX//vtC+fXvB3NxcsLe3F7799lvlsAX/rfPSpUtC586dBTs7O8HExERwc3MTunXrJkRFRSnXye+t4Pk93tm3gj979kxle3Xv8+bNG2HEiBGCnZ2dULx4cSEgIEB48OBBvodTyMrKEmbNmiVUqVJFMDExEWxtbYXatWsLU6dOFVJSUpTr7dq1S6hevbpgamoquLu7C7NmzRJWr16tdr937dolNGzYUDAzMxOsrKyEevXqCZs2bVIuz+3fNCgoSHBzc/tozfnd55cvXwrBwcGCvb29YGFhIfj7+ws3b94U3NzchKCgIJXXXLFihVCuXDnB0NBQ5XNw4sQJoUGDBoKZmZlQqlQpYezYsUJkZORHP9OZmZnCmDFjBG9vb8HS0lIoXry44O3tLfzyyy851s3PZ0wQBGHatGmCi4uLYGBgwNvCP5FMEHTkXDMRERGRBrDPDREREUkKww0RERFJCsMNERERSQrDDREREUkKww0RERFJSpEb50ahUODRo0ewtLTkSJBERER6QhAEpKWloVSpUsrxrHJT5MLNo0ePlPOoEBERkX558OABSpcunec6RS7cZM/t8+DBA5VZk4mIiEh3paamwtXVVWWOvtwUuXCTfSnKysqK4YaIiEjP5KdLCTsUExERkaQw3BAREZGkMNwQERGRpBS5PjdERKRf5HI53r17J3YZpAXGxsYfvc07PxhuiIhIJwmCgCdPnuDVq1dil0JaYmBggLJly8LY2PizXofhhoiIdFJ2sHF0dIS5uTkHXpW47EF2Hz9+jDJlynzWvzfDDRER6Ry5XK4MNnZ2dmKXQ1ri4OCAR48e4f379zAyMvrk12GHYiIi0jnZfWzMzc1FroS0KftylFwu/6zXYbghIiKdxUtRRYum/r1FDTdHjx5FQEAASpUqBZlMhh07dnx0m5iYGNSqVQsmJiaoUKECIiIiCr1OIiIi0h+ihpuMjAx4e3tjyZIl+Vr/3r17+PLLL+Hn54fLly9j5MiRGDBgACIjIwu5UiIiItIXonYobtOmDdq0aZPv9ZctW4ayZcti7ty5AAAvLy8cP34c8+fPh7+/f2GVSUREpBOaNWuGGjVqYMGCBflaPyIiAiNHjixyt9PrVZ+bU6dOoWXLlipt/v7+OHXqVK7bZGZmIjU1VeVBRERE0qVX4ebJkydwcnJSaXNyckJqairevHmjdpsZM2bA2tpa+XB1ddVGqURERCQSvQo3n2LChAlISUlRPh48eCB2SUREJDHNmjXDN998g5EjR8LW1hZOTk5YsWIFMjIyEBwcDEtLS1SoUAH79+9XbnPkyBHUq1cPJiYmKFmyJMaPH4/3798rl2dkZCAwMBAWFhYoWbKkskvGv2VmZmL06NFwcXFB8eLFUb9+fcTExGhjl3WaXoUbZ2dnPH36VKXt6dOnsLKygpmZmdptTExMYGVlpfLQZTc8vdQ+iIhIt61duxb29vY4e/YsvvnmG3z99dfo2rUrGjZsiIsXL6JVq1bo06cPXr9+jYcPH6Jt27aoW7curly5gqVLl2LVqlWYPn268vXGjBmDI0eOYOfOnTh48CBiYmJw8eJFlfccPnw4Tp06hc2bN+Pq1avo2rUrWrdujbi4OG3vvk7Rq3Dj4+ODqKgolbZDhw7Bx8dHpIqIiIg+8Pb2xg8//AAPDw9MmDABpqamsLe3x8CBA+Hh4YHJkycjOTkZV69exS+//AJXV1csXrwYnp6e6NixI6ZOnYq5c+dCoVAgPT0dq1atwpw5c9CiRQtUq1YNa9euVTmzk5iYiDVr1mDbtm1o0qQJypcvj9GjR6Nx48ZYs2aNiEdCfKLeLZWeno47d+4on9+7dw+XL19GiRIlUKZMGUyYMAEPHz7EunXrAABDhgzB4sWLMXbsWPTr1w+HDx/G1q1bsXfvXrF2gYiICABQvXp15f8bGhrCzs4O1apVU7Zl9xlNSkrCjRs34OPjozJoXaNGjZCeno5//vkHL1++RFZWFurXr69cXqJECVSqVEn5PDY2FnK5HBUrVlSpIzMzs8hPWSFquDl//jz8/PyUz0NCQgAAQUFBiIiIwOPHj5GYmKhcXrZsWezduxejRo3CwoULUbp0aaxcuZK3gRMRkej+OxeSTCZTacsOMgqFQiPvl56eDkNDQ1y4cAGGhoYqyywsLDTyHvpK1HDTrFkzCIKQ63J1ow83a9YMly5dKsSqNGyKdS7tKdqtg4iIdIaXlxd+//13CIKgDD0nTpyApaUlSpcujRIlSsDIyAhnzpxBmTJlAAAvX77E7du34evrCwCoWbMm5HI5kpKS0KRJE9H2RRdxVnA9sWTI4Rxtw5Y1F6ESIiL6XEOHDsWCBQvwzTffYPjw4bh16xZCQ0MREhICAwMDWFhYoH///hgzZgzs7Ozg6OiIiRMnwsDgf11lK1asiN69eyMwMBBz585FzZo18ezZM0RFRaF69er48ssvRdxDcTHcEBERaZmLiwv27duHMWPGwNvbGyVKlED//v3xww8/KNeZPXs20tPTERAQAEtLS3z33XdISVE9679mzRpMnz4d3333HR4+fAh7e3s0aNAA7dq10/Yu6RSZkNd1IQlKTU2FtbU1UlJStHNbeAEvS+V22/fhZjnn3+KZGyKSqrdv3+LevXsoW7YsTE1NxS6HtCSvf/eC/P7mmRuRVFtbTW37Vi3XQUREJDV6Nc4NERER0ccw3BAREZGkMNwQERGRpDDcEBERkaQw3BAREZGkMNwQERGRpPBWcAn6Z/wxte2lZ3J4biIikj6GGz02t7v6ESi7lx2n5UqIiIh0B8MNERHpFffxe7X2Xgkzi+78TPqMfW6IiIhIUhhuiIiISFIYboiIiDTowIEDaNy4MWxsbGBnZ4d27dohPj5eufzBgwfo1q0bbGxsUKJECXTo0AEJCQniFSxBDDdEREQalJGRgZCQEJw/fx5RUVEwMDBAp06doFAo8O7dO/j7+8PS0hLHjh3DiRMnYGFhgdatWyMrK0vs0iWDHYqJiIg06KuvvlJ5vnr1ajg4OODvv//G5cuXoVAosHLlSshkMgDAmjVrYGNjg5iYGLRq1UqMkiWH4YaoqJlinUt7inbrIJKouLg4TJ48GWfOnMHz58+hUCgAAImJibhy5Qru3LkDS0tLlW3evn2rcumKPg/DjYbkdmtigqmWCyEiIlEFBATAzc0NK1asQKlSpaBQKFC1alVkZWUhPT0dtWvXxoYNG3Js5+DgIEK10sRwQ0REpCHJycm4desWVqxYgSZNPowKf/z4ceXyWrVqYcuWLXB0dISVlZVYZUoeOxQTERFpiK2tLezs7LB8+XLcuXMHhw8fRkhIiHJ57969YW9vjw4dOuDYsWO4d+8eYmJiMGLECPzzzz8iVi4tPHNDRER6RZdHDTYwMMDmzZsxYsQIVK1aFZUqVUJ4eDiaNWsGADA3N8fRo0cxbtw4dO7cGWlpaXBxcUGLFi14JkeDGG6IiIg0qGXLlvj7779V2gRBUP6/s7Mz1q5dq+2yihReliIiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIknhCMVERKRfplhr8b1StPdepDE8c0NERESSwnBDREREksJwQ0REpEEKhQJhYWGoUKECTExMUKZMGfz0008AgHHjxqFixYowNzdHuXLlMGnSJLx7905l+507d6JWrVowNTVFuXLlMHXqVLx//16MXdFb7HNDRESkQRMmTMCKFSswf/58NG7cGI8fP8bNmzcBAJaWloiIiECpUqUQGxuLgQMHwtLSEmPHjgUAHDt2DIGBgQgPD0eTJk0QHx+PQYMGAQBCQ0NF2yd9w3BDRESkIWlpaVi4cCEWL16MoKAgAED58uXRuHFjAMAPP/ygXNfd3R2jR4/G5s2bleFm6tSpGD9+vHLbcuXKYdq0aRg7dizDTQEw3BAREWnIjRs3kJmZiRYtWqhdvmXLFoSHhyM+Ph7p6el4//49rKyslMuvXLmCEydOKC9jAYBcLsfbt2/x+vVrmJubF/o+SAHDDRERkYaYmZnluuzUqVPo3bs3pk6dCn9/f1hbW2Pz5s2YO3eucp309HRMnToVnTt3zrG9qalpodQsRQw3REREGuLh4QEzMzNERUVhwIABKstOnjwJNzc3TJw4Udl2//59lXVq1aqFW7duoUKFClqpV6oYboiIiDTE1NQU48aNw9ixY2FsbIxGjRrh2bNnuH79Ojw8PJCYmIjNmzejbt262Lt3L/7880+V7SdPnox27dqhTJky6NKlCwwMDHDlyhVcu3YN06dPF2mv9A/DDRFpxT/jj6ltLz2ziZYrIb2n46MGT5o0CcWKFcPkyZPx6NEjlCxZEkOGDEH//v0xatQoDB8+HJmZmfjyyy8xadIkTJkyRbmtv78/9uzZgx9//BGzZs2CkZERPD09c5wForwx3BAREWmQgYEBJk6cqHL5KVtYWBjCwsJU2kaOHKny3N/fH/7+/oVZouRxED8iIiKSFIYbIiIikhReliK9Vm1tNbXtsUGxWq6EiIh0Bc/cEBERkaQw3BAREZGkMNwQERGRpDDcEBERkaQw3BAREZGkMNwQERGRpPBWcCIi0iu5DQFRGDishH5iuCGiPN3w9FLbfrjZErXtw5Y1L8xyiIg+iuGGJCm3X8heN29ouRIiItI29rkhIiLSoGbNmmH48OEYPnw4rK2tYW9vj0mTJkEQBABAZmYmxo0bB1dXV5iYmKBChQpYtWoVAEAul6N///4oW7YszMzMUKlSJSxcuFDM3dFLPHNDRESkYWvXrkX//v1x9uxZnD9/HoMGDUKZMmUwcOBABAYG4tSpUwgPD4e3tzfu3buH58+fAwAUCgVKly6Nbdu2wc7ODidPnsSgQYNQsmRJdOvWTeS90h8MN0QEIPdOmlu1XAeRFLi6umL+/PmQyWSoVKkSYmNjMX/+fPj6+mLr1q04dOgQWrZsCQAoV66ccjsjIyNMnTpV+bxs2bI4deoUtm7dynBTAKJfllqyZAnc3d1hamqK+vXr4+zZs3muv2DBAlSqVAlmZmZwdXXFqFGj8PbtWy1VS0RE9HENGjSATCZTPvfx8UFcXBwuXboEQ0ND+Pr65rrtkiVLULt2bTg4OMDCwgLLly9HYmKiNsqWDFHDzZYtWxASEoLQ0FBcvHgR3t7e8Pf3R1JSktr1N27ciPHjxyM0NBQ3btzAqlWrsGXLFnz//fdarpyIiKjgTE1N81y+efNmjB49Gv3798fBgwdx+fJlBAcHIysrS0sVSoOo4WbevHkYOHAggoODUblyZSxbtgzm5uZYvXq12vVPnjyJRo0aoVevXnB3d0erVq3Qs2fPPM/2ZGZmIjU1VeVBRERUmM6cOaPy/PTp0/Dw8IC3tzcUCgWOHDmidrsTJ06gYcOGGDp0KGrWrIkKFSogPj5eGyVLimjhJisrCxcuXFBecwQAAwMDtGzZEqdOnVK7TcOGDXHhwgVlmLl79y727duHtm3b5vo+M2bMgLW1tfLh6uqq2R0hIqLPN8Va9bG4jtgVfZbExESEhITg1q1b2LRpExYtWoRvv/0W7u7uCAoKQr9+/bBjxw7cu3cPMTEx2Lr1Q+82Dw8PnD9/HpGRkbh9+zYmTZqEc+fOibw3+ke0DsXPnz+HXC6Hk5OTSruTkxNu3rypdptevXrh+fPnaNy4MQRBwPv37zFkyJA8L0tNmDABISEhyuepqakMOEREeuxs7U1q282qVtVyJbkLDAzEmzdvUK9ePRgaGuLbb7/FoEGDAABLly7F999/j6FDhyI5ORllypRR/h4bPHgwLl26hO7du0Mmk6Fnz54YOnQo9u/fL+bu6B29ulsqJiYGP//8M3755RfUr18fd+7cwbfffotp06Zh0qRJarcxMTGBiYmJlislIqKizMjICAsWLMDSpUtzLDM1NcW8efMwb968HMtMTEywZs0arFmzRqV9xowZhVarFIkWbuzt7WFoaIinT5+qtD99+hTOzs5qt5k0aRL69OmDAQMGAACqVauGjIwMDBo0CBMnToSBgeg3f5GOWzLksNp2ThlARCQdoqUBY2Nj1K5dG1FRUco2hUKBqKgo+Pj4qN3m9evXOQKMoaEhAChHfiQiIqKiTdTLUiEhIQgKCkKdOnVQr149LFiwABkZGQgODgbw4Zqli4uL8nRcQEAA5s2bh5o1ayovS02aNAkBAQHKkENERCSmmJgYsUso8kQNN927d8ezZ88wefJkPHnyBDVq1MCBAweUnYwTExNVztT88MMPkMlk+OGHH/Dw4UM4ODggICAAP/30k1i7QERERDpG9A7F2ZOLqfPf9FusWDGEhoYiNDRUC5URERGRPmIPXCIiIpIUhhsiIiKSFNEvSxHl2xTrnG1ly2i/DiIi0mk8c0NERESSwjM3RESkVxK6dNXae3ndvKHx13R3d8fIkSMxcuTIfK0fERGBkSNH4tWrVxqvRap45oaIiEiHde/eHbdv3xa7DL3CMzdEREQ6zMzMDGZmZmKXoVd45oaIiEiDmjVrphzDzdraGvb29pg0aVKu0wTNmzcP1apVQ/HixeHq6oqhQ4ciPT1duTwiIgI2NjbK51OmTEGNGjWwfv16uLu7w9raGj169EBaWlph75reYLghIiLSsLVr16JYsWI4e/YsFi5ciHnz5mHlypVq1zUwMEB4eDiuX7+OtWvX4vDhwxg7dmyerx8fH48dO3Zgz5492LNnD44cOYKZM2cWxq7oJV6WIiIi0jBXV1fMnz8fMpkMlSpVQmxsLObPn4+BAwfmWPffHYvd3d0xffp0DBkyBL/88kuur69QKBAREQFLS0sAQJ8+fRAVFcXpiP4/nrkhIiLSsAYNGkAmkymf+/j4IC4uDnK5PMe6f/31F1q0aAEXFxdYWlqiT58+SE5OxuvXr3N9fXd3d2WwAYCSJUsiKSlJszuhxxhuiIiIRJKQkIB27dqhevXq+P3333HhwgUsWbIEAJCVlZXrdkZGRirPZTIZFApFodaqT3hZioiISMPOnDmj8vz06dPw8PCAoaGhSvuFCxegUCgwd+5cGBh8ON+wdetWrdUpVTxzQ0REpGGJiYkICQnBrVu3sGnTJixatAjffvttjvUqVKiAd+/eYdGiRbh79y7Wr1+PZcuWiVCxtPDMDRGRltzw9FLbXhij4EqZ+/ZtatvNqlbVciW5CwwMxJs3b1CvXj0YGhri22+/xaBBg3Ks5+3tjXnz5mHWrFmYMGECmjZtihkzZiAwMFCEqqWD4YaIiEjDjIyMsGDBAixdujTHsoSEBJXno0aNwqhRo1Ta+vTpo/z/vn37om/fvsrnU6ZMwZQpU1TWL8h0DvmRdD9Vbbujm5XG3qMw8bIUERERSQrDDREREUkKL0sRERFpUExMjNglFHk8c0NERESSwnBDREREksJwQ0RERJLCPjdERER65uo/r9S2Vy9to9U6dBXP3BAREZGkMNwQERGRpPCyFBER6ZXVi5NyWXJY4+81bFlzjb8mFT6euSEiIiJJYbghIiIqRFlZWWKXUOQw3BAREWlQs2bNMHz4cIwcORL29vbw9/eHTCbD0qVL0aZNG5iZmaFcuXLYvn272KVKFvvcEBGR1riP36u2PcFUy4UUsrVr1+Lrr7/GiRMnAACenp6YNGkSZs6ciYULF2L9+vXo0aMHYmNj4eXlJXK10sMzN0RERBrm4eGBsLAwVKpUCZUqVQIAdO3aFQMGDEDFihUxbdo01KlTB4sWLRK5UmliuCEiItKw2rVr52jz8fHJ8fzGjRvaKqlIYbghIiLSsOLFi4tdQpHGPjdERCJbMiTn+CwcX0V6Tp8+jcDAQJXnNWvWFLEi6WK4ISIi0oJt27ahTp06aNy4MTZs2ICzZ89i1apVYpclSQw3RESkV/oNd1Tbbla1qpYrKZipU6di8+bNGDp0KEqWLIlNmzahcuXKYpclSQw3REREGhQTE6O2vVSpUjh48KB2iymiGG6IiDSs2tpqatu3arkOoqKKd0sRERGRpPDMDRERUSETBEHsEooUhhsiIqIi6s21a+oXWJbRbiEaxstSREREJCkMN0RERCQpDDdEREQkKexzQ5SHf8YfU9teemYTLVdCRET5xTM3REREJCk8c0NERHrll2njtfZe323Zo7HX6tu3L169eoUdO3Z88mvExMTAz88Px64lwMraGju3bsTsqRNw/Pp9jdUpBTxzQ0RERJLCcENERESSwnBDRPSpplirf1CRtn37dlSrVg1mZmaws7NDy5YtkZGRoVw+Z84clCxZEnZ2dhg2bBjevXunXLZ+/XrUqVMHlpaWcHZ2Rq9evZCUlFSg91+6dCnKly8PY2NjVKpUCevXr1cuGz16NNq1a6d8vnj9ephXq4aDx48r26q2bYvfNq/96PucO3cOX3zxBezt7WFtbQ1fX19cvHhRZZ1Xr15hwIABcHBwgJWVFZo3b44rV64UaH8+BcMNERGRhjx+/Bg9e/ZEv379cOPGDcTExKBz587K6Reio6MRHx+P6OhorF27FhEREYiIiFBu/+7dO0ybNg1XrlzBjh07kJCQgL59++b7/f/88098++23+O6773Dt2jUMHjwYwcHBiI6OBgD4+vri+PHjkMvlAIBj58/D3tYWx86dAwA8fPoUdx88QKMGjT/6XmlpaQgKCsLx48dx+vRpeHh4oG3btkhLS1Ou07VrVyQlJWH//v24cOECatWqhRYtWuDFixf53qdPwQ7FREREGvL48WO8f/8enTt3hpubGwCgWrX/zRJva2uLxYsXw9DQEJ6envjyyy8RFRWFgQMHAgD69eunXLdcuXIIDw9H3bp1kZ6eDgsLi4++/5w5c9C3b18MHToUABASEoLTp09jzpw58PPzQ5MmTZCWloZLly6hdu3aOHHhAkb27Yvd/z/8HDt3DqUcHVHWvfxH36t58+Yqz5cvXw4bGxscOXIE7dq1w/Hjx3H27FkkJSXBxMREWd+OHTuwfft2DBo06KPv8al45oaIiEhDvL290aJFC1SrVg1du3bFihUr8PLlS+XyKlWqwNDQUPm8ZMmSKpedLly4gICAAJQpUwaWlpbw9fUFACQmJubr/W/cuIFGjRqptDVq1Ag3btwAANjY2MDb2xsxMTGIjY2FkZER+nXtiis3biD99WscO38eTerUydd7PX36FAMHDoSHhwesra1hZWWF9PR0Za1XrlxBeno67OzsYGFhoXzcu3cP8fHx+XqPT8UzN0RERBpiaGiIQ4cO4eTJkzh48CAWLVqEiRMn4syZMwAAIyMjlfVlMhkUCgUAICMjA/7+/vD398eGDRvg4OCAxMRE+Pv7IysrS2M1NmvWDDExMTAxMUGTOnVQwtoalcqVw8mLF3H8wgWMCAzM1+sEBQUhOTkZCxcuhJubG0xMTODj46OsNT09HSVLlkRMTEyObW1sbDS2P+ow3BARkU56lP4I71++h4GRfl1kkMlkaNSoERo1aoTJkyfDzc0Nf/7550e3u3nzJpKTkzFz5ky4uroCAM6fP1+g9/YqXwYnDu1E0BfVlW0n/toL1wquuP78OgCgbM2yWLlqJd4o3qBD3Q9neZrWrYtt+/cjLiEBTevWzdd7nThxAr/88gvatm0LAHjw4AGeP3+uXF6rVi08efIExYoVg7u7e4H243Pp1yeGiIhIh505cwY///wzzp8/j8TERPzxxx949uwZvLy8PrptmTJlYGxsjEWLFuHu3bvYtWsXpk2bVqD3H/N1ICK27sbStdsQdzcR8379DX/sP4y+w/oq16ntUxsZ6Rk4cvAImv7/S1BN6tTB5r174ezgAI98BhEPDw+sX78eN27cwJkzZ9C7d2+YmZkpl7ds2RI+Pj7o2LEjDh48iISEBJw8eRITJ04scGgrKJ65ISIivTJ00ky17WmWZdS2O7pZFWY5KqysrHD06FEsWLAAqampcHNzw9y5c9GmTRts2bIlz20dHBwQERGB77//HuHh4ahVqxbmzJmD9u3b5/v9O7b2w8KpYzDn1/X4NnQ2yrq6YM28UNRqVE+5jrWNNTy8PJD8LBmVypUDADSuUwcKhSLf/W0AYNWqVRg0aBBq1aoFV1dX/Pzzzxg9erRyuUwmw759+zBx4kQEBwfj2bNncHZ2RtOmTeHk5JTv9/kUDDdEREQa4uXlhQMHDqhd9u9bvrMtWLBA5XnPnj3Rs2dPlbbs28iBD/1lBEHA1X9eAQA6dOuFDt16qaz/dVBXfB3UVaXt+n/e9/eY3z/8z5MPr13C2hoZV6+qrTs3NWvWxLn/fwt5ti5duqg8t7S0RHh4OMLDwwv02p+Ll6WIiIhIUnjmhoiIiHIoW7lUjjaZ7MN/9+/fjyZNmmi5ovwT/czNkiVL4O7uDlNTU9SvXx9nz57Nc/1Xr15h2LBhKFmyJExMTFCxYkXs27dPS9USEREVDYf3HcvxuHz5Mi5fvow6BeibIwZRz9xs2bIFISEhWLZsGerXr48FCxbA398ft27dgqOjY471s7Ky8MUXX8DR0RHbt2+Hi4sL7t+/X+j3yxMRERU16kYp1mbn7M8hariZN28eBg4ciODgYADAsmXLsHfvXqxevRrjx4/Psf7q1avx4sULnDx5UjkQ0sfunc/MzERmZqbyeWpqquZ2gIiIiHSOaJelsrKycOHCBbRs2fJ/xRgYoGXLljh16pTabXbt2gUfHx8MGzYMTk5OqFq1Kn7++WflBGDqzJgxA9bW1spH9sBIREREJE2ihZvnz59DLpfnuNfdyckJT548UbvN3bt3sX37dsjlcuzbtw+TJk3C3LlzMX369FzfZ8KECUhJSVE+Hjx4oNH9ICIiIt2iV3dLKRQKODo6Yvny5TA0NETt2rXx8OFDzJ49G6GhoWq3MTExUc5GSkRERNInWrixt7eHoaEhnj59qtL+9OlTODs7q92mZMmSMDIyUplR1cvLC0+ePEFWVhaMjY0LtWYiIiLSfaKFG2NjY9SuXRtRUVHo2LEjgA9nZqKiojB8+HC12zRq1AgbN26EQqGAgcGHK2q3b99GyZIlGWyIiIqI5N9e5rJEffs/n/FepWfq7lgulDtRL0uFhIQgKCgIderUQb169bBgwQJkZGQo754KDAyEi4sLZsyYAQD4+uuvsXjxYnz77bf45ptvEBcXh59//hkjRowQczeIdJL7+L1q2xNMtVwIEZGWiRpuunfvjmfPnmHy5Ml48uQJatSogQMHDig7GScmJirP0ACAq6srIiMjMWrUKFSvXh0uLi749ttvMW7cOLF2gSRibvd2atu7l+Vni4hI34g+QvHw4cNx//59ZGZm4syZM6hfv75yWUxMTI6Jxnx8fHD69Gm8ffsW8fHx+P7771X64BAREYll+fLlKFWqFBQKhUp7hw4d0K9fP8THx6NDhw5wcnKChYUF6tati7/++ktl3czMTIwePRouLi4oXrw46tevj5iYGC3uhf4TPdwQERFJRdeuXZGcnIzo6Ghl24sXL3DgwAH07t0b6enpaNu2LaKionDp0iW0bt0aAQEBSExMVK4/fPhwnDp1Cps3b8bVq1fRtWtXtG7dGnFxcWLskl5iuCEiItIQW1tbtGnTBhs3blS2bd++Hfb29vDz84O3tzcGDx6MqlWrwsPDA9OmTUP58uWxa9cuAB+6Y6xZswbbtm1DkyZNUL58eYwePRqNGzfGmjVrxNotvcNwQ0REpEG9e/fG77//rpz6Z8OGDejRowcMDAyQnp6O0aNHw8vLCzY2NrCwsMCNGzeUZ25iY2Mhl8tRsWJFWFhYKB9HjhxBfHy8mLulV/RqED8iIiJdFxAQAEEQsHfvXtStWxfHjh3D/PnzAQCjR4/GoUOHMGfOHFSoUAFmZmbo0qULsrKyAADp6ekwNDTEhQsXcvQntbCw0Pq+6CuGGyIiIg0yNTVF586dsWHDBty5cweVKlVCrVq1AAAnTpxA37590alTJwAfwkxCQoJy25o1a0IulyMpKQlNmnCMnU/FcENERKRhvXv3Rrt27XD9+nX83//9n7Ldw8MDf/zxBwICAiCTyTBp0iSVO6sqVqyI3r17IzAwEHPnzkXNmjXx7NkzREVFoXr16vjyyy/F2B29w3BDRER6xe7/bNW2p1mWUdvu6GZVmOWo1bx5c5QoUQK3bt1Cr169lO3z5s1Dv3790LBhQ9jb22PcuHFITU1V2XbNmjWYPn06vvvuOzx8+BD29vZo0KAB2rVTPx4X5cRwQ0REpGEGBgZ49OhRjnZ3d3ccPnxYpW3YsGEqz42MjDB16lRMnTq1UGuUMt4tRURERJKS73Dz6NEjjB49OsfpMwBISUnBmDFjcszwTURERKRt+Q438+bNQ2pqKqyscl67tLa2RlpaGubNm6fR4oiIiIgKKt/h5sCBAwgMDMx1eWBgIPbs2aORooiIiDTlSXyc2gdJV77Dzb1791CmjPqe6ABQunRplXv1iYiIPpmggAABEMQuhLRJEDTzD57vu6XMzMyQkJCQa8BJSEiAmZmZRooiIqKizfj1U6RkpcAi2QLmNuaQGcoA2YdlmQr1vwDfvc9S266Qy9W2v337ViO1ikHIZV/fGqg/NgqZQm17QY9lYR4zQRDw7NkzyGQyGBkZfdZr5Tvc1K9fH+vXr0fTpk3VLl+3bh3q1av3WcUQEREBgIHwHgsSFuArp6/g9doLxQz+9+tKyHlfCwDgbar6ECMo1G+QLtff00JJL9+obTeWPVO/fjH1v+4LeixTs0w/XtxnkMlkKF26dI6pJwoq3+Fm9OjR+OKLL2BtbY0xY8bAyckJAPD06VOEhYUhIiICBw8e/KxiiEj/ze2ufqCx7mXHabkS0nev3r/C6oerYWFoAXNDcxj8/54U85e/V7t+bL3JatuzUiPUtgfPX6aROsUw4I8Yte1RJqPVtn/rUkpte0GPZe+pXh8v7jMYGRl9drABChBu/Pz8sGTJEnz77beYP38+rKysIJPJkJKSAiMjIyxatAjNmzf/7IKIiIiyCRCQJk9DmjxN2WbwWP0v5Lcp6i+9vH35XG27qWnhnoUoTA/T1J9ZMX33QG374yyZ2vaCHkt9OWYFGqF48ODBaNeuHbZu3Yo7d+5AEARUrFgRXbp0QenSpQurRiIiIqJ8K/D0Cy4uLhg1alRh1EJERB/xz/hjattLz+QM0kTZ8h1uwsPD1bZbW1ujYsWK8PHx0VhRRES6xH38XrXtCfpxhp6oyMl3uJk/f77a9levXiElJQUNGzbErl27UKJECY0VR0RERFRQBRrET93j5cuXuHPnDhQKBX744YfCrJWIiIjoozQyK3i5cuUwc+ZM3gpOREREotNIuAGAMmXK4MmTJ5p6OSIiIqJPorFwExsbCzc3N029HBEREdEnyXeH4tRU9WM0p6Sk4MKFC/juu+8QFBSkscKIiIiIPkW+w42NjQ1kMvUjHMpkMgwYMADjx4/XWGFEREREnyLf4SY6Olptu5WVFTw8PGBhYYFr166hatWqGiuOiIiosHBAROnKd7jx9fVV256WloaNGzdi1apVOH/+POS5TC1PlF8cMI2IE5CSbiro51KsoPjJHYqPHj2KoKAglCxZEnPmzIGfnx9Onz6tydqIiIiICqxAc0s9efIEERERWLVqFVJTU9GtWzdkZmZix44dqFy5cmHVSERERJRv+T5zExAQgEqVKuHq1atYsGABHj16hEWLFhVmbUREREQFlu8zN/v378eIESPw9ddfw8PDozBrIiIiIvpk+T5zc/z4caSlpaF27dqoX78+Fi9ejOfPnxdmbUREREQFlu9w06BBA6xYsQKPHz/G4MGDsXnzZpQqVQoKhQKHDh1CWlpaYdZJRERElC8FvluqePHi6NevH44fP47Y2Fh89913mDlzJhwdHdG+ffvCqJGIiIgo3z5rbqlKlSohLCwM//zzDzZt2qSpmoiIiIg+mUYmzjQ0NETHjh2xa9cuTbwcERER0SfT2KzgRERERLqA4YaIiIgkheGGiIiIJIXhhoiIiCSF4YaIiIgkheGGiIiIJIXhhoiIiCSF4YaIiIgkheGGiIiIJIXhhoiIiCSF4YaIiIgkheGGiIiIJIXhhoiIiCSF4YaIiIgkheGGiIiIJIXhhoiIiCSF4YaIiIgkheGGiIiIJIXhhoiIiCSF4YaIiIgkheGGiIiIJIXhhoiIiCRFJ8LNkiVL4O7uDlNTU9SvXx9nz57N13abN2+GTCZDx44dC7dAIiIi0huih5stW7YgJCQEoaGhuHjxIry9veHv74+kpKQ8t0tISMDo0aPRpEkTLVVKRERE+kD0cDNv3jwMHDgQwcHBqFy5MpYtWwZzc3OsXr06123kcjl69+6NqVOnoly5cnm+fmZmJlJTU1UeREREJF2ihpusrCxcuHABLVu2VLYZGBigZcuWOHXqVK7b/fjjj3B0dET//v0/+h4zZsyAtbW18uHq6qqR2omIiEg3iRpunj9/DrlcDicnJ5V2JycnPHnyRO02x48fx6pVq7BixYp8vceECROQkpKifDx48OCz6yYiIiLdVUzsAgoiLS0Nffr0wYoVK2Bvb5+vbUxMTGBiYlLIlREREZGuEDXc2Nvbw9DQEE+fPlVpf/r0KZydnXOsHx8fj4SEBAQEBCjbFAoFAKBYsWK4desWypcvX7hFExERkU4T9bKUsbExateujaioKGWbQqFAVFQUfHx8cqzv6emJ2NhYXL58Wflo3749/Pz8cPnyZfanISIiIvEvS4WEhCAoKAh16tRBvXr1sGDBAmRkZCA4OBgAEBgYCBcXF8yYMQOmpqaoWrWqyvY2NjYAkKOdiIiIiibRw0337t3x7NkzTJ48GU+ePEGNGjVw4MABZSfjxMREGBiIfsc6ERER6QnRww0ADB8+HMOHD1e7LCYmJs9tIyIiNF8QERER6S2eEiEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIknRiXCzZMkSuLu7w9TUFPXr18fZs2dzXXfFihVo0qQJbG1tYWtri5YtW+a5PhERERUtooebLVu2ICQkBKGhobh48SK8vb3h7++PpKQktevHxMSgZ8+eiI6OxqlTp+Dq6opWrVrh4cOHWq6ciIiIdJHo4WbevHkYOHAggoODUblyZSxbtgzm5uZYvXq12vU3bNiAoUOHokaNGvD09MTKlSuhUCgQFRWldv3MzEykpqaqPIiIiEi6RA03WVlZuHDhAlq2bKlsMzAwQMuWLXHq1Kl8vcbr16/x7t07lChRQu3yGTNmwNraWvlwdXXVSO1ERESkm0QNN8+fP4dcLoeTk5NKu5OTE548eZKv1xg3bhxKlSqlEpD+bcKECUhJSVE+Hjx48Nl1ExERke4qJnYBn2PmzJnYvHkzYmJiYGpqqnYdExMTmJiYaLkyIiIiEouo4cbe3h6GhoZ4+vSpSvvTp0/h7Oyc57Zz5szBzJkz8ddff6F69eqFWSYRERHpEVEvSxkbG6N27doqnYGzOwf7+Pjkul1YWBimTZuGAwcOoE6dOtoolYiIiPSE6JelQkJCEBQUhDp16qBevXpYsGABMjIyEBwcDAAIDAyEi4sLZsyYAQCYNWsWJk+ejI0bN8Ld3V3ZN8fCwgIWFhai7QcRERHpBtHDTffu3fHs2TNMnjwZT548QY0aNXDgwAFlJ+PExEQYGPzvBNPSpUuRlZWFLl26qLxOaGgopkyZos3SiYiISAeJHm4AYPjw4Rg+fLjaZTExMSrPExISCr8gIiIi0luiD+JHREREpEkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpDDdEREQkKQw3REREJCkMN0RERCQpOhFulixZAnd3d5iamqJ+/fo4e/Zsnutv27YNnp6eMDU1RbVq1bBv3z4tVUpERES6TvRws2XLFoSEhCA0NBQXL16Et7c3/P39kZSUpHb9kydPomfPnujfvz8uXbqEjh07omPHjrh27ZqWKyciIiJdJHq4mTdvHgYOHIjg4GBUrlwZy5Ytg7m5OVavXq12/YULF6J169YYM2YMvLy8MG3aNNSqVQuLFy/WcuVERESki4qJ+eZZWVm4cOECJkyYoGwzMDBAy5YtcerUKbXbnDp1CiEhISpt/v7+2LFjh9r1MzMzkZmZqXyekpICAEhNTf3M6lUpMl+rbU+VCWrb5W/katvT5erb32Rl5GjLfPdO7bppmTnXBTS/z4WlIMdSE8cRkOaxFOMzCfBYAvx+54Xfb80oit/v7NcSBPX7qEIQ0cOHDwUAwsmTJ1Xax4wZI9SrV0/tNkZGRsLGjRtV2pYsWSI4OjqqXT80NFQAwAcffPDBBx98SODx4MGDj+YLUc/caMOECRNUzvQoFAq8ePECdnZ2kMlkIlaWt9TUVLi6uuLBgwewsrISuxy9xeOoOTyWmsNjqRk8jpqjD8dSEASkpaWhVKlSH11X1HBjb28PQ0NDPH36VKX96dOncHZ2VruNs7NzgdY3MTGBiYmJSpuNjc2nF61lVlZWOvtB0yc8jprDY6k5PJaaweOoObp+LK2trfO1nqgdio2NjVG7dm1ERUUp2xQKBaKiouDj46N2Gx8fH5X1AeDQoUO5rk9ERERFi+iXpUJCQhAUFIQ6deqgXr16WLBgATIyMhAcHAwACAwMhIuLC2bMmAEA+Pbbb+Hr64u5c+fiyy+/xObNm3H+/HksX75czN0gIiIiHSF6uOnevTuePXuGyZMn48mTJ6hRowYOHDgAJycnAEBiYiIMDP53gqlhw4bYuHEjfvjhB3z//ffw8PDAjh07ULVqVbF2oVCYmJggNDQ0xyU1KhgeR83hsdQcHkvN4HHUHKkdS5kg5OeeKiIiIiL9IPogfkRERESaxHBDREREksJwQ0RERJLCcENERESSwnBDREREksJwQ0RERJLCcEOSdPToUbx//z5H+/v373H06FERKtJP69atQ2ZmZo72rKwsrFu3ToSKiIg+juFGhxgaGiIpKSlHe3JyMgwNDUWoSH/5+fnhxYsXOdpTUlLg5+cnQkX6KTg4GCkpKTna09LSlKOIU/7w+605PJaa0bx5c7x69SpHe2pqKpo3b679gjSI4UaH5DaeYmZmJoyNjbVcjX4TBEHtrO/JyckoXry4CBXpp9yO4z///JPvCezoA36/NYfHUjNiYmKQlZWVo/3t27c4duyYCBVpjujTLxAQHh4OAJDJZFi5ciUsLCyUy+RyOY4ePQpPT0+xytMrnTt3BvDhWPbt21dlKHG5XI6rV6+iYcOGYpWnN2rWrAmZTAaZTIYWLVqgWLH//aiQy+W4d+8eWrduLWKF+oPfb83hsdSMq1evKv//77//xpMnT5TP5XI5Dhw4ABcXFzFK0xiGGx0wf/58AB/+Glm2bJnKaVVjY2O4u7tj2bJlYpWnV7LPJgiCAEtLS5iZmSmXGRsbo0GDBhg4cKBY5emNjh07AgAuX74Mf39/lV8i2Z/Jr776SqTq9Au/35rDY6kZNWrUUP7xou7yk5mZGRYtWiRCZZrDuaV0iJ+fH/744w/Y2tqKXYremzp1KkaPHs1LUJ9p7dq16N69O0xNTcUuRe/x+605PJaf5/79+xAEAeXKlcPZs2fh4OCgXGZsbAxHR0e977vEcKND3r59m+svkcePH6NkyZJarkh/3bx5M9fT05GRkfD399dyRfopOjo61w7Yv/76KwYPHqzlivQXv9+aw2NZ+HLrb6cv2KFYh9SqVQuXL1/O0f7777+jevXq2i9Ij9WqVQtLlixRacvMzMTw4cPRoUMHkarSP61bt8aYMWPw7t07Zdvz588REBCA8ePHi1iZ/uH3W3N4LDWjb9++yMjIyNGekJCApk2bilCR5jDc6JBmzZqhQYMGmDVrFgAgIyMDffv2RZ8+ffD999+LXJ1+iYiIwOTJk9G2bVs8ffoUly9fRs2aNfHXX3/p/V0A2hQdHY0///wTdevWxd9//429e/eiatWqSE1NVfvLhXLH77fm8FhqxpUrV1C9enWcOnVK2bZ27Vp4e3vD3t5exMo0QCCdsmfPHsHZ2Vlo3LixUL58ecHb21uIjY0Vuyy99ODBA6Fly5aCnZ2dYGpqKgwZMkTIyMgQuyy9k5aWJvTu3VswMTERjIyMhJkzZwoKhULssvQSv9+aw2P5+bKysoTRo0cLxsbGwoQJE4SuXbsKFhYWwvLly8Uu7bPxbikd06ZNG3Tu3BlLly5FsWLFsHv3blStWlXssvRWVlYW5HI55HI5SpYsyY6xn+D27ds4f/48SpcujUePHuHWrVt4/fo1O2t/An6/NYfH8vMZGRlh9uzZMDc3x7Rp01CsWDEcOXIEPj4+Ypf22XhZSofEx8fDx8cHe/bsQWRkJMaOHYv27dtj7NixKn0e6OM2b96MatWqwdraGrdv38bevXuxfPlyNGnSBHfv3hW7PL0xc+ZM+Pj44IsvvsC1a9dw9uxZXLp0KcepbPo4fr81h8dSM969e4fvvvsOs2bNwoQJE+Dj44POnTtj3759Ypf2+cQ+dUT/Y2FhIXTv3l14+fKlsu3EiRNC+fLlhRo1aohXmB4yNzcXfvnlF5W2Fy9eCF27dhUsLS1Fqkr/ODs7C/v27VNp+/epbMo/fr81h8dSM6pXry5UqFBBOHXqlCAIgqBQKISZM2cKJiYmwtdffy1ydZ+H4UaHrFu3Tm17amqq0K9fPy1Xo99u3ryZ67LcjjPl9OzZs1yXxcTEaLES/cfvt+bwWGpGv379hPT09BztFy9eFKpUqSJCRZrDy1I6xM3NTe1M1mZmZggKChKhIv21ZcsWvH79Okf7mzdvcO/ePREq0k9jx45FWlpajvaMjAysXbtWhIr0F7/fmsNjqRmrVq1S23euZs2auHjxoggVaQ4H8dMhhoaGePz4MRwdHVXak5OT4ejoCLlcLlJl+ofHUjNyO47Pnz+Hs7Oz2l8wpB4/k5rDY6kZUj6OvFtKhwicyVpjcjuWV65cQYkSJUSoSL+kpqZC+HDZGmlpaSp3mcnlcuzbty/HD0TKG7/fmsNjqRm5nduQwuzqDDc6gDNZa46tra1yQriKFSuq/ACUy+VIT0/HkCFDRKxQP9jY2Kgcx/+SyWSYOnWqCJXpH36/NYfHUjOKwuzqDDc6gDNZa86CBQsgCAL69euHqVOnKo8t8L9Zg6UwhkNhi46OhiAIaN68OX7//XeVs13GxsZwc3NDqVKlRKxQf/D7rTk8lppRFGZXZ58bHcKZrDXnyJEjaNSoEYoVY37/HPfv30eZMmX0egI9XcHvt+bwWGqGlGdXZ7ghIiIiSeGt4Hrg+++/R79+/cQuQxJatmyJcuXKiV2G3vPy8lI5lU2fjt9vzeGx1IydO3di3bp1YpfxWXjOXg88fPgQDx48ELsMSejUqROeP38udhl6b8aMGUhJSRG7DL1y9+5dtcGa32/N4bHUjHHjxiEuLg6BgYFil/LJeFmKiEgLDAwM4Ovri/79+6NLly6cxJWoEPGylI579eqV2CVQEff8+XOcP38eFy5cQHJystjl6K2LFy+ievXqCAkJgbOzMwYPHoyzZ8+KXRaRJPHMjQ6ZNWsW3N3d0b17dwBAt27d8Pvvv8PZ2Rn79u2Dt7e3yBXqtuwxMPLjjz/+KMRKpOH69ev4+uuvceLECZV2X19fLF26FJUqVRKpMv32/v177Nq1CxEREThw4AAqVqyIfv36oU+fPnBwcBC7PJ2nUCgQERGBP/74AwkJCZDJZChbtiy6dOmCPn368M6+AoiLi8POnTtVjmPHjh0l0S+R4UaHlC1bFhs2bEDDhg1x6NAhdOvWDVu2bMHWrVuRmJiIgwcPil2iTgsODs73umvWrCnESvTfkydPULVqVTg4OGDIkCHw9PSEIAj4+++/sWLFCiQnJ+PatWscpfgzZGZm4pdffsGECROQlZUFY2NjdOvWDbNmzULJkiXFLk8nCYKAgIAA5R972Z/LGzduIDY2Fu3bt8eOHTvELlMvzJgxA5MnT4ZCoYCjoyMEQcCzZ89gaGiIn3/+GaNHjxa7xM+jlek5KV9MTU2FxMREQRAEYcSIEcKgQYMEQRCEW7duCTY2NmKWRkXM2LFjhVq1aglv3rzJsez169dCrVq1hPHjx4tQmf47d+6c8PXXXwu2trZC6dKlhYkTJwp3794Vjh49KrRo0UKoW7eu2CXqrNWrVwuWlpbC4cOHcyyLiooSLC0thbVr14pQmX45fPiwYGBgIISGhgovXrxQticnJwuTJk0SDA0NhSNHjohY4edjuNEhJUuWFE6cOCEIgiBUrFhR2Lp1qyAIgnDz5k3B0tJSzNKoiKlZs6awZcuWXJdv2rRJqFmzphYr0n9z584VqlatKhgZGQkdOnQQdu/eLcjlcpV1Hjx4IBgaGopUoe774osvhBkzZuS6/KeffhJatWqlxYr0U7du3ZR/PKszcOBAoUePHlqsSPN4K7gO6dy5M3r16gUPDw8kJyejTZs2AIBLly6hQoUKIlenf7Zv3668pJeVlaWy7OLFiyJVpR/u3r2LWrVq5bq8Tp06uHv3rhYr0n9Lly5Fv3790Ldv31wvOzk6OmLVqlVarkx/XL16FWFhYbkub9OmjXLeJMrd2bNnsX79+lyX9+nTR69vAwd4t5ROmT9/PoYPH47KlSvj0KFDysnMHj9+jKFDh4pcnX4JDw9HcHAwnJyccOnSJdSrVw92dna4e/euMjRS7tLS0mBlZZXrcktLS6Snp2uxIv136NAhjBs3LkewEQQBiYmJAD7M6xMUFCRGeXrhxYsXcHJyynW5k5MTXr58qcWK9NPTp0/h7u6e6/KyZcviyZMn2iuoELBDMUmSp6cnQkND0bNnT1haWuLKlSsoV64cJk+ejBcvXmDx4sVil6jTDA0Ncfv27Vzv3nn69Ck8PT0hl8u1XJn+MjQ0xOPHj3N0wk5OToajoyOPZT4YGhriyZMneX4uS5UqxWP5EQYGBnjy5EmuNwRI4TjyspTIdu3ahTZt2sDIyAi7du3Kc9327dtrqSr9l5iYiIYNGwIAzMzMkJaWBuDD6dYGDRow3HyEIAioWLFinst5y23B5PZ3ZHp6Ogf0yydBENC3b1+YmJioXZ6ZmanlivTXypUrlVcH/iv756U+Y7gRWceOHZUJumPHjrmuJ5PJ9DpFa5uzszNevHgBNzc3lClTBqdPn4a3tzfu3buX6y8Z+p/o6GixS5CMkJAQAB++w5MnT4a5ublymVwux5kzZ1CjRg2RqtMv+blkp+99RbShTJkyWLFixUfX0WcMNyJTKBRq/58+T/PmzbFr1y7UrFkTwcHBGDVqFLZv347z588XaLC/osrX11fsEiTj0qVLAD6cdYiNjYWxsbFymbGxMby9vfV/TBEt4fhUmpGQkCB2CYWOfW503KtXr2BjYyN2GXpHoVBAoVCgWLEP+X3z5s04efIkPDw8MHjwYJVfMJQ/giAgOjoab968QcOGDWFrayt2SXolODgYCxcuzLOjNhFpBu+W0iGzZs3Cli1blM+7du2KEiVKwMXFBVeuXBGxMv3zzz//wNDQUPm8R48eCA8Px/Dhw/X+LgBtePXqFYKCglCtWjUMHDgQqampaNKkCVq2bImAgAB4eXnh6tWrYpepV9asWcNgowFXrlzB9OnT8csvv+D58+cqy1JTU9GvXz+RKtMvb968wfHjx/H333/nWPb27VusW7dOhKo0h2dudAinX9Ac3pnyeQYMGICjR48iKCgIu3fvhoGBAQRBwIIFC2BgYICxY8fCwsICu3fvFrtUnda5c2dERETAysrqo5dDOd/Zxx08eBABAQHw8PBAWloaMjIysG3bNvj5+QGQxl0+2nD79m20atUKiYmJkMlkaNy4MTZv3qwcpkAKx5F9bnTIkydP4OrqCgDYs2cPunXrhlatWsHd3R3169cXuTr9ktvdPLwzJX/279+PjRs3wtfXF3379oWrqysOHz6s/BzOmjWLd+/lg7W1tfJzaG1tLXI1+m/KlCkYPXo0fvrpJwiCgNmzZ6N9+/bYtm0bWrduLXZ5emPcuHGoWrUqzp8/j1evXmHkyJFo1KgRYmJi9L4jcTaGGx1ia2uLBw8ewNXVFQcOHMD06dMBfPhFrc8JWpv+fWfKpEmTeGfKJ3r69KnyVnAXFxeYmpoqgzfw4U6KZ8+eiVWe3vh3B1h2hv18169fV46sK5PJMHbsWJQuXRpdunTB5s2bUbduXZEr1A8nT57EX3/9BXt7e9jb22P37t0YOnQomjRpgujoaBQvXlzsEj8bw40O4fQLn493pmiGQqFQ6bNkaGiociaMY9wU3Js3byAIgjJw379/H3/++ScqV66MVq1aiVydfjAxMcGrV69U2nr16gUDAwN0794dc+fOFacwPfPmzRvlzRbAh+/z0qVLMXz4cPj6+mLjxo0iVqcZDDc6ZP78+XB3d8eDBw8QFhbG6Rc+Qfb4LLwz5fP9e5Cv9+/fIyIiAvb29gCkMciXtnXo0AGdO3fGkCFD8OrVK9SrVw/GxsZ4/vw55s2bh6+//lrsEnVejRo1EB0djdq1a6u09+jRA4IgcOqKfPL09MT58+fh5eWl0p49uKkULjmzQzFJUkpKCuRyOUqUKKHS/uLFCxQrVoyh5yPc3d3zdXbm3r17WqhGGuzt7XHkyBFUqVIFK1euxKJFi3Dp0iX8/vvvmDx5Mm7cuCF2iTrvzz//xNGjRzF//ny1yzdu3IgVK1ZwEMqPmDFjBo4dO4Z9+/apXT506FAsW7ZMr8deY7jRMfHx8ViwYIHyB13lypUxcuRIlCtXTuTK9EubNm0QEBCQ44zXsmXLsGvXrly/1ESFxdzcHDdv3kSZMmXQrVs3VKlSBaGhoXjw4AEqVaqE169fi12izjt69Ch8fHxgZGQkdimk4zjOjQ6JjIxE5cqVcfbsWVSvXh3Vq1fHmTNnlLOEU/6dOXNGeXvovzVr1gxnzpwRoSIq6ipUqIAdO3bgwYMHiIyMVPazSUpK4pnEfPLz8+Os3xpgaGiIpKQkscsoVOxzo0PGjx+PUaNGYebMmTnax40bhy+++EKkyvRPZmYm3r9/n6P93bt3ePPmjQgV6Zfw8PB8rTdixIhCrkQ6Jk+ejF69emHUqFFo3rw5fHx8AHwYu6VmzZoiV6cfeKFBM4rCceRlKR1iamqK2NhYeHh4qLTfvn0b1atXx9u3b0WqTP/4+fmhatWqWLRokUr7sGHDcPXqVRw7dkykyvRD2bJlP7qOTCbD3bt3tVCNdDx58gSPHz+Gt7c3DAw+nDg/e/YsrKys4OnpKXJ1us/AwABPnz6Fg4OD2KXoNQMDA+WEzVLFMzc6xMHBAZcvX84Rbi5fvizpD2FhmD59Olq2bIkrV66gRYsWAICoqCicO3eOIz3nAzsKFw5nZ2ekp6fj0KFDaNq0KczMzFC3bl3eWl8Affv2hYmJSZ7rcLTnj/v33ZC50eczsww3OmTgwIEYNGgQ7t69i4YNGwIATpw4gVmzZikHp6P8adSoEU6dOoWwsDBs3boVZmZmqF69OlatWpUjPNLnq1atGvbt26cy0B+pSk5ORrdu3RAdHQ2ZTIa4uDiUK1cO/fv3h62tLcdoySdLS0uYmZmJXYbeW7ZsmcpYVv8lk8n0OtzwspQOyZ67Z+7cuXj06BEAoFSpUhgzZgxGjBjBv+5IZ1laWuLKlSu8qy8PgYGBSEpKwsqVK+Hl5aU8XpGRkQgJCcH169fFLlHnFYXLKdpQFI4jz9zoEJlMhlGjRmHUqFHKQdIsLS1Frkp/xcfHY82aNbh79y4WLFgAR0dH7N+/H2XKlEGVKlXELo+KmIMHDyIyMhKlS5dWaffw8MD9+/dFqkq/8A88zSgKx5G3gusoS0tLBpvPcOTIEVSrVg1nzpzB77//jvT0dADAlStXEBoaKnJ1VBRlZGSozHWW7cWLFx/tQ0If8EKDZhSF48hwo0OePn2KPn36oFSpUihWrBgMDQ1VHpR/48ePx/Tp03Ho0CGV+aWaN2+O06dPi1gZFVVNmjTBunXrlM9lMhkUCgXCwsLUjslEOUVHR+cYdTwv1apVw4MHDwqxIv0UGhr60c7E+o6XpXRI3759kZiYiEmTJqFkyZJF4tRhYYmNjVU7+ZujoyOeP38uQkVU1IWFhaFFixY4f/48srKyMHbsWFy/fh0vXrzAiRMnxC5PL/j6+hZo/YSEBLx7966QqtFfq1evRkJCAvz8/ODn5yfJGwEYbnTI8ePHcezYMdSoUUPsUvSejY0NHj9+nGO8lkuXLsHFxUWkqqgoq1q1Km7fvo3FixfD0tIS6enp6Ny5M4YNG4aSJUuKXR4VIcHBwYiJicHmzZuRlZWFsmXLws/PD82bN4efnx+cnZ3FLvGzMdzoEFdX1yJxLVQbevTogXHjxmHbtm3K0/8nTpzA6NGjERgYKHZ5euHdu3do3bo1li1b9tHb53/99Vc4OTlpqTL9ZW1tjYkTJ4pdBhVxU6ZMAfBhJPcTJ07gyJEjiImJwfr16/Hu3TtUrFgRzZs3x5IlS8Qt9DPwVnAdcvDgQcydOxe//vor3N3dxS5Hr2VlZWHYsGGIiIiAXC5HsWLFIJfL0atXL0RERLAPUz45ODjg5MmTHBtIQ16+fIlVq1apTIwbHBxcoH4klH8coqBgXr58iblz52LRokVIT0+HXC4Xu6RPxnAjMltbW5W+NRkZGXj//j3Mzc1zzHz74sULbZenlwRBwIMHD+Dg4IDnz58jNjYW6enpqFmzJn9JF9CoUaNgYmKSY74zKrijR48iICAA1tbWqFOnDgDgwoULePXqFXbv3o2mTZuKXKH0MNzkLSsrC6dOnUJMTAxiYmJw5swZuLi4oGnTpvD19dXrs9wMNyJbu3ZtvtcNCgoqxEqkQ6FQwNTUFNevX2eY+UzffPMN1q1bBw8PD9SuXRvFixdXWT5v3jyRKtM/1apVg4+PD5YuXao8cyiXyzF06FCcPHkSsbGxIlcoPQw36v3444/KMOPm5qYMM76+vihVqpTY5WkEw42OePfuHQYPHoxJkybla9JCyluVKlWwatUqNGjQQOxS9FpetyjLZDIcPnxYi9XoNzMzM1y+fBmVKlVSab916xZq1KjB2erzqSB9wTZu3IgOHTrkCOVFnYGBAcqUKYPx48eja9eusLOzE7skjWO40SHW1ta4fPkyw40G7N69G2FhYVi6dCmqVq0qdjlEaNSoEcaMGYOOHTuqtO/YsQMzZ87k+EsFwL5gnycyMhLR0dGIiYnBpUuXULFiRTRr1kx59kYKs64z3OiQoKAg1KhRA6NGjRK7FL1na2uL169f4/379zA2Ns4x0R77LxXMnTt3EB8fr5zJWhAEjsOUD1evXlX+/40bNzB27Fh88803yjOKp0+fxpIlSzBz5kx0795drDL1DvuCaU5aWhqOHTuGI0eOIDo6GleuXEGFChXg5+eHxYsXi13eJ2O40SHTp0/H3Llz0aJFC7X9G/R5hlZt+1hfJvZfyp/cZrLu168fZ7LOBwMDA8hkso8O8SCTyfT6zhRtY18wzZPL5Th79ix27dqFX375hXdLkebkdTlKJpPh7t27WqyGiDNZf66CTIjp5uZWiJVIC/uCfT6FQoHz588rL0+dOHECGRkZKF26tHLkYn3+I5DhRkcIgoDExEQ4OjrmuIRCn0Yul2PHjh3KMUWqVKmC9u3bc4ybAnB2dkZkZCS8vb1V7jy5e/cuqlevrpyQlIj0R5s2bXDy5EmkpaWhVKlS8PPzQ7NmzeDn5yeZO8s4QrGOEAQBHh4evH1ZQ+7cuYO2bdvi4cOHyrtTZsyYAVdXV+zduxfly5cXuUL9wJmsNWfbtm3YtGkTbt++DQCoWLEievXqhS5duohcmf5iX7BPY2Njg9mzZ8PPz0+yv284K7iOMDAwgIeHB5KTk8UuRRJGjBiB8uXL48GDB7h48SIuXryIxMRElC1bln2XCoAzWX8+hUKB7t27o3v37vj7779RoUIFVKhQAdevX0f37t3Ro0cPTrtSQMnJyWjRogUqVqyItm3b4vHjxwCA/v3747vvvhO5Ot23adMmDBo0KN/BRi9nVxdIZ+zatUto3LixEBsbK3Ypes/c3Fy4evVqjvbLly8LxYsXF6Ei/RQbGys4OjoKrVu3FoyNjYUuXboIXl5egpOTk3Dnzh2xy9ML8+bNE0qUKCHs3r07x7KdO3cKJUqUEObPn6/9wvRYnz59BH9/f+HBgweChYWFEB8fLwiCIBw4cECoXLmyyNVJz7+Psb7gmRsdEhgYiLNnz8Lb2xtmZmYoUaKEyoPyz8TEBGlpaTna09PTYWxsLEJF+il7JuvGjRujQ4cOyMjIQOfOnXHp0iVe2sunNWvWYPbs2WjXrl2OZe3bt0dYWBhWr14tQmX66+DBg5g1axZKly6t0u7h4VGgTtwkXexzo0MWLFggdgmS0a5dOwwaNAirVq1CvXr1AABnzpzBkCFD0L59e5Gr0x/R0dHw8/NTO5P1kiVLMGzYMBGq0i9xcXFo2bJlrstbtmyJ4cOHa7Ei/ce+YPQxvFuKJOnVq1cICgrC7t27lROQvn//Hu3bt0dERASsra1FrlA/2Nra4q+//kLt2rVV2hcuXIhJkyYhNTVVpMr0R4kSJRATE4Pq1aurXR4bG4umTZvi5cuXWq5Mf7Vt2xa1a9fGtGnTYGlpiatXr8LNzQ09evSAQqHA9u3bxS5RUvRxji6euRFZamoqrKyslP+fl+z16ONsbGywc+dOxMXF4ebNmwAALy8vVKhQQeTK9Mvs2bPRpk0bHD16FJ6engCAuXPn4scff8TevXtFrk4/ZE+WuXTpUrXLlyxZAh8fHy1Xpd/CwsLQokULnD9/HllZWRg7diyuX7+OFy9e4MSJE2KXRzqA4UZktra2ePz4MRwdHWFjY6P2Nkbh/9/eqM+jRYrFw8NDsrc6asOAAQPw4sULtGzZEsePH8eWLVvw888/Y9++fWjUqJHY5emFiRMnolmzZkhOTsbo0aPh6ekJQRBw48YNzJ07Fzt37kR0dLTYZeqV7L5gixcvhqWlJdLT09G5c2cMGzYMJUuWFLs80gG8LCWyI0eOoFGjRihWrBiOHDmS57q+vr5aqko/hYSE5HtdDs9eMOPGjcOqVasgl8uxf/9+zrZeQH/++ScGDRqUY04zW1tb/Prrr/jqq69Eqkw/ZfcFU4d9wfJH6rOrM9yQZOR33BUOz5638PBwte1z5sxB06ZNlR20Ac53VhCvX79GZGQk4uLiAHwYxK9Vq1ZqO8ZS3tgXTDOkPLs6w40OiIuLw+TJk/Hrr7/m6FeTkpKCr7/+GtOnT9erzlykv/Ka4+zfON9Z4ahWrRr27dsHV1dXsUvRWStXrsT333+vti/Ynj170KRJE5Er1A9Snl2dfW50wOzZs+Hq6qq2w7C1tTVcXV0xe/bsXDskUt7++ecfAMgxJgapd+/ePbFLKNISEhLw7t07scvQaewLphnv37/H6tWrlWfBpDS7OsONDjhy5Ah+++23XJd369YNvXr10mJF+k+hUGD69OmYO3eucnJHS0tLfPfdd5g4cSIMDDh+5ce8e/cOnp6e2LNnD7y8vMQuh0jF2LFjkZycjDp16kAulyMyMpJ9wQro2rVrqFWrFgAo5zzLpu9zdDHc6IDs2cBzY29vr3/zeohs4sSJWLVqFWbOnKn8S+748eOYMmUK3r59i59++knkCnWfkZER3r59K3YZRADU9wVzcXGBubk5mjZtirNnz+Ls2bMA2Bcsv6R8lx773OgAZ2dnbNy4Ec2bN1e7PCoqCr1798aTJ0+0XJn+KlWqFJYtW5ZjNOKdO3di6NChePjwoUiV6Zeff/4Zt2/fxsqVK1GsGP8W0gZ9HDBNG9gXrPBIcXZ1/rTSAU2bNsWiRYtyDTfh4eHsIFdAL168UHY0/DdPT88ct+NS7s6dO4eoqCgcPHgQ1apVy3FN/o8//hCpMipq2BdM85KTk9GtWzdER0dDJpMhLi4O5cqVQ//+/WFra4u5c+eKXeInY8cDHTBhwgTs378fXbp0wdmzZ5GSkoKUlBScOXMGX331FSIjIzFhwgSxy9Qr3t7eWLx4cY72xYsXw9vbW4SK9JONjQ2++uor+Pv7o1SpUrC2tlZ5EGnbu3fvUL58edy4cUPsUvTeqFGjYGRkhMTERJUhCbp3744DBw6IWNnn45kbHVCzZk1s374d/fr1w59//qmyzM7ODlu3blV2+qL8CQsLw5dffom//vpLObT9qVOn8ODBA+zbt0/k6vTHmjVrxC5BEgoyYNqvv/4KJycnLVWmf9gXTHMOHjyIyMhISc6uzj43OuTNmzc4cOAA7ty5A0EQOMjXZ3r06BGWLFmiMrfU0KFDUapUKZEro6JIygOmaRv7gmmGpaUlLl68CA8PD5W+XufPn4e/vz+Sk5PFLvGTMdzoIQ7yRdq0fft2bN26FYmJicjKylJZdvHiRZGq0j9SHjBN2zp16oSoqChYWFiwL9hnkPLs6oy8eoiDfOWOoz1rVnh4OCZOnIi+ffti586dCA4ORnx8PM6dO8f5ewpIygOmaVt2XzD6PFKeXZ1nbvQQbxXN3aBBg2BjY4OwsDC1y8eNG4fU1FSO9pxPnp6eCA0NRc+ePVU+d5MnT8aLFy/Udtom9fKa+4zznZFYUlJSsHjxYly5cgXp6emoVauWJGZXZ7jRQww3uatUqRJ+++031K1bV+3yCxcuoFevXrh165aWK9NP5ubmuHHjBtzc3ODo6IhDhw7B29sbcXFxaNCggV5fkycq6qQ8uzpvBSdJ4WjPmuXs7KwcF6hMmTI4ffo0gA9jjvDvok9z584dREZG4s2bNwDA4/iJtm/fjm7duqFBgwaoVauWyoPyp3Pnzrhw4UKO9oULF+r98CMMNyQp1tbWiI+Pz3X5nTt31E5QSuo1b94cu3btAgAEBwdj1KhR+OKLL9C9e3d06tRJ5Or0S3JyMlq0aIGKFSuibdu2ePz4MQCgf//++O6770SuTr+Eh4cjODgYTk5OuHTpEurVqwc7OzvcvXsXbdq0Ebs8vTF79my0adNGeUcp8GF29cmTJ2Pv3r0iVvb5eFlKD/GyVO66deuGd+/e5RgvKFuHDh1gbGyMbdu2abky/aRQKKBQKJS3227ZsgUnTpyAh4cHhgwZAiMjI5Er1B+BgYFISkrCypUr4eXlpfwOR0ZGIiQkBNevXxe7RL3BvmCaExYWhvDwcMnNrs5woyMKMsjXxo0b0aFDhxx3WxBw6dIl+Pj4oF27dhg7diwqVaoEALh58ybCwsKwd+9enDx5kqeuC+Dt27e4evUqkpKSoFAolO0ymQwBAQEiVqZfnJ2dERkZCW9vb5VfyHfv3kX16tWVs9fTx7EvmGaNGzcOq1atglwux/79+yUxuzpvBdcRRkZGuHr1ar7W7dWrVyFXo7842rNmHThwAH369FH7y0Imk0Eul4tQlX7KyMhQOyDnixcvYGJiIkJF+iu7L5ibm5uyL5i3tzf7guVDUZldnWdudAgH+dIcjvasGR4eHmjVqhUmT57MKQE+k5QHTNO2AQMGwNXVFaGhoViyZAnGjBmDRo0a4fz58+jcuTNWrVoldok6q6jMrs5wo0O++eYbrFu3Dh4eHhzkS0s42nPerKyscOnSJZQvX17sUvTetWvX0KJFC9SqVQuHDx9G+/btVQZM4zHOP/YFo49huNEhHORL+9g5O2/9+vVDo0aN0L9/f7FLkQSpDpgmBvYF+zzv3r2Dp6cn9uzZAy8vL7HL0TiGGyrSGG7y9vr1a3Tt2hUODg6oVq1ajr+I9fmavLZJecA0bWNfMM1wcXHBX3/9xXBD2nHnzh3Ex8ejadOmMDMzgyAIkMlkYpclSQw3eVu1ahWGDBkCU1NT2NnZqXwO9f2avLbZ2toq55X6t4ULF2LSpElITU0VqTL9w75gmiHl2dUZbnRIcnIyunXrhujoaMhkMsTFxaFcuXLo168fbG1tMXfuXLFLlByGm7w5OztjxIgRGD9+PAwMOObn51i5ciW+//57HD16FJ6engA+DJj2448/Ys+ePWjSpInIFeoP9gXTDCnPri6tqKbnRo0aBSMjIyQmJqqcJuzevTtCQkIYbkjrsrKy0L17dwYbDRgwYABevHiBli1bSm7ANG3r0qULYmJiGG4+k5RnV+eZGx3CQb60j2du8jZq1Cg4ODjg+++/F7sUyZDigGnaxr5g9DE8c6NDOMiXZhRktOdff/2V1+zzIJfLERYWhsjISFSvXj3HLxEOT5C3ojJgmrZt2rQJBw8ehKmpKWJiYnL0BeOxJJ650SEc5EtzHBwccPLkyY+GG8obhyf4PEVlwDRtY18wzdm+fTu2bt2KxMREZGVlqSy7ePGiSFV9PoYbHcJBvjSHoz0TSVeJEiVw7tw5/kz8TOHh4Zg4cSL69u2L5cuXIzg4GPHx8Th37hyGDRuGn376SewSPxnDjY7hIF+awdGeSZdIfcA0bWNfMM2Q8uzq7HOjQ7IH+Zo4cWKOZRzkq2CuXbumnCDz9u3bKss4ZhBpm5GREd6+fSt2GZLBvmCakZiYiIYNGwIAzMzMkJaWBgDo06cPGjRowHBDmtG5c+c8B/liuMm/6OhosUsgUjFs2DDMmjVLkgOmaVtsbCxq1qwJ4MMfMv/GP17yT8qzq/MbpkNmz56NNm3aqB3ka+/evSJXp5842jPpinPnziEqKgoHDx6U3IBp2sY/XjSjefPm2LVrF2rWrIng4GCMGjUK27dvV86urs/Y50bHhIWFITw8nIN8fSaO9ky6Jjg4OM/la9as0VIlRB9IeXZ1hhsdxEG+Pl9gYCCSkpKwcuVKeHl5KTvKRUZGIiQkBNevXxe7RCIi0Ul1dnVelhIZB/kqHAcPHkRkZCRKly6t0u7h4YH79++LVBURke6Q8uzqPHMjMg7yVTgsLS1x8eJFeHh4qNzieP78efj7+6v9MhMVNqkOmEb6Scqzq3NoR5Hdu3cvXw8Gm4Jp0qQJ1q1bp3wuk8mgUCgQFhaW56i7RIUlPDwcwcHBcHJywqVLl1CvXj3Y2dnh7t27aNOmjdjlURH09OlThISESC7YADxzozM4yJdmcbRn0jVSHjCN9FO/fv3QqFEj9O/fX+xSNI7hRoe4uLjgr7/+YrjREI72TLrE3NwcN27cgJubGxwdHXHo0CF4e3sjLi4ODRo04KVS0jopz67ODsU6hIN8aQ5HeyZdI+UB00g/SXl2dZ650SGdOnVCVFQULCwsOMjXZ7K1tc1ztOfU1FSRKqOiasCAAXB1dUVoaCiWLFmCMWPGoFGjRsoB01atWiV2iVTESHl2dZ4e0CE2Njb46quvxC5DEjjaM+ma5cuXK8cRGTZsGOzt7XHixAm0b98eQ4YMEbk6KoqysrLQvXt3yQUbgGduSMI42jPpGqkOmEb6Scqzq/PMDUnW2LFjkZycjDp16kAulyMyMpKjPZNopDxgGuknKc+uzjM3OoaDfH06daM9A8CcOXPQtGlT1KtXT9mmzx3lSD9JecA00k95jfklk8lw+PBhLVajWQw3OiQ8PBwTJ05E3759sXz5cgQHByM+Ph7nzp3DsGHD8NNPP4ldok7jaM+ky6ysrHDp0iWOsUSkBQw3OoSDfBFJl5QHTCPSNQw3OoSDfGkGR3smXSTlAdOIdA07FOsQDvKlGUZGRnj79q3YZRCpkPKAaUS6Rno3t+ux5s2bY9euXQCA4OBgjBo1Cl988QW6d++OTp06iVydfske7fn9+/dil0IEAJg4cSKmTp2KlJQUJCQkcGJcokLEy1I6RKFQQKFQKKde2LJlC06cOAEPDw8MGTIkx2lsyh1HeyZdU6JECZw7d44diom0gOFGx3CQL80IDg7Oc/maNWu0VAnRB1IeMI1I1zDc6BAO8kUkXSNGjMC6devg7e0tuQHTiHQNw40O4SBfRNIl5QHTiHQNw40O4SBfmsXRnomIiibeLaVDunTpgpiYGLHLkITw8HAEBwfDyckJly5dQr169WBnZ4e7d++iTZs2YpdHRESFiGdudAgH+dIcjvZMRFR0MdzokFWrVmHIkCEwNTWFnZ1djkG+OBZG/nG0ZyKioouXpXQIB/nSnOzRngEoR3sGwNGeiYiKAIYbHZKVlYXu3bvDwID/LJ+Loz0TERVdvCylQzjIl+ZwtGcioqKL4UaHcJAvzeJoz0RERRPDjQ7hIF+aw9GeiYiKLoYbkiSO9kxEVHQx3JAkcbRnIqKii7flkCRxtGcioqKLZ25IkjjaMxFR0cVwQ5LE0Z6JiIouhhuSJGdnZ4wYMQLjx4/noIhEREUMf+qTJHG0ZyKioos/+UmSgoKCsGXLFrHLICIiERQTuwCiwiCXyxEWFobIyEiO9kxEVMSwzw1JEkd7JiIquhhuiIiISFLY54aIiIgkheGGiIiIJIXhhoiIiCSF4YaIiIgkheGGiPRGs2bNMHLkSLHLICIdx3BDRJIUExMDmUyGV69eiV0KEWkZww0RERFJCsMNEemkjIwMBAYGwsLCAiVLlsTcuXNVlq9fvx516tSBpaUlnJ2d0atXLyQlJQEAEhISlAM52traQiaToW/fvgCAAwcOoHHjxrCxsYGdnR3atWuH+Ph4re4bERUuhhsi0kljxozBkSNHsHPnThw8eBAxMTG4ePGicvm7d+8wbdo0XLlyBTt27EBCQoIywLi6uuL3338HANy6dQuPHz/GwoULAXwITSEhITh//jyioqJgYGCATp06QaFQaH0fiahwcIRiItI56enpsLOzw2+//YauXbsCAF68eIHSpUtj0KBBWLBgQY5tzp8/j7p16yItLQ0WFhaIiYmBn58fXr58CRsbm1zf6/nz53BwcEBsbCyqVq1aSHtERNrEMzdEpHPi4+ORlZWF+vXrK9tKlCiBSpUqKZ9fuHABAQEBKFOmDCwtLeHr6wsASExMzPO14+Li0LNnT5QrVw5WVlZwd3fP13ZEpD8YbohI72RkZMDf3x9WVlbYsGEDzp07hz///BMAkJWVlee2AQEBePHiBVasWIEzZ87gzJkz+dqOiPQHww0R6Zzy5cvDyMhIGTwA4OXLl7h9+zYA4ObNm0hOTsbMmTPRpEkTeHp6KjsTZzM2NgYAyOVyZVtycjJu3bqFH374AS1atICXlxdevnyphT0iIm1iuCEinWNhYYH+/ftjzJgxOHz4MK5du4a+ffvCwODDj6wyZcrA2NgYixYtwt27d7Fr1y5MmzZN5TXc3Nwgk8mwZ88ePHv2DOnp6bC1tYWdnR2WL1+OO3fu4PDhwwgJCRFjF4moEDHcEJFOmj17Npo0aYKAgAC0bNkSjRs3Ru3atQEADg4OiIiIwLZt21C5cmXMnDkTc+bMUdnexcUFU6dOxfjx4+Hk5IThw4fDwMAAmzdvxoULF1C1alWMGjUKs2fPFmP3iKgQ8W4pIiIikhSeuSEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSWG4ISIiIklhuCEiIiJJYbghIiIiSfl/iHf0QWixHKEAAAAASUVORK5CYII=","text/plain":["<Figure size 640x480 with 1 Axes>"]},"metadata":{},"output_type":"display_data"}],"source":["# compare each model best auc for each data set\n","df.groupby(['data', 'model'])['AUC'].max().unstack().plot(kind='bar')\n","plt.ylabel('AUC')\n","plt.title('Best AUC for each model and each data set')\n","plt.show()"]},{"cell_type":"markdown","metadata":{"id":"qH75TNU71eRH"},"source":["# Discussion\n","\n","The goal of the original paper was to find the best prediction method for each dataset, comparing their performance on each task (in particular, the authors of DeepMicro wanted to see whether autoencoders would have superior performance to traditional dimension reduction or classification-only techniques).\n","\n","Below is a summary of DeepMicro's findings of the best methods, compared to ours.\n","\n","| Dataset | DeepMicro Best Results | Our Best Results | Our Best Results (excluding Plain) |\n","|------|-------------------|-------------|-------------|\n","| Marker IBD | SAE/SVM; AUC=0.955   | CAE/RF; AUC=0.929 | CAE/RF; AUC=0.929   |\n","| Marker EW-T2D | SAE/RF; AUC=0.899   | RP/RF; AUC=0.737 | RP/RF; AUC=0.737   |\n","| Marker C-T2D | CAE/MLP; AUC=0.763 | Plain/MLP; AUC=0.807 | CAE/SVM; AUC=0.807 |\n","| Marker Obesity | DAE/RF; AUC=0.659 | Plain/RF; AUC=0.739 | RP/RF; AUC=0.685 |\n","| Marker Cirrhosis | CAE/SVM; AUC=0.940 | Plain/RF; AUC=0.992 | CAE/SVM; AUC=0.98 |\n","| Marker Colorectal | Plain/RF; AUC=0.811 | MLP/PCA; AUC=0.846 | MLP/PCA; AUC=0.846 |\n","\n","In general, the findings of the original paper differ significantly from our own.\n","\n","None of the original paper's most performant models perfectly replicated in our own study.  In all 6 marker datasets, the model we found that gave the best results was different than theirs;  in some cases, such as IBD, we measured worse performance (AUC=0.929 vs our AUC=0.955),  while in other cases, ours outperformed the original's, such as in the Cirrhosis dataset (AUC=0.940 vs our AUC=0.992).\n","\n","Furthermore, while the original paper claimed that the DeepMicro (autoencoder-based) methods outperformed other classification methods in 5 out of 6 datasets, our conclusion was quite different:  in only 1 out of 5 datasets (Marker IBD) did we find that an autoencoder (in this case, CAE with RF classification) had superior results.\n","\n","The original paper claimed that \"none of the autoencoders dominate across the datasets in terms of getting the best representation for classification\".  However, in our reproduction, we observed the CAE consistently outperformed the other 3 autoencoder methods in all of the datasets except for Marker EW-T2D.\n","\n","The training time benchmarks of the original paper also diverged from ours slightly, but this is less worth discussing due to the fact that we ported our code over to pytorch and used different hardware settings for our training runs.  In general, we confirmed that MLP takes the longest to run with the plain method, followed by SVM, and then RF.\n","\n","The orginal paper suggests repeating the entire training process for each method 5 times, then taking the average of the AUC to get the final result.  Due to the long training times for each dataset (roughly 5 hours each), this aspect was less practical.  Otherwise, most aspects of the paper were fairly easy to *attempt* to reproduce, given the extensive, existing codebase, though very little of the findings were able to be reproduced in actuality.\n","\n","While the codebase itself was fairly well-organized and easy to run, the specific parameters of each experiment had to be reconstructed, as the original simply had a CLI that took dozens of arguments to configure each experiment.  It would have been more helpful if the paper also gave the exact arguments they gave when running each method.\n","\n","Further studies might investigate the high variance and discrepency of the original findings.  For example, perhaps the datasets we found where the plain method had the highest AUC have something in common; further research could be done into what exactly this commonality is.\n","\n","Our conclusion is that the central findings of the paper, namely that autoencoders consistently outperform other methods on microbiome data for disease prediction, are not necessarily reproducible.  In 3 out of 6 datasets, non-representation-learning-based \"plain\" methods outperformed all other methods we tested, while in 5 out of 6 datsets, non-autoencoder-based methods outperformed autoencoder-based methods.  "]},{"cell_type":"markdown","metadata":{"id":"SHMI2chl9omn"},"source":["# References\n","\n","1. Cho, I., & Blaser, M. J. (2012). The human microbiome: at the interface of health and disease. Nature Reviews Genetics, 13(4), 260-270.\n","2. Huttenhower, C. et al. Structure, function and diversity of the healthy human microbiome. nature 486, 207 (2012).\n","3. McQuade, J. L., Daniel, C. R., Helmink, B. A. & Wargo, J. A. Modulating the microbiome to improve therapeutic response in cancer. The Lancet Oncology 20, e77–e91 (2019).\n","4. Eloe-Fadrosh, E. A. & Rasko, D. A. The human microbiome: from symbiosis to pathogenesis. Annual review of medicine 64, 145–163 (2013).\n","5. Scholz, M. et al. Strain-level microbial epidemiology and population genomics from shotgun metagenomics. Nature methods 13, 435 (2016).\n","6. Truong, D. T. et al. MetaPhlAn2 for enhanced metagenomic taxonomic profiling. Nature methods 12, 902 (2015).\n","7. Oh, M., & Zhang, L. (2020). DeepMicro: deep representation learning for disease prediction based on microbiome data. Scientific reports, 10(1), 6026.\n","8. Pasolli, E., Truong, D. T., Malik, F., Waldron, L. & Segata, N. Machine learning meta-analysis of large metagenomic datasets: tools and biological insights. PLoS computational biology 12, e1004977 (2016).\n","9. Cawley, G. C. & Talbot, N. L. On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research 11, 2079–2107 (2010).\n","10. Varma, S. & Simon, R. Bias in error estimation when using cross-validation for model selection. BMC bioinformatics 7, 91 (2006).\n"]}],"metadata":{"accelerator":"GPU","colab":{"gpuType":"L4","machine_shape":"hm","provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.6"}},"nbformat":4,"nbformat_minor":0}