notebooks/Working with unbalanced data part 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are involved in building predictive models for biological end points in pharma research, you've certainly encountered the problem that our datasets tend to be very highly unbalanced - we usually have many, many more inactive molecules than active ones. The imbalance in the data can present challenges when evaluating model performance and \"confuse\" some of the machine-learning algorithms we use. \n",
    "\n",
    "Let's start with evaluating models. Although at first glance a classifier (we'll focus on classifiers here) that shows 99% accuracy on hold-out data sounds pretty good, it's not going to be practically useful if the data set is 99% inactive and the model is just always predicting \"inactive\". There are a number of balanced accuracy metrics available to solve this problem; for this post we will use [Cohen's kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa), which is readily available in scikit-learn. As an example of what a difference it makes, the normal accuracy for this confusion matrix:\n",
    "```\n",
    "[[400   0]\n",
    " [ 19   1]]\n",
    "```\n",
    "is 95.4% accurate but kappa, which can adopt values from 0 (bad) to 1 (good) is only 0.091. Paul Czodrowski has written an nice article discussing this in more detail that's well worth reading: [https://link.springer.com/article/10.1007/s10822-014-9759-6](https://link.springer.com/article/10.1007/s10822-014-9759-6). Paul has also done a [presentation](https://www.researchgate.net/publication/258240105_TheKappaStatistic_PaulCzodrowski) summarizing this.\n",
    "\n",
    "There are multiple strategies available for improving the performance of predictive models built on unbalanced data sets. Most of these focus on making the training set more balanced by either oversampling the active examples or undersampling the inactive examples. There's a broad literature on this and you can find plenty of other information online and if I end up having time I'll do a post looking at these approaches later. Here we're going to explore a different approach that I first learned about from [Dean Abbott](http://www.abbottanalytics.com/data-mining-consulting-services-about.php) at a [workshop](https://files.knime.com/sites/default/files/inline-images/strategies_for_predictive_analytics_-_dean_abbott_feb2014.pdf) he held at a KNIME User Group Meeting several years ago: adjusting the threshold a model uses to make its predictions. I've talked about this in a few venues but this is the first time that I've actually been somewhat systematic and written stuff down. There is also a body of literature available about the approach, which I will call adjusting the decision threshold, but since I've never formally written this up, I haven't yet done a real literature review. It looks like this article from Provost and Fawcett is one of the first publications: [https://link.springer.com/article/10.1023/A:1007601015854](https://link.springer.com/article/10.1023/A:1007601015854)\n",
    "\n",
    "Most classification algorithms can provide predicted probabilities for each class; the actual class predicted for any given new instance is the one which has the highest predicted probability. For a two-class problem this means that the class with a predicted probability of more than 0.5 is the final prediction. The key idea I learned from Dean (and that you can find in the literature) is that you often get much better balanced accuracy (as measured by things like Cohen's kappa) on unbalanced datasets by shifting the decision boundary and allowing the model to predict the minority class even when the predicted probability of that class is not the highest. To make that concrete: if I'm building a model to distinguish actives from inactives based on a training set where there are far more inactives than actives, I may benefit by assigning \"active\" to data points where the predicted probability of being active is smaller, often considerably smaller, than 0.5. For example, if I set the decision threshold for the model that produced the confusion matrix above to 0.1, I get the following confusion matrix:\n",
    "```\n",
    "[[384  16]\n",
    " [  6  14]]\n",
    "```\n",
    "This model is clearly producing far more false positives (16 instead of none) but finds far more of the true positives and has a kappa of 0.533. That's not great, but it is significantly better than the original 0.091. \n",
    "\n",
    "It's possible to imagine many different approaches for picking the \"best\" decision threshold for a model as well as multiple definitions of what \"best\" means at all. The lower the threshold is set, the more true actives that will be retrieved, together with more false positives. Since I wanted to consider a large number of models here, I chose an automated approach: I try a number of different decision thresholds and calculate kappa using the out-of-bag prediction probabilities for the training set; the threshold that produces the best kappa value is picked as the best.\n",
    "\n",
    "To demonstrate why this method works and when it doesn't work particularly well, here are two more examples taken from the ChEMBL datasets presented below; both are PubChem qHTS confirmatory assays. In the first example, assay ID CHEMBL1614421,the balancing approach works well: we start with a kappa of 0.033 on the holdout data, but by using the OOB predictions to pick a threshold of 0.200 we arrive at a significantly better kappa of 0.451. \n",
    "\n",
    "```\n",
    "assay_id CHEMBL1614421, description: PUBCHEM_BIOASSAY: qHTS for Inhibitors of Tau Fibril Formation, Thioflavin T Binding. (Class of assay: confirmatory) [Related pubchem assays: 596 ]\n",
    "--------- Default -----------\n",
    "ratio: 0.129 kappa: 0.033, AUC: 0.860, OOB_AUC: 0.850\n",
    "[[8681    4]\n",
    " [1101   22]]\n",
    "--------- Balanced -----------\n",
    "thresh: 0.200 kappa: 0.451\n",
    "[[8261  424]\n",
    " [ 595  528]]\n",
    "```\n",
    "\n",
    "For assay CHEMBL1614249, on the other hand, kappa starts at 0.000 (no compounds from the holdout set are classified as active) and the best the balancing procedure (again based on OOB predictions) can do is a kappa of 0.102 when the threshold is set to 0.050.\n",
    "\n",
    "```\n",
    "assay_id CHEMBL1614249, description: PUBCHEM_BIOASSAY: qHTS Assay for Identification of Novel General Anesthetics. In this assay, a GABAergic mimetic model system, apoferritin and a profluorescent 1-aminoanthracene ligand (1-AMA), was used to construct a competitive binding assay for identification of novel general anesthetics (Class of assay: confirmatory) [Related pubchem assays: 2385 (Probe Development Summary for Identification of Novel General Anesthetics), 2323 (Validation apoferritin assay run on SigmaAldrich LOPAC1280 collection)]\n",
    "--------- Default -----------\n",
    "ratio: 0.006 kappa: 0.000, AUC: 0.725, OOB_AUC: 0.741\n",
    "[[8356    0]\n",
    " [  51    0]]\n",
    "--------- Balanced -----------\n",
    "thresh: 0.050 kappa: 0.102\n",
    "[[8302   54]\n",
    " [  45    6]]\n",
    "```\n",
    "\n",
    "What's the difference here? Why does the balancing procedure produce reasonable results for one of the datasets and poor quality results for the other? A quite obvious difference between the two models can be seen in the ROC curves generated for the OOB predictions on the training sets. Note that these curves, which are generated solely using the predicted probabilities of being active, are not sensitive to the decision threshold.\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd0VOX28PHvzqRCICGhiPTeixIBRQEbol4rFhSxYQFERSygqCg/UETFi1ICXMuLDbuiV1GuBWwoRYog0oUABkIJkJA2s98/zhBCTMIAmcxMsj9rzSKnzs5hcvacp4qqYowxxhQnLNABGGOMCW6WKIwxxpTIEoUxxpgSWaIwxhhTIksUxhhjSmSJwhhjTIksURhjjCmRJYoKRESuF5FFInJARLaLyBcicqZ32xMi8kYRx6iINC2wT673+EOvvYX2TRWR8ALrwkVkh4hogXXfiUiW9/h0EZkvIu0KbC8yFu+2Id7fIVtEXitieyURmSIiaYfOXWDb2SLyrXf9phKuUw/v7zKmmO3feLeHe5drisjbIrLNe+4fRaRLgf1ri8hs73YVkYbFvXdJvNdtj4hEFbH+tkLreopISoFlEZF7ROR3EckQkRQRea/gdfcxBhGRZ0Rkl/c1XkSkhP3vFpGNIrLP+/92ZhH7RIrI6oLxetdf4o33gIj8JCKtjyVWU3osUVQQIjIM+DfwFFALqA9MAS47xlO9o6qxBV7xhbbvBS4ssHwRsKeI8wxR1VggEfgOeN3H998GjAFeKWb7dCABaOX9974C2zK8xz1Y3MlFJAKYCPxSzPZ+QHih1bHAQqCT9z3/H/BfEYn1bvcAc4A+xb3v0XiTy1mAApcexykmAvcC93hjbA58DFx8jOe5A7gc6AC0B/4F3FlMzF2AccBVQBzwMvCRiLgK7fogsKPQsc2AN4GBQDzwKTC74JcQU3YsUVQAIhIHjAbuUtUPVTVDVXNV9VNVLfameZxeB24ssHwjMLO4nVU1D5gF+PRt0Rv/x8CuwttEpAXOTfQOVd2pqm5VXVzg2F9V9XVgQwlvcT/wFbC6iPPHAaOAhwrFtEFVJ6jqdu97TgcigRbe7amqOgUnmRyvG4EFwGvATcdyoPemexdwnap+o6rZqpqpqm+q6rhjjOMm4HlVTVHVrcDzwM3F7NsQWKmqi9UZAmImUB2oWSC2RsANwNOFjr0A+F5Vf/B+Rp4B6gA9jjFeUwosUVQMpwPRwEdl8F4fA91FJF5E4nG+BX9S3M4iEgn0w7kJnqguwF/Ak96ipxUi4vO3eBFpANyKk1SL8hQwFfj7KOfpiJMo1vn4viNE5LOj7HYjzjfsN4ELRKSWL+f2OhdIUdVfjxLD3uJeBXZtAywrsLzMu64oXwAuEenifYq4FVjKkdfvJeAR4GDhkLyvwstti/81jb9YoqgYEoE07zezklxTwg2iuH2+LbQ9C6eY4FqgLzDbu66wF73nPwAMAZ48pt+oaHVxbiTpwMne8/4/EWnl4/EvAo+p6oHCG0QkCeiGc2MrlohUxXmqelJV0315U1Udp6r/KuGcZwINgHe9T0jrget9ObdXIrDdhxjii3sV2DUW5/oekg7EFlNPsR/4APgByMZ5GrvD+3SBiFwBhKtqUV9g5gI9vHUtkTjJJBKo5NNvbEqVJYqKYRdQ3Yfy3XdLuEEUt8/ZRewzE+cbcEnFTvd4zx+NU879voi09/H3Kc5BIBcYo6o5qjoP+BbodbQDReQSoIqqvlPEtjCc+px7S0q2IhKDkyQXqGrhopQTcRPwlaqmeZff4sjipzwgotAxETjXApz//9qlFMsBoGqB5arAgUM3/0Juw3mKaINzk78B+ExEThaRysB44O6i3kRVV+P8jpNwklx1YBWQUtT+xr+sYqhi+BnnW/3lwPtl8H7f49yYFOfbZJPidlRVD/C9iKzDuaEvP4H3PZFjzwWSRORQsUgc4Pa2CroJSALe8X5xPlQZmyIiV6vq996WSB8DWymmcvd4eJPPNThFOIdiiwLiRaSDqi4DNuPUBxTUCKcYDuBrYLKIJKnqomLe5xGcb+1F8jY8AFiJU5F9qBirg3ddUToAn6rqGu/yHBHZDpyBUyzXEOf/HpxEEuf9Hbuq6iZVfR/v59VbjHkrJ1bPY46TPVFUAN4ikMdxbhaXi9OENEJELhSR8X54PwUuAS4t5pvmEUTkdJzK7II3nDARiS7wivLuGy4i0Tg3a5d326EvPPNxbpoPe/frBvQEvvQeG+Y9NsJZlGhvsQbAYzgtgTp6X7OBGcAtHC7KOrTtIu8xnYBfvC2l3sd5ornRm/wK/47RODd4gCjvsi8uB9ze63Po/VvhJONDjQbeAW4Rkc7e5qvNcVp7zQJQ1bU4T0RvHyrK8f7ufUVkhHefpwq1ZjviVSCemcAwEakjIifjVP6/VkzsC4GLRaSxN67zca7x795XvQK/021AqvfnLd5r1klEXCJSA5iGk3T+0cjAlAFVtVcFeeFUGi/CaSb6N/Bf4AzvtieAN4o4RoGmBfbJxSl+KPiqWXjfQudoijd/eJe/w3nCOXT8OuC+Atuf8J6r4CulhG1PFDi2Dc4TVAZOUcUVBbb1LOLY74q5Vq/hFGEVta2h99hw73IP73JmoetyVqHreMSrwLZHgC+Kea85OK2MCq+/xvt/eCiGW3ES7T7v9RwBhBXYX3Cax670xrkVJ8G0OcbPkOAUGe32vsYDUmB7/u/t3Xc0TvLeD/wB9C/mvD0P/R8XWPeD97jdOImicqD/hirqS7z/IcYYY0yRrOjJGGNMiSxRGGOMKZElCmOMMSWyRGGMMaZEIdePonr16tqwYcNAh2GMMSFl8eLFaapa43iODblE0bBhQxYtKrLPkDHGmGKIyF9H36toVvRkjDGmRJYojDHGlMgShTHGmBJZojDGGFMiSxTGGGNKZInCGGNMifyWKETkFRHZISK/F7NdRORFEVknIstF5FR/xWKMMeb4+bMfxWs4s1MVN8PZhUAz76sLzlzEXfwYjzHGBDW3R8nzePB4yP/Xrc66fQfzUFXcqng84FHF7XGWVZWd+3OIiij6u39uTs4JxeW3RKGq80WkYQm7XAbMVGec8wUiEi8itVW1xLl9jTHG31SVPI+Sme1mV0Y2uW4l1+1hb2YuuzKyWZt6gJhIF+t3ONOru8IEt0fJ9Si7M7LZvjeLmlWj8m/4h27ov2/dx0lVnTmrnBu+8z4ej7I/+2hT2h+fPd++Qk7q+hM6RyB7ZtfBO5OVV4p33T8ShYjcAdwBUL9+/TIJzhgTurbuPciW3Zls3pVJWJiQlevmj+37SKgcSXaeh6Wb91I7PpqcPA+rtu8j/WAuOXke8jxKntuDpxSm6dmQllHk+r/3ZR312KjwMMLDhLAwITxMcIUJaQdyiIuJoFbVKMJECBNnfViY4BLwKGzenUm7OnFHnGv9zrb8vOTTE/pdApkopIh1Rf73qOp0YDpAUlKSzbRkTAWjquzJzCUr103KnoOk7Mlk5bZ9uD3Kp8u2UaNKFKv/3l+q73noRp2T56FZzVjCXWFEuoTMHDfNT6pCVo6bDvXiycjOo25CJSpFuAh3CeFhYeR5PFSNjqByVDiuMHCFheHy3tgjXELlqPD887tEcLmcfyPDw4hwnVjV8apVq1iyZAk33HADAHrLafz12C00atTo+K/FCUV0YlJw5sw9pC6wLUCxGGPKiKqyNzOXvQdz+Ts9C48quzJySNufjUeV7DwPK1LSqRTp4qOlW6kU4SIjx13iOXdlHFkGHx4m5HmU7s1rkJqexakNqhEeJhzIzqN5rSpEhodxMCePRtVjiQwPwxUGzWtVoXpsVP43eJGivssGr8zMTMaMGcOzzz6Ly+Wia9euNG3aFBHhRAdSDWSimA0MEZFZOJXY6VY/YUzoyMnzkLovi41pGbg9Snaem/U7M/KLcTyqfL82jexcNzGRLjJz3GxPP3qxS2EFk0TNKlFER7jYsT+Lbk2qUyU6nHZ140msHEm7unFUjgwnoXIkkeEVq+X/F198wV133cXGjRsBGDBgAImJiaV2fr8lChF5G2fC9OoikgKMAiIAVDUZ+By4CGci+EzgFn/FYow5Oo9H2Z2Zw4GsPDbtymD9zgw2pWUQE+lif1YeO/Zl8d2anSRUjmTn/uxSec9DN/4mNSrjUTg5Poa4mAgiXUJ2noeO9eJpc3IcteOjT7hIpjzaunUrQ4cO5f333wegffv2JCcnc/rpp5fq+/iz1dN1R9muwF3+en9jzD/ty8rly9//ZnlKOvuyclm1bR+ZOW72ZeWyP8u3VjcFk0R0RBhZuR4a16hM0xqxRLjC2JeVS91qMdRPqEx8pQhcIrSpU5UqURFEhoeRGBtpN/1Sctddd/HJJ59QqVIlRo8ezb333kt4eOnf1kNuPgpjTMk8HmVb+kFWbdvH8pR0NqQdYMXWdLbsPujT8XXiY6hWOYLEylFEhodRJTqcDnXjqRoTzslxMcRXiqR6bCQJlSNDrhy/PMjLy8tPBs888wwRERE8//zzfm0RaonCmBDm9ijLUvby3qIU5q/Zyda9viUDgBu61qfNyXHEx0TQqnZV6laLIdy+6Qet9PR0Hn30UdasWcOcOXMQEVq0aMF7773n9/e2RGFMCPF4lFXb9/HDujQW/7WHuatSi923fd04snLd9GxRk+7NapAYG0nzWlVwhdlTQChRVd577z2GDh3K9u3bcblcLF26lFNOOaXMYrBEYUwQSdmTyfKUdHLdHjamZbBk815SdmciAjv2ZRfbe7d6bBRXnHIyl3Q4mcY1YomNsj/t8mD9+vUMGTKEOXPmAHD66aeTnJxM+/btyzQO+zQZEyCqys4D2Xy+fDtPfLrK5+MqR7qoGhPB2S1rcnG72nRrWt2PUZpAee6553jsscfIysoiPj6eZ555httuu42wsLIvHrREYUwZyczJY9mWdKbOW89vm/eU2MqoWc1YOjWoRlxMBHUTKlG3WgwtT6pCrSrRhFnRUYWQmZlJVlYW/fv357nnnqNmzZoBi8UShTGlTFXZsvsgv23Zw+yl2/hl424OlDDgW9OasZzZtDrnt65Fl0YJVqFcQe3cuZM///yTM888E4Dhw4fTs2dPunfvHuDILFEYc9w8HmXe2p1sSsvgp/W7+MM7uNzR+iP0bFGDns1r0L15DRomVrYnhArO4/Hwyiuv8NBDDxEeHs7q1atJSEggKioqKJIEWKIwxmeL/9rNV6tS+XzFdqpGR7By274S94+LiaBJjcq0qxPHvzqcTPu6cUSFu8ooWhMKfv/9dwYOHMiPP/4IwPnnn09mZiYJCQkBjuxIliiMKcGB7Dxe+mYt0+ZtKLTlcH+FdnXiaF6rCgmVI+jZoiatalcloXJk2QZqQkpGRgajR49mwoQJ5OXlUatWLf79739z7bXXBmUnRksUxnjty8rlp3W7+GBJCgvW7yqyKerpjROpn1CJDvXiOa91TWpWiQ5ApCbUXXXVVfmd5gYPHszYsWOJj48PdFjFskRhKqytew8y6Zu1bN2bxfw1O0vc97rO9bnvvGbUrGqJwZy44cOHk5qaytSpU+nSJfhngLZEYSqUQ0VJb/z8V5FzHLjChJtOb0jdajE0qRlL54YJxERavYI5fnl5ebz00kts2rSJiRMnAtCzZ08WLVoUkD4Rx8MShSn3dmfk8OyXq5n35062FZoPoWaVKC5sexJ9O9e34S1Mqfv111+58847Wbp0KQB33HEHbdq0AQiZJAGWKEw5tXDTbn5ev4uXf9hI+sHcI7ZFusLo0aIGg3o24dT61QIUoSnP9u7dyyOPPEJycjKqSoMGDZg0aVJ+kgg1lihMubA7I4fPlm/jgyVbWbZl7z+2V450cXbLmgw7vzmNa8QGIEJTUcyaNYuhQ4eSmppKeHg4999/P4899hiVK1cOdGjHzRKFCTmqypLNe5i7ageL/9rNwk17itwv0hXGbWc1ou9p9amfWKmMozQV1VdffUVqairdunVj6tSptGvXLtAhnTBLFCZk7MvK5Y0FfzF+zp/F7tO1cQJDzm7GaY2qWec2Uyays7PZunUrjRs3BmD8+PGcddZZ3HTTTSFVD1ESSxQmqKkqr/y4ibH/XYVHj9zWuEZlujWpTouTqnD5KXVsaG1T5r755hsGDRpEWFgYy5YtIzIykurVq3PLLbcEOrRSZX9ZJiipKuO+WM20+Uf2iHaFCRe1q81DF7SgXoIVJ5nASE1N5YEHHuCNN94AoGXLlqSkpOQ/VZQ3lihM0EjZk0nyvPV8/ccOthdqxnp+61o8enErGiSGboWgCX0ej4cZM2YwYsQI9u7dS3R0NI8++igPPvggkZHld9gWSxQmoLLz3Pzn+428u2gLf+3K/Mf2qzrVZfRlbagUaR9VE3hXXHEFs2fPBuCCCy5g8uTJNGnSJMBR+Z/99Zkytycjh0+Xb+M/329k8+4jk0ODxEr079qADvXibbRVE3SuvPJKfv31VyZOnMjVV18dlAP4+YOo6tH3CiJJSUm6aNGiQIdhjoPbowz/YDnvL075x7bru9Rn6HnNbJA9E1Rmz55NSkoKgwcPBpy6swMHDlClSpUAR3bsRGSxqiYdz7H2RGH8TlV5d9EWhn+w4oj1vducRPt6cQzq0aTCfDMzoWHz5s3cc889fPLJJ0RFRdG7d28aN26MiIRkkjhRliiM3+w6kM38tTu5751lR6y/8fQGPHlpG0sOJujk5uby4osvMmrUKDIyMqhSpQpjxoyhQYMGgQ4toCxRmFK1PGUvo2av5LfN/xxGo1vTRF6+6TSiI6zewQSfBQsWcOedd7J8+XIArr76al544QXq1KkT4MgCzxKFOWFujzJt/vpie0zf0LU+g3o2pU58TBlHZozvHnvsMZYvX06jRo2YNGkSF110UaBDChqWKMxxW5u6n7vf/o0/U/dTuE3E8N4t6dOpjlVOm6Clquzfv5+qVasCMGnSJGbOnMnIkSOpVMk6cxZkrZ7MMcvKdTNh7hqmF+o1fUGbWgw7vwUtTqp4lX0mtPz5558MHjwYEWHu3LkVor7MWj2ZMuH2KA+9v5wPlhxu3hoZHsZ7d55O+7pxFeKPzYS2rKwsnn76acaNG0dOTg6JiYls2rSJRo0aBTq0oGaJwhxV2oFsJsxdw1u/bM5fVz+hEkkNq/HUFe2sctqEhLlz5zJ48GDWrVsHwK233sr48eNJTEwMcGTBz6+JQkR6AxMBF/AfVR1XaHt94P8B8d59Rqjq5/6MyfjO7VHunfUbny3ffsT6+85rzr3nNQtQVMYcG1VlwIABvPrqqwC0bt2a5ORkzjrrrABHFjr8lihExAVMBs4HUoCFIjJbVVcV2O1R4F1VnSoirYHPgYb+isn4bvaybUz46k82FRh/6d/XduTSDicTZvNKmxAiIjRs2JCYmBgef/xxhg0bVq4H8PMHfz5RdAbWqeoGABGZBVwGFEwUClT1/hwHbPNjPMYHf2zfx/APlrM8JT1/3S3dGvJArxZUtvkeTIhYunQp27dv58ILLwRg+PDh9O/f3+oijpM///LrAFsKLKcAXQrt8wTwlYjcDVQGzivqRCJyB3AHQP369Us9UAMHsvN47OPf+ei3rfnrmtaMZXr/TjbHtAkZ+/fvZ9SoUUycOJHExERWr15NQkICUVFRliROgD8TRVHlE4Xb4l4HvKaqz4vI6cDrItJWVT1HHKQ6HZgOTvNYv0RbQe3Yn8Wkb9Yx8+e/jlj/+L9ac0u3htaSyYQEVeXjjz/mnnvuISUlhbCwMK6//noiIiICHVq54M9EkQLUK7Bcl38WLQ0AegOo6s8iEg1UB3b4MS4DrNtxgCc/Xcn3a9OOWH9eq1pM6XcqkeHlY65fU/799ddfDBkyhM8++wyApKQkpk2bxqmnnhrgyMoPfyaKhUAzEWkEbAX6AtcX2mczcC7wmoi0AqKBnX6MqcI7kJ3HddMXsGJr+hHrx/dpz9VJde0JwoQUVaVPnz4sXryYqlWr8tRTTzFw4EBcLmuyXZr8lihUNU9EhgBf4jR9fUVVV4rIaGCRqs4G7gdmiMh9OMVSN2uodRUPEekHc3nkwxX8d8WRTV3/c2MS57WuFaCojDk+Ho+HsLAwRITnnnuO5ORkXnjhBWrXrh3o0MolG8KjAvht8x6umPJT/nL9hEr0bnsSj1zUKoBRGXPsdu3axYgRIwCYMWNGgKMJLTaEhylSVq6bu95cwterD1f5/L9bO9OjeY0ARmXMsVNVZs6cyQMPPEBaWhqRkZGMGjWKunXrBjq0CsESRTm1MS2Ds5/7Ln+5ea1YXrulMyfbUN8mxPzxxx8MGjSIefPmAdCzZ0+mTp1qSaIMWaIoZ3LyPDzx6cojxmV6oFdz7jq7qVVUm5Ciqjz++OM888wz5ObmUr16dZ5//nn69+9vn+UyZomiHNmYlsH1MxawPT0rf93PD59D7Th7ijChR0TYunUrubm53H777YwbN46EhIRAh1UhWWV2ObE2dT99pv7Evqw8AB68oAWDezaxb14mpGzbto20tDTat28PQFpaGn/++SfdunULcGShzyqzK7hZv25mxIcrAIiOCOPr+3vatKMmpLjdbqZOncrIkSOpU6cOS5cuJTIykurVq1O9evVAh1fhWaIIYbluD49+9DvvLDo8pNa8B8+mVlWbftSEjiVLlnDnnXdyqKSge/fu7Nu3zxJEEPEpUYhIJFBfVdf5OR7jo8Ktmro2TmBa/yTiYmxsGxMa9u3bx2OPPcakSZPweDzUrVuXF198kcsvv9yKTIPMUROFiFwMTAAigUYi0hEYpapX+Ds4U7QtuzO5YsqP+ctPXdGO67vYqLomdKgq3bt3Z9myZbhcLoYNG8YTTzxBlSo233ow8uWJYjTO8ODfAqjqUhFp6teoTLG+WLGdQW8uAaBKVDhz7utu9REm5IgI9913H1OmTGHatGl07Ngx0CGZEviSKHJVdW+hR8HQaipVTjz/1Z+89M3h0r//3nOWJQkTEnJycpgwYQIul4sHH3wQgBtvvJEbbrjBBvALAb4kij9E5BogzDsS7L3AAv+GZQpbsGFXfpKIjghj6eO9iI6wPzAT/L7//nsGDhzIqlWriIqK4sYbb6RWrVqIiCWJEOHLpANDgE6AB/gQyMJJFqaMrE3dT9/pTm7u0iiBVU/2tiRhgl5aWhq33nor3bt3Z9WqVTRr1ozPPvuMWrVstOJQ40uiuEBVh6vqKd7XCOBCfwdmHD+uS+P8F+bnL8+4KYmwMGsRYoKXqvLqq6/SsmVLXn311fwB/JYvX8555xU527EJcr4kikeLWDeytAMxR/J4lBfmrqHff37JX/fN/T2oGm3NX03we+ONN9i1axfnnHMOy5cv54knniA62vr3hKpi6yhE5AKcaUrriMiEApuq4hRDGT85mOPmgfeW5U8y1LVxAlP7daJa5cgAR2ZM0TIzM0lPT6d27dqICFOmTGHhwoX069fP+kSUAyVVZu8Afsepk1hZYP1+YIQ/g6rItu09SI9nvyXX7TQsu/ucpgw9rzkuK24yQeqLL77grrvuonHjxsydOxcRoUWLFrRo0SLQoZlSUmyiUNXfgN9E5E1VzSpuP1N65q5KZdg7S8l1K5GuMB7q3YLbzmoc6LCMKdLWrVsZOnQo77//PgBVqlRh165dNvRGOeRL89g6IjIWaA3kFzKqanO/RVUBjftiNcnz1gPgChO+fdAG9jPBye12M3nyZB599FH2799P5cqVGT16NPfccw/h4TZ8XHnky//qa8AY4Dmc1k63YHUUpSbP7aHzU1+zOyMHgKQG1Uju34nqsVEBjsyYf/J4PPTo0YMff3SGkLn88suZOHEi9evbEDLlmS+tniqp6pcAqrpeVR8FzvZvWBWDx6P866Uf8pNEm5Or8v6gMyxJmKAVFhZGr169qFevHp988gkfffSRJYkKwJcnimxxmi2sF5GBwFagpn/DKv92HcjmjHHfkJ3nPJwNPa8ZQ8+z0jwTXFSVd999l/DwcPr06QPA8OHDGTZsGLGxsQGOzpQVXxLFfUAscA8wFogDbvVnUOWdqvL47JX5SWLkRa24vbtVWpvgsn79egYPHsxXX31FjRo1OOecc6hWrRpRUVFERdlTb0Vy1EShqod6fO0H+gOISF1/BlXezV2Vyn+XO30kZt3Rla6NEwMckTGHZWdn8+yzzzJ27FiysrKoVq0aY8eOJS4uLtChmQApMVGIyGlAHeAHVU0TkTbAcOAcwJLFcViyeQ93vL4YgJvPaGhJwgSV7777jkGDBrF69WoA+vfvz3PPPUfNmlbaXJEVW5ktIk8DbwL9gDkiMhJnToplgBWmH4dlW/Zy5ZSfAKiXEMOIC1sGOCJjDnO73QwePJjVq1fTokULvvnmG2bOnGlJwpT4RHEZ0EFVD4pIArDNu/xn2YRWvvz5935ufW1h/vLHg7vZCLAm4DweD1lZWVSqVAmXy8XUqVOZP38+Dz30kNVDmHwlJYosVT0IoKq7RWS1JYnj883qVG59zZk4vmp0OF/f35NEawJrAmzFihUMHDiQli1b8vLLLwPQo0cPevToEeDITLApKVE0FpEPvT8L0LDAMqp6pV8jKyfe/OUvRn70O+BMXTrrjtOpUcWShAmcjIwMRo8ezYQJE8jLy2Pjxo3s2bOHatWqBTo0E6RKShR9Ci1P8mcg5dEPa9Pyk8R5rWoy6fpTrbjJBNSnn37KkCFD2Lx5MyLC4MGDGTt2LPHx8YEOzQSxkgYF/LosAylvVm5Lz6+TOK9VLWbc2MmGWzYBk5eXx7XXXsuHHzqFAh07dmTatGl07tw5wJGZUODLEB7mGG1PP8iNL/9KjttD18YJTOl3qiUJE1Dh4eHExcURGxvLCy+8wMKFCy1JGJ+Jqvrv5CK9gYmAC/iPqo4rYp9rgCcABZap6vUlnTMpKUkXLVrkh2hLR57bQ6cx/yP9YC7VKkXw44hzqBRpI2qasvfLL05f2S5dugCwa9cuDh48SN261gWqIhKRxaqadDzH+nwHE5EoVc0+hv1dwGTgfCAFWCgis1V1VYF9mgEPA91UdY+IhHSDbVXl2ukLSD+YC8Ck60+1JGHK3N69e3n9DIeRAAAgAElEQVT44YeZNm0aLVu2ZOnSpURGRpKYaJ07zfE5atGTiHQWkRXAWu9yBxF5yYdzdwbWqeoGVc0BZuH0zSjodmCyqu4BUNUdxxR9EFFVrpn2M4v/2gPAtP6d6NbUJnAxZUdVeeutt2jZsiXJycm4XC4uvfRS3G53oEMzIc6XOooXgX8BuwBUdRm+DTNeB9hSYDnFu66g5kBzEflRRBZ4i6pCjqrS7z+/sHCTkyRu7daIC9qcFOCoTEWydu1aevXqRb9+/UhNTaVbt2789ttvjBs3jpgYmwDLnBhfykXCVPWvQpWxvnxFKar2tnCFSDjQDOiJM3bU9yLSVlX3HnEikTuAO4CgG/s+/WAul076gb92ZQI2Eqwpe7m5uZxzzjmkpKSQkJDA+PHjueWWWwgLs7YqpnT48knaIiKdARURl4gMBdb4cFwKUK/Acl2cYUAK7/OJquaq6kbgT5zEcQRVna6qSaqaVKNGDR/euuw88tGK/CTx5KVtLEmYMnOoIUpERARjx47l5ptvZvXq1QwYMMCShClVvnyaBgHDgPpAKtDVu+5oFgLNRKSRiEQCfYHZhfb5GG8xlohUxymK2uBb6IE3+tNV+cOFv3xTEjed0TCwAZkKITU1lf79+zNmzJj8dTfeeCOvvvoqwfZFypQPvhQ95alq32M9sarmicgQ4Euc5rGvqOpKERkNLFLV2d5tvURkFU5x1oOquutY3ysQFv+1m1d+3AjAA72ac26rWgGOyJR3Ho+HGTNmMGLECPbu3Ut8fDxDhw6lSpUqgQ7NlHNH7UchIutxioTeAT5U1f1lEVhxgqEfxfKUvVw6yZlc/pqkuoy/qkNA4zHl37Jlyxg4cCALFiwAoHfv3kyePJnGja2o0/jmRPpRHLXoSVWbAGOATsAKEflYRI75CaO8yHN7uG6688daKdLFIxe1CnBEpjzLzc3lgQceoFOnTixYsIDatWvz7rvv8vnnn1uSMGXGpxovVf1JVe8BTgX24UxoVCHd9dYSMnKcRl9f3HsW8ZUiAxyRKc/Cw8P57bff8Hg83H333fzxxx9cffXVNiSMKVNHraMQkVicjnJ9gVbAJ8AZfo4rKC3ZvIcvV6YC8PqAzjRIrBzgiEx5tHnzZtxuN40aNUJESE5OJj09naSk4yo1MOaE+fJE8TtOS6fxqtpUVe9X1V/8HFdQemGu0yq4X5f6nNXMWpeY0pWbm8tzzz1Hq1atuP322/ObvzZr1syShAkoX1o9NVZVj98jCXJbdmfy/do0AK5JqneUvY05Nj///DMDBw5k+fLlACQkJJCZmUnlyvbUagKv2EQhIs+r6v3AByLyj6ZRFWmGu31ZuZw1/lsAOjWoRod6NsmLKR179uxhxIgRTJ8+HYBGjRoxefJkLrzwwgBHZsxhJT1RvOP9t0LPbJeT5+HqqT/nLz95aZsARmPKk+zsbDp27MjmzZuJiIjgwQcfZOTIkVSqVCnQoRlzhJJmuPvV+2MrVT0iWXg70lWIGfDuf28Zf6Y6XUfeGNCFtnXiAhyRKS+ioqIYMGAAX3/9NVOnTqV169aBDsmYIvlSmX1rEesGlHYgwWjxX7v5dJkzPNVL153Cmc1s2HBz/LKyshg1ahRvvfVW/rpHHnmE7777zpKECWol1VFci9MktpGIfFhgUxVgb9FHlR95bg9Pfpo/xxIXtasdwGhMqJs7dy6DBw9m3bp11KxZkyuuuIKYmBjCw21iKxP8SvqU/oozB0VdnJnqDtkP/ObPoAItJ89Dq8fn4PYoMREufhpxDq4w6+Bkjt3ff//NsGHDePvttwFo06YNycnJNkeECSkl1VFsBDYC/yu7cAIvK9fN+S/Mw+1xGnqNv6o91Spb72tzbNxuN9OmTeORRx4hPT2dmJgYRo0axX333UdkpH2eTGgpqehpnqr2EJE9HDnhkACqqgl+j66M5bk99J2+gC27DxIm8NzVHbikw8mBDsuEILfbzUsvvUR6ejoXXXQRkyZNolGjRoEOy5jjUlLR06HpTitMDe4bC/5i6Za9RLiECdd0tCRhjsn+/ftxu93Ex8cTGRnJjBkzSE1N5corr7SxmUxIK7bVU4He2PUAl6q6gdOBO4Fy2V30CW/l9fDeLS1JGJ+pKh9++CGtWrXi/vvvz19/5pln0qdPH0sSJuT50jz2Y5xpUJsAM3EGBnyr5ENCz6FmsAD9ujQIYCQmlGzatIlLL72UPn36sHXrVn7//XeysrICHZYxpcqXROFR1VzgSuDfqno3UMe/YZWtdTsOcPfbTkOu6zrXJybSFeCITLDLzc3lmWeeoXXr1nz22WdUrVqVSZMm8dNPPxEdHR3o8IwpVT5NhSoiVwP9gcu96yL8F1LZUlXOmzAvf3nUJdbxyZQsMzOTrl27smLFCgD69u3LhAkTqF3b+tqY8smXRHErMBhnmPENItIIeNu/YZWd+d4RYQEm9u1IdIQ9TZiSVapUiaSkJDIzM5kyZQq9evUKdEjG+NVR58wGEJFwoKl3cZ2q5vk1qhKU9pzZrR6bw8FcNzed3oAnL2tbauc15YeqMnPmTJo0acKZZ54JQHp6OpGRkdZxzoSME5kz25cZ7s4CXge24vShOElE+qvqj8fzhsHkz7/3k5XnTGva/3SrwDb/9McffzBo0CDmzZtHq1atWLp0KZGRkcTF2eCQpuLwpejpBeAiVV0FICKtcBJHyE+5NejNxahCUoNqNK1ZJdDhmCBy8OBBxo4dy/jx48nNzaVGjRo8/PDDRESUm+o5Y3zmS6KIPJQkAFT1DxEJ+TEIvl29gw07M3CFCdNvDPmcZ0rRnDlzuOuuu9iwYQMAt99+O+PGjSMhodwNRmCMT3xJFEtEZBrOUwRAP8rBoIBv/7oZgOs61yPBxnIyXgcOHKB///6kpaXRtm1bkpOT6datW6DDMiagfEkUA4F7gIdw6ijmAy/5Myh/25SWwVerUgEYcGbjAEdjAs3tduPxeIiIiCA2NpaJEyeSkpLCfffdZ0VNxnCURCEi7YAmwEeqOr5sQvIvVc3vXNejeQ0aVS+Xo5EYHy1evJg777yTyy67jMceewyA66+/PsBRGRNciu2ZLSKP4Azf0Q+YKyJFzXQXchb9tYcVW9MBePrKdgGOxgTKvn37uPfee+ncuTOLFy/m9ddfJzc3N9BhGROUShrCox/QXlWvBk4DBpVNSP711i9O3cS1SfU4Od7awFc0qsp7771Hy5YtefHFFxERhg0bxpIlS6yYyZhilFT0lK2qGQCqulNEfBkXKqit3JbOx0u3AjDgLJsboKLZv38/1157LV988QUAXbp0ITk5mY4dOwY4MmOCW0mJonGBubIFaFJw7mxVvdKvkZUyt0cZ+dHvqMJlHU+meS3rN1HRxMbGkp2dTVxcHOPGjeOOO+4gLCzkv/8Y43clJYo+hZYn+TMQf3t/8RaWbtkLwMiLWgU4GlNW5s+fT+3atWnWrBkiwiuvvEJ0dDS1atUKdGjGhIyS5sz+uiwD8bdXf9wEwOCeTahZ1YaBLu/S0tJ46KGHePXVVzn33HOZO3cuIkKDBjZUizHHqkI8dy9P2cvqv/dTOdLFPec2C3Q4xo88Hg+vvPIKLVq04NVXXyUyMpKzzjoLt9sd6NCMCVl+TRQi0ltE/hSRdSIyooT9rhIRFRG/jKXx8W/O7HUXtqttw4iXYytXrqRnz54MGDCA3bt3c+6557JixQpGjRpFeLgvfUuNMUXx+a9HRKJUNfsY9ncBk4HzgRRgoYjMLjhulHe/Kjg9v3/x9dzHwuNRPliSAsCVp5ariflMAenp6XTt2pUDBw5Qs2ZNJkyYwPXXX2/zVRtTCo76RCEinUVkBbDWu9xBRHwZwqMzztwVG1Q1B5gFXFbEfv8HjAf8MtHw9+vSSD+YS1xMBKc3TvTHW5gAOjSfSlxcHMOHD2fgwIGsXr2afv36WZIwppT4UvT0IvAvYBeAqi4DzvbhuDrAlgLLKRSaa1tETgHqqepnJZ1IRO4QkUUismjnzp0+vPVhHyx2niZuOr2B3TjKka1bt3LVVVfxxhtv5K8bOXIkU6dOpVq1agGMzJjyx5dEEaaqfxVa50vNYFF35fzp9Lwd+F4A7j/aiVR1uqomqWpSjRo1fHhrR57bw7d/7gDgovY2n3F5kJeXx8SJE2nZsiUffPABo0aNyq+oti8CxviHL4lii4h0BlREXCIyFFjjw3EpQL0Cy3WBbQWWqwBtge9EZBPQFZhdmhXaP67fxf6sPBokVqLlSVVL67QmQBYuXEiXLl0YOnQoBw4c4PLLL2fevHm4XNZAwRh/8iVRDAKGAfWBVJwbui/jPi0EmolII+9ER32B2Yc2qmq6qlZX1Yaq2hBYAFyqqqU2IfbrP28CoHfbk0rrlCYAMjIyGDJkCF26dGHJkiXUr1+fTz75hI8++oh69eod/QTGmBNy1FZPqroD5yZ/TFQ1T0SGAF8CLuAVVV0pIqOBRao6u+QznJjdGTn87w+n2OmaJLuZhLLw8HD+97//ERYWxrBhwxg1ahSVK9vw8MaUlaMmChGZQYG6hUNU9Y6jHauqnwOfF1r3eDH79jza+Y7Fgg278n9uUiO2NE9tysD69euJj48nMTGRqKgoXn/9daKjo2nXzoaGN6as+VL09D/ga+/rR6Am4HN/ikD5xZsobjvTRokNJdnZ2YwZM4a2bdsyfPjw/PWnnXaaJQljAsSXoqd3Ci6LyOvAXL9FVApUlbcXOi1zz2lZM8DRGF999913DBo0iNWrVwNOCye3222V1cYE2PEM4dEICOqR1eauSiUnz0N8pQi6WCe7oLdjxw5uuukmzj77bFavXk2LFi345ptveO211yxJGBMEfKmj2MPhOoowYDdQ7LhNwWDMf/8AoFuT6rjCrG19MEtLS6NVq1bs3r2bqKgoRo4cyUMPPURUVFSgQzPGeJWYKMTpwdQB2Opd5dFDYyYEKVVlb2YOAMN7twxwNOZoqlevzmWXXUZKSgpTpkyhadOmgQ7JGFNIiYlCVVVEPlLVTmUV0Ilat+MA+7LyiHAJ9RJsTuxgk5GRwejRo7n44ovp3r07AFOmTCEqKsp6VhsTpHypo/hVRE71eySl5OvVTt+Jc1rWtBtPkPn0009p3bo148ePZ/DgwXg8HgCio6Pt/8qYIFbsE4WIhKtqHnAmcLuIrAcycMZwUlUNyuTxn+83AHBWM9/HhDL+tWXLFu69914++ugjAE455RSmTZtm81UbEyJKKnr6FTgVuLyMYjlh2Xludmc49RPntrJmsYGWl5fHiy++yOOPP05GRgaxsbGMGTOGu+66yyYSMiaElPTXKgCqur6MYjlhP63bhUehcY3K1I6z+olA27dvH08//TQZGRn06dOHf//739StWzfQYRljjlFJiaKGiAwrbqOqTvBDPCfkZ29v7J7N7WkiUPbu3UtMTAxRUVEkJCQwbdo0oqKiuPjiiwMdmjHmOJVUSOwCYnGGAy/qFXQOTVKU1NAmrilrqspbb71FixYtGD9+fP76K6+80pKEMSGupCeK7ao6uswiOUFuj5KV60xg0/bkuABHU7GsWbOGwYMH8/XXXwMwf/58VNVaMhlTTpT0RBFSf+Wzl20lI8dNjSpR1n+ijGRlZfHkk0/Srl07vv76axISEnj55Zf58ssvLUkYU46U9ERxbplFUQq+X5MGwLnWf6JM/P3333Tv3p21a9cCcPPNN/Pss89SvXr1AEdmjCltxSYKVd1dloGcqCWb9wDQ1QYBLBO1atWiXr16hIeHM3XqVHr06BHokIwxflIuGrOn7sti065MosLDuLCdTXvqDx6PhxkzZnD22WfTvHlzRIS33nqLatWqERkZGejwjDF+VC66xv7vj1QATq1fjahwG5a6tC1btoxu3boxcOBABg8ezKFxIWvVqmVJwpgKoFwkinU7DgDQsLrNo1yaDhw4wAMPPECnTp1YsGABJ598MgMHDgx0WMaYMlYuip7mrdkJ2Gx2penjjz/m7rvvJiUlhbCwMO6++27GjBlD1apVAx2aMaaMlYtEsWFnBgDt6lj/idKwdetW+vbtS3Z2Np06dSI5OZmkpKRAh2WMCZCQTxSHJikCqFXVZkU7Xrm5uYSHhyMi1KlTh7FjxxIZGcngwYNtOlJjKriQr6NYtMlpFtukRmXrP3GcfvrpJzp16sQbb7yRv+7+++/n7rvvtiRhjAn9RPHDOqejXfNaQTn8VFDbvXs3d955J926dWPFihVMmTKFIJ/p1hgTACGfKP7Yvg+Aa5LqBTiS0KGqvP7667Rs2ZLp06cTERHByJEj+eabb+ypzBjzDyFfR3GoaWzzk+yJwhepqalcd911fPvttwD06NGDqVOn0qpVqwBHZowJViH9RLEvK5ddGTnERLg4OS460OGEhPj4eLZv30716tV57bXX+Pbbby1JGGNKFNJPFJt3ZQJOaycrMine3LlzOfXUU0lMTCQqKor33nuP2rVrk5ho42IZY44upJ8oFm5yxi1skGg9souyfft2rrvuOnr16sXw4cPz17dt29aShDHGZyGdKNIP5gJQOcqacBbkdruZMmUKLVu2ZNasWcTExNCiRQtr0WSMOS4hXfR0qEd2t6Y2B8IhS5YsYeDAgSxcuBCAiy++mEmTJtGwYcPABmaMCVkhnShWbksHoFVtG38IYNOmTXTu3Bm3202dOnV48cUXueKKK6z+xhhzQvyaKESkNzARcAH/UdVxhbYPA24D8oCdwK2q+pcv5/Z4lC17DgLQrGZsaYYdsho2bMgtt9xClSpVePLJJ6lSxZoMG2NOnN/qKETEBUwGLgRaA9eJSOtCu/0GJKlqe+B9YLyv50/Zc5CcPA/VYyOpEh1RWmGHlE2bNnHJJZcwb968/HXTp09nwoQJliSMMaXGn08UnYF1qroBQERmAZcBqw7toKrfFth/AXCDrydfmrIXqJhDd+Tm5jJhwgSefPJJDh48SFpaGj///DOAFTMZY0qdP1s91QG2FFhO8a4rzgDgi6I2iMgdIrJIRBbt3OnMPbEpzanIPqlqxepo98MPP3DKKacwYsQIDh48SN++ffnwww8DHZYxphzzZ6Io6qttke0zReQGIAl4tqjtqjpdVZNUNalGjRrA4T4U8ZUqxlSce/bs4bbbbuOss85i5cqVNGnShC+//JK3336b2rVrBzo8Y0w55s9EkQIUHKmvLrCt8E4ich4wErhUVbN9PfnuDGceiq6NE04syhDh8Xj45JNPiIiI4LHHHmPFihX06tUr0GEZYyoAf9ZRLASaiUgjYCvQF7i+4A4icgowDeitqjt8PbHHo2z0Fj2d2qBaqQUcbFavXk2jRo2IiooiMTGRN998k/r169OyZctAh2aMqUD89kShqnnAEOBL4A/gXVVdKSKjReRS727PArHAeyKyVERm+3LutAPZZOa4qRzpIrFy+St6yszMZOTIkbRv357x4w83BOvVq5clCWNMmfNrPwpV/Rz4vNC6xwv8fN7xnDftgFPslJHjLnetfObMmcPgwYPZuHEjAGlpaQGOyBhT0YVkz+wte5xRY7s3rxHgSErPtm3bGDp0KO+99x4A7dq1Izk5mTPOOCPAkRljKrqQTBQ/r98FQIOESgGOpHSsWbOGpKQk9u/fT6VKlXjiiScYOnQoEREVsyOhMSa4hGSiCPMWN5WXUqdmzZpx2mmnUblyZV566SUaNGgQ6JCMMSZfSA4zvivDaUUbqoMB7tu3j6FDh7JmzRrA6U09e/ZsZs+ebUnCGBN0QvKJYtteZzDAutViAhzJsVFV3n//fe699162b9/O6tWrmTNnDgCVK9vkS8aY4BSiiSILgNpxoZMoNmzYwJAhQ/jiC2eUkq5du/LMM88EOCpjjDm6kCx6OjSzXY3YqABHcnQ5OTk89dRTtGnThi+++IL4+HiSk5P58ccf6dChQ6DDM8aYowq5JwqPwoHsPCJdYVSNCf7wt2zZwujRo8nOzqZfv348//zz1KpVK9BhGWOMz4L/TltIrtsDQK24qKDtbLdnzx7i4+MREZo0acLEiRNp2rQp5557bqBDM8aYYxZyRU85eU6iCMb6CY/HwyuvvELTpk1544038tffeeedliSMMSEr5BJFnsdJFME2D8XKlSvp2bMnAwYMYPfu3fmV1sYYE+pCLlFk5TqJIr5ScPRazszM5OGHH6Zjx458//331KxZkzfffJM333wz0KEZY0ypCLk6ijy3hwicCu1AW7NmDRdccAGbNm1CRBg4cCBPPfUU1aqV36HPjTEVT8glCo93jrzWQdAru0GDBkRHR9OhQweSk5Pp2rVroEMyxphSF3JFT/uznT4UgRi+Iy8vj0mTJrFrlzMoYVRUFHPmzGHRokWWJIwx5VbIJYrwMCfkWmVcmf3rr7/SuXNn7r77boYPH56/vkGDBoSHh9yDmTHG+CzkEsWhfhRlNbNdeno6Q4YMoWvXrvz222/Ur1+fyy67rEze2xhjgkHIJYpDqkT791u8qjJr1ixatmzJ5MmTcblcPPTQQ6xatYpLLrnEr+9tjDHBJCTLTGIiXIS7/Jvjli1bxnXXXQfAGWecQXJyMu3atfPrexpjTDAKyURxMNftl/O63W5cLhcAHTt25L777qN169bceuuthIWF7MOXMcackJC8+zVMLP0pUL/99lvatm3L/Pnz89dNmDCB2267zZKEMaZCC8k7YGkWO+3YsYObbrqJc845h9WrVzNhwoRSO7cxxpQHIZkodh3IPuFzeDweZsyYQcuWLZk5cyZRUVH83//9H++8804pRGiMMeVHSNZRtK8bf0LHb9y4kRtuuIGffvoJgF69ejF58mSaNm1aGuEZY0y5EpJPFHsyc07o+KpVq7JmzRpOOukkZs2axZw5cyxJGGNMMULyiaJZzSrHfMyXX35Jz549iYqKIjExkdmzZ9O6dWvi4uL8EKExxpQfIflEcSxToG7ZsoUrrriC3r178+yzz+avP/300y1JGGOMD0IzUUQffS6KvLw8JkyYQKtWrfj444+JjY0lISGhDKIzxpjyJSSLnhJjSx7nacGCBQwcOJBly5YB0KdPHyZOnEidOnXKIjxjjClXQjJRREe4it32yy+/cMYZZ6CqNGzYkEmTJnHxxReXYXTGGFO+hGSiqFstpthtnTt35oILLuCUU07h0UcfpVKl0u/FbYwxFUlI1lHERh3Ob2vXruVf//oXa9asAUBE+O9//8tTTz1lScIYY0pBSD5RVIp0kZ2dzbhx43j66afJzs4mOjqa999/H8DGZjLGmFLk1zuqiPQWkT9FZJ2IjChie5SIvOPd/ouINPTlvL/+OI/27dvzxBNPkJ2dzS233EJycnJph2+MMQYQVfXPiUVcwBrgfCAFWAhcp6qrCuwzGGivqgNFpC9whapeW9J5XTFV1JN1AIBWrVqRnJxM9+7d/fI7GGNMeSEii1U16XiO9ecTRWdgnapuUNUcYBZQeA7Ry4D/5/35feBcEZGSTurJyiA6OpqnnnqKpUuXWpIwxhg/8+cTxVVAb1W9zbvcH+iiqkMK7PO7d58U7/J67z5phc51B3CHd7Et8Ltfgg491YG0o+5VMdi1OMyuxWF2LQ5roarHPv4R/q3MLurJoHBW8mUfVHU6MB1ARBYd7+NTeWPX4jC7FofZtTjMrsVhIrLoeI/1Z9FTClCvwHJdYFtx+4hIOBAH7PZjTMYYY46RPxPFQqCZiDQSkUigLzC70D6zgZu8P18FfKP+KgszxhhzXPxW9KSqeSIyBPgScAGvqOpKERkNLFLV2cDLwOsisg7nSaKvD6ee7q+YQ5Bdi8PsWhxm1+IwuxaHHfe18FtltjHGmPLBujAbY4wpkSUKY4wxJQraROGv4T9CkQ/XYpiIrBKR5SLytYg0CEScZeFo16LAfleJiIpIuW0a6cu1EJFrvJ+NlSLyVlnHWFZ8+BupLyLfishv3r+TiwIRp7+JyCsissPbR62o7SIiL3qv03IROdWnE6tq0L1wKr/XA42BSGAZ0LrQPoOBZO/PfYF3Ah13AK/F2UAl78+DKvK18O5XBZgPLACSAh13AD8XzYDfgGre5ZqBjjuA12I6MMj7c2tgU6Dj9tO16A6cCvxezPaLgC9w+rB1BX7x5bzB+kThl+E/QtRRr4Wqfquqmd7FBTh9VsojXz4XAP8HjAeyyjK4MubLtbgdmKyqewBUdUcZx1hWfLkWClT1/hzHP/t0lQuqOp+S+6JdBsxUxwIgXkRqH+28wZoo6gBbCiyneNcVuY+q5gHpQGKZRFe2fLkWBQ3A+cZQHh31WojIKUA9Vf2sLAMLAF8+F82B5iLyo4gsEJHeZRZd2fLlWjwB3CAiKcDnwN1lE1rQOdb7CRC881GU2vAf5YDPv6eI3AAkAT38GlHglHgtRCQMeAG4uawCCiBfPhfhOMVPPXGeMr8XkbaqutfPsZU1X67FdcBrqvq8iJyO03+rrap6/B9eUDmu+2awPlHY8B+H+XItEJHzgJHApaqaXUaxlbWjXYsqOINGficim3DKYGeX0wptX/9GPlHVXFXdCPyJkzjKG1+uxQDgXQBV/RmIxhkwsKLx6X5SWLAmChv+47CjXgtvccs0nCRRXsuh4SjXQlXTVbW6qjZU1YY49TWXqupxD4YWxHz5G/kYp6EDIlIdpyhqQ5lGWTZ8uRabgXMBRKQVTqLYWaZRBofZwI3e1k9dgXRV3X60g4Ky6En9N/xHyPHxWjwLxALveevzN6vqpQEL2k98vBYVgo/X4kugl4isAtzAg6q6K3BR+4eP1+J+YIaI3IdT1HJzefxiKSJv4xQ1VvfWx4wCIgBUNRmnfuYiYB2QCdzi03nL4bUyxhhTioK16MkYY0yQsERhjDGmRJYojDHGlMgShTHGmBJZojDGGFMiSxQm6IiIW0SWFng1LGHfhsWNlHmM7/mdd/TRZb/GPgoAAAPeSURBVN4hL1ocxzkGisiN3p9vFpGTC2z7j4i0LuU4F4pIRx+OGSoilU70vU3FZYnCBKODqtqxwGtTGb1vP1XtgDPY5LPHerCqJqvqTO/izcDJBbbdpqqrSiXKw3FOwbc4hwKWKMxxs0RhQoL3yeF7EVnifZ1RxD5tRORX71PIchFp5l1/Q4H100TEdZS3mw809R57rncOgxXesf6jvOvHyeE5QJ7zrntCRB4Qkatwxtx60/ueMd4ngST5/+3dTYhVdRjH8e9vUWRFgosiEHohKIgm6Q3BRdgbRQQl4S1EahGRFEHhJmwTtIigRWYlETEuKgZFIXqhJAYXMlPNorREEqxFEOVCQmSCsF+L5z9xut25M9eVw/w+u/u/95z//x6457nnOYfnkbZIerWz5sclvXGW65yiU9BN0tuSZlS9J15qY89SAWtS0mQbu0fSVDuOuyVdvMA8scwlUMS5aEUn7bSvjf0O3G37JqAHbB+w3VPA67bXUCfqX1q5hh6wro2fATYtMP8DwGFJFwDjQM/2DVQlgy2SVgEPAdfbHgNe7m5sew8wQ/3zX2N7tvP2HmBD53UPmDjLdd5LlemYs832LcAYcLukMdvbqVo+622vb6U8XgTuasdyBnh+gXlimTsnS3jEsjfbTpZd5wE7Wk7+DFW3qN8UsE3SamCv7WOS7gRuBr5p5U1WUEFnkPclzQI/U2WorwV+sv1je38X8DSwg+p18a6kT4BFlzS3fULS8VZn51ib42Db7yjrvIgqV9HtULZR0pPU7/pyqkHPob5t17bxg22e86njFjGvBIpYKp4DfgNupK6E/9eUyPYHkr4C7gc+l/QEVVZ5l+0XFjHHpm4BQUkD+5u02kK3UUXmHgGeAe4Y4btMABuBo8A+21adtRe9TqqL2yvAm8AGSVcBW4FbbZ+UNE4VvusnYL/tR0dYbyxzST3FUrES+LX1D9hM/Zv+D0lXA8dbuuUjKgXzJfCwpEvbZ1Zp8T3FjwJXSrqmvd4MHGg5/ZW2P6VuFA968ugUVfZ8kL3Ag1SPhIk2NtI6bf9FpZDWtrTVJcBp4A9JlwH3zbOWaWDd3HeSdKGkQVdnEf9KoIil4i3gMUnTVNrp9IDP9IDvJX0LXEe1fDxCnVC/kHQI2E+lZRZk+0+quuZuSYeBv4Gd1En347a/A9TVTr9xYOfczey+/Z4EjgBX2P66jY28znbv4zVgq+3vqP7YPwDvUemsOe8An0matH2CeiLrwzbPNHWsIuaV6rERETFUrigiImKoBIqIiBgqgSIiIoZKoIiIiKESKCIiYqgEioiIGCqBIiIihvoHvPibnfpHFTsAAAAASUVORK5CYII=\" />\n",
    "        </td>\n",
    "        <td>\n",
    "<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XuczXX+wPHX24wZd2LiZ8mlyCVJNeuSQrUrZTeVWkoqKca4JF1oJbIIRZHLYJdWNy1bURtlW6VtEyqXSFJZjaxyv4+5vH9/fL+j0zRz5gxzzvd8z7yfj8d5mO/9fb4zzvt8Pp/v5/MRVcUYY4wpSCmvAzDGGBPdLFEYY4wJyhKFMcaYoCxRGGOMCcoShTHGmKAsURhjjAnKEoUxxpigLFGUICJym4isFZEjIrJLRJaKyOXutlEi8kI+x6iINAjYJ9M9Pvd1IM++u0UkPmBdvIj8ICIasO49ETnhHn9QRFaKyIUB2/ONxd02wH0PGSLyXD7by4nIDBHZk3vugG1XisgKd/32IPepvftexgSsu1NEPhGRQyKSLiITA99nwH4N3ff2QsA6EZHhIrLDPX6BiFQq6PpB4npPRPaLSGI+6+/Js66DiKTniWGQiHwuIkfd97Aw8L6HGIOIyAQR2eu+JoqIFLDvH/P8rRwXkRwRScqzX1UR+VFE/p1nfYG/SxNZlihKCBEZAjwDjANqAHWAGUCXIp7qFVWtEPCqkmf7AeDagOXrgP35nGeAqlYAqgHvAc+HeP3vgTHA3AK2zwaqAk3cf+8P2HbUPe6hgk4uIqWBKcDHeTaVAwYDSUAr4GrgwXxOMR1Yk2fdHUBPoC3wK6As8GxBMRQQVz3gCkCB64tyrGsKcB8wCOe+nA+8DnQu4nn6ADcAFwHNgd8BffPbUVXHBf6tABOA91R1T55dJwBf5HOKYL9LE0GWKEoAEakMjAb6q+qrqnpUVTNV9Q1VLfBD8zQ9j/PBmOsOYH5BO6tqFrAAaBrKyd34Xwf25t0mIo1wPkT7qOqPqpqtqp8EHLtaVZ8HvglyiQeAd4Atea47U1U/UNWTqroTeBHngz/w+t1xEuW7ec75e+Avqvqdqh7B+WDsJiLlQnnPrjuAVcBzwJ1FOA4RaQj0B25V1X+paoaqHlPVF1V1fFHO5V57kqqmu/dhEnBXCDEITrL8a571bYBmwLw864P+Lk1kWaIoGdoAZYDXInCt14F2IlJFRKrgfAteXNDOIpIA9MD5EDxTrYD/Ao+71RUbRaRrqAeLSF3gbpykWph2wKaAYyu5xz2Q36ndV+ByItDQPXaYiLxZyPXuwElOLwLXiEiNEGLMdTWQrqqrC9rBjeFAQa+AXS8A1gcsr3fXFeYKnJLs3wOuGYdTAhuAU1IKdEa/S1O8LFGUDNWAPe6392D+EOQDoqB9VuTZfgJ4A+gGdAeWuOvymuqe/wjOB8XjRXpH+auN8+30IE4VzwDgryLSJMTjpwIj3G/9BRKRXkAy8FTA6j/hlhryOWQpcI+I1HNLd0Pd9eUAVHW8qv4uyPUuB+oCf3O/VX8N3BbiewLn978r2A5uDFUKegXsWgHn/uY6CFQoqJ0iwJ3Aojz3dhDwcQElhTP9XZpiZImiZNgLJOXX+JrH34J8QBS0z5X57DMf5xtwsGqnQe75y+DUcy8SkeYhvp+CHAcygTFuFdH7wAqgY2EHisjvgYqq+koh+90AjAeuza1rF5EWwG+Apws4bC7wMk5bzCY3JoD0AvbP607gnYC6/Zf4efVTFlA6zzGlce4FOL//miFeqzBHgMCG+ErAEQ0yuqiIlAVuIaDaSUR+hZMohhdw2Gn/Lk3xK+yDw8SGj3C+1d8ALIrA9T7A+WBS4N/AeQXtqKo5wAcisg3nQ2DDGVz3TI69GkgWkf+5y5WBbBG5UFW7AIhIJ2AO0FlVNwYc2wGoB+xwv1hXAOJEpKmqXuK+x5HuCxHpCOx0X0G5H7J/cM+XG1siUEVELlLV9cAO9/qB6uNU3YDTZjJdRJJVdW0B1/kj8MeC4nAbo8FJdBcBudVYFxFQBVeAm4B9OIkyV0ucv5HN7j0rC5R132Mtzux3aYqZlShKAFU9CDyG82Fxg/vYYWkRuVZEJobheorTgHt9sG+audwGzab8/AOnlIiUCXgluvvGi0gZIA7nw7NMQElpJc6H5iPufm1xPsTfdo8t5R5b2lmUMm4bCcAInCeBWrivJThJoZd77FU47QNd86nrn42TDHOPTQP+AVzjHltVRM5zHy1tCkwGRrsJpDA3ANnu/ck9fxOcZJz70MArQC8Raele43ycJ4QWAKjqVzhPuL0szmOzCe577y4iw9x9fvaEUt5XQDzzgSEiUsstFTyA08AezJ3A/Dx/C0txklvue3oM+AxooarZFPK7NBGmqvYqIS+cRuO1OI+J/g/nw+wyd9so4IV8jlGgQcA+mTjVD4Gv6nn3zXOOBrj5w11+D6eEk3v8NuD+gO2j3HMFvtKDbBsVcOwFOCWoo8Bm4MaAbR3yOfa9Au7VczjVHrnLK3CqeALf99ICjv3ZvcRJQF8Cx3C+5Q/Js/8fg5xrGc5TRnnX/8H9Hca7y3fjJNpD7v0cBpQK2F9wHo/d5MaxEyfBXFDEvyEBJuKUEPa5P0vA9iPAFQHLtdz79ou/izznvQv4d551Bf4u7RXZl7i/EGOMMSZfVvVkjDEmKEsUxhhjgrJEYYwxJihLFMYYY4LyXT+KpKQkrVevntdhGGOMr3zyySd7VPXs0znWd4miXr16rF2bb58hY4wxBRCR/xa+V/6s6skYY0xQliiMMcYEZYnCGGNMUJYojDHGBGWJwhhjTFCWKIwxxgQVtkQhInNF5AcR+byA7SIiU0Vkm4hsEJFLwhWLMcaY0xfOEsVzQKcg26/FmTO4IdAHmBnGWIwxpsQ6efLkGR0ftkShqitxxqsvSBfcyUxUdRXOjF3FNV2jMcaUeL3mraZyq65UPr/lGZ3Hy57ZtYDAiejT3XW/mAReRPrglDqoU6dORIIzxpjT1WvealZ8+aPXYQBQOqkuhz5Zckbn8DJRSD7r8p1FSVVn40w3SXJyss20ZIzxTDQlgfyc3LODk7u/psIFVwLwu67dGDV9EPXr1z/tc3qZKNKBcwKWawPfexSLMcb8zJkmhCsbnc28XmdW5VMUx44dY8yYMTz51yeJi4tj1dMpNGjQoFjO7WWiWAIMEJEFQCvgoKr+otrJGGPCrahJIdJJoDBLly6lf//+fPvttwD07t2batWqFdv5w5YoRORlnMnsk0QkHRgJlAZQ1TTgLeA6nIngjwG9whWLMcYEU1CSiLaEkNfOnTsZPHgwixYtAqB58+akpaXRpk2bYr1O2BKFqt5ayHYF+ofr+sYYE4pe81af+nn7+M4eRlJ0/fv3Z/HixZQrV47Ro0dz3333ER9f/B/rvpuPwhhjQlWUKqUrG53WnD4Rl5WVdSoZTJgwgdKlSzNp0qSwPhFqicIYEzNOtwE62quYAA4ePMijjz7K1q1bWbZsGSJCo0aNWLhwYdivbYnCGONboSQGPySBYFSVhQsXMnjwYHbt2kVcXBzr1q3j4osvjlgMliiMMb5UUJLwe2II9PXXXzNgwACWLVsGQJs2bUhLS6N58+YRjcMShTHGN/JLDrGUGAI99dRTjBgxghMnTlClShUmTJjAPffcQ6lSkR/02xKFMSYqnE77QqwmCXA60J04cYKePXvy1FNPUb16dc9isURhjPGU3zu7FZcff/yRL7/8kssvvxyAoUOH0qFDB9q1a+dxZJYojDEeypskYjUJBJOTk8PcuXN5+OGHiY+PZ8uWLVStWpXExMSoSBJgicIY45HAJFESEwTA559/TkpKCh9++CEAv/3tbzl27BhVq1b1OLKfs0RhjAmLUKuUSmKSOHr0KKNHj2by5MlkZWVRo0YNnnnmGbp164ZIfgNre8sShTGm2Fh7Q2huvvnmU53mUlNTGTt2LFWqVPE6rAJZojDGnLGS0KehOA0dOpTdu3czc+ZMWrVq5XU4hbJEYYw5YyW9QTqYrKwsnn32WbZv386UKVMA6NChA2vXrvWkT8TpsERhjCmygkoQfht9NdxWr15N3759WbduHQB9+vThggsuAPBNkgDwT6TGmKgQrJrJOA4cOEBqaiqtW7dm3bp11K1blzfeeONUkvAbK1EYY/JVWMO0VTHlb8GCBQwePJjdu3cTHx/PAw88wIgRIyhfvrzXoZ02SxTGmHxZkjg977zzDrt376Zt27bMnDmTCy+80OuQzpglCmPML/h51rdIy8jIYOfOnZx77rkATJw4kSuuuII777zTV+0QwcTGuzDGFJu8PaZNwf71r3/RvHlzOnfuzMmTJwFISkqiV69eMZMkwEoUxpRY1gZx+nbv3s2DDz7ICy+8AEDjxo1JT08/VaqINZYojCkhijp/tCWJX8rJyWHOnDkMGzaMAwcOUKZMGR599FEeeughEhISvA4vbCxRGFMCWM/p4nHjjTeyZMkSAK655hqmT5/Oeeed53FU4WeJwhifs5JC5Nx0002sXr2aKVOmcMstt0TlAH7hIKrqdQxFkpycrGvXrvU6DGOigiWJ8FqyZAnp6emkpqYCoKocOXKEihUrehxZ0YnIJ6qafDrHWonCGB8r6fM5hMuOHTsYNGgQixcvJjExkU6dOnHuueciIr5MEmcqdp7fMqYEsyRRPDIzM5k0aRJNmzZl8eLFVKxYkYkTJ1K3bl2vQ/OUlSiM8anATnHmzK1atYq+ffuyYcMGAG655RaefvppatWq5XFk3rNEYUyUOZ3Jf8yZGzFiBBs2bKB+/fpMmzaN6667zuuQooYlCmOiRFETBFjbxJlQVQ4fPkylSpUAmDZtGvPnz2f48OGUK1fO4+iiiz31ZIzH8ksQlgDC68svvyQ1NRURYfny5SXiMVd76skYH7IEEXknTpzgiSeeYPz48Zw8eZJq1aqxfft26tev73VoUc0ShTERZgnCG8uXLyc1NZVt27YBcPfddzNx4kSqVavmcWTRL6yJQkQ6AVOAOODPqjo+z/Y6wF+BKu4+w1T1rXDGZEykBWt7sAQRfqpK7969mTdvHgBNmzYlLS2NK664wuPI/CNsiUJE4oDpwG+BdGCNiCxR1c0Buz0K/E1VZ4pIU+AtoF64YjIm0myMJe+JCPXq1aNs2bI89thjDBkyJKYH8AuHcJYoWgLbVPUbABFZAHQBAhOFApXcnysD34cxHmPCykoO0WPdunXs2rWLa6+9FoChQ4fSs2dPa4s4TeHsmV0L+C5gOd1dF2gUcLuIpOOUJgbmdyIR6SMia0Vk7Y8/Fu3xQWMiwZJEdDh8+DBDhgzh0ksv5c4772Tfvn0AJCYmWpI4A+EsUeT3vFneZ3FvBZ5T1Uki0gZ4XkSaqWrOzw5SnQ3MBufx2LBEa8xpyjsjnCWFyFNVXn/9dQYNGkR6ejqlSpXitttuo3Tp0l6HFhPCmSjSgXMClmvzy6ql3kAnAFX9SETKAEnAD2GMy5hikbcUYUnCG//9738ZMGAAb775JgDJycnMmjWLSy65xOPIYkc4q57WAA1FpL6IJADdgSV59tkBXA0gIk2AMoDVLRlfsCThPVWla9euvPnmm1SqVIlp06axatUqSxLFLGwlClXNEpEBwNs4j77OVdVNIjIaWKuqS4AHgDkicj9OtdRd6reu4iamnM4wGtvHdw5TNKYgOTk5lCpVChHhqaeeIi0tjaeffpqaNWt6HVpMsiE8TIl3Oskhl5UkImvv3r0MGzYMgDlz5ngcjb/YEB7GFJE9peQvqsr8+fN58MEH2bNnDwkJCYwcOZLatWt7HVqJYInClBiWHPzpiy++oF+/frz//vsAdOjQgZkzZ1qSiCBLFCYmhVKdZMkhuqkqjz32GBMmTCAzM5OkpCQmTZpEz549S8Ror9HEEoWJSVZy8D8RYefOnWRmZnLvvfcyfvx4qlat6nVYJZIlChPT7Ikkf/n+++/Zs2cPzZs3B2DixIn07t2btm3behxZyWaJwsSUM3mCyXgnOzubmTNnMnz4cGrVqsW6detISEggKSmJpKQkr8Mr8SxRGN8JNRnYXNL+8Omnn9K3b19yH3tv164dhw4dsgQRRUJKFG7P6jqqui3M8RhTKGukjg2HDh1ixIgRTJs2jZycHGrXrs3UqVO54YYbrLE6yhSaKESkMzAZSADqi0gLYKSq3hju4EzJFUqpwdof/EtVadeuHevXrycuLo4hQ4YwatQoKlas6HVoJh+hjPU0GmgFHABQ1XVAg3AGZUq2UB9tNf4lItx///20bNmStWvXMmnSJEsSUSyUqqdMVT2Qpyjor3E/jG/YkN2x6eTJk0yePJm4uDgeeughAO644w5uv/124uLiPI7OFCaURPGFiPwBKCUi9YH7gFXhDcuURJYkYtMHH3xASkoKmzdvJjExkTvuuIMaNWogIpYkfCKUqqcBwKVADvAqcAInWRhTrCxJxJY9e/Zw9913065dOzZv3kzDhg158803qVGjhtehmSIKpURxjaoOBYbmrhCRm3CShjFnJL/2CEsS/qaqPPfcczz00EPs3buXhIQEHnnkEYYNG0aZMmW8Ds+chlBKFI/ms254cQdiSqa8ScIaqWPDCy+8wN69e7nqqqvYsGEDo0aNsiThYwWWKETkGpxpSmuJyOSATZVwqqGMKVSonePsUVd/O3bsGAcPHqRmzZqICDNmzGDNmjX06NHD+kTEgGBVTz8An+O0SWwKWH8YGBbOoIz/FWUoDStF+NvSpUvp378/5557LsuXL0dEaNSoEY0aNfI6NFNMCkwUqvoZ8JmIvKiqJyIYk/G5vEnCGqdj086dOxk8eDCLFi0CoGLFiuzdu9eG3ohBoTRm1xKRsUBT4FQlo6qeH7aojK/Z00uxLTs7m+nTp/Poo49y+PBhypcvz+jRoxk0aBDx8TZ8XCwK5bf6HDAGeAq4FuiFtVGYAvSat/rUz5YkYk9OTg7t27fnww8/BOCGG25gypQp1KlTx+PITDiF8tRTOVV9G0BVv1bVR4ErwxuW8avA0oSJPaVKlaJjx46cc845LF68mNdee82SRAkQSokiQ5zHFr4WkRRgJ1A9vGEZv7PSRGxQVf72t78RHx9P165dARg6dChDhgyhQoUKHkdnIiWURHE/UAEYBIwFKgN3hzMo40+B1U7G/77++mtSU1N55513OPvss7nqqqs466yzSExMJDEx0evwTAQVmihU9WP3x8NATwARqR3OoIy/5PeUk/GvjIwMnnzyScaOHcuJEyc466yzGDt2LJUrV/Y6NOORoIlCRH4N1AL+rap7ROQCnKE8rgIsWRgAexQ2hrz33nv069ePLVu2ANCzZ0+eeuopqle32uaSLFjP7CeArsB64FEReQ1nMMAJQEpkwjPRLG9JwnpX+1t2djapqals2bKFRo0aMXPmTK680p5bMcFLFF2Ai1T1uIhUBb53l7+MTGgmmll1U2zIycnhxIkTlCtXjri4OGbOnMnKlSt5+OGHrR3CnBIsUZxQ1eMAqrpPRLZYkijZ8huWw6qa/Gvjxo2kpKTQuHFj/vKXvwDQvn172rdv73FkJtoESxTnikjuUOIC1AtYRlVvCmtkJupYkogNR48eZfTo0UyePJmsrCy+/fZb9u/fz1lnneV1aCZKBUsUXfMsTwtnICa6BT76am0R/vXGG28wYMAAduzYgYiQmprK2LFjqVKlitehmSgWbFDAdyMZiIlO1hYRG7KysujWrRuvvupUCrRo0YJZs2bRsqWVCE3hQhnCw5RQNgps7IiPj6dy5cpUqFCBp59+mjVr1liSMCETVQ3fyUU6AVOAOODPqjo+n33+AIwCFFivqrcFO2dycrKuXbs2DNEasAbrWPLxx05f2VatWgGwd+9ejh8/Tu3a1gWqJBKRT1Q1+XSODXlMYBFJVNWMIuwfB0wHfgukA2tEZImqbg7YpyHwCNBWVfeLiPXq8ZglCf87cOAAjzzyCLNmzaJx48asW7eOhIQEqlWr5nVoxqcKTRQi0hL4C84YT3VE5CLgHlUdWMihLYFtqvqNe54FOH0zNgfscy8wXVX3A6jqD0V/CyYcrMHaf1SVl19+mSFDhrB7927i4+O5/vrryc7O9jo043OhtFFMBX4H7AVQ1fWENsx4LeC7gOV0d12g84HzReRDEVnlVlUZj9igfv711Vdf0bFjR3r06MHu3btp27Ytn332GePHj6ds2bJeh2d8LpSqp1Kq+t88E6SH8hUlvxnV8zaIxAMNgQ44Y0d9ICLNVPXAz04k0gfoA9jY92ES2DZhTzb5S2ZmJldddRXp6elUrVqViRMn0qtXL0qVsmdVTPEI5S/pO7f6SUUkTkQGA1tDOC4dOCdguTbOMCB591msqpmq+i3wJU7i+BlVna2qyaqafPbZ9iEWDjZ9qf/kPohSunRpxo4dy1133cWWLVvo3bu3JQlTrEL5a+oHDAHqALuB1u66wqwBGopIfRFJALoDS/Ls8zpuNZaIJOFURX0TWuimuNj0pf6ye/duevbsyZgxY06tu+OOO5g3bx72RcqEQyhVT1mq2r2oJ1bVLBEZALyN83jsXFXdJCKjgbWqusTd1lFENuNUZz2kqnuLei1zZqzKyR9ycnKYM2cOw4YN48CBA1SpUoXBgwdTsWJFr0MzMS6URLFGRL4EXgFeVdXDoZ5cVd8C3sqz7rGAnxWntDIk1HOa8LHSRPRav349KSkprFq1CoBOnToxffp0SxImIkKZ4e48EbkMp+rocRFZByxQ1QVhj86ETX4d60z0yczM5JFHHuGZZ54hOzubmjVrMmXKFG6++WbyPGBiTNiE1OKlqv9R1UHAJcAh4MWwRmXCLr+OdSb6xMfH89lnn5GTk8PAgQP54osvuOWWWyxJmIgKpcNdBZyOct2BJsBi4LIwx2XCIL9ShHWsiz47duwgOzub+vXrIyKkpaVx8OBBkpNPa/QFY85YKG0UnwNvABNV9YMwx2PCoKBqJitFRJfMzEymTJnCyJEjadOmDcuXL0dEaNjwF0+MGxNRoSSKc1U1J+yRmGJRWNuD9ZOITh999BEpKSls2LABgKpVq3Ls2DHKly/vcWTGBEkUIjJJVR8A/i4ivxhi1ma4iz7BkoQliOi0f/9+hg0bxuzZswGoX78+06dP59prr/U4MmN+EqxE8Yr7r81s5xPWu9pfMjIyaNGiBTt27KB06dI89NBDDB8+nHLlynkdmjE/E2yGu9zuuk1U9WfJwu1IZzPgeaygEoQlCX9ITEykd+/evPvuu8ycOZOmTZt6HZIx+Sp04iIR+VRVL8mz7jNVvTiskRXAJi6yKia/OnHiBE888QSNGjXittuc+bmysrKIi4uzx11N2IVl4iIR6YbzSGx9EXk1YFNF4ED+R5niUJTOcJYY/GH58uWkpqaybds2qlevzo033kjZsmWJjw957jBjPBPsr3Q1zhwUtXFmqst1GPgsnEGVdKEkCUsQ/vC///2PIUOG8PLLLwNwwQUXkJaWZnNEGF8J1kbxLfAt8M/IhWMCWWc4/8rOzmbWrFn88Y9/5ODBg5QtW5aRI0dy//33k5CQ4HV4xhRJsKqn91W1vYjs5+cTDgnOeH5Vwx5dCWSzzMWG7Oxsnn32WQ4ePMh1113HtGnTqF+/vtdhGXNaglU95U53mhSJQIzNMud3hw8fJjs7mypVqpCQkMCcOXPYvXs3N910kzVWG18LVvWU2xv7HOB7VT0pIpcDzYEXcAYHNGcg2NAa1v7gH6rKa6+9xqBBg7jmmmv4y1/+AsDll1/ucWTGFI9QRo99HWca1POA+TgDA74U1qhKCEsS/rd9+3auv/56unbtys6dO/n88885ceKE12EZU6xCeTYvR1UzReQm4BlVnSoi9tTTGchbkrBGa//JzMxk8uTJPP744xw/fpxKlSoxbtw4UlJSiIuL8zo8Y4pVSFOhisgtQE/gBndd6fCFFPsCk4S1RfjPsWPHaN26NRs3bgSge/fuTJ48mZo1a3ocmTHhEUqiuBtIxRlm/BsRqQ+8HN6wYpOVJGJDuXLlSE5O5tixY8yYMYOOHTt6HZIxYRXKVKifi8ggoIGINAa2qerY8IcWe6wk4U+qyvz58znvvPNONVA//fTTJCQkWMc5UyKEMsPdFcDzwE6cPhT/JyI9VfXDcAcXC2xWOX/74osv6NevH++//z5NmjRh3bp1JCQkULlyZa9DMyZiQql6ehq4TlU3A4hIE5zEYfMyhsDmpvan48ePM3bsWCZOnEhmZiZnn302jzzyCKVLW/OcKXlCSRQJuUkCQFW/EBEbg6CIrBThH8uWLaN///588803ANx7772MHz+eqlVtMAJTMoWSKD4VkVk4pQiAHtiggIUqygiwJnocOXKEnj17smfPHpo1a0ZaWhpt27b1OixjPBVKokgBBgEP47RRrASeDWdQfpc3SVh1U3TLzs4mJyeH0qVLU6FCBaZMmUJ6ejr333+/VTUZQyGJQkQuBM4DXlPViZEJyf9sSlL/+OSTT+jbty9dunRhxIgRAKcmFTLGOAocwkNE/ogzfEcPYLmI3B2xqHyq17zV1Bv2j1PLliSi16FDh7jvvvto2bIln3zyCc8//zyZmZleh2VMVAo21lMPoLmq3gL8GugXmZD8yaqb/EFVWbhwIY0bN2bq1KmICEOGDOHTTz+1aiZjChCs6ilDVY8CqOqPIhLKAIIlgs1Z7U+HDx+mW7duLF26FIBWrVqRlpZGixYtPI7MmOgWLFGcGzBXtgDnBc6drao3hTWyKGZJwp8qVKhARkYGlStXZvz48fTp04dSpez7jzGFCZYouuZZnhbOQPwicAY66xsR/VauXEnNmjVp2LAhIsLcuXMpU6YMNWrU8Do0Y3wj2MRF70YyEL+wGej8Yc+ePTz88MPMmzePq6++muXLlyMi1K1b1+vQjPEdK3cXQWBpwqqYolNOTg5z586lUaNGzJs3j4SEBK644gqys7O9Ds0Y3wprohCRTiLypYhsE5FhQfa7WURURKJ6/CgrTUS3TZs20aFDB3r37s2+ffu4+uqr2bhxIyNHjiQ+PpS+pcaY/IT8v0dEElU1owgw6Xy4AAAWJklEQVT7xwHTgd8C6cAaEVkSOG6Uu19FnJ7fH4d6bi9YaSK6HTx4kNatW3PkyBGqV6/O5MmTue222xARr0MzxvcKLVGISEsR2Qh85S5fJCKhDOHREmfuim9U9SSwAOiSz35/AiYCUTvRcODjsFaaiC6qCkDlypUZOnQoKSkpbNmyhR49eliSMKaYhFKimAr8DqeXNqq6XkSuDOG4WsB3AcvpQKvAHUTkYuAcVX1TRB4s6EQi0gfoA1CnTp0QLl088utEZ6WJ6LBz507uu+8+unTpQs+ePQEYPny4JQdjwiCUNopSqvrfPOtCaRnM73+sntrodOB7GnigsBOp6mxVTVbV5LPPjtw3eksS0ScrK4spU6bQuHFj/v73vzNy5MhTDdWWJIwJj1BKFN+JSEtA3XaHgcDWEI5LB84JWK4NfB+wXBFoBrzn/gf/P2CJiFyvqmtDCT5cbG7r6LRmzRpSUlL49NNPAbjhhhuYOnUqcXFxHkdmTGwLpUTRDxgC1AF2A60JbdynNUBDEanvTnTUHViSu1FVD6pqkqrWU9V6wCrA8yQBNrd1tDl69CgDBgygVatWfPrpp9SpU4fFixfz2muvcc455xR+AmPMGSm0RKGqP+B8yBeJqmaJyADgbSAOmKuqm0RkNLBWVZcEP4M3rOd19ImPj+ef//wnpUqVYsiQIYwcOZLy5ct7HZYxJUahiUJE5hDQtpBLVfsUdqyqvgW8lWfdYwXs26Gw84WbPd0UPb7++muqVKlCtWrVSExM5Pnnn6dMmTJceOGFXodmTIkTStXTP4F33deHQHUg5P4UfmITDnkvIyODMWPG0KxZM4YOHXpq/a9//WtLEsZ4JJSqp1cCl0XkeWB52CLyiHWo8957771Hv3792LJlC+A84ZSdnW2N1cZ47HTGNagPxMzIajbhkPd++OEHHnroIebPnw9Ao0aNmDlzJldeGUp3HWNMuIXSRrGfn9ooSgH7gALHbfIb6yvhrT179tCkSRP27dtHYmIiw4cP5+GHHyYxMdHr0IwxrqCJQpwODhcBO91VOZo7ZkIMsCecvJeUlESXLl1IT09nxowZNGjQwOuQjDF5BE0Uqqoi8pqqXhqpgCLJnnCKvKNHjzJ69Gg6d+5Mu3btAJgxYwaJiYnWs9qYKBXKU0+rReSSsEcSYdZ4HXlvvPEGTZs2ZeLEiaSmppKTkwNAmTJlLEkYE8UKLFGISLyqZgGXA/eKyNfAUZwxnFRVfZc88jZcg5UmIuG7777jvvvu47XXXgPg4osvZtasWTZftTE+EazqaTVwCXBDhGIJu/yShJUmwicrK4upU6fy2GOPcfToUSpUqMCYMWPo37+/TSRkjI8E+98qAKr6dYRiiRhruI6MQ4cO8cQTT3D06FG6du3KM888Q+3atb0OyxhTRMESxdkiMqSgjao6OQzxhE1gm4QJnwMHDlC2bFkSExOpWrUqs2bNIjExkc6dLTkb41fBKonjgAo4w4Hn9/ING8Mp/FSVl156iUaNGjFx4sRT62+66SZLEsb4XLASxS5VHR2xSMIkb5KwNonit3XrVlJTU3n33XcBWLlyJapqTzIZEyMKbaPwo4KebrIkUbxOnDjBhAkTGDduHCdPnqRq1ao8+eST3HXXXZYkjIkhwRLF1RGLophZkgi///3vf7Rr146vvvoKgLvuuosnn3ySpKQkjyMzxhS3AhOFqu6LZCDFxYbliIwaNWpwzjnnEB8fz8yZM2nfvr3XIRljwiTmHma3RuvwyMnJYc6cOVx55ZWcf/75iAgvvfQSZ511FgkJCV6HZ4wJo5jtGmtVTcVn/fr1tG3blpSUFFJTU8kdF7JGjRqWJIwpAWI2UZgzd+TIER588EEuvfRSVq1axa9+9StSUlK8DssYE2ExVfVkneqKz+uvv87AgQNJT0+nVKlSDBw4kDFjxlCpUiWvQzPGRFhMJQprnygeO3fupHv37mRkZHDppZeSlpZGcnKy12EZYzwSM4nChg0/M5mZmcTHxyMi1KpVi7Fjx5KQkEBqaqrNWW1MCRczbRRWmjh9//nPf7j00kt54YUXTq174IEHGDhwoCUJY0zsJIpcVpoI3b59++jbty9t27Zl48aNzJgxgxia6dYYU0xiLlGYwqkqzz//PI0bN2b27NmULl2a4cOH869//cuG3jDG/ELMtFGY0OzevZtbb72VFStWANC+fXtmzpxJkyZNPI7MGBOtrERRwlSpUoVdu3aRlJTEc889x4oVKyxJGGOCiokShfWfCG758uVccsklVKtWjcTERBYuXEjNmjWpVq2a16EZY3wgJkoU9sRT/nbt2sWtt95Kx44dGTp06Kn1zZo1syRhjAmZ7xOF9Z/4pezsbGbMmEHjxo1ZsGABZcuWpVGjRvZEkzHmtPi66smmOP2lTz/9lJSUFNasWQNA586dmTZtGvXq1fM2MGOMb/k6UdgUpz+3fft2WrZsSXZ2NrVq1WLq1KnceOON9sirMeaMhDVRiEgnYAoQB/xZVcfn2T4EuAfIAn4E7lbV/xb1OpYkHPXq1aNXr15UrFiRxx9/nIoVK3odkjEmBoStjUJE4oDpwLVAU+BWEWmaZ7fPgGRVbQ4sAiaGK55YtH37dn7/+9/z/vvvn1o3e/ZsJk+ebEnCGFNswlmiaAlsU9VvAERkAdAF2Jy7g6quCNh/FXB7qCcvyY/EZmZmMnnyZB5//HGOHz/Onj17+OijjwCsmskYU+zC+dRTLeC7gOV0d11BegNL89sgIn1EZK2IrP3xR6ddoqQ2Yv/73//m4osvZtiwYRw/fpzu3bvz6quveh2WMSaGhTNR5PfVNt/nM0XkdiAZeDK/7ao6W1WTVTX57LN/nhhKSvvE/v37ueeee7jiiivYtGkT5513Hm+//TYvv/wyNWvW9Do8Y0wMC2eiSAfOCViuDXyfdycR+Q0wHLheVTPCGI+v5eTksHjxYkqXLs2IESPYuHEjHTt29DosY0wJEM42ijVAQxGpD+wEugO3Be4gIhcDs4BOqvpDGGPxpS1btlC/fn0SExOpVq0aL774InXq1KFx48Zeh2aMKUHCVqJQ1SxgAPA28AXwN1XdJCKjReR6d7cngQrAQhFZJyJLwhWPnxw7dozhw4fTvHlzJk786UGwjh07WpIwxkRcWPtRqOpbwFt51j0W8PNvwnl9P1q2bBmpqal8++23AOzZs8fjiIwxJZ2ve2bHku+//57BgwezcOFCAC688ELS0tK47LLLPI7MGFPSWaKIAlu3biU5OZnDhw9Trlw5Ro0axeDBgyldurTXoRljjD8TRax1tmvYsCG//vWvKV++PM8++yx169b1OiRjjDnFl8OM+72z3aFDhxg8eDBbt24FnN7US5YsYcmSJZYkjDFRx5clilx+62ynqixatIj77ruPXbt2sWXLFpYtWwZA+fLlPY7OGGPy5+tE4SfffPMNAwYMYOlSZ5SS1q1bM2HCBI+jMsaYwvmy6slPTp48ybhx47jgggtYunQpVapUIS0tjQ8//JCLLrrI6/CMMaZQVqIIs++++47Ro0eTkZFBjx49mDRpEjVq1PA6LGOMCZklijDYv38/VapUQUQ477zzmDJlCg0aNODqq6/2OjRjjCkyq3oqRjk5OcydO5cGDRrwwgsvnFrft29fSxLGGN+yRFFMNm3aRIcOHejduzf79u071WhtjDF+57tEsX3PUa9D+Jljx47xyCOP0KJFCz744AOqV6/Oiy++yIsvvuh1aMYYUyx810ZxOCOLCkRHZ7utW7dyzTXXsH37dkSElJQUxo0bx1lnneV1aMYYU2x8lyhyRUNnu7p161KmTBkuuugi0tLSaN26tdchGWNMsfNd1ZOXsrKymDZtGnv37gUgMTGRZcuWsXbtWksSxpiYZYkiRKtXr6Zly5YMHDiQoUOHnlpft25d4uN9WzAzxphCWaIoxMGDBxkwYACtW7fms88+o06dOnTp0sXrsIwxJmIsURRAVVmwYAGNGzdm+vTpxMXF8fDDD7N582Z+//vfex2eMcZEjNWZFGD9+vXceuutAFx22WWkpaVx4YUXehyVMcZEniWKANnZ2cTFxQHQokUL7r//fpo2bcrdd99NqVJW+DLGlEz26edasWIFzZo1Y+XKlafWTZ48mXvuuceShDGmRCvxn4A//PADd955J1dddRVbtmxh8uTJXodkjDFRpcQmipycHObMmUPjxo2ZP38+iYmJ/OlPf+KVV17xOjRjjIkqJbKN4ttvv+X222/nP//5DwAdO3Zk+vTpNGjQwOPIjDEm+pTIEkWlSpXYunUr//d//8eCBQtYtmyZJQljjClAiSlRvP3223To0IHExESqVavGkiVLaNq0KZUrV/Y6NGOMiWoxX6L47rvvuPHGG+nUqRNPPvnkqfVt2rSxJGGMMSGI2USRlZXF5MmTadKkCa+//joVKlSgatWqXodljDG+48uqp8Lmoli1ahUpKSmsX78egK5duzJlyhRq1aoVifCMMSamiKp6HUORJNZsqBm7vipw+8cff0ybNm1QVerVq8e0adPo3LlzBCM0xpjoIyKfqGry6RzryxJFMC1btuSaa67h4osv5tFHH6VcuXJeh2SMMb7m+zaKr776it/97nds3boVABHhH//4B+PGjbMkYYwxxcC3JYqMjAzGjx/PE088QUZGBmXKlGHRokUANjaTMcYUo7B+oopIJxH5UkS2iciwfLYnisgr7vaPRaReKOd99913ad68OaNGjSIjI4NevXqRlpZW3OEbY4whjI3ZIhIHbAV+C6QDa4BbVXVzwD6pQHNVTRGR7sCNqtot2HnjylbUnBNHAGjSpAlpaWm0a9cuLO/BGGNixZk0ZoezRNES2Kaq36jqSWABkHcO0S7AX92fFwFXi4gEO2nOiaOUKVOGcePGsW7dOksSxhgTZuEsUdwMdFLVe9zlnkArVR0QsM/n7j7p7vLX7j578pyrD9DHXWwGfB6WoP0nCdhT6F4lg92Ln9i9+Indi580UtWKp3NgOBuz8ysZ5M1KoeyDqs4GZgOIyNrTLT7FGrsXP7F78RO7Fz+xe/ETEVl7useGs+opHTgnYLk28H1B+4hIPFAZ2BfGmIwxxhRROBPFGqChiNQXkQSgO7Akzz5LgDvdn28G/qV+6ypujDExLmxVT6qaJSIDgLeBOGCuqm4SkdHAWlVdAvwFeF5EtuGUJLqHcOrZ4YrZh+xe/MTuxU/sXvzE7sVPTvte+G6sJ2OMMZFlXZiNMcYEZYnCGGNMUFGbKMI1/IcfhXAvhojIZhHZICLvikhdL+KMhMLuRcB+N4uIikjMPhoZyr0QkT+4fxubROSlSMcYKSH8H6kjIitE5DP3/8l1XsQZbiIyV0R+cPuo5bddRGSqe582iMglIZ1YVaPuhdP4/TVwLpAArAea5tknFUhzf+4OvOJ13B7eiyuBcu7P/UryvXD3qwisBFYByV7H7eHfRUPgM+Asd7m613F7eC9mA/3cn5sC272OO0z3oh1wCfB5AduvA5bi9GFrDXwcynmjtUQRluE/fKrQe6GqK1T1mLu4CqfPSiwK5e8C4E/AROBEJIOLsFDuxb3AdFXdD6CqP0Q4xkgJ5V4oUMn9uTK/7NMVE1R1JcH7onUB5qtjFVBFRGoWdt5oTRS1gO8CltPddfnuo6pZwEGgWkSii6xQ7kWg3jjfGGJRofdCRC4GzlHVNyMZmAdC+bs4HzhfRD4UkVUi0ili0UVWKPdiFHC7iKQDbwEDIxNa1Cnq5wkQvfNRFNvwHzEg5PcpIrcDyUD7sEbknaD3QkRKAU8Dd0UqIA+F8ncRj1P91AGnlPmBiDRT1QNhji3SQrkXtwLPqeokEWmD03+rmarmhD+8qHJan5vRWqKw4T9+Esq9QER+AwwHrlfVjAjFFmmF3YuKOINGvici23HqYJfEaIN2qP9HFqtqpqp+C3yJkzhiTSj3ojfwNwBV/QgogzNgYEkT0udJXtGaKGz4j58Uei/c6pZZOEkiVuuhoZB7oaoHVTVJVeupaj2c9prrVfW0B0OLYqH8H3kd50EHRCQJpyrqm4hGGRmh3IsdwNUAItIEJ1H8GNEoo8MS4A736afWwEFV3VXYQVFZ9aThG/7Dd0K8F08CFYCFbnv+DlW93rOgwyTEe1EihHgv3gY6ishmIBt4SFX3ehd1eIR4Lx4A5ojI/ThVLXfF4hdLEXkZp6oxyW2PGQmUBlDVNJz2meuAbcAxoFdI543Be2WMMaYYRWvVkzHGmChhicIYY0xQliiMMcYEZYnCGGNMUJYojDHGBGWJwkQdEckWkXUBr3pB9q1X0EiZRbzme+7oo+vdIS8ancY5UkTkDvfnu0TkVwHb/iwiTYs5zjUi0iKEYwaLSLkzvbYpuSxRmGh0XFVbBLy2R+i6PVT1IpzBJp8s6sGqmqaq893Fu4BfBWy7R1U3F0uUP8U5g9DiHAxYojCnzRKF8QW35PCBiHzqvi7LZ58LRGS1WwrZICIN3fW3B6yfJSJxhVxuJdDAPfZqdw6Dje5Y/4nu+vHy0xwgT7nrRonIgyJyM86YWy+61yzrlgSSRaSfiEwMiPkuEXn2NOP8iIAB3URkpoisFWfuicfddYNwEtYKEVnhrusoIh+593GhiFQo5DqmhLNEYaJR2YBqp9fcdT8Av1XVS4BuwNR8jksBpqhqC5wP6nR3uIZuQFt3fTbQo5Dr/x7YKCJlgOeAbqp6Ic5IBv1EpCpwI3CBqjYHxgQerKqLgLU43/xbqOrxgM2LgJsClrsBr5xmnJ1whunINVxVk4HmQHsRaa6qU3HG8rlSVa90h/J4FPiNey/XAkMKuY4p4aJyCA9T4h13PywDlQamuXXy2TjjFuX1ETBcRGoDr6rqVyJyNXApsMYd3qQsTtLJz4sichzYjjMMdSPgW1Xd6m7/K9AfmIYz18WfReQfQMhDmqvqjyLyjTvOzlfuNT50z1uUOMvjDFcROEPZH0SkD87/65o4E/RsyHNsa3f9h+51EnDumzEFskRh/OJ+YDdwEU5J+BeTEqnqSyLyMdAZeFtE7sEZVvmvqvpICNfoETiAoIjkO7+JO7ZQS5xB5roDA4CrivBeXgH+AGwBXlNVFedTO+Q4cWZxGw9MB24SkfrAg8CvVXW/iDyHM/BdXgIsV9VbixCvKeGs6sn4RWVglzt/QE+cb9M/IyLnAt+41S1LcKpg3gVuFpHq7j5VJfQ5xbcA9USkgbvcE3jfrdOvrKpv4TQU5/fk0WGcYc/z8ypwA84cCa+464oUp6pm4lQhtXarrSoBR4GDIlIDuLaAWFYBbXPfk4iUE5H8SmfGnGKJwvjFDOBOEVmFU+10NJ99ugGfi8g6oDHOlI+bcT5Q3xGRDcBynGqZQqnqCZzRNReKyEYgB0jD+dB90z3f+zilnbyeA9JyG7PznHc/sBmoq6qr3XVFjtNt+5gEPKiq63Hmx94EzMWpzso1G1gqIitU9UecJ7Jedq+zCudeGVMgGz3WGGNMUFaiMMYYE5QlCmOMMUFZojDGGBOUJQpjjDFBWaIwxhgTlCUKY4wxQVmiMMYYE9T/A4joN6rOZTrtAAAAAElFTkSuQmCC\" />\n",
    "        </td>\n",
    "    </tr>\n",
    "    </table>\n",
    "    \n",
    "The ROC curve for assay CHEMBL1614249 shows significantly better early enrichment than that for CHEMBL1614421 and an overall better AUC. The nice-looking ROC curve indicates that the model for CHEMBL1614249 is actually pretty good at distinguishing actives from inactives, adjusting the decision threshold allows us to take advantage of this.\n",
    "\n",
    "In order to try and determine how general the method is, I've applied the same balancing procedure to a collection of diverse datasets pulled from ChEMBL:\n",
    "\n",
    "  1. 13 Ki datasets for Serotonin receptors. [download](https://rdkit.org/datasets/chembl_5HT.csv.gz)\n",
    "  2. The 80 \"Dataset 1\" datasets from our benchmarking set. [download](https://github.com/rdkit/benchmarking_platform/tree/master/compounds/ChEMBL)\n",
    "  3. 8 large PubChem qHTS confirmatory assays. [download](https://rdkit.org/datasets/ChEMBL_PubChem_HTS.csv.gz)\n",
    "  4. 44 DrugMatrix assays. [download](https://rdkit.org/datasets/ChEMBL_DrugMatrix.csv.gz)\n",
    "\n",
    "Here's a summary graphic plotting the balanced kappa values (obtained by shifting the decision threshold) against the original kappa values for all the data sets. The points are colored by the out-of-bag AUROC. You can see that the balancing procedure results in better kappa values, often dramatically better values, in almost all cases and that the datasets with low AUROC values tend to show less improvement.\n",
    "\n",
    "<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4FOXawOHfG4oUaYLSEUhCSYBQD02QJuLhKHpEVFTUJDQD4ieKIAcEjYKARxFRWhDLodgQpEsLJYCEGgiEhB4glAChl7DP90eSNWWTTJKdLcl7XxcX2dnZmWe2zDPzViUiaJqmaRqAh7MD0DRN01yHTgqapmmalU4KmqZpmpVOCpqmaZqVTgqapmmalU4KmqZpmpVOCpqmaZqVTgqapmmalU4KmqZpmlVhZweQUxUqVJCaNWs6OwxN0zS3smPHjgsi8mB267ldUqhZsybh4eHODkPTNM2tKKWOG1lPFx9pmqZpVjopaJqmaVY6KWiapmlWOilomqZpVjopaJqmaVY6KWiapmlWOilomqZpVqYlBaXUbKXUOaXUvkyeV0qpL5VSMUqpvUqppmbFommaphljZue1OcBXwPeZPP8E4J38ryXwTfL/DmWxWNi4P5rI46fxebgK7Xy98fDQN1DuRn+Ors0VPp/ExESmLd/AjkPHaFanJgOeaE/hwhlPgUZitcfxZLcNZ71npiUFEdmglKqZxSo9gO9FRICtSqmySqnKInLGrJjSs1gsDPp6Lmv3HLQu6+RXj6/e6K1PKG5Ef46uzRU+n8TERDq8N4kLV64BsOXgEeav/4v1n76TJjEYidUex5PdNlI/L5Z7KI9CDnvPnPmLqQqcTPU4NnmZw2zcH53mQwFYu+cgG/dHOzIMLY/05+jaXOHzmbZ8gzUhpLhw5RrTlm9Is8xIrPY4nuy2sXF/NKu2hHN6zW+cCV2Sq33kljOTgrKxTGyuqFQ/pVS4Uir8/Pnzdgsg8vhpm8sPnHDYzYpmB/pzdG2u8PnsOHTM5vKd0WmHAzISqz2OJ6ttiAjfzvmOo7/O4NrxQ9xXrgJJBSqOec+cOSBeLFA91eNqgM13SkRmADMAmjdvbjNx5IbPw1VsLq9fo7K9dqE5gP4cXZsrfD7N6tRky8EjGZY39X44zWMjsdrjeDLbRvmiiieffJKlS5dS7KGqVGr3L+4rVyFX+8gtZ94pLAb6JLdCagUkOLI+AaCdrzed/OqlWdbJrx7tfL0dGYaWR672OVosFkIjovhmyTpCI6KwWCwOfX1O5WZ/OXlNO19vOjaqm2ZZx0Z17fb5JCYm8tUfa3n9s9l89cdaEhMTM6wz4In2VCh9f5plFUrfz4An2meINbvvktHjySqu9PsRESpdO01gz6dYt24d//3vf3l15CdpEoKjvtOm3SkopeYBHYAKSqlY4AOgCICITAOWAf8EYoAbwOtmxZIZDw8PvnqjNxv3R3PgxBnq16isW624IVf6HPNaCenoStnc7C9XMSqV9eNcMlqBXLhwYdZ/+g7Tlm9gZ/Rxmno/bLP1keHvUjbHk11cqfezYVs4C2dMIXT7Njp16sTMmTOpXbu2tfWRo7/TZrY+ejGb5wUIMmv/Rnl4ePBow7o82rBu9itrLstVPsesKhCNxJbX1+dUbvaX09ds3B/NunTrr7PTMWVVgTzoyU5plhcuXDjDMluy+y4ZOR4jcYkIO1cvY9zIkRQpUoSZM2cSEBCASk4wzvpO60tiTbOjvFZCOrpSNjf7y+lrzDwmoxXI9mTkeLKLKzIykkceeYS3336bzp07s3//fgIDA60JwZl0UtA0O8prJaSjK2Vzs7+cvsbMY2pWp6bN5ekrkO3JyPFkFpdfraoEBwfTpEkToqOj+d///sfixYupVq2aGaHmik4KmpaKkUrLrOS10tvRlea52V87X286pntNxyxek9P1wXhFttEKZHsy8p7ZiqvYrSt8O3YYo0aN4plnniEyMpLevXu7xN1Bam43R7OmmcVopWVW8lrp7eHhwZcDXkgajiH6GM28k4ZjMKuCMdfximT9OA/r56Qi22gFsj0Zec9Sx/VXZAxx4etZu/AnHnzwQRYuXMjTTz9tWnx5pSS7D9PFNG/eXMLDw50dhpYPffXHWr5avDbD8kFPdTJUQWkPrjAkRHZCI6Lo/+UPGZZPf/MVm5WiZq/vysLCwggICODgwYO8/vrrfPbZZ5QrV84psSildohI8+zWc41vmaa5AGdUWqbnCkNCZMfsimZX6AGdV9evX2fIkCE88sgj3Lx5k5UrVzJ79mynJYSc0ElB05I5o9IyPXc4IZpd0ewKPaDzYvXq1TRo0IAvv/ySoKAg9u3bR9euXZ0dlmE6KWhaMmdUWqaX1xOi2b2TIeeV0znt0ZzT7Tu6B3hmLl++TGBgII899hhFixZl48aNTJkyhfvvvz/7F7sQXdGsack8PDxoULMq6/dGWZc1qFnVoWX5KS11UneOyq6lTgpH9U7OVeV0Dno052T7rlIH88cffzBgwADi4uJ47733+OCDDyhevLjD9m9P+k5B05Jt3B+dJiEArN8b5fjy/Jy27EmWm/qI3NZhpPS2HdC9A482rJvlCTirHsB53b6z62DOnz9P7969eeqppyhfvjzbtm1j/PjxbpsQQCcFTbNyhfL8jfujWZcuMa0zmJgc0Ts5N8zch7M+MxFh/vz5+Pj48MsvvzB27FjCw8Np3jzbxj0uTycFTUvmChWceTnJOaJ3cm6YuQ9nfGanT5/m6aef5sUXX6R27drs3LmT0aNHU7RoUdP26Ug6KWhaMlcYgrteJiezutUrZfva3PZONvuYzdyHIz8zESEkJAQfHx9WrVrFpEmTCAsLo0GDBnbflzPpzmualoqzhitOsW7PQQZ+9WOG5d8MejnDUBG25CZ+RxyzmftwRPxHjx6lX79+rF69mkcffZRZs2bh5eVl132YzWjnNZ0UNENSfniRx0/j83AVPe+ESb5Zso7Ji9ZkWP7W010Y0L2D4wMq4CwWC1OnTmXEiBEopZg4cSL9+vVzy+++0aSgm6Rq2XKVZn8FQb3qtouP3KXjVn4SFRVFQEAAmzdvplu3bkyfPp0aNWo4OyzT6V+0li1nN/srKCwWCz9t2J5hudF+Cpp9JCYmMn78ePz8/IiMjOS7775j2bJlBSIhgE4KmgGu0FSzILDVHBWgV/vmhu/IXKV3r7vas2cPLVu2ZMSIEfzrX/8iMjKSPn36uNzw1mbSxUdatlyhqWZBkFnyjToZR8dGxiqZdTFf7ty+fZvg4GDGjx9P+fLl+eWXX3j22WedHZZT6G+Klq229T1tjgnUtr6nkyLKKD9cIec1+epivtzZtm0bTZs2JTg4mN69exMZGVlgEwLoOwXNgM0HDtuchHzzgcMuMb59frlCTmlzn/44jNYnZFXM5wqfk6u5ceMGo0aN4osvvqBKlSosW7aMJ554wtlhOV2BTwq6qWX29h87ZXv58dMucbLJ6grZFeIzKq+ztuliPuPWr19PYGAghw8fZsCAAXz66aeULl3a2WG5hAKdFPLLFabZ7mXSl+VeHoto7JWQ89MVsoeHh/XOIOW4jL4veb3TKAiuXLnCsGHDmD59Op6enqxbt44OHTo4OyyXUqCTQn65wjTbyXPxtpefv5jrbdozIeenK+S8vC95vdPI75YtW0b//v05ffo0Q4cO5cMPP6REiRLODsvlFOhvS35ramlaZWsmzfHy0kzPnpWirjBmkb3k9X3JyZDWBUV8fDx9+vShe/fulClThi1btjBp0iSdEDJRoO8U9BWmMU80b8jirXsyLO/WLPcDgdmzyMeeV8jOrmPKT0VhruCXX34hKCiIixcvMnr0aN5//33uu+8+Z4fl0gp0UshPZbBmFoW18/WiQun707RAqlD6ftr55n5AMHsn5JQr5LwcqyvUMeWnCxVniouLIygoiN9++41mzZqxatUq/Pz8nB2WWyjQ95YpV5jT33yFt57uwvQ3X3HbSmYzi8KyapKaW65Y5OMK7fxd8X1xJyLCd999h4+PD0uXLmX8+PFs3bpVJ4QcKNB3Cqm522ix6Zl5hWlGkYYrVoq6QtGNK74vrix1cV/5ojD7v5+ycuVK2rZtS0hICHXr6iK3nCrQScEVigvsxcyiMLMSjj2KfOzJVYpuXO19cVUpv981uw9w+cBOzm9fSyGl+PLLLwkKCnK737CrMDUpKKW6AZOBQsAsERmf7vkawHdA2eR1hovIMjNjSi0/NUk18wozP9W9ZCU/HKezK8odaeP+aFZsCCNu41Juxp2gRNVaVHrknzTq0DXfHrMjmJYUlFKFgKnAY0AssF0ptVhEIlOt9h/gJxH5RinlAywDapoVU3quUFxgT2ZdYRaUIg13P878dOebncTERCZ//jnHfgtBFSpMpfb/orR3I5RSbvv7dRVm3in8A4gRkSMASqn5QA8gdVIQIKVveRnA9lnaJK5SXOBO3L3uJTvuXHSTn+58s7Jv3z78/f3Zvn079z9ch4ptulG4ZCnr8/r3mzdmJoWqwMlUj2OBlunWGQOsUkoNBkoCXWxtSCnVD+gH2HWii/xQXOAIOb0CLUhFGK7E3ne+rvY53rlzh3HjxvHxxx9TtmxZ5s6dy4YLiWnmoNC/37wzMynY6u6a/jLzRWCOiHymlGoN/KCUaiAiabriisgMYAYkzdFsrwAdWVzgaj+wnMjJFWhBKsJwNfa883W1zzE8PBx/f38iIiLo3bs3kydPpkKFCjyf/Ltyx+I+V2VmUogFqqd6XI2MxUMBQDcAEdmilCoGVADOmRhXGo4oLnC1H1hO5eQKtKAUYbgie975usrnePPmTcaMGcOkSZOoVKkSixcv5sknn7Q+787Ffa7KzKSwHfBWStUCTgEvAL3TrXMC6AzMUUrVB4oB502MySlc5QeWWzm5As1vlffuxJ53vq7wOW7cuJGAgACio6Pp27cvEyZMoGzZsg7Zd0Fm2mWqiCQCg4CVwAGSWhntV0p9qJR6Knm1oUBfpdQeYB7wmji4JtMRM3a5+8B7Oellm5MEkh9mS3M19hoQz5mNMK5evcqgQYNo3749iYmJrFmzhhkzZuiE4CCm9lNI7nOwLN2y0an+jgTamhlDVhxVrOPurZxycgVqtAjD3YvU8jtnNcJYtWoVffv25eTJkwwZMoSPP/6YkiVLmrpPLa0C3aPZUcU6+aGVk9GyW6MJxFWL1Ny5QYA9ObrPxqVLl3j77beZM2cO9erVY9OmTbRp08aUfWlZK9BJwVHTTKb+gUUeP02lsqWocH9Jjpw+h2fVinmal8AVpU4gmZ1kM3vvI504xae+e0nLUZW4Cxcu5I033uD8+fO8//77jBo1iqJFixIaEVXgk7MzFOikkFkJtsWEag0PDw/aN6jD2XMXiYw5YV3uXb0yz3Vq5fKJITdX0FmdZDN77zOb+tMRXPXuJb86d+4cgwcP5qeffqJx48YsW7aMJk2a6OTsZAU6KahMTkBmfe0OnzpL9Mm0lcvRJ89w+NRZvKpVMmmveZfbH2lWJ9nMXuXhxOToCi1uCgIRYe7cuQwZMoSrV6/y8ccf8+6771KkSBFAJ2dnK9BpVzI5AZnVBiYu/nKOlueUWa15cjvPQFYnWd+aVW0+55tJpbwjuHuDAHcQGxvLk08+ycsvv0ydOnXYvXs377//vjUhgPu31nN3BfpOwZFXqxaLhfNXr9l8rlL5vDe1M/OWO7dX0FmdZF2x8t0VY3InWRUxiggzZ87k3XffJTExkc8//5zBgwdTqFChDNvRydm5CnRScNTVauoTdod6tameKgl4V6+MZ9WKed6Hmbfcuf2RZnWSdcURSV0xJneR1UXJ0aNH6du3L+vWraNTp07MnDmT2rVrZ7otnZydq0AnBUd9+VKfsNcfPEKVcqV5oGQJXurciqfbNrNLJbOZ5eG5fZ+yO8m64hAFrhiTO7B1UbJmVyRvDh/J7K8mU6RIEWbMmEFgYGC233ednJ2rQCcFR3350p+wT1+6wulLVzibcM1urY7MvOXOy/ukT7IFQ/rv+O1LF4jbuISp507RvXt3pk2bRrVq1QxvT39vnKdAJwVwzJfPEWWkZt/16B+plpWU77hY7nFxzxbid23Co0hR/jNuIh++N9Tlm1xrf8s2KSilPAA/oApwE9gvImfNDsxRHNGD1RHFVO54y+1uvYfdLV5HaufrTcMHivFHyFfcjj9Lqdo+/DvwDcYOG6gTgpvJNCkopTyB90ia+CaapNFLiwF1lFI3gOnAd+nnPnAnjuok4+gTtjvMjuZuHZTcLV5HunXrFh9++CELP5tAmXIP8NrIj3jx+ed00nRTWd0pBAPfAP3Tj1yqlHqIpGGwXwG+My88czmyk4zZxS8Wi4Wgr+eyLtXxdPSrx1QXPWm5WwclM+MVEQ6fOktc/GUqlS/rVkOfhIWFERAQwMGDB3n99df57LPPKFeunLPD0vIg07OFiLwoIhtsDWUtIudE5AsRcduEAJmPfZRZSx5XFhpxKE1CAFi35yChEYecFFHW3K2Dklnxigg/r93KgtVhhO6KZMHqMH5eu9Xl7/auX7/OkCFDeOSRR7h58yYrV65k9uzZOiHkA1kVH/07qxeKyG/2D8exXHH8ndxaHh5hc/mKHfvomG4uBFfgbh2UzIrXHYc+WbNmDX379uXo0aMEBQUxbtw4SpUq5eywNDvJqvgoZc67h4A2wNrkxx2B9YDbJwVH92h2SiWli+Y3d+ugZFa8mQ1xcubCJU5dSnCpSu2EhATeeecdZs2ahbe3Nxs2bKBdu3ZOjUmzv0yTgoi8DqCUWgL4iMiZ5MeVgamOCc9czujRnMJIJWVOEskTLRqyeOueDMu7tWiQ9wMwgbu1ljIr3syGOPltyy6W74y0PnZ2pfYff/zBgAEDiIuLY9iwYYwZM4bixYs7JRbNXEb6KdRMSQjJzgJ1TIrHodr5etOxUV3W7Y2yLuvYqK6pPZpTZFdJmdNE8miDOjaP5dEGrv9RuXr5eQozGgt4Vq2Id/XKaYqQHihTih8270yznrMq4c+fP8+QIUOYN28eDRs2ZNGiRTRv3tyhMWiOZSQprFdKrSRpDmUBXgDWmRqVI6UvKjKh6Cg3Q1DkNJF4eHgwNeglt7ny1k08kyileK5TqzStj1btirS5riOH8BYRFixYwODBg0lISGDs2LEMHz6cokWLOmT/mvNkmxREZJBS6hmgffKiGSKy0NywHGPj/mibLXbsfUWWm0rK3CQSd+p17G5NUs2klMKrWiVrxfKpSwk213NUJfzp06cZOHAgixcvpkWLFsyePZsGDVyzGFKzP6OXZDuBpSLyf8BKpVS+aGqQ1XSc9pRSTJVadsVU7tY6J6fcrUmqI6VUaqfmiEp4EWH27Nn4+PiwatUqJk6cSFhYmE4IBYyRYS76Av2ABwBPoCowDehsbmjmS2mSWqVcacqXLEH89RucvnTFlOk4c1pM5W6tc3Iqvye9vHBGJfyxY8fo168ff/75J+3bt7e2MNIKHiN1CkHAP4BtACISndyj2e0pkQzzG5yMv2z36ehyU0zlbq1zciq/J728clRRoMViYerUqYwYMQKlFF9//TX9+/fPN98zLeeMJIXbInInpdu9UqowLtv6PWeqPFCGy+maBFYvX5bK5UrbdT+5nevAneoIciq/Jz13EBUVRWBgIJs2baJbt25Mnz6dGjVqODsszcmMJIVQpdT7QHGl1GPAG8Af5oblGBXuL5mj5bmli0psy89Jz5UlJiby2Wef8cEHH1CiRAm+++47XnnlFbcZb0kzl5HLsuEkjZAaAfQHlgH/MTMoR6lcwfY4LZktzy1nVRxqWnp79uyhZcuWDB8+nH/9619ERkbSp08fnRA0KyN3CsWB2SIyE0ApVSh52Q0zA3MEWx2H7DVncmq6qERzttu3b/Pxxx8zbtw4HnjgAX7++Wd69uzp7LA0F6Sy602qlNoKdBGRa8mP7wdWiUgbB8SXQfPmzSU8PNxu23PnYYs1zYht27bh7+9PZGQkr7zyCp9//jnly5d3dliagymldohItt3RjVyqFktJCADJf5fIS3CuREQ4dSmBiJNnOHUpwbQhFywWC6ERUXyzZB2hEVFYLG47N5HmJm7cuMHQoUNp06YNV65cYenSpXz//fc6IWhZMlJ8dF0p1VREdgIopZqRNC1ntpRS3YDJQCFgloiMt7FOL2AMSS2a9ohIb4Ox55mjhlrQQzpojhYaGkpAQACHDx9mwIABfPrpp5Qubd9WdVr+ZOSM9Bbws1Jqo1JqI7AAGJTdi5LrHqYCTwA+wItKKZ9063gDI4C2IuKbvC+HyWqoBXfcj6ZduXKFgQMH0qFDBwDWrVvHN998oxOCZli2SUFEtgP1gIEkNUetLyI7DGz7H0CMiBwRkTvAfKBHunX6AlNF5FLyvs7lJPi8ctRQC3pIB80Rli1bhq+vLzNmzGDo0KHs3bvXmhw0zSijZRctgEZAE5Ku+PsYeE1V4GSqx7HJy1KrA9RRSm1WSm1NLm5yGEf1H9D9FDQzxcfH06dPH7p3707p0qUJCwtj0qRJlCiRb6r+NAfKNikopX4AJgGPkJQcWgBGBlS31YQnfS1uYcAb6AC8CMxSSmWYdUQp1U8pFa6UCj9//ryBXRvjqP4Dup+CZpZffvkFHx8f5s2bx6hRo9i5cyctW7Z0dliaGzNS0dycpJnXctosJxaonupxNSB9OUossFVE7gJHlVJRJCWJ7alXEpEZwAxIapKawzgy5aj+A0b347QpOzW3ExcXR1BQEL/99htNmzZl1apV+Pn5OTssLR8wkhT2AZWAnBaAbwe8lVK1gFMkTc6TvmXR7yTdIcxRSlUgqTjpSA73kyeOGmohu/3oFkqaESLCDz/8wFtvvcWNGzcYP348Q4cOpXBhIz9lTcuekW9SBSBSKfUXcDtloYg8ldWLRCRRKTUIWElSk9TZIrJfKfUhEC4ii5Of66qUigTuAe+KSHwuj8Wt6UlntOycOHGC/v37s2LFCtq2bUtISAh16+rvhmZfRpLCmNxuXESWkTRWUuplo1P9LcDbyf8cIn0RTdv6nmw+cNjpRTa5HUlVy/8sFgvTp09n2LBhiAhTpkzhjTfe0HeQmimMTMcZ6ohAHMFWEU2F0vdz4Yq1wzYd/eox1UFFNqkT1L1MqmwqlrmfTXsO6iE4Cqjo6GgCAwPZsGEDXbp0YebMmdSsWdPZYWn5mJGZ11oBU4D6QFGSioKui4jb9YaxVUSTOiFA0uQ3oRGH6JiutZC9GUlQPVs2IjLmhPWxd/XKPNeplU4MBcC9e/f4/PPPGTVqFPfddx8hISG8/vrr+rPXTGek+OgrkiqJfyapJVIfkloIuZ3MimjSW7Fjn+lJIXWCSj0daL/HHwGLUKHM/Zw8k7b5bfTJMxw+ddY6wbsj6AEDHW/fvn34+/uzfft2evTowddff02VKrb7umiavRlqsiAiMUqpQiJyD/hWKRVmclymyKwTWQYOmFcuJUGlnw705rWkEclP3rA9vFRc/GWHJQUR4ee1WzMMLa7vVsxx584dxo8fT3BwMGXLlmX+/Pn06tVLv9eaQxlJCjeUUkWB3UqpCSQ1TbXv1GQOYmteYFu6NvO1+77TX3HXr1GZKuVKp0kIRlTK4fp5cfjU2TQJAZxzt1IQhIeH4+/vT0REBL1792by5MlUqFDB2WFpBZCRpPAKSfUIg4D/I6lD2rNmBmWW9J3I7t67x9Q/1tlYz75XZjavuKtVooVnzubDNWMCoKzExV/OdLlOCvZx8+ZNxowZw6RJk6hUqRKLFy/mySefdHZYWgFmpPXR8eQ/bwJjzQ3HfKk7kX2zJGNCAIg6GUfHRvarU7B5xR0bxz9bNGTT3qgsX9uucX08lHJKeX5mdyWOvFvJzzZu3EhAQIC1hdHEiRMpW1a/t5pzZZoUlFIRZFG6LiKNTInIgeplMiBd3er2vQrO7Iq7UKFCGaYDTc27emXa+dVzWpmyo6YrLWiuXr3KiBEjmDp1KrVq1WL16tV07tzZ2WFpGpD1ncK/HBaFs2SW8uxc0ZzVFXfbRnWtdQ0VHygDwNmLCS7R0kcpxXOdWunWR3a0atUq+vbty8mTJxkyZAgff/wxJUu6ZRWdlk9lmhRSFRvlWwczuUKPio2za5NUz6oV8a5WiejYOOsy72qVqFX5QTbsO2TtTe1ZtSIeHh54V3edIbWVUnhVq6TrEPLo0qVLDB06lG+//ZZ69eqxadMm2rRxyjTnmpalAtV5LT1HzXMgIqw/eISDJ07zQMkSXLx+g9hrN1h34DDrUtUp6AHw8qfff/+dgQMHcv78ed5//31GjRpFsWLFnB2WptmU285rXmYG5SjtfL3p2KhumhNzx0Z17T7PQeqOaqcvXUnzf2oHT5zmf6vDaO3jpYtp8oFz584xePBgfvrpJxo3bsyyZcto0qSJs8PStCwVqM5rNqU/8ZpwIt5/7FS266R0Yjtx+hwnTp9zeCcx3XPZfkSEuXPnMmTIEK5evUpwcDDDhg2jSJEizg5N07JVoDqvpbdxfzTr0nVkW2fCcNWWbJ631YnNkZ3EdM9l+4mNjWXAgAEsXbqUVq1aERISgo+Pj7PD0jTDjBRev5K83iDgOm7ceS29rIartqfM3uRaFcsDUL6k7bl0M2vKCkkD6oVGRPHNknWERkRhsWSXejKXVc9lzRgRYebMmfj6+rJ27Vo+//xzNm3apBOC5nYMdV5LvlOoCfwGRInIHbMDcwRHVTT71qxqc/mw557Aw0OxJ/o4V9ON1gqZN2W19yxtuudy3hw5coS+ffuydu1aOnbsyMyZM/H09HR2WJqWK9meQZRS3YHDwJckVTrHKKWeMDswR2hb35MKpe9Ps6xC6ftpW9++P+iUMZdS6+RXj0cb1uHRhnUZ/MxjGZqhZtVJLKtZ2nJD91zOnXv37vHFF1/QsGFDwsPDmTFjBmvWrNEJQXNrRuoUPgM6ikgMgFLKE1gKLDczMEfYfOBwhvkULly5xuYDh+1ap5B+zKX6NSqnmeEtp53EspulLaeVxrrncs4dOHCAgIAAtmzZQvfu3Zk2bRrVqlVzdlialmdGksK5lIRO1ye/AAAgAElEQVSQ7AhwzqR4HMqRU2CmHnMp/ZSgKQnCaCexrIq9clNprHsuG3f37l0mTJjAhx9+SKlSpfjxxx/p3bu3fq+0fCOrsY/+nfznfqXUMuAnkgaAeA7Y7oDYTOeoOoXU7FEfYGsI8E5+9Wjn653r4a6VUtSu8hCnLiXw5+4DnLqU4LT5ql3Vrl278Pf3Z/fu3fTq1YspU6bw0EMPOTssTbOrrO4UUo/fexZ4NPnv80A50yJyoKxOrmbJqj7A6N1JVsVRua00tnfldX5y69YtPvroIz799FMefPBBFi5cyNNPP+3ssDTNFFmNffS6IwNxhuzK+s1g7yIrkbSj9+W20tgeySo/2rJlC/7+/hw8eJDXXnuN//73v5Qrly+uiTTNpqyKj/4DTBWRS5k83wkoISJLzArOEVKX9dti756+9iiyyuqqPreVxo6oX8msLsUVXb9+nZEjR/Lll19SvXp1VqxYweOPP+7ssDTNdFkVH0UAS5RSt4CdJBUbFQO8gcbAauAT0yM0WVYnKjN6+tqjyCq7q/rcVBqbXb/iTsVTa9asoW/fvhw9epSgoCDGjRtHqVKlnB2WpjlEVsVHi4BFSilvoC1QGbgC/Aj0ExHbM8u7kexOVGbMUWyPIqvsrupzM9y12fUr7lA8lZCQwDvvvMOsWbPw9vZmw4YNtGvXztlhaZpDGenRHA3krleUi8vuRGVWT9/siqyyk91VfW6KacyuX3Fk89/c+OOPPxgwYABxcXEMGzaMMWPGULx4cWeHpWkOZ2iU1PwquxOVq/b0zeqqPi/FNHlNVllxRvNfIy5cuMCQIUOYO3cuDRs2ZNGiRTRv3typMWmaMxXopJDdicpVe/pmdVUfGhHlksU0zmj+mxUR4aeffmLw4MFcvnyZMWPGMGLECIoWLeqUeDTNVRTopJDdicoZPX2NtnbK7KreVYtp0ieyetUrUaVsacIiDjm8B/Xp06d54403WLRoES1atGD27Nk0aNDAIfvWNFdnZDrOYkAA4EtS6yMARMTfxLgcwkg5uiPnKLZHa6fcFtM4orloSiJr36AOP6/dyuadkdbnHDF/g4jw7bff8vbbb3P79m0mTpzIW2+9ReHCBfraSNPSMPKr/wGoBDwOhALVgKtGNq6U6qaUilJKxSilhmexXk+llCilHF6Yq5SiarkyNKhWiarlyjh1DBt7zGuQ2YisWRXTpNRD9P/yByYvWkP/L39g0Ndz8zRHQ1acMX/DsWPHePzxxwkICMDPz4+9e/fyzjvv6ISgaekY+UV4ichzSqkeIvKdUmousDK7FymlCgFTgceAWGC7UmqxiESmW68U8CawLefh542rzThmj9ZOuWlF5Ojmoo6cv8FisTB16lRGjBiBUoqvv/6a/v37u1zfCE1zFUaSwt3k/y8rpRoAcSRNuJOdfwAxInIEQCk1H+gBRKZb7yNgAvCOkYDzKnWZvUXE7v0Q8sJerZ1y2orI0fUQjmrVFRUVRWBgIJs2baJbt25Mnz6dGjVq2HUfmpbfGEkKM5RS5YBRwGLg/uS/s1MVOJnqcSzQMvUKSqkmQHURWaKUMj0p2LozsMVZM445q7VT6noIJUKjq5epefM6vpfjEYsFZeerarOPMzExkc8++4wPPviAEiVKMGfOHPr06aOHt9Y0A4x0XpuV/GcoUDsH27b1C7SO3qaU8gA+B17LdkNK9QP6AXm60rNVlm2Ls/ohOGteg5R6iHW7DzDkWBTNriQPdzX5a2J27cZrwli7JgYzj3Pv3r34+/uzY8cO/v3vfzN16lQqVdJTimqaUUam4yyvlJqilNqplNqhlPpCKVXewLZjgeqpHlcDUpdTlAIaAOuVUseAVsBiW5XNIjJDRJqLSPMHH3zQwK5ty6wsOzWvapWc2g8hpbXTI3718KpWySFXtyn1EDM6Nv07ISS7vCGMhC32nz7D3sd5+/ZtRo8eTbNmzTh58iQ///wzv/76q04ImpZDRi7/5pM009qzQE/gArDAwOu2A95KqVpKqaLACyQVPwEgIgkiUkFEaopITWAr8JSIhOfwGAwzcgdQUEsYPDw88LxlezirG1ExNpe7im3bttGsWTM++ugjXnzxRSIjI+nZs6ezw9I0t2QkKTwgIh+JyNHkf8FAtmdXEUkEBpHUUukA8JOI7FdKfaiUeipvYeeOZ9WKeGdTVxB9Ms7UppGurEQ9281WS9T1cnAkxty4cYOhQ4fSpk0bEhISWLp0Kd9//z3lyxu5kdU0zRYjFc3rlFIvkDQdJyTdLSw1snERWQYsS7dsdCbrdjCyzbwQEdYfPMLBE6d5oGQJShe7D8+KGU8gzqpodrYyrVtQtn0bLm8Isy4r274NZVq3cGJUtoWGhhIQEMDhw4cZMGAAn376KaVLl3Z2WJrm9owkhf7A2yQNmQ1JdxfXlVJvAyIibvNLTN0e//SlK1QpV9pmUnD2gHfOojw88JowloQt27kRFUOJul6Uad3C7q2P8uLKlSu89957TJs2DU9PT9atW0eHDh2cHZam5RtGWh/lm9lF0rfHP33pCjfv3KV40SLWZSWL3UftKgV3Mnbl4UHZti0p27Zl9is72PLly+nfvz+nTp3i7bff5qOPPqJEiRLODkvT8hVDffyVUv8GHiGpSelGEfnd1KhMkn5coCrlSqdJCADXb90mJjaOOjVsjyGUE+40/aQru3jxIm+99RY//PADPj4+hIWF0bKl6yUtTcsPjAyI9zXgBcxLXjRAKfWYiASZGpkJ0o+KWr6k7avMvw4eyXNScKfpJ13Zr7/+SlBQEPHx8YwaNYqRI0dy3333OTssTcu3jNwpPAo0EBEBUEp9R9L8zW4n/bhAJ+LO21wv/tqNPO/LHaafdGVxcXEMGjSIX3/9laZNm7Jy5Ur8/PycHZam5XtGLlmjgNTdiKsDe80Jx3wp4wIN6N6B5nVq2lyn+oPl8ryfrMYT0jInInz//ff4+PiwZMkSxo8fz7Zt23RC0DQHMXKnUB44oJT6K/lxC2CLUmoxgIg4pc+BPTxY6n4OZLI8r1x1+klXduLECfr378+KFSto27YtISEh1K2r76o0zZGMJAWb/Qryg4oPlMnR8pwwMv2k0VnW8juLxcL06dMZNmwYIsKXX35JUFCQrnvRNCcw0iQ11BGBOMOZTMZCOhN/Oc8VzVnNa2CxWNiw7xB/7Yvm7p271tc4cy4HZ4mJiSEwMJDQ0FC6dOnCzJkzqVmzprPD0rQCy0jro1bAFKA+UBQoBFx3p05rmYk9F5+j5Tlla16DlFZJB0+cprNP2uEjnDmXg6Pdu3ePL774glGjRlG0aFFCQkJ4/fXXC1RC1DRXZOT+/CvgRSAaKA4EJi9ze9UrVsjRcntIaZWUWXNYIyO5urt9+/bRpk0b3nnnHbp27UpkZCT+/v46IWiaCzBUaCsiMUAhEbknIt8CHUyNykEeaVSXksXStnkvWew+HmlkXuVmSquk+Ou2m73m5yE27ty5w4cffkjTpk05cuQI8+fPZ+HChVSpkveOgpqm2YeRiuYbyUNf71ZKTQDOACXNDctxos5f5PbNWzxYqiTnr17nvuLFTN1fSquk05eucDL+MtVTJQFHzLLmLOHh4fj7+xMREcGLL77I5MmTycvcGJqmmcPIncIryesNAq6T1E/hWTODcpT1EVHInbt4PlSe0sWL4flQeeTOXdZHRJm2z5RWSQDrDx5hTWQMCbfv8FynVvmykvnmzZu89957tGzZkvj4eBYvXszcuXN1QtA0F2XkTqGCiBwHbgFjAZRSTwKuPfOKAavD96W5UgeoXr4sa8L308mvvin7zKpVUn6zadMmAgICOHToEIGBgUycOJGyZfNv8Zim5QdGzkQzlVINUx4kz63wH/NCcpwimVyVFzb5Yj11r+pHG9bNdwnh2rVrDB48mPbt23P37l1Wr17NzJkzdULQNDdg5GzUE/hOKVVfKdUXCAK6mhuWY1TJpFK3aj6u7DXbqlWraNCgAVOnTuXNN98kIiKCzp07OzssTdMMyjYpiMgRkuZX/pWkBNFVRBLMDswROjROVUQkQoXjx6gVHk4byx3EYnFeYG7o0qVL+Pv78/jjj1OsWDE2bdrEF198QcmS+aZNgqYVCJnWKSilIkiaPyHFAyR1XNumlEJEGpkdnNnq1KiCV9WKxMTG0Xj5Mh46dgyAm39tI2bzFrwmjHWpWcdc1e+//87AgQM5f/48I0aMYPTo0RQrZm4rLk3TzJFVRfO/HBaFkyil6NWlDQcXr+RackJIcXlDGAlbtrvkDGSu4ty5cwwePJiffvoJPz8/li5dStOmTZ0dlqZpeZBpUkhucZTvKaUoc+EC12w8dyMqRicFG0SEefPm8eabb3L16lWCg4MZNmwYRYoUyf7Fmqa5NEPTceZ3Jep5215e18vm8oLs1KlTDBgwgCVLltCqVStCQkLw8fFxdliaptmJLjAHyrRuQdn2bdIsK9u+DWVat3BSRK5HRJg5cyY+Pj6sWbOGzz//nE2bNumEoGn5jL5TAJSHB14TxpKwZTs3omIoUdeLMq1b6ErmZEeOHKFv376sXbuWjh07MnPmTDw9PZ0dlqZpJtBJIZny8KBs25a6DiGVe/fuMWXKFEaOHEmhQoWYPn06ffv2zXdDcWia9jedFDSbDhw4QEBAAFu2bKF79+5MmzaNatWqOTssTdNMVuDLR8Ri4fLmbZwO+ZHLm7cV+E5rd+/e5ZNPPqFx48ZERUXx448/8scff+iEoGkFRIG+UxCLhZhhH3B5Q5h1Wdn2bQpsp7Vdu3bh7+/P7t276dWrF1OmTOGhhx5ydliapjlQwTvzpZKwZXuahAB/d1orSG7dusXIkSNp0aIFcXFxLFy4kAULFuiEoGkFUIFOCjcORtteHuX2o4IbtmXLFpo0acInn3zCK6+8QmRkJE8//bSzw9I0zUlMTQpKqW5KqSilVIxSariN599WSkUqpfYqpdYopR42M5703LnTmogQExvHpj0HiYmNQ0Syf1Eq169f56233qJt27bcuHGDFStW8O2331KuXDmTItY0zR2YVqeglCoETAUeA2KB7UqpxSISmWq1XUBzEbmhlBoITACeNyum9FI6raWvU3D1Tmsiws9rtxJ98ox1mXf1yoZnbluzZg19+/bl6NGjBAUFMW7cOEqVKmVmyJqD3b17l9jYWG7duuXsUDQHK1asGNWqVcv1sDNmVjT/A4hJHnobpdR8oAdgTQoisi7V+luBl02MJwN37bR2+NTZNAkBIPrkGQ6fOotXtUqZvi4hIYF3332XmTNn4u3tTWhoKO3btzc7XM0JYmNjKVWqFDVr1tT9SgoQESE+Pp7Y2Fhq1aqVq22YefarCpxM9Tg2eVlmAoDltp5QSvVTSoUrpcLPnz9vxxD/7rRWxf8lyrZt6fIJASAu/nKOlgMsWbIEX19fQkJCePfdd9mzZ49OCPnYrVu3KF++vE4IBYxSivLly+fpDtHMM6Ctb6PNgm+l1MtAc2CiredFZIaINBeR5nrCd6iUycxwtpZfuHCBl156iSeffJIHHniArVu3MmHCBIoXL252mJqT6YRQMOX1czczKcQC1VM9rgacTr+SUqoLMBJ4SkRumxhPvuFZtSLe1SunWeZdvTKeVStaH4sICxYswMfHh59//pkxY8YQHh5OixauXV+i5S8LFy5EKcXBgwcBWL9+Pf/6V9qpWl577TV++eUXADp06EDdunXx8/OjRYsW7N6927peQkICffr0wdPTE09PT/r06UNCwt+TQB46dIh//vOfeHl5Ub9+fXr16sXZs2cNxblixQrq1q2Ll5cX48ePt7nO8ePH6dy5M40aNaJDhw7Exsbm6L1wGyJiyj+S6iuOALWAosAewDfdOk2Aw4C30e02a9ZMNBGLxSLRJ8/Ixt0HJPrkGbFYLNbnTp06JT169BBAWrRoIREREU6MVHOGyMhIZ4cgIiLPPfecPPLII/LBBx+IiMi6deuke/fuadZ59dVX5eeffxYRkUcffVS2b98uIiKzZ8+WLl26WNd79tlnrdsRERk9erT07NlTRERu3rwpXl5esnjxYuvza9eulYiICLl27Zrcvn070xgTExOldu3acvjwYbl9+7Y0atRI9u/fn2G9nj17ypw5c0REZM2aNfLyyy/n4J1wLFufPxAuBs6xpt0piEgiMAhYCRwAfhKR/UqpD5VSTyWvNhG4H/hZKbVbKbXYrHiyiDNPTTudRSmFV7VKPOJXD69qlUieIpXZs2fj4+PDypUrmThxImFhYTRo0MDZ4WouzmKxEBoRxTdL1hEaEYXFDsO9XLt2jc2bNxMSEsL8+fNz/PrWrVtz6tQpAGJiYtixYwejRo2yPj969GjCw8M5fPgwc+fOpXXr1jz55JPW5zt27EiDBg04dOgQdevWZejQoRw4cCDDfv766y+8vLyoXbs2RYsW5YUXXmDRokUZ1ouMjKRz587WbdtaJz8wdZgLEVkGLEu3bHSqv7uYuf/sSB6bdrqSY8eO0a9fP/7880/atWtHSEgI3t62+2FoWmoWi4VBX89l7Z6D1mWd/Orx1Ru98chDw4vff/+dbt26UadOHR544AF27tyZo9evWLHC2pEyMjKSxo0bU6hQIevzhQoVonHjxuzfv599+/bRrFkzm9tp0qQJe/fuZcGCBQQGBqKUIiAggF69elGyZElOnTpF9ep/l3RXq1aNbdu2ZdiOn58fv/76K0OGDGHhwoVcvXqV+Ph4ypcvn6PjcnWu39TGRFk17XQXFouFr776igYNGrBlyxamTp3K+vXrdULQDNu4PzpNQgBYu+cgG/fb7vFv1Lx583jhhRcAeOGFF5g3b16mF1upl7/00ktUq1aNTz/9lMGDBwNJF3C2XpvZ8vRKlSpFYGAgmzdvZsaMGcycOZPKlStbt5FVPCkmTZpEaGgoTZo0ITQ0lKpVq1K4cP4bPi7/HVEOZNW0M6v2/q7i0KFDBAQEsGnTJh5//HGmT5/Oww87tFO4lg9EHs/Q/gOAAyfO8GjDurnaZnx8PGvXrmXfvn0opbh37x5KKfr06cOlS5fSrHvx4kUqVKhgffy///0PPz8/hg8fTlBQEL/99hu+vr7s2rULi8VivXuxWCzs2bOH+vXrc+7cOUJDQ7OM6fjx48yZM4d58+bh5+fHmDFjgKQ7g5Mn/249HxsbS5UqVTK8vkqVKvz2229AUtHYr7/+SpkyZXL1/riyAn2nkJOmna4kMTGRCRMm0KhRI/bv38+cOXNYvny5Tgharvg8nPEECFC/RmWby4345Zdf6NOnD8ePH+fYsWOcPHmSWrVqcfHiRU6fPm0t2z9+/Dh79uyhcePGaV5fpEgRgoOD2bp1KwcOHMDLy4smTZoQHBxsXSc4OJimTZvi5eVF7969CQsLY+nSpdbnV6xYQUREBMeOHaNLly706NGDsmXLsnnzZhYsWEDXrl0BaNGiBdHR0Rw9epQ7d+4wf/58nnrqKdK7cOGCta5l3Lhx+Pv75/r9cWUFOil4Vq2Id9WKVDh+jFrh4VQ4fgzvqhXTNO10NXv37qVVq1a89957dO/encjISF599VW3qwPRXEc7X286+dVLs6yTXz3a+ea+CHLevHk888wzaZY9++yzzJ8/nx9//JHXX3+dxo0b07NnT2bNmmXzirt48eIMHTqUSZMmARASEsKhQ4fw8vLC09OTQ4cOERISYl13yZIlTJkyBW9vb3x8fJgzZw4PPfQQhQoV4pNPPmH37t0MGTIkQx1A4cKF+eqrr3j88cetTVl9fX2BpMrsxYuT2r+sX7+eunXrUqdOHc6ePcvIkSNz/f64MuUurW1SNG/eXMLDw+2yLbFYiHl3NJc3brEuK9uuNV4TP3S5ns23b9/m448/Zty4cTzwwANMnTqVnj17OjsszUUdOHCA+vXrG17fYrGwcX80B06coX6NyrTz9c5TJbPmXLY+f6XUDhFpnt1rC3SdQsKW7WkSAsDljVtI2LLdpeZq/uuvv/D392f//v28/PLLfPHFF/muxYPmXB4eHjzasG6u6xC0/KNAXwq4+nwKN27c4J133qF169YkJCSwdOlSfvjhB50QNE0zTYG+U3Dl+RRCQ0MJDAwkJiaG/v37M2HCBEqXLu3ssDRNy+cK9J1CynwKqTl7PoUrV67wxhtv0KFDB0SEdevWMW3aNJ0QNE1ziAJ9p+Bq8yksX76c/v37c+rUKd5++20++ugjSpQo4ZRYNE0rmAp0UkjDia2wLl68yFtvvcUPP/yAj48PYWFhtGzpOhXdmqYVHAW6+CilSWr0/73PqenfEv1/7xPz7mjEDoOBGfXrr7/i4+PDvHnz+M9//sPOnTt1QtDyheyGoz5x4gQdO3akSZMmNGrUiGXLltnYSubSD8sNrjs0d3YuXrzIY489hre3N4899liGXt8p3nvvPRo0aECDBg1YsGCBXfadXoFOCpfDttlskno5LONgWPYWFxdHz5496dmzJ1WrViU8PJyPPvqI++67z/R9a5rZ7t27R1BQEMuXLycyMpJ58+YRGRmZZp3g4GB69erFrl27mD9/Pm+88UaW20x/opw3bx6PPPJIjkdg/d///seePXt44403ePfdd63LAwICqF27NocPH+bw4cPUqlWLwMBAIGkmu+7duzNw4EBiYmI4cOAAAwcOJKuZIDM7sdsyfvx4OnfuTHR0NJ07d7aZRJcuXcrOnTvZvXs327ZtY+LEiVy5ciUHR25MgU4Kl/5cb3v56qzHUMkLEeH777/Hx8eHJUuWMG7cOLZt24afn59p+9S07Nh7CHkjw1ErpawntYSEBJvjDYkIa9eupXfv3jRv/ne/q7wOyw32G5rbSLzZWbRoEa+++ioAr776Kr///nuGdSIjI3n00UcpXLgwJUuWxM/PjxUrVuTomI3QdQq2mFS9cPLkSfr378/y5ctp27Yts2bNol69etm/UNNMZMYQ8kaGox4zZgxdu3ZlypQpXL9+ndWrV1ufO336NHPmzLHWs/n7+/PDDz9Yn7c1LHfTpk1zFKO9huY2Em+7du24evVqhtdNmjSJLl26cPbsWeuorZUrV+bcuXMZ1vXz82Ps2LG8/fbb3Lhxg3Xr1uHj45OjYzaiQCeFcl06EL98tY3lj9p1PxaLhRkzZjBs2DAsFgtffvklQUFBehgBzSVkNYR8bkcLNjIc9bx583jttdcYOnQoW7Zs4ZVXXmHfvn2Eh4fTpk0bAgMD2bhxY5oRVFO/9q233gL+Hpa7adOmhofmvn79Ovfu3bPO8ZCXobn/+uuvbOPduHFjltswomvXrmzfvp02bdrw4IMP0rp1a1OG7i7QZ6WybVtSpl3rNMvKtGtt1yEuYmJi6NSpEwMHDqRly5bs27ePwYMH64SguYyshpDPLSPDUYeEhNCrVy8gqSjn1q1bXLhwgUaNGhESEkJkZCQ9evRg5syZacrOU4blDgwMpGbNmkycOJEFCxYgIpQvX97Q0NxHjx6ld+/eBAUFAaQZmjtF6qG5fX192bFjh81jzS5eSLpTaNy4cYZ/KXdHFStW5MyZpMR85swZHnroIZv7GjlyJLt37+bPP/9ERMyZN8XInJ2u9M/eczRb7t2TS5u2yqmQH+XSpq1iuXfPLttNTEyUSZMmSfHixaVMmTIya9asNPMoa5qZcjJHc/TJMxL87a8Z/kWfPJPr/d+9e1dq1aolR44csc57vG/fvjTrdOvWTb799ltrvJUrV87wGzl06JC89957Urt2beucyNOmTZN+/fqlWa99+/ayYcMGuXXrltSsWdN6/MeOHZMaNWrI5cuXRSTtHNA3btyQypUrW9d95plnZOzYsdZtjh07Vv79739b1/X09JQlS5ZYn1++fLns3bs323iNeOedd2TcuHEiIjJu3Dh59913M6yTmJgoFy5cEBGRPXv2iK+vr9y9e9fm9vIyR7PTT/I5/WfvpGCGffv2yT/+8Q8B5KmnnpJTp045OyStgMlJUrBYLLJgdViahLBgdVieL2KWLl0q3t7eUrt2bQkODhYRkVGjRsmiRYtERGT//v3Spk0badSokfj5+cnKlSsz3dbdu3fl999/F5GkE/vy5cvTPD958mQZMGCAiIhs2rRJWrZsKX5+ftK8eXNZtWqVdb3USUFEZNKkSeLv7y8iIhcvXpSXXnpJPD09pXbt2vLSSy/JpUuXrOseOHBAHn/8cfHy8pL69evL888/L3FxcdnGa8SFCxekU6dO4uXlJZ06dZL4+HgREdm+fbsEBASIiMjNmzelfv36Ur9+fWnZsqXs2rUr0+3lJSkU6KGz7e3OnTuMHz+e4OBgypQpw5QpU3j++ef1XAeaw+V06GwR4fCps8TFX6ZS+bJ4Vq2ov7duTA+d7QJ27NiBv78/e/fu5cUXX2Ty5Mk8+OCDzg5L0wxRSuFVrZJbTEOrmUvXdubRzZs3GT58OC1btuTChQssXryYuXPn6oSgaZpb0ncKebBp0yYCAgI4dOgQgYGBTJw4kbJlXXt+Z03TtKzoO4VcuHbtGoMHD6Z9+/bcuXOHP//8k5kzZ+qEoGma29NJIYf+/PNPGjRowNSpUxk8eDARERF06dLF2WFpmqbZhU4KBl26dAl/f3+6du1KsWLF2LhxI5MnT+b+++93dmiapml2o5OCAYsWLcLX15fvv/+eESNGsHv3btq2bevssDTN5aUf3jq/D209bNgwfH19qV+/Pm+++WaeBxZ0Bp0UsnD+/HleeOEFnn76aR566CH++usvPvnkE4oVK+bs0DTNLeRmeGt3Hdo6LCyMzZs3s3fvXvbt28f27dsJDTVvxGWz6KRgg4gwd+5c6tevz8KFCwkODmb79u05HoVR09yFWCxc3ryN0yE/cnnzNrtMNJXX4a3dbWhrpRS3bt3izp073L59m7t371KxYsUcHbMrMLVJqlKqGzAZKATMEpHx6e1ONf8AAA0HSURBVJ6/D/geaAbEA8+LyDEzY8rOqVOnGDBgAEuWLKFly5bMnj3blOFpNc1ViMVCzLAPuLwhzLqsbPs2eE0Ym6f5ym0Nb50T7ja0devWrenYsSOVK1dGRBg0aFCOepW7CtOSglKqEDAVeAyIBbYrpRaLSOrplwKASyLipZR6AfgUeN6smGxJ6d5/5sIlNqxayoSPP+Lu3bv897//5c0330zzJdS0/Chhy/Y0CQHg8oYwErZsz9OIwbaGt05fn5AiPwxtnVJsFRsbC8Bjjz3Ghg0baN++fZ637Uhm3in8A4gRkSMASqn5QA8gdVLoAYxJ/vsX4CullBIH1c5I8uQi23bs4vc50zhyIALfxk35/ecFeHl5OSIETXO6GwejbS+Pisl1UkgZ3nrfvn0opbh37x5KKfr06WNoaGs/Pz+GDx9OUFAQv/32W5qhrVOGnU89tPW5c+cyLb9PGdo6JCSEHj168Nprr/H8889TunRp6zrZ3SmkDG1duXLlTIe2XrhwIa1atbK2SHziiSfYunWr2yUFM+sUqgInUz2OTV5mcx0RSQQSgPImxpTG4VNnWbs+lCmj3ubU0Rh6vNqf5wePgGK6malWcJSoZ3tM/hJ1c39h9Msvv9CnTx+OHz/OsWPHOHnyJLVq1eLixYucPn2aAwcOAHD8+HH27NlD48aN07y+SJEiBAcHs3XrVg4cOICXlxdNmjQhODjYuk5wcDBNmzbFy8uL3r17ExYWxtKlS63Pr1ixgoiICIoVK8arr77Khg0bmDNnDocPH6ZJkya88sor1nU3btzI7t27M/xL6YP01FNP8d133wHw3Xff0aNHjwzHXKNGDUJDQ0lMTOTu3buEhoa6ZfGRmUnB1j1d+jsAI+uglOqnlApXSoVn1Zogp+LiL1P54Vo0a9+ZwcFf0KJDVzw8PPI0uYimuZsyrVtQtn2bNMvKtm9DmdYtcr3NefPm8cwzz6RZ9uyzzzJ//nx+/PFHXn/9dRo3bkzPnj2ZNWsWZcqUybCN4sWLM3ToUCZNmgQkTcpz6NAhvLy88PT05NChQ4SEhFjXXbJkCVOmTMHb2xsfHx/mzJmT4Yre29ub8ePHExUVRc+ePQ0fz/Dhw/nzzz/x9vbmzz//ZPjw4QCEh4dbW0D17NkTT09PGjZsiJ+fH35+fmkqvt2FaUNnK6VaA2NE5PHkxyMARGRcqnVWJq+zRSlVGIgDHsyq+MieQ2fHxMaxYHVYhuXPd2mjR4vU3FqOh862WEjYsp0bUTGUqOtFmdYt8lTJrDmXqw6dvR3wVkrVAk4BLwC9062zGHgV2AL0BNY6qj4BwLNqRbyrV84wYblnVfdrRqZpeaE8PCjbtqVdp6LV3JNpSUFEEpVSg4CVJDVJnS0i+5VSH5I0A9BiIAT4QSkVA1wkKXE4jFKK5zq10pOLaJqmJTO1n4KILAOWpVs2OtXft4DnzIwhO3pyEU3TtL/pQkNNy6fccdwdLe/y+rnrpKBp+VCxYsWIj4/XiaGAERHi4+PzND6bnnlN0/KhatWqERsbm+WAcFr+VKxYMapVq5br1+ukoGn5UJEiRahVq5azw9DckC4+0jRN06x0UtA0TdOsdFLQNE3TrEwb5sIsSqnzwHETNl0BuGDCdl2RPtb8SR9r/mSvY31YRB7MbiW3SwpmUUqFGxkXJD/Qx5o/6WPNnxx9rLr4SNM0TbPSSUHTNE2z0knhbzOcHYAD6WPNn/Sx5k8OPVZdp6BpmqZZ6TsFTdM0zapAJQWlVDelVJRSKkYpNdzG8/cppRYkP79NKVXT8VHah4FjfVspFamU2quUWqOUetgZcdpDdseaar2eSilRSrl1qxUjx6uU6pX8+e5XSs11dIz2YuB7XEMptU4ptSv5u/xPZ8SZV0qp2Uqpc0qpfZk8r5RSXya/D3uVUk1NC0ZECsQ/kib6OQzUBooCewCfdOu8AUxL/vsFYIGz4zbxWDsCJZL/HpifjzV5vVLABmAr0NzZcZv82XoDu4ByyY8fcnbcJh7rDGBg8t8+wDFnx53LY20PNAX2ZfL8P4HlJM1r3wrYZlYsBelO4R9AjIgcEZE7wHygR7p1egDfJf/9C9BZuec0bNkeq4isE5EbyQ+3ArkfVtG5jHyuAB8BE4BbjgzOBEaOty8wVUQuAYjIOQfHaC9GjlWA0sl/lwFOOzA+uxGRDSTNPpmZHsD3kmQrUFYpVdmMWApSUqgKnEz1ODZ5mc11RCQRSADKOyQ6+zJyrKkFkHQV4o6yPValVBOguogscWRgJjHy2dYB6iilNiultiqlujksOvsycqxjgJeVUrEkzfI42DGhOVxOf9O5VpCGzrZ1xZ++6ZWRddyB4eNQSr0MNAceNTUi82R5rEopD+Bz4DVHBWQyI59tYZKKkDqQdAe4USnVQEQumxybvRk51heBOSLymVKqNUlzvjcQEYv54TmUw85NBelOIRaonupxNTLealrXUUoVJul2NKtbOldl5FhRSnUBRgJPichtB8Vmb9kdaymgAbBeKXWMpPLYxW5c2Wz0e7xIRO6KyFEgiqQk4W6MHGsA8BOAiGwBipE0VlB+Y+g3bQ8FKSlsB7yVUrWUUkVJqkhenG6dxcCryX/3BNZKci2Pm8n2WJOLVKaTlBDctcwZsjlWEUkQkQoiUlNEapJUf/KUiIQ7J9w8M/I9/p2khgQopSqQVJx0xKFR2oeRYz0BdAZQStUnKSnkx+nmFgN9klshtQISROSMGTsqMMVHIpKolBoErCSpVcNsEdmvlPoQCBeRxUAISbefMSTdIbzgvIhzz+CxTgTuB35Orks/ISJPOS3oXDJ4rPmGweNdCXRVSkUC94B3RSTeeVHnjsFjHQrMVEr9H0nFKa+544WcUmoeScV9FZLrRz4AigCIyDSS6kv+CcQAN4DXTYvFDd8/TdM0zSQFqfhI0zRNy4ZOCpqmaZqVTgqapmmalU4KmqZpmpVOCpqmaZqVTgqaS1BK1cxshEhHU0o9rZQanYfXH0vuH2B0/deUUl/ldn95kRKrUqqoUmpDcqdNrQDTSUHTMvr/9s4uxKoqiuO//8AoVNPXMEgUUSYWmajFDJU9mBQRGIVWktLXQ9SDWEEJPYQGFb2IWVMU9SBUiOCMWviQOaSTIM7QlH1gvdgHiWRBVHMJrWn1sNe5bO+cGS4NdO/c1g8Od5911v66l3vW3vue+9/rgFcb3Yj/EhecGwBWNrotQWOJoBA0HZJmuz5+t88gPpI04sf17rPER7Y7fN+A11znCEmjkja6/4CkLrc/JGlY0mFJfZLOKKl7LnDSzH728y2S7syuj2b175O0XdJXkt6pUdR9UtKQH3M8z21K+3R8ImmvpFkl9Zf6SNqgpLm/T9JRSWuzPPe5xv5hSW+5rcv7OOzHYrd3Strj5b/O6Zo6O4HV/+IjC1qICApBUyHpcqAPeNDMhoETwM1mdjVpFPtS5t5D+kfrfOAyYLnbzwRGPM9+0r9DAfrNrNvMFgBHSLo5tSwGRups7iLgMZKO/2zPW/CbmfUAvcCLbjsAXGtmi0gy0OtKypzM5wrgFu/3ekntkuaR9KuWer8edd/NwCYz6wZWAG+6fT1wwMt/F7g4K/8LoLvOvgctSqwfBs1EF7ALWGFmX7qtHeiVtJAk2TA38x8ys6NQlQm4gbQPxt/ANvd5G+j39FWSngXOJUl8vF/ShguoXztnyMx+8Po/BS4h3dQBtmavmzx9EbBNSQd/BvBNSZmT+ex24cKTkk4As4ClwPZiZmNmhYDjTcCV2eTlbEkdpM1clrvvbkm/FA5mNibplKQOM/u9zvcgaDFiphA0E7+SNOPzEffjwI/AApLE94zsWq1Gy0SaLYV9C7DGzOYDz5DE02r5o8b+F/498eWhvP5cWXaM0wdZVpJ+Gej1+h+eoP7JfMrqE+X9bgOuM7OFflyY3egn07aZyfTfiCiYAhEUgmbiFHAHSQ1yldvOAY67Pv69JGG0gh5X0GwjLS0Vo/Q2ksotwKrM3gEcl9TOxGvnR4A52fm3wDWevh0XKauDldnrwawvxzx9/7gc9fvkDAB3S+oEkHS+2/cAawonn2lB2pJ0tdtuBc7LfDqBn8zszzrqDVqUWD4Kmgozq0haBnwgqUJ6CqhP0l3Ah0Alcz8IvED6TWEQ2OH2CjBP0sek2Udxg34aOAR8B3xOChK1DAIbJcnVNt8AdkkaIt2AKyV5ypgp6RApQN3jtg0kVdpjJAnvS0vy1eNTxVVDnwP2Sxoj7c38ALAWeEXSZ6Tv+SDwCGmGtFXSCOn3lu+z4m4kqXEG/2NCJTWYlkhaAjxhZstKro2a2VlTKHsz8J6Z7Z1CE6cdkvqBp8zs60a3JWgcsXwUBON5Hhj3uGoro7SJzc4ICEHMFIIgCIIqMVMIgiAIqkRQCIIgCKpEUAiCIAiqRFAIgiAIqkRQCIIgCKpEUAiCIAiq/AOf9CnQfyenRwAAAABJRU5ErkJggg==\" />\n",
    "\n",
    "The full details, along with the code to run the experiment, are in the [jupyter notebook](https://github.com/greglandrum/rdkit_blog/blob/master/notebooks/Working%20with%20unbalanced%20data%20part%201.ipynb) associated with this post, view it [here](https://nbviewer.jupyter.org/github/greglandrum/rdkit_blog/blob/master/notebooks/Working%20with%20unbalanced%20data%20part%201.ipynb). Since a couple of the datasets are a bit large, I haven't checked them all into github with this jupyter notebook, there are links above to download the data. I'm also happy to provide the KNIME workflows that I used to construct the datasets.\n",
    "\n",
    "I plan a couple more blog posts in this series. One will show how to adjust the decision threshold when making predictions within KNIME, one will look at using machine learning methods other than random forests, and I'd also like to include a comparison to the results obtained using standard resampling methods. \n",
    "\n",
    "I think this approach should be a standard part of our toolbox when building predictive models for unbalanced datasets. What do you think?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "from rdkit import Chem, DataStructs\n",
    "from rdkit.Chem import Draw\n",
    "from rdkit.Chem import rdFingerprintGenerator\n",
    "from rdkit.Chem.Draw import IPythonConsole\n",
    "from collections import defaultdict\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import gzip\n",
    "import pickle\n",
    "import matplotlib.pyplot as plt\n",
    "%pylab inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the function to build and evaluate a random forest for an assay. This won't build the best possible model for the assay - we aren't doing any parameter optimization and are always using the same fingerprints - but that's beside the point for this analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "import random\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn import metrics\n",
    "\n",
    "def run_it2(acts,inacts):\n",
    "    # generate fingerprints for the actives and inactives and convert them to numpy arrays so that\n",
    "    # we can learn from them. This uses the generalized fingerprinter added in the RDKit 2018.09 release\n",
    "    act_data = []\n",
    "    for i,fp in enumerate(rdFingerprintGenerator.GetFPs([Chem.MolFromSmiles(x) for x in acts['canonical_smiles']],fpType=rdFingerprintGenerator.MorganFP)):\n",
    "        bv = np.zeros((len(fp),),np.int16)\n",
    "        DataStructs.ConvertToNumpyArray(fp,bv)\n",
    "        act_data.append((acts['compound_chembl_id'].iloc[i],bv,1))\n",
    "    inact_data = []\n",
    "    for i,fp in enumerate(rdFingerprintGenerator.GetFPs([Chem.MolFromSmiles(x) for x in inacts['canonical_smiles']],fpType=rdFingerprintGenerator.MorganFP)):\n",
    "        bv = np.zeros((len(fp),),np.int16)\n",
    "        DataStructs.ConvertToNumpyArray(fp,bv)\n",
    "        inact_data.append((inacts['compound_chembl_id'].iloc[i],bv,0))\n",
    "\n",
    "    # do a random, stratified 80/20 training/holdout split:\n",
    "    random.seed(0xf00d)\n",
    "    nAct = len(act_data)\n",
    "    actis = list(range(nAct))\n",
    "    nInact = len(inact_data)\n",
    "    inactis = list(range(nInact))\n",
    "    random.shuffle(actis)\n",
    "    random.shuffle(inactis)\n",
    "\n",
    "    train_pct = 0.80\n",
    "    train = [act_data[x] for x in actis[:int(train_pct*nAct)]] + [inact_data[x] for x in inactis[:int(train_pct*nInact)]]\n",
    "    test = [act_data[x] for x in actis[int(train_pct*nAct):]] + [inact_data[x] for x in inactis[int(train_pct*nInact):]]\n",
    "\n",
    "    random.shuffle(train)\n",
    "    random.shuffle(test)\n",
    "\n",
    "    # build the random forest:\n",
    "    cls = RandomForestClassifier(n_estimators=500,max_depth=15,min_samples_leaf=2,n_jobs=4,oob_score=True)\n",
    "    cls.fit([y for x,y,z in train],[z for x,y,z in train])\n",
    "    oob_auroc = metrics.roc_auc_score([z for x,y,z in train],[x[1] for x in cls.oob_decision_function_])\n",
    "\n",
    "    # generate and show some evaluation stats for the model on the holdout data:\n",
    "    scores = cls.predict([y for x,y,z in test])\n",
    "    probs = cls.predict_proba([y for x,y,z in test])[:,1]\n",
    "    auroc = metrics.roc_auc_score([z for x,y,z in test], probs)\n",
    "    print('ratio: %.3f kappa: %.3f, AUC: %.3f, OOB_AUC: %.3f'%(nAct/nInact,metrics.cohen_kappa_score([z for x,y,z in test],scores),auroc,oob_auroc))\n",
    "    print(metrics.confusion_matrix([z for x,y,z in test],scores))\n",
    "    print(metrics.classification_report([z for x,y,z in test],scores))\n",
    "    \n",
    "    # to allow further analysis we return a tuple with:\n",
    "    #  - predicted probabilities of being active (class 1) for the holdout set\n",
    "    #  - actual values for the holdout set\n",
    "    #  - (actual values for the training set, OOB predicted probabilities for the training set)\n",
    "    return probs,[z for x,y,z in test],([z for x,y,z in train],cls.oob_decision_function_)\n",
    "\n",
    "def run_it_(assay_id,acts,inacts,thresholds,archive):\n",
    "    print(\"--------- Default -----------\")\n",
    "    ps,tgt,oob_tpl = run_it2(acts,inacts)\n",
    "    oob_tgt,oob_probs = oob_tpl\n",
    "    oob_probs = [x[1] for x in oob_probs]\n",
    "    auroc = metrics.roc_auc_score(oob_tgt, oob_probs)\n",
    "    scores = [1 if x>=0.5 else 0 for x in ps]\n",
    "    kappa = metrics.cohen_kappa_score(tgt,scores)\n",
    "    confusion = metrics.confusion_matrix(tgt,scores)\n",
    "    archive[assay_id].append((0.5,kappa,confusion,auroc))\n",
    "    print(\"--------- Balanced -----------\")\n",
    "    if thresholds is None:\n",
    "        # pick the best threshold using the OOB kappas\n",
    "        threshs = np.arange(0.05,0.55,0.05)\n",
    "        tscores = []\n",
    "        for thresh in threshs:\n",
    "            scores = [1 if x>=thresh else 0 for x in oob_probs]\n",
    "            kappa = metrics.cohen_kappa_score(oob_tgt,scores)\n",
    "            tscores.append((np.round(kappa,3),thresh))\n",
    "        tscores.sort(reverse=True)\n",
    "        thresh = tscores[0][-1]\n",
    "        thresholds = [thresh]\n",
    "    for thresh in thresholds:\n",
    "        scores = [1 if x>=thresh else 0 for x in ps]\n",
    "        kappa = metrics.cohen_kappa_score(tgt,scores)\n",
    "        confusion = metrics.confusion_matrix(tgt,scores)\n",
    "        archive[assay_id].append((thresh,kappa,confusion,None))\n",
    "        print('thresh: %.3f kappa: %.3f'%(thresh,kappa))\n",
    "        print(confusion)        \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The serotonin Ki datasets\n",
    "This is a set of data about binding to serotonin receptors exported from ChEMBL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>standard_relation</th>\n",
       "      <th>standard_value</th>\n",
       "      <th>standard_type</th>\n",
       "      <th>pchembl_value</th>\n",
       "      <th>assay_chembl_id</th>\n",
       "      <th>target_chembl_id</th>\n",
       "      <th>canonical_smiles</th>\n",
       "      <th>compound_chembl_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&gt;</td>\n",
       "      <td>10000.0</td>\n",
       "      <td>Ki</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CHEMBL615807</td>\n",
       "      <td>CHEMBL214</td>\n",
       "      <td>CCN(C)C1CCCc2ccncc12.OC(=O)\\C=C/C(=O)O.OC(=O)\\...</td>\n",
       "      <td>CHEMBL1794855</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>=</td>\n",
       "      <td>168.0</td>\n",
       "      <td>Ki</td>\n",
       "      <td>6.78</td>\n",
       "      <td>CHEMBL615460</td>\n",
       "      <td>CHEMBL214</td>\n",
       "      <td>CCCN(CCC)[C@@H]1CCc2c(OC)cccc2[C@@H]1C</td>\n",
       "      <td>CHEMBL278751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>=</td>\n",
       "      <td>181.0</td>\n",
       "      <td>Ki</td>\n",
       "      <td>6.74</td>\n",
       "      <td>CHEMBL615809</td>\n",
       "      <td>CHEMBL214</td>\n",
       "      <td>C(N1CCN(CC1)c2ncccn2)c3c[nH]c(n3)c4ccccc4</td>\n",
       "      <td>CHEMBL103772</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>=</td>\n",
       "      <td>3.9</td>\n",
       "      <td>Ki</td>\n",
       "      <td>8.41</td>\n",
       "      <td>CHEMBL615460</td>\n",
       "      <td>CHEMBL214</td>\n",
       "      <td>CCCN1CC[C@@H]2[C@H]1CCc3ccc4ccoc4c23</td>\n",
       "      <td>CHEMBL328107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>=</td>\n",
       "      <td>1.6</td>\n",
       "      <td>Ki</td>\n",
       "      <td>8.80</td>\n",
       "      <td>CHEMBL615756</td>\n",
       "      <td>CHEMBL214</td>\n",
       "      <td>COc1cccc2OC[C@H](CN3[C@@H]4CC[C@H]3C[C@@](O)(C...</td>\n",
       "      <td>CHEMBL148860</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  standard_relation  standard_value standard_type  pchembl_value  \\\n",
       "0                 >         10000.0            Ki            NaN   \n",
       "1                 =           168.0            Ki           6.78   \n",
       "2                 =           181.0            Ki           6.74   \n",
       "3                 =             3.9            Ki           8.41   \n",
       "4                 =             1.6            Ki           8.80   \n",
       "\n",
       "  assay_chembl_id target_chembl_id  \\\n",
       "0    CHEMBL615807        CHEMBL214   \n",
       "1    CHEMBL615460        CHEMBL214   \n",
       "2    CHEMBL615809        CHEMBL214   \n",
       "3    CHEMBL615460        CHEMBL214   \n",
       "4    CHEMBL615756        CHEMBL214   \n",
       "\n",
       "                                    canonical_smiles compound_chembl_id  \n",
       "0  CCN(C)C1CCCc2ccncc12.OC(=O)\\C=C/C(=O)O.OC(=O)\\...      CHEMBL1794855  \n",
       "1             CCCN(CCC)[C@@H]1CCc2c(OC)cccc2[C@@H]1C       CHEMBL278751  \n",
       "2          C(N1CCN(CC1)c2ncccn2)c3c[nH]c(n3)c4ccccc4       CHEMBL103772  \n",
       "3               CCCN1CC[C@@H]2[C@H]1CCc3ccc4ccoc4c23       CHEMBL328107  \n",
       "4  COc1cccc2OC[C@H](CN3[C@@H]4CC[C@H]3C[C@@](O)(C...       CHEMBL148860  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with gzip.open('../data/chembl_5HT.csv.gz') as inf:\n",
    "    serotonin_d = pd.read_csv(inf)\n",
    "serotonin_d.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get some info about those assays from the ChEMBL REST services:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CHEMBL3371 3180 Homo sapiens Serotonin 6 (5-HT6) receptor\n",
      "CHEMBL224 3068 Homo sapiens Serotonin 2a (5-HT2a) receptor\n",
      "CHEMBL214 2672 Homo sapiens Serotonin 1a (5-HT1a) receptor\n",
      "CHEMBL3155 1636 Homo sapiens Serotonin 7 (5-HT7) receptor\n",
      "CHEMBL225 1580 Homo sapiens Serotonin 2c (5-HT2c) receptor\n",
      "CHEMBL1833 922 Homo sapiens Serotonin 2b (5-HT2b) receptor\n",
      "CHEMBL1898 542 Homo sapiens Serotonin 1b (5-HT1b) receptor\n",
      "CHEMBL1983 525 Homo sapiens Serotonin 1d (5-HT1d) receptor\n",
      "CHEMBL1899 491 Homo sapiens Serotonin 3a (5-HT3a) receptor\n",
      "CHEMBL1875 369 Homo sapiens Serotonin 4 (5-HT4) receptor\n",
      "CHEMBL3426 227 Homo sapiens Serotonin 5a (5-HT5a) receptor\n",
      "CHEMBL2182 86 Homo sapiens Serotonin 1e (5-HT1e) receptor\n",
      "CHEMBL1805 36 Homo sapiens Serotonin 1f (5-HT1f) receptor\n"
     ]
    }
   ],
   "source": [
    "import requests,json\n",
    "assay_lookup = {}\n",
    "tpls = sorted([(len(v),k) for k,v in serotonin_d.groupby('target_chembl_id').groups.items()],reverse=True)\n",
    "for v,k in tpls:\n",
    "    resp=requests.get(f'https://www.ebi.ac.uk/chembl/api/data/target/{k}?format=json')\n",
    "    info = json.loads(resp.text)\n",
    "    assay_lookup[k] = info['organism'],info['pref_name']\n",
    "    print(k,v,info['organism'],info['pref_name'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../data/serotonin_data.pkl','wb+') as outf:\n",
    "    pickle.dump((serotonin_d,assay_lookup),outf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../data/serotonin_data.pkl','rb') as inf:\n",
    "    serotonin_d,assay_lookup = pickle.load(inf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the distributions of activities for some of the assays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFjVJREFUeJzt3X+w3XV95/HnS1MqEiX80LuURKM19UfJiHAH6bplbsSxgN0GOzLFogbEpn9g1660Y3R2xnHGbnFa1upsx25WtLFVImItLOAPGr2ttoVKKho0ugQaIQSDFohGaG123/vH+Wa9hpvcc+89N+fcT56PmTvn+/18P+f7fX9y7n2dbz7ne85JVSFJateThl2AJGlhGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6DXykkwmedMROlYled4htl2a5EsLeOwF3b+OXga9JDXOoJekxhn0OqKS7Ezy9iTfSPJIkg8neUq3bW2SO5N8P8k9Sc6bctdnJ/nbJD9I8rkkJ0/Z59lJ/i7Jo0m+mmRiyrbJJO/utu9L8r+SnJTko91xvpxk5UFlXpDk3iTfS/IHSfr+O0nyJ0n+8KC2G5K8tVve0I3tB92/wasPsZ+V3TTSkoPG8qYp629Msr37d/xskmf3W6eOLga9huES4JeAnwV+DvgvSc4CPgL8LrAMOAfYOeU+vw5cBjwTOAb4HYAkpwI3A+8GTuzaP5nkGVPuezHweuDU7ph/D3y4678deOdB9b0aGAfOANYCb5zF2D4G/FqSdPWdALwS2Nxtvwf4ReB44F3Anyc5ZRb7p9vvhcA7gF8FngF8Ebh2tvvR0cGg1zD896q6v6oeBn4PeC1wOfChqrq1qv5vVT1QVd+ccp8PV9X/rqrHgeuA07v21wG3VNUt3f1uBe4ALjjovvdU1V7g08A9VfVXVbUf+ATwkoPqe09VPVxV9wF/1NXXry8CRS/MAV4D/H1V7Qaoqk9U1e6u1o8DdwNnzWL/B/wm8PtVtb0bx38FTvesXtMx6DUM909Z/jbwM8AKeme7h/KdKcuPAUu75WcDF3XTNo8meRT4D8DUs+Q9U5Yfn2Z9KT9puvr6Ur1PCdzMj58cfh346IHtSd7QTU8dqPU04OQn7mlGzwbeN2U/DwOh978W6ScY9BqGFVOWnwXspheuPzuHfd0P/FlVLZvyc1xVXTXg+mbjWuA13dn1S4FPAnTr/xN4M3BSVS0D7qIX0Af7YXf71Clt/27K8v3Abx407mOr6u9mWauOAga9huGKJMuTnEhvnvnjwDXAZUnOTfKkJKcmeUEf+/pz4D8m+aUkT07ylCQTSZbPo77fTXJCkhXAW7r6+lZVXwG+C3wQ+GxVPdptOo7etM53AZJcRu+Mfrp9fBd4AHhdN6438pNPhH8CvD3Jz3f7Oj7JRbOpU0cPg17D8DHgc8C93c+7q+of6L3Y+l5gL/DX9KYnDquq7qf3guk76AXo/fRe0J3P7/YNwFbgTnov9F4zh31cC7yC3lgP1PoN4Gp6LwbvAVYDf3uYffwGvbH8M/DzwP8/W6+qTwHvATYn+T69/xmcP4c6dRSIXzyiIynJTuBNVfVXw65FOlp4Ri9JjVsycxdJUyX5RXqXaT5BVR18BY80dE7dSFLjnLqRpMaNxNTNySefXCtXrhx2GdP64Q9/yHHHHTfsMgbG8Yw2xzPaRm08W7du/V5VPWOmfiMR9CtXruSOO+4YdhnTmpycZGJiYthlDIzjGW2OZ7SN2niSfLuffk7dSFLjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS40binbGSnmjlhpuHctydV71qKMfVwvGMXpIa11fQJ1mW5Pok30yyPckvJDkxya1J7u5uT+j6Jsn7k+xI8rUkZyzsECRJh9PvGf37gM9U1QuAFwPbgQ3AlqpaBWzp1qH3vZWrup/1wAcGWrEkaVZmDPokTwfOofuC5Kr6Ufet9muBTV23TcCF3fJa4CPVcxuwLMkpA69cktSXGb9hKsnpwEbgG/TO5rcCbwEeqKplU/o9UlUnJLkJuKqqvtS1bwHeVlV3HLTf9fTO+BkbGztz8+bNgxvVAO3bt4+lS9v5djjHM9qmjmfbA3uHUsPqU48f2L5afnxGwZo1a7ZW1fhM/fq56mYJcAbwW1V1e5L38eNpmulkmrYnPJtU1UZ6TyCMj4/XKH3G81Sj9vnT8+V4RtvU8Vw6rKtuLpkY2L5afnwWk36Cfhewq6pu79avpxf0e5KcUlUPdlMzD03pv2LK/ZcDuwdVsHQkHelLHK9cvX9oAa92zThHX1XfAe5P8vyu6Vx60zg3Auu6tnXADd3yjcAbuqtvzgb2VtWDgy1bktSvft8w9VvAR5McA9wLXEbvSeK6JJcD9wEXdX1vAS4AdgCPdX0lSUPSV9BX1Z3AdBP+507Tt4Ar5lmXJGlAfGesJDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDWur6BPsjPJtiR3Jrmjazsxya1J7u5uT+jak+T9SXYk+VqSMxZyAJKkw5vNGf2aqjq9qsa79Q3AlqpaBWzp1gHOB1Z1P+uBDwyqWEnS7M1n6mYtsKlb3gRcOKX9I9VzG7AsySnzOI4kaR5SVTN3Sv4JeAQo4H9U1cYkj1bVsil9HqmqE5LcBFxVVV/q2rcAb6uqOw7a53p6Z/yMjY2duXnz5oENapD27dvH0qVLh13GwDie2dn2wN4F2/d0xo6FPY8f0UM+wepTjx/Yvvx9W1hr1qzZOmWW5ZCW9Lm/l1XV7iTPBG5N8s3D9M00bU94NqmqjcBGgPHx8ZqYmOizlCNrcnKSUa1tLhzP7Fy64eYF2/d0rly9n6u39ftnuTB2XjIxsH35+zYa+vqNqqrd3e1DST4FnAXsSXJKVT3YTc081HXfBayYcvflwO4B1ixpAa0c4JPblav3z+rJcudVrxrYsfVjM87RJzkuydMOLAOvBO4CbgTWdd3WATd0yzcCb+iuvjkb2FtVDw68cklSX/o5ox8DPpXkQP+PVdVnknwZuC7J5cB9wEVd/1uAC4AdwGPAZQOvWpLUtxmDvqruBV48Tfs/A+dO017AFQOpTpI0b74zVpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDVuuO+1lvp0qHdrzvadl9LRyDN6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJalzfQZ/kyUm+kuSmbv05SW5PcneSjyc5pmv/6W59R7d95cKULknqx2zO6N8CbJ+y/h7gvVW1CngEuLxrvxx4pKqeB7y36ydJGpK+gj7JcuBVwAe79QAvB67vumwCLuyW13brdNvP7fpLkoYgVTVzp+R64PeBpwG/A1wK3NadtZNkBfDpqjotyV3AeVW1q9t2D/DSqvreQftcD6wHGBsbO3Pz5s0DG9Qg7du3j6VLlw67jIFZrOPZ9sDeadvHjoU9jx/hYhbQ0T6e1acev3DFDMCo/f2sWbNma1WNz9RvyUwdkvwy8FBVbU0ycaB5mq7Vx7YfN1RtBDYCjI+P18TExMFdRsLk5CSjWttcLNbxXLrh5mnbr1y9n6u3zfhrvGgc7ePZecnEwhUzAIv176efR+BlwK8kuQB4CvB04I+AZUmWVNV+YDmwu+u/C1gB7EqyBDgeeHjglUuS+jLjHH1Vvb2qllfVSuBi4PNVdQnwBeA1Xbd1wA3d8o3dOt32z1c/80OSpAUxn+vo3wa8NckO4CTgmq79GuCkrv2twIb5lShJmo9ZTQZW1SQw2S3fC5w1TZ9/AS4aQG2SpAHwnbGS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1LgZgz7JU5L8Q5KvJvl6knd17c9JcnuSu5N8PMkxXftPd+s7uu0rF3YIkqTD6eeM/l+Bl1fVi4HTgfOSnA28B3hvVa0CHgEu7/pfDjxSVc8D3tv1kyQNyYxBXz37utWf6n4KeDlwfde+CbiwW17brdNtPzdJBlaxJGlWUlUzd0qeDGwFngf8MfAHwG3dWTtJVgCfrqrTktwFnFdVu7pt9wAvrarvHbTP9cB6gLGxsTM3b948uFEN0L59+1i6dOmwyxiYxTqebQ/snbZ97FjY8/gRLmYBHe3jWX3q8QtXzACM2t/PmjVrtlbV+Ez9lvSzs6r6P8DpSZYBnwJeOF237na6s/cnPJtU1UZgI8D4+HhNTEz0U8oRNzk5yajWNheLdTyXbrh52vYrV+/n6m19/RovCkf7eHZeMrFwxQzAYv37mdVVN1X1KDAJnA0sS3LgEVwO7O6WdwErALrtxwMPD6JYSdLs9XPVzTO6M3mSHAu8AtgOfAF4TddtHXBDt3xjt063/fPVz/yQJGlB9PN/qlOATd08/ZOA66rqpiTfADYneTfwFeCarv81wJ8l2UHvTP7iBahbktSnGYO+qr4GvGSa9nuBs6Zp/xfgooFUJ0maN98ZK0mNM+glqXHtXMelI2LlIS5zlDS6PKOXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1Dg/AkHSyBjWR2zsvOpVQznukeIZvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGzRj0SVYk+UKS7Um+nuQtXfuJSW5Ncnd3e0LXniTvT7IjydeSnLHQg5AkHVo/Z/T7gSur6oXA2cAVSV4EbAC2VNUqYEu3DnA+sKr7WQ98YOBVS5L6NmPQV9WDVfWP3fIPgO3AqcBaYFPXbRNwYbe8FvhI9dwGLEtyysArlyT1ZVZz9ElWAi8BbgfGqupB6D0ZAM/sup0K3D/lbru6NknSEPT9McVJlgKfBH67qr6f5JBdp2mrafa3nt7UDmNjY0xOTvZbyhG1b9++ka1tLuY7nitX7x9cMQMwduzo1TQfjmc4+v2bWKx50FfQJ/kpeiH/0ar6i655T5JTqurBbmrmoa59F7Biyt2XA7sP3mdVbQQ2AoyPj9fExMTcRrDAJicnGdXa5mK+47l0SJ8XfihXrt7P1dva+VoFxzMcOy+Z6KvfYs2Dfq66CXANsL2q/tuUTTcC67rldcANU9rf0F19czaw98AUjyTpyOvnqfZlwOuBbUnu7NreAVwFXJfkcuA+4KJu2y3ABcAO4DHgsoFWLEmalRmDvqq+xPTz7gDnTtO/gCvmWZcOYz5ft3bl6v0jN/0iaWH5zlhJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJatyMQZ/kQ0keSnLXlLYTk9ya5O7u9oSuPUnen2RHkq8lOWMhi5ckzayfM/o/Bc47qG0DsKWqVgFbunWA84FV3c964AODKVOSNFczBn1V/Q3w8EHNa4FN3fIm4MIp7R+pntuAZUlOGVSxkqTZS1XN3ClZCdxUVad1649W1bIp2x+pqhOS3ARcVVVf6tq3AG+rqjum2ed6emf9jI2Nnbl58+YBDGfw9u3bx9KlS4ddxk/Y9sDeOd937FjY8/gAixkyxzPaFst4Vp96fF/9Ri0P1qxZs7Wqxmfqt2TAx800bdM+k1TVRmAjwPj4eE1MTAy4lMGYnJxk1Gq7dMPNc77vlav3c/W2QT/sw+N4RttiGc/OSyb66jeKedCPuV51s+fAlEx3+1DXvgtYMaXfcmD33MuTJM3XXJ9qbwTWAVd1tzdMaX9zks3AS4G9VfXgvKscUSvncWYtSUfKjEGf5FpgAjg5yS7gnfQC/roklwP3ARd13W8BLgB2AI8Bly1AzZKkWZgx6KvqtYfYdO40fQu4Yr5FSZIGx3fGSlLjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxo/+NADNY6I8KvnL1/nl90YckDZtn9JLUOINekhq36KduJGm++p0CXoip3J1XvWqg+5uOZ/SS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktS4BQn6JOcl+VaSHUk2LMQxJEn9GXjQJ3ky8MfA+cCLgNcmedGgjyNJ6s9CnNGfBeyoqnur6kfAZmDtAhxHktSHVNVgd5i8Bjivqt7Urb8eeGlVvfmgfuuB9d3q84FvDbSQwTkZ+N6wixggxzPaHM9oG7XxPLuqnjFTp4X49MpM0/aEZ5Oq2ghsXIDjD1SSO6pqfNh1DIrjGW2OZ7Qt1vEsxNTNLmDFlPXlwO4FOI4kqQ8LEfRfBlYleU6SY4CLgRsX4DiSpD4MfOqmqvYneTPwWeDJwIeq6uuDPs4RNPLTS7PkeEab4xlti3I8A38xVpI0WnxnrCQ1zqCXpMYZ9IeRZGeSbUnuTHLHsOuZryTLklyf5JtJtif5hWHXNFdJnt89Lgd+vp/kt4dd13wk+c9Jvp7kriTXJnnKsGuaqyRv6cbx9cX6uCT5UJKHktw1pe3EJLcmubu7PWGYNfbLoJ/Zmqo6fTFeOzuN9wGfqaoXAC8Gtg+5njmrqm91j8vpwJnAY8CnhlzWnCU5FfhPwHhVnUbvQoaLh1vV3CQ5DfgNeu+SfzHwy0lWDbeqOflT4LyD2jYAW6pqFbClWx95Bv1RIsnTgXOAawCq6kdV9ehwqxqYc4F7qurbwy5knpYAxyZZAjyVxfv+kxcCt1XVY1W1H/hr4NVDrmnWqupvgIcPal4LbOqWNwEXHtGi5sigP7wCPpdka/eRDYvZc4HvAh9O8pUkH0xy3LCLGpCLgWuHXcR8VNUDwB8C9wEPAnur6nPDrWrO7gLOSXJSkqcCF/CTb6JczMaq6kGA7vaZQ66nLwb94b2sqs6g90mcVyQ5Z9gFzcMS4AzgA1X1EuCHLJL/dh5O96a8XwE+Mexa5qOb610LPAf4GeC4JK8bblVzU1XbgfcAtwKfAb4K7B9qUUc5g/4wqmp3d/sQvfnfs4Zb0bzsAnZV1e3d+vX0gn+xOx/4x6raM+xC5ukVwD9V1Xer6t+AvwD+/ZBrmrOquqaqzqiqc+hNf9w97JoGZE+SUwC624eGXE9fDPpDSHJckqcdWAZeSe+/pItSVX0HuD/J87umc4FvDLGkQXkti3zapnMfcHaSpyYJvcdn0b5YnuSZ3e2zgF+ljccIeh/nsq5bXgfcMMRa+uY7Yw8hyXP58VUcS4CPVdXvDbGkeUtyOvBB4BjgXuCyqnpkuFXNXTf/ez/w3KraO+x65ivJu4BfozfN8RXgTVX1r8Otam6SfBE4Cfg34K1VtWXIJc1akmuBCXofTbwHeCfwl8B1wLPoPTlfVFUHv2A7cgx6SWqcUzeS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXu/wHZRqMlL7yA5wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFGxJREFUeJzt3X+w3XV95/Hnq2QpP1IJv8zaJCVuZbWWjBTuIK1bJhG38sM12IEpFhUQG/9AS0vaNTo7w+yMu4VZKdp2x07WqLgiUUEHVqhKkdTaVlqitAGiJdAUEjBggWiATpv1vX+cb9bLzU3uyb335Nzz4fmYuXO+3+/5fL/f17lz87rf+zk/kqpCktSunxh2AEnSYFn0ktQ4i16SGmfRS1LjLHpJapxFL0mNs+g15yXZkORdB+lcleQV+7jvkiTfGOC5B3p8vXhZ9JLUOItekhpn0eugSrI1yfuTPJDk6SSfSHJYd9/KJPcm+UGSh5KcNW7XE5L8RZIfJvlqkuPGHfP0JH+Z5Jkkf5tk+bj7NiT5YHf/riT/J8mxSW7ozvM3SZZOiHlOkoeTfD/J/0jS97+TJH+c5EMTtt2S5MpueU332H7YfQ/eso/jLO2mkeZNeCzvGrf+ziSbu+/jV5Kc0G9OvbhY9BqGi4A3Aj8L/HvgvyQ5DfgU8LvAAuAMYOu4fX4duBR4KXAo8DsASRYBtwEfBI7ptt+c5Phx+14IvB1Y1J3zr4BPdOM3A1dNyPcWYAw4BVgJvPMAHttngF9Lki7f0cCvAOu7+x8Cfhk4CvivwKeTvOwAjk933POADwC/ChwP/Dlw44EeRy8OFr2G4Y+q6tGqegr4b8BbgcuAj1fVHVX1o6raXlXfGbfPJ6rq76vqeeBzwMnd9rcBt1fV7d1+dwD3AOdM2PehqtoJ/AnwUFX9aVXtBj4P/MKEfNdU1VNV9Qjw4S5fv/4cKHplDnA+8FdV9RhAVX2+qh7rsn4WeBA47QCOv8e7gd+rqs3d4/jvwMle1WsyFr2G4dFxy/8I/DSwhN7V7r58b9zyc8D8bvkE4IJu2uaZJM8A/wEYf5W8Y9zy85Osz+eFJsvXl+p9SuB6fvzL4deBG/bcn+Qd3fTUnqwnAcftfaQpnQB8ZNxxngJC768W6QUseg3DknHLPwM8Rq9cf3Yax3oU+N9VtWDc15FVdfUs5zsQNwLnd1fXrwVuBujW/xfwHuDYqloA3EevoCd6trs9Yty2fztu+VHg3RMe9+FV9ZcHmFUvAha9huHyJIuTHENvnvmzwDrg0iRnJvmJJIuSvKqPY30a+E9J3pjkkCSHJVmeZPEM8v1ukqOTLAGu6PL1raq+DTwJfAz4SlU90911JL1pnScBklxK74p+smM8CWwH3tY9rnfywl+Efwy8P8nPd8c6KskFB5JTLx4WvYbhM8BXgYe7rw9W1V/Te7L1OmAn8Gf0pif2q6oepfeE6QfoFeij9J7QncnP9i3ARuBeek/0rpvGMW4E3kDvse7J+gBwLb0ng3cAy4C/2M8xfoPeY/kn4OeB/3+1XlVfBK4B1if5Ab2/DM6eRk69CMT/eEQHU5KtwLuq6k+HnUV6sfCKXpIaN2/qIZLGS/LL9F6muZeqmvgKHmnonLqRpMY5dSNJjZsTUzfHHXdcLV26dFr7Pvvssxx55JGzG2iARinvKGWF0co7SllhtPKOUlaYWd6NGzd+v6qOn3JgVQ3969RTT63puuuuu6a97zCMUt5Rylo1WnlHKWvVaOUdpaxVM8sL3FN9dKxTN5LUOItekhpn0UtS4yx6SWqcRS9JjbPoJalxFr0kNc6il6TGWfSS1Lg58REI0ly1dM1tAznu6mW7uWSKY2+9+tyBnFsvPl7RS1LjLHpJapxFL0mNs+glqXEWvSQ1zqKXpMZZ9JLUOItekhpn0UtS4yx6SWqcRS9JjbPoJalxFr0kNc6il6TGWfSS1DiLXpIaZ9FLUuMseklqnEUvSY3rq+iT/HaS+5Pcl+TGJIcleXmSu5M8mOSzSQ7txv5kt76lu3/pIB+AJGn/piz6JIuA3wTGquok4BDgQuAa4LqqOhF4Gris2+Uy4OmqegVwXTdOkjQk/U7dzAMOTzIPOAJ4HHg9cFN3//XAed3yym6d7v4zk2R24kqSDtSURV9V24EPAY/QK/idwEbgmara3Q3bBizqlhcBj3b77u7GHzu7sSVJ/UpV7X9AcjRwM/BrwDPA57v1q7rpGZIsAW6vqmVJ7gfeWFXbuvseAk6rqn+acNxVwCqAhQsXnrp+/fppPYBdu3Yxf/78ae07DKOUd5SywmDybtq+c1aPt8fCw2HH8/sfs2zRUQM593SM0s/CKGWFmeVdsWLFxqoam2rcvD6O9QbgH6rqSYAkXwB+CViQZF531b4YeKwbvw1YAmzrpnqOAp6aeNCqWgusBRgbG6vly5f3EWVvGzZsYLr7DsMo5R2lrDCYvJesuW1Wj7fH6mW7uXbT/v/5bb1o+UDOPR2j9LMwSlnh4OTtZ47+EeD0JEd0c+1nAg8AdwHnd2MuBm7plm/t1unu/1pN9WeDJGlg+pmjv5vek6rfAjZ1+6wF3gdcmWQLvTn4dd0u64Bju+1XAmsGkFuS1Kd+pm6oqquAqyZsfhg4bZKx/wxcMPNokqTZ4DtjJalxFr0kNc6il6TGWfSS1DiLXpIaZ9FLUuMseklqnEUvSY2z6CWpcRa9JDXOopekxln0ktQ4i16SGmfRS1LjLHpJapxFL0mNs+glqXEWvSQ1zqKXpMZZ9JLUOItekhpn0UtS4yx6SWqcRS9JjbPoJalxFr0kNc6il6TGWfSS1DiLXpIaZ9FLUuMseklqnEUvSY2z6CWpcRa9JDXOopekxln0ktQ4i16SGtdX0SdZkOSmJN9JsjnJLyY5JskdSR7sbo/uxibJHyTZkuTvkpwy2IcgSdqffq/oPwJ8uapeBbwG2AysAe6sqhOBO7t1gLOBE7uvVcBHZzWxJOmATFn0SV4CnAGsA6iqf6mqZ4CVwPXdsOuB87rllcCnquebwIIkL5v15JKkvqSq9j8gORlYCzxA72p+I3AFsL2qFowb93RVHZ3kS8DVVfWNbvudwPuq6p4Jx11F74qfhQsXnrp+/fppPYBdu3Yxf/78ae07DKOUd5SywmDybtq+c1aPt8fCw2HH8/sfs2zRUQM593SM0s/CKGWFmeVdsWLFxqoam2rcvD6ONQ84BXhvVd2d5CP8eJpmMplk216/TapqLb1fIIyNjdXy5cv7iLK3DRs2MN19h2GU8o5SVhhM3kvW3Darx9tj9bLdXLtp///8tl60fCDnno5R+lkYpaxwcPL2M0e/DdhWVXd36zfRK/4de6Zkutsnxo1fMm7/xcBjsxNXknSgpiz6qvoe8GiSV3abzqQ3jXMrcHG37WLglm75VuAd3atvTgd2VtXjsxtbktSvfqZuAN4L3JDkUOBh4FJ6vyQ+l+Qy4BHggm7s7cA5wBbguW6sJGlI+ir6qroXmGzC/8xJxhZw+QxzSZJmie+MlaTGWfSS1DiLXpIaZ9FLUuMseklqnEUvSY2z6CWpcRa9JDXOopekxln0ktQ4i16SGmfRS1LjLHpJaly/H1MsDdXSPv6np9XLdg/sf4SSRplX9JLUOItekhpn0UtS4yx6SWqcRS9JjbPoJalxFr0kNc6il6TGWfSS1DiLXpIaZ9FLUuMseklqnEUvSY2z6CWpcRa9JDXOz6OX5qh+PoN/ELZefe5QzqvB8Ypekhpn0UtS45y60QEZ1nSCpOnzil6SGmfRS1LjLHpJapxFL0mN67vokxyS5NtJvtStvzzJ3UkeTPLZJId223+yW9/S3b90MNElSf04kCv6K4DN49avAa6rqhOBp4HLuu2XAU9X1SuA67pxkqQh6avokywGzgU+1q0HeD1wUzfkeuC8bnllt053/5ndeEnSEKSqph6U3AT8HvBTwO8AlwDf7K7aSbIE+JOqOinJfcBZVbWtu+8h4LVV9f0Jx1wFrAJYuHDhqevXr5/WA9i1axfz58+f1r7DMEp5J8u6afvOIaWZ2sLDYcfzw07Rn7mcddmio/baNuo/t3PZTPKuWLFiY1WNTTVuyjdMJXkT8ERVbUyyfM/mSYZWH/f9eEPVWmAtwNjYWC1fvnzikL5s2LCB6e47DKOUd7Ksl8zhN0ytXrabazeNxnsA53LWrRct32vbqP/czmUHI28/P2mvA96c5BzgMOAlwIeBBUnmVdVuYDHwWDd+G7AE2JZkHnAU8NSsJ5ck9WXKOfqqen9VLa6qpcCFwNeq6iLgLuD8btjFwC3d8q3dOt39X6t+5ockSQMxk9fRvw+4MskW4FhgXbd9HXBst/1KYM3MIkqSZuKAJgmragOwoVt+GDhtkjH/DFwwC9kkSbPAd8ZKUuMseklqnEUvSY2z6CWpcRa9JDXOopekxln0ktQ4i16SGmfRS1LjLHpJapxFL0mNs+glqXEWvSQ1zqKXpMZZ9JLUOItekhpn0UtS4yx6SWqcRS9JjbPoJalxFr0kNc6il6TGWfSS1DiLXpIaZ9FLUuMseklqnEUvSY2bN+wAOnBL19x2UM6zetluLjlI55I0OF7RS1LjLHpJapxFL0mNs+glqXEWvSQ1zqKXpMZZ9JLUOItekhpn0UtS46Ys+iRLktyVZHOS+5Nc0W0/JskdSR7sbo/utifJHyTZkuTvkpwy6AchSdq3fq7odwOrq+rngNOBy5O8GlgD3FlVJwJ3dusAZwMndl+rgI/OempJUt+mLPqqeryqvtUt/xDYDCwCVgLXd8OuB87rllcCn6qebwILkrxs1pNLkvqSqup/cLIU+DpwEvBIVS0Yd9/TVXV0ki8BV1fVN7rtdwLvq6p7JhxrFb0rfhYuXHjq+vXrp/UAdu3axfz586e17zDMRt5N23fOUpr9W3g47Hj+oJxqVoxS3rmcddmio/baNkr/zkYpK8ws74oVKzZW1dhU4/r+9Mok84Gbgd+qqh8k2efQSbbt9dukqtYCawHGxsZq+fLl/UZ5gQ0bNjDdfYdhNvIerE+UXL1sN9duGp0POB2lvHM666Zn99q0etn/5dpv7L19tm29+twZH+PF2AlT6etVN0n+Db2Sv6GqvtBt3rFnSqa7faLbvg1YMm73xcBjsxNXknSg+nnVTYB1wOaq+v1xd90KXNwtXwzcMm77O7pX35wO7Kyqx2cxsyTpAPTzt+PrgLcDm5Lc2237AHA18LkklwGPABd0990OnANsAZ4DLp3VxJKkAzJl0XdPqu5rQv7MScYXcPkMc0mSZonvjJWkxln0ktQ4i16SGmfRS1LjLHpJapxFL0mNs+glqXEWvSQ1zqKXpMZZ9JLUOItekhpn0UtS4yx6SWqcRS9JjbPoJalxFr0kNc6il6TGWfSS1DiLXpIaZ9FLUuMseklqnEUvSY2z6CWpcRa9JDXOopekxln0ktS4ecMOMMqWrrntgPdZvWw3l0xjP0maLq/oJalxFr0kNc6pG0lzxnSmQyeazvTo1qvPnfF55zKv6CWpcRa9JDXOopekxln0ktS4kX8ydtP2nb4uXZL2wyt6SWqcRS9JjRtI0Sc5K8l3k2xJsmYQ55Ak9WfWiz7JIcD/BM4GXg28NcmrZ/s8kqT+DOLJ2NOALVX1MECS9cBK4IEBnEuSZmw23pE7XZ8868iBnyNVNbsHTM4Hzqqqd3XrbwdeW1XvmTBuFbCqW30l8N1pnvI44PvT3HcYRinvKGWF0co7SllhtPKOUlaYWd4Tqur4qQYN4oo+k2zb67dJVa0F1s74ZMk9VTU20+McLKOUd5SywmjlHaWsMFp5RykrHJy8g3gydhuwZNz6YuCxAZxHktSHQRT93wAnJnl5kkOBC4FbB3AeSVIfZn3qpqp2J3kP8BXgEODjVXX/bJ9nnBlP/xxko5R3lLLCaOUdpawwWnlHKSschLyz/mSsJGlu8Z2xktQ4i16SGjfyRZ/kkCTfTvKlYWfZnyRbk2xKcm+Se4adZypJFiS5Kcl3kmxO8ovDzjSZJK/svqd7vn6Q5LeGnWt/kvx2kvuT3JfkxiSHDTvTviS5ost5/1z8vib5eJInktw3btsxSe5I8mB3e/QwM+6xj6wXdN/bHyUZ2EssR77ogSuAzcMO0acVVXXyiLzG9yPAl6vqVcBrmKPf46r6bvc9PRk4FXgO+OKQY+1TkkXAbwJjVXUSvRcsXDjcVJNLchLwG/Te7f4a4E1JThxuqr18EjhrwrY1wJ1VdSJwZ7c+F3ySvbPeB/wq8PVBnnikiz7JYuBc4GPDztKSJC8BzgDWAVTVv1TVM8NN1ZczgYeq6h+HHWQK84DDk8wDjmDuvs/k54BvVtVzVbUb+DPgLUPO9AJV9XXgqQmbVwLXd8vXA+cd1FD7MFnWqtpcVdP9VIC+jXTRAx8G/jPwo2EH6UMBX02ysfv4h7ns3wFPAp/opsU+lmTwH8gxcxcCNw47xP5U1XbgQ8AjwOPAzqr66nBT7dN9wBlJjk1yBHAOL3wz5Fy1sKoeB+huXzrkPEM3skWf5E3AE1W1cdhZ+vS6qjqF3qd6Xp7kjGEH2o95wCnAR6vqF4BnmTt//k6qe3Pem4HPDzvL/nTzxSuBlwM/DRyZ5G3DTTW5qtoMXAPcAXwZ+Ftg91BDaVpGtuiB1wFvTrIVWA+8Psmnhxtp36rqse72CXpzyKcNN9F+bQO2VdXd3fpN9Ip/Ljsb+FZV7Rh2kCm8AfiHqnqyqv4V+ALwS0POtE9Vta6qTqmqM+hNOzw47Ex92JHkZQDd7RNDzjN0I1v0VfX+qlpcVUvp/cn+taqak1dGSY5M8lN7loFfofdn8ZxUVd8DHk3yym7Tmcz9j5l+K3N82qbzCHB6kiOShN73dk4+0Q2Q5KXd7c/Qe9JwFL7HtwIXd8sXA7cMMcucMPL/OfiIWAh8sffvmnnAZ6rqy8ONNKX3Ajd0UyIPA5cOOc8+dfPH/xF497CzTKWq7k5yE/AtetMg32Zuv2X/5iTHAv8KXF5VTw870HhJbgSWA8cl2QZcBVwNfC7JZfR+sV4wvIQ/to+sTwF/CBwP3Jbk3qp646yf249AkKS2jezUjSSpPxa9JDXOopekxln0ktQ4i16SGmfRS1LjLHpJatz/A2T4dp9im+g7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFVJJREFUeJzt3X+U3XV95/HnS1MrEkv45SwN0WhlrZYcEeYgu249E/EoQrfBHjnFogJi0z/wrF1p1+jZczyeY9d4WtfqtsduVtTYKhGxLqz4i0ZTtS1WUtGgsUugEUJo0ALRCP2R7Xv/uN/oMJlw78zcyZ374fk4Z879fj/fz3y+708m9zXf+dzvnUlVIUlq1+NGXYAkaXEZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoteQl2ZbkdUfpXJXkmUc4dlmSryziuRd1fD12GfSS1DiDXpIaZ9DrqEqyO8mbk3w7yQNJPpjkid2xdUluTfKDJHckOW/apz4tyV8k+WGSzyc5adqY5yT5yyQPJvlGkqlpx7YleXt3/ECS/5PkxCQf6c7ztSSrZ5R5fpI7k3w/ye8mGfh5kuSPkvzejLbrk7yx297Qze2H3b/By48wzupuGWnZjLm8btr+a5Ps7P4dP5fkaYPWqccWg16jcAnwUuDngH8L/NckZwMfBn4bWAG8ENg97XN+DbgceArwBOC3AJKsBG4E3g6c0LV/IsnJ0z73YuDVwMrunH8FfLDrvxN464z6Xg5MAmcC64DXzmFuHwV+NUm6+o4HXgJs6Y7fAfwicBzwNuBPkpwyh/Hpxr0QeAvwK8DJwJeBa+Y6jh4bDHqNwh9U1d1VdT/wO8ArgSuAD1TVTVX1r1V1T1V9Z9rnfLCq/m9VPQxcC5zRtb8K+HRVfbr7vJuAW4DzZ3zuHVW1H/gMcEdV/VlVHQQ+DjxvRn3vrKr7q+ou4Pe7+gb1ZaDohTnAK4C/qqq9AFX18ara29X6MeB24Ow5jH/IbwDvqKqd3Tz+G3CGV/WajUGvUbh72vZ3gZ8FVtG72j2Sv5+2/RCwvNt+GnBRt2zzYJIHgf8ATL9K3jdt++FZ9pfzSLPVN5Dq/ZbALfzkm8OvAR85dDzJa7rlqUO1ng6cdPhIfT0NeM+0ce4HQu+nFukRDHqNwqpp208F9tIL15+bx1h3A39cVSumfRxbVRuHXN9cXAO8oru6fj7wCYBu/38BrwdOrKoVwG30AnqmH3WPT5rW9m+mbd8N/MaMeR9TVX85x1r1GGDQaxSuTHJqkhPorTN/DLgauDzJuUkel2Rlkp8fYKw/Af5jkpcmeXySJyaZSnLqAur77STHJ1kFvKGrb2BV9XXge8D7gc9V1YPdoWPpLet8DyDJ5fSu6Gcb43vAPcCrunm9lkd+I/wj4M1JfqEb67gkF82lTj12GPQahY8Cnwfu7D7eXlV/Te/F1ncD+4E/p7c88aiq6m56L5i+hV6A3k3vBd2F/N++HtgO3Ervhd6r5zHGNcCL6c31UK3fBt5F78XgfcAa4C8eZYxfpzeXfwB+Afjx1XpVfRJ4J7AlyQ/o/WTwsnnUqceA+IdHdDQl2Q28rqr+bNS1SI8VXtFLUuOW9e8iabokv0jvNs3DVNXMO3ikkXPpRpIa59KNJDVuSSzdnHTSSbV69epRl/FjP/rRjzj22GNHXca8jXv9MP5zGPf6YfznMO71Q/85bN++/ftVdfIRO3SWRNCvXr2aW265ZdRl/Ni2bduYmpoadRnzNu71w/jPYdzrh/Gfw7jXD/3nkOS7g4zj0o0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDVuSbwzVupn9YYbR3Le3RsvGMl5pWHyil6SGmfQS1LjBgr6JLuT7Ehya5JburYTktyU5Pbu8fiuPUnem2RXkm8mOXMxJyBJenRzuaJfW1VnVNVkt78B2FpVpwFbu33o/YHi07qP9cD7hlWsJGnuFrJ0sw7Y3G1vBi6c1v7h6rkZWJHklAWcR5K0AAP9KcEkfwc8ABTwP6tqU5IHq2rFtD4PVNXxST4FbKyqr3TtW4E3VdUtM8ZcT++Kn4mJibO2bNkytEkt1IEDB1i+fHz/9Oe41w+Hz2HHPftHUsealcfN6/Na/BqMm3GvH/rPYe3atdunrbIc0aC3V76gqvYmeQpwU5LvPErfzNJ22HeTqtoEbAKYnJyspfQHAsb9DxaMe/1w+BwuG9XtlZdM9e0zmxa/BuNm3OuH4c1hoKWbqtrbPd4HfBI4G9h3aEmme7yv674HWDXt008F9i64UknSvPQN+iTHJnnyoW3gJcBtwA3ApV23S4Hru+0bgNd0d9+cA+yvqnuHXrkkaSCDLN1MAJ9Mcqj/R6vqs0m+Blyb5ArgLuCirv+ngfOBXcBDwOVDr1o6Sub7jtyr1hxc8HKT78rVsPQN+qq6E3juLO3/AJw7S3sBVw6lOknSgvnOWElqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapx/HFxzcrT+SPcwfoWApB6v6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaN3DQJ3l8kq8n+VS3//QkX01ye5KPJXlC1/7T3f6u7vjqxSldkjSIuVzRvwHYOW3/ncC7q+o04AHgiq79CuCBqnom8O6unyRpRAYK+iSnAhcA7+/2A7wIuK7rshm4sNte1+3THT+36y9JGoFUVf9OyXXAO4AnA78FXAbc3F21k2QV8JmqOj3JbcB5VbWnO3YH8Pyq+v6MMdcD6wEmJibO2rJly9AmtVAHDhxg+fLloy5j3haz/h337F+UcWeaOAb2PXxUTrUohlH/mpXHDaeYefJ5MHr95rB27drtVTXZb5xl/Tok+SXgvqranmTqUPMsXWuAYz9pqNoEbAKYnJysqampmV1GZtu2bSyleuZqMeu/bMONizLuTFetOci7dvT977lkDaP+3ZdMDaeYefJ5MHrDmsMg/xNfAPxykvOBJwI/A/w+sCLJsqo6CJwK7O367wFWAXuSLAOOA+5fcKWSpHnpu0ZfVW+uqlOrajVwMfCFqroE+CLwiq7bpcD13fYN3T7d8S/UIOtDkqRFsZD76N8EvDHJLuBE4Oqu/WrgxK79jcCGhZUoSVqIOS0iVtU2YFu3fSdw9ix9/hG4aAi1SZKGwHfGSlLjDHpJapxBL0mNM+glqXEGvSQ1bnzfeig1bvVRehfyTLs3XjCS82rxeEUvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY3rG/RJnpjkr5N8I8m3kryta396kq8muT3Jx5I8oWv/6W5/V3d89eJOQZL0aAa5ov8n4EVV9VzgDOC8JOcA7wTeXVWnAQ8AV3T9rwAeqKpnAu/u+kmSRqRv0FfPgW73p7qPAl4EXNe1bwYu7LbXdft0x89NkqFVLEmak1RV/07J44HtwDOBPwR+F7i5u2onySrgM1V1epLbgPOqak937A7g+VX1/RljrgfWA0xMTJy1ZcuW4c1qgQ4cOMDy5ctHXca8LWb9O+7ZvyjjzjRxDOx7+KicalGMc/1rVh4H+DxYCvrNYe3atdurarLfOMsGOVlV/T/gjCQrgE8Cz56tW/c429X7Yd9NqmoTsAlgcnKypqamBinlqNi2bRtLqZ65Wsz6L9tw46KMO9NVaw7yrh0D/fdcksa5/t2XTAE+D5aCYc1hTnfdVNWDwDbgHGBFkkP/k08F9nbbe4BVAN3x44D7F1ypJGleBrnr5uTuSp4kxwAvBnYCXwRe0XW7FLi+276h26c7/oUaZH1IkrQoBvnZ8hRgc7dO/zjg2qr6VJJvA1uSvB34OnB11/9q4I+T7KJ3JX/xItQtSRpQ36Cvqm8Cz5ul/U7g7Fna/xG4aCjVSZIWzHfGSlLjDHpJatx43v/1GLe6zy2OV605eNRug5S09HlFL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMb1Dfokq5J8McnOJN9K8oau/YQkNyW5vXs8vmtPkvcm2ZXkm0nOXOxJSJKObJAr+oPAVVX1bOAc4MokzwE2AFur6jRga7cP8DLgtO5jPfC+oVctSRpY36Cvqnur6m+67R8CO4GVwDpgc9dtM3Bht70O+HD13AysSHLK0CuXJA0kVTV452Q18CXgdOCuqlox7dgDVXV8kk8BG6vqK137VuBNVXXLjLHW07viZ2Ji4qwtW7YscCrDc+DAAZYvXz7qMo5oxz37H/X4xDGw7+GjVMwiGfc5jHP9a1YeByz950E/414/9J/D2rVrt1fVZL9xlg16wiTLgU8Av1lVP0hyxK6ztB323aSqNgGbACYnJ2tqamrQUhbdtm3bWEr1zHTZhhsf9fhVaw7yrh0Df2mXpHGfwzjXv/uSKWDpPw/6Gff6YXhzGOiumyQ/RS/kP1JVf9o17zu0JNM93te17wFWTfv0U4G9C65UkjQvg9x1E+BqYGdV/fdph24ALu22LwWun9b+mu7um3OA/VV17xBrliTNwSA/W74AeDWwI8mtXdtbgI3AtUmuAO4CLuqOfRo4H9gFPARcPtSKJUlz0jfouxdVj7Qgf+4s/Qu4coF1SRqR1d1rQFetOdj39aBh273xgqN6vscK3xkrSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1Lj+gZ9kg8kuS/JbdPaTkhyU5Lbu8fju/YkeW+SXUm+meTMxSxektTfsgH6fAj4A+DD09o2AFuramOSDd3+m4CXAad1H88H3tc9Nmn1hhtHXYIk9dX3ir6qvgTcP6N5HbC5294MXDit/cPVczOwIskpwypWkjR3812jn6iqewG6x6d07SuBu6f129O1SZJGJFXVv1OyGvhUVZ3e7T9YVSumHX+gqo5PciPwjqr6Ste+FfgvVbV9ljHXA+sBJiYmztqyZcsQpjMcBw4cYPny5X377bhn/1GoZu4mjoF9D4+6ioUZ9zmMe/0wmjmsWXnc0MYa9Hm8lPWbw9q1a7dX1WS/cQZZo5/NviSnVNW93dLMfV37HmDVtH6nAntnG6CqNgGbACYnJ2tqamqepQzftm3bGKSey5boGv1Vaw7yrh3z/dIuDeM+h3GvH0Yzh92XTA1trEGfx0vZsOYw36WbG4BLu+1Lgeuntb+mu/vmHGD/oSUeSdJo9P12neQaYAo4Kcke4K3ARuDaJFcAdwEXdd0/DZwP7AIeAi5fhJolSXPQN+ir6pVHOHTuLH0LuHKhRUmShsd3xkpS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXF9/zj4Urd6w41DH/OqNQe5bBHGlfTohvl8nsvzePfGC4Z23qXIK3pJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxi/K7bpKcB7wHeDzw/qrauBjnkaRhWIzfmTWoo/F7doZ+RZ/k8cAfAi8DngO8Mslzhn0eSdJgFmPp5mxgV1XdWVX/DGwB1i3CeSRJA0hVDXfA5BXAeVX1um7/1cDzq+r1M/qtB9Z3u88C/naohSzMScD3R13EAox7/TD+cxj3+mH85zDu9UP/OTytqk7uN8hirNFnlrbDvptU1SZg0yKcf8GS3FJVk6OuY77GvX4Y/zmMe/0w/nMY9/pheHNYjKWbPcCqafunAnsX4TySpAEsRtB/DTgtydOTPAG4GLhhEc4jSRrA0JduqupgktcDn6N3e+UHqupbwz7PIluSS0pzMO71w/jPYdzrh/Gfw7jXD0Oaw9BfjJUkLS2+M1aSGmfQS1LjDPppkuxOsiPJrUluGXU985FkRZLrknwnyc4k/27UNQ0qybO6f/tDHz9I8pujrmuukvznJN9KcluSa5I8cdQ1zUWSN3S1f2tc/v2TfCDJfUlum9Z2QpKbktzePR4/yhr7OcIcLuq+Dv+aZN63WRr0h1tbVWeM8f237wE+W1U/DzwX2DniegZWVX/b/dufAZwFPAR8csRlzUmSlcB/Aiar6nR6NyRcPNqqBpfkdODX6b3D/bnALyU5bbRVDeRDwHkz2jYAW6vqNGBrt7+UfYjD53Ab8CvAlxYysEHfkCQ/A7wQuBqgqv65qh4cbVXzdi5wR1V9d9SFzMMy4Jgky4AnMV7vI3k2cHNVPVRVB4E/B14+4pr6qqovAffPaF4HbO62NwMXHtWi5mi2OVTVzqpa8G8NMOgfqYDPJ9ne/YqGcfMM4HvAB5N8Pcn7kxw76qLm6WLgmlEXMVdVdQ/we8BdwL3A/qr6/GirmpPbgBcmOTHJk4DzeeQbIMfJRFXdC9A9PmXE9YyMQf9IL6iqM+n95s0rk7xw1AXN0TLgTOB9VfU84Ecs/R9XD9O90e6XgY+Pupa56taB1wFPB34WODbJq0Zb1eCqaifwTuAm4LPAN4CDIy1KC2bQT1NVe7vH++itDZ892ormbA+wp6q+2u1fRy/4x83LgL+pqn2jLmQeXgz8XVV9r6r+BfhT4N+PuKY5qaqrq+rMqnohvaWE20dd0zztS3IKQPd434jrGRmDvpPk2CRPPrQNvITej7Fjo6r+Hrg7ybO6pnOBb4+wpPl6JWO4bNO5CzgnyZOShN7XYGxeEAdI8pTu8an0Xggc16/FDcCl3falwPUjrGWkfGdsJ8kz+MkdHsuAj1bV74ywpHlJcgbwfuAJwJ3A5VX1wGirGly3Lnw38Iyq2j/qeuYjyduAX6W35PF14HVV9U+jrWpwSb4MnAj8C/DGqto64pL6SnINMEXv1/ruA94K/G/gWuCp9L4BX1RVM1+wXTKOMIf7gf8BnAw8CNxaVS+d89gGvSS1zaUbSWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIa9/8BO+1U0OBDrA0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAF/hJREFUeJzt3X+UXGWd5/H3R0BF2iUg2BOTDO2OGUckx0D6ILPMcLrBVcDZCcyRWRhUfjlhz8FdZzfjGjx7Vj0ju/GMyDhnXGajAcLwo0GQAwv4AyOtw4yICSIBIkuACEmYAEMINLDMBD/7R904RWxSVV1VXV1PPq9z6tS9Tz333u+TdH361tNVdWWbiIgo1+t6XUBERHRXgj4ionAJ+oiIwiXoIyIKl6CPiChcgj4ionAJ+pjxJI1L+tg0HcuS3vEaj50p6Y4uHrur+489V4I+IqJwCfqIiMIl6GNaSdoo6XxJD0jaJulSSW+sHlss6R5Jz0l6WNLxdZseIunvJD0v6TuSDqrb51GS/l7Ss5J+Kmmk7rFxSZ+vHp+Q9H8kvUXSldVxfixpaJcyT5T0iKSnJf25pKafJ5L+WtIXd2m7UdJ/qZaXVWN7vvo3OPk19jNUTSPtvctYPla3frak9dW/47clHdJsnbFnSdBHL5wOfAD4DeA3gf8m6UjgcuCTwCzgGGBj3TZ/BJwFvBV4PfCnAJLmALcAnwcOrNqvl3Rw3banAh8B5lTH/CFwadV/PfCZXeo7GRgGjgAWA2e3MLargH8vSVV9BwDvB8aqxx8GfhfYH/gccIWk2S3sn2q/JwGfBv4AOBj4W+DqVvcTe4YEffTCX9l+3PYzwAXAacA5wCW2b7P9C9ubbf+sbptLbf9f2y8B1wILq/YPA7favrXa7jZgDXDiLts+bHs78E3gYdvftb0D+Dpw+C71fcH2M7YfA/6iqq9ZfwuYWpgDfAj4oe0tALa/bntLVes1wEPAkS3sf6dzgf9pe301jv8BLMxZfUwmQR+98Hjd8s+BtwHzqJ3tvpZ/qFt+ERiolg8BTqmmbZ6V9CzwO0D9WfLWuuWXJlkf4NUmq68prn1L4Bj/8svhj4Ardz4u6aPV9NTOWg8DDvrVPTV0CPDluv08A4jaq5aIV0nQRy/Mq1v+dWALtXD9jSns63Hgb2zPqrvtZ3t5h+trxdXAh6qz6/cC1wNU618FPg68xfYs4D5qAb2rF6r7N9W1/Vrd8uPAubuMe1/bf99irbEHSNBHL5wnaa6kA6nNM18DrATOknScpNdJmiPpt5rY1xXAv5P0AUl7SXqjpBFJc9uo75OSDpA0D/hEVV/TbP8EeAr4GvBt289WD+1HbVrnKQBJZ1E7o59sH08Bm4EPV+M6m1f/Ivxr4HxJ7672tb+kU1qpM/YcCfrohauA7wCPVLfP276L2h9bLwK2A9+nNj2xW7Yfp/YH009TC9DHqf1Bt52f7RuBtcA91P7Qu3IK+7gaeB+1se6s9QHgQmp/DN4KLAD+bjf7+GNqY/lH4N3AL8/Wbd8AfAEYk/QctVcGJ0yhztgDKBceiekkaSPwMdvf7XUtEXuKnNFHRBRu78ZdIqKepN+l9jbNX2F713fwRPRcpm4iIgqXqZuIiMLNiKmbgw46yENDQy1t88ILL7Dffvt1p6BpVtJYoKzxZCwzU8ZSs3bt2qdtH9yo34wI+qGhIdasWdPSNuPj44yMjHSnoGlW0ligrPFkLDNTxlIj6efN9MvUTURE4RL0ERGFS9BHRBQuQR8RUbiGQV99SdRd1ZV77pf0uar9MkmPVl+5eo+khVW7JP2lpA2S7pV0RLcHERERr62Zd928DBxre0LSPsAdknZ+KvCTtq/bpf8JwPzq9l7g4uo+IiJ6oOEZvWsmqtV9qtvuPk67GLi82u5OYNZULpUWERGd0dRXIEjai9rXtr4D+IrtT0m6DPhtamf8q4Fltl+WdDOw3PYd1bargU/ZXrPLPpcASwAGBwcXjY2N0YqJiQkGBsr4WpGSxgJljSdjmZkylprR0dG1tocbdrTd9I3aRZtvp3axhNnUrozzBmAV8N+rPrcAv1O3zWpg0e72u2jRIrfq9ttvb3mbmaqksdhljSdjmZkylhpgjZvI7pY+GWv7WUnjwPG2v1g1vyzpUuBPq/VNvPpSbHNp/VJsETPG0LJbenLcjcs/2JPjRnmaedfNwZJmVcv7Urtqzs92zrtLEnAStSvcANwEfLR6981RwHbbT3Sl+oiIaKiZM/rZwKpqnv51wLW2b5b0PUkHU5u+uQf4D1X/W4ETgQ3Ai9QuDxcRET3SMOht3wscPkn7sa/R38B57ZcWERGdkE/GRkQULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4BH1EROES9BERhWvpmrERMX06fa3apQt2cGYT+8y1asuTM/qIiMIl6CMiCpegj4goXII+IqJwDYNe0hsl3SXpp5Lul/S5qv3tkn4k6SFJ10h6fdX+hmp9Q/X4UHeHEBERu9PMGf3LwLG23wMsBI6XdBTwBeAi2/OBbcA5Vf9zgG223wFcVPWLiIgeaRj0rpmoVvepbgaOBa6r2lcBJ1XLi6t1qsePk6SOVRwRES2R7cadpL2AtcA7gK8Afw7cWZ21I2ke8E3bh0m6Dzje9qbqsYeB99p+epd9LgGWAAwODi4aGxtrqfCJiQkGBgZa2mamKmksUNZ4JiYmeHT7K70uoyMG94WtLzXut2DO/t0vpk2l/YxNdSyjo6NrbQ836tfUB6ZsvwIslDQLuAF412TdqvvJzt5/5beJ7RXACoDh4WGPjIw0U8ovjY+P0+o2M1VJY4GyxjM+Ps6Fd7zQ6zI6YumCHVy4rvFTfuPpI90vpk2l/Yx1eywtvevG9rPAOHAUMEvSzp+aucCWankTMA+genx/4JlOFBsREa1r5l03B1dn8kjaF3gfsB64HfhQ1e0M4MZq+aZqnerx77mZ+aGIiOiKZqZuZgOrqnn61wHX2r5Z0gPAmKTPAz8BVlb9VwJ/I2kDtTP5U7tQd0RENKlh0Nu+Fzh8kvZHgCMnaf9/wCkdqS4iItqWT8ZGRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQulxKMvtDpy+o1a+mCHeRpEv0uZ/QREYVL0EdEFC5BHxFRuAR9REThEvQREYVL0EdEFC5BHxFRuAR9REThEvQREYVL0EdEFC5BHxFRuAR9REThEvQREYVL0EdEFK5h0EuaJ+l2Sesl3S/pE1X7ZyVtlnRPdTuxbpvzJW2Q9KCkD3RzABERsXvNfNH2DmCp7bslvRlYK+m26rGLbH+xvrOkQ4FTgXcDbwO+K+k3bb/SycIjIqI5Dc/obT9h++5q+XlgPTBnN5ssBsZsv2z7UWADcGQnio2IiNa1NEcvaQg4HPhR1fRxSfdKukTSAVXbHODxus02sftfDBER0UWy3VxHaQD4PnCB7W9IGgSeBgz8GTDb9tmSvgL80PYV1XYrgVttX7/L/pYASwAGBwcXjY2NtVT4xMQEAwMDLW0zU5U0FujOeNZt3t7R/TVrcF/Y+lJPDt1xzY5lwZz9u19Mm0p6zrQzltHR0bW2hxv1a+pimJL2Aa4HrrT9DQDbW+se/ypwc7W6CZhXt/lcYMuu+7S9AlgBMDw87JGRkWZK+aXx8XFa3WamKmks0J3xnNnDa8ZeuK6Ma8Y2O5aNp490v5g2lfScmY6xNPOuGwErgfW2v1TXPruu28nAfdXyTcCpkt4g6e3AfOCuzpUcERGtaOZU5WjgI8A6SfdUbZ8GTpO0kNrUzUbgXADb90u6FniA2jt2zss7biIieqdh0Nu+A9AkD926m20uAC5oo66IiOiQfDI2IqJwCfqIiMIl6CMiCpegj4goXII+IqJwCfqIiMIl6CMiCpegj4goXII+IqJwCfqIiMIl6CMiCpegj4goXII+IqJwCfqIiMIl6CMiCpegj4goXII+IqJwCfqIiMIl6CMiCpegj4goXII+IqJwezfqIGkecDnwa8AvgBW2vyzpQOAaYAjYCPyh7W2SBHwZOBF4ETjT9t3dKT+m29CyWxr2WbpgB2c20S8ipkczZ/Q7gKW23wUcBZwn6VBgGbDa9nxgdbUOcAIwv7otAS7ueNUREdG0hkFv+4mdZ+S2nwfWA3OAxcCqqtsq4KRqeTFwuWvuBGZJmt3xyiMioimy3XxnaQj4AXAY8JjtWXWPbbN9gKSbgeW276jaVwOfsr1ml30toXbGz+Dg4KKxsbGWCp+YmGBgYKClbWaqfhrLus3bG/YZ3Be2vjQNxUyDPXEsC+bs3/1i2tRPz5lG2hnL6OjoWtvDjfo1nKPfSdIAcD3wJ7afq03FT951krZf+W1iewWwAmB4eNgjIyPNlgLA+Pg4rW4zU/XTWJqZe1+6YAcXrmv6R2tG2xPHsvH0ke4X06Z+es40Mh1jaepdN5L2oRbyV9r+RtW8deeUTHX/ZNW+CZhXt/lcYEtnyo2IiFY1DPrqXTQrgfW2v1T30E3AGdXyGcCNde0fVc1RwHbbT3Sw5oiIaEEzr0mPBj4CrJN0T9X2aWA5cK2kc4DHgFOqx26l9tbKDdTeXnlWRyuOiIiWNAz66o+qrzUhf9wk/Q2c12ZdERHRIflkbERE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RoGvaRLJD0p6b66ts9K2izpnup2Yt1j50vaIOlBSR/oVuEREdGcZs7oLwOOn6T9ItsLq9utAJIOBU4F3l1t878k7dWpYiMionUNg972D4BnmtzfYmDM9su2HwU2AEe2UV9ERLRJtht3koaAm20fVq1/FjgTeA5YAyy1vU3SXwF32r6i6rcS+Kbt6ybZ5xJgCcDg4OCisbGxlgqfmJhgYGCgpW1mqn4ay7rN2xv2GdwXtr40DcVMgz1xLAvm7N/9YtrUT8+ZRtoZy+jo6Frbw4367T2lvcPFwJ8Bru4vBM4GNEnfSX+T2F4BrAAYHh72yMhISwWMj4/T6jYzVT+N5cxltzTss3TBDi5cN9UfrZllTxzLxtNHul9Mm/rpOdPIdIxlSu+6sb3V9iu2fwF8lX+ZntkEzKvrOhfY0l6JERHRjimdqkiabfuJavVkYOc7cm4CrpL0JeBtwHzgrrarjIhpM9TEq7Zu2bj8gz07dskaBr2kq4ER4CBJm4DPACOSFlKbltkInAtg+35J1wIPADuA82y/0p3SIyKiGQ2D3vZpkzSv3E3/C4AL2ikqIiI6J5+MjYgoXII+IqJwCfqIiMIl6CMiCpegj4goXII+IqJwCfqIiMIl6CMiCpegj4goXII+IqJwCfqIiMIl6CMiCpegj4goXII+IqJwCfqIiMIl6CMiCpegj4goXII+IqJwCfqIiMIl6CMiCpegj4goXMOgl3SJpCcl3VfXdqCk2yQ9VN0fULVL0l9K2iDpXklHdLP4iIhorJkz+suA43dpWwastj0fWF2tA5wAzK9uS4CLO1NmRERMVcOgt/0D4JldmhcDq6rlVcBJde2Xu+ZOYJak2Z0qNiIiWifbjTtJQ8DNtg+r1p+1Pavu8W22D5B0M7Dc9h1V+2rgU7bXTLLPJdTO+hkcHFw0NjbWUuETExMMDAy0tM1M1U9jWbd5e8M+g/vC1pemoZhpkLFMrwVz9m+qXz89ZxppZyyjo6NrbQ836rf3lPb+2jRJ26S/SWyvAFYADA8Pe2RkpKUDjY+P0+o2M1U/jeXMZbc07LN0wQ4uXNfpH63eyFim18bTR5rq10/PmUamYyxTfdfN1p1TMtX9k1X7JmBeXb+5wJaplxcREe2aatDfBJxRLZ8B3FjX/tHq3TdHAdttP9FmjRER0YaGr+MkXQ2MAAdJ2gR8BlgOXCvpHOAx4JSq+63AicAG4EXgrC7UHBERLWgY9LZPe42Hjpukr4Hz2i0qIiI6J5+MjYgoXII+IqJwCfqIiMIl6CMiCjezPz0REXuUoSY+kAe1D3818+G9Zm1c/sGO7WsmStD3oWafDBERkKmbiIjiJegjIgqXoI+IKFyCPiKicAn6iIjCJegjIgqXoI+IKFyCPiKicAn6iIjCJegjIgqXoI+IKFyCPiKicAn6iIjCJegjIgqXoI+IKFxb30cvaSPwPPAKsMP2sKQDgWuAIWAj8Ie2t7VXZkRETFUnzuhHbS+0PVytLwNW254PrK7WIyKiR7oxdbMYWFUtrwJO6sIxIiKiSbI99Y2lR4FtgIH/bXuFpGdtz6rrs832AZNsuwRYAjA4OLhobGyspWNPTEwwMDAw5dpnklbHsm7z9i5W077BfWHrS72uojMylpmp02NZMGf/zu2sRe1k2ejo6Nq62ZTX1O41Y4+2vUXSW4HbJP2s2Q1trwBWAAwPD3tkZKSlA4+Pj9PqNjNVq2Pp5EWRu2Hpgh1cuK6MyxFnLDNTp8ey8fSRju2rVdORZW1N3djeUt0/CdwAHAlslTQboLp/st0iIyJi6qYc9JL2k/TmncvA+4H7gJuAM6puZwA3tltkRERMXTuvfQaBGyTt3M9Vtr8l6cfAtZLOAR4DTmm/zIiImKopB73tR4D3TNL+j8Bx7RQVERGdk0/GRkQULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4BH1EROHK+OKLHhnq0HfOLF2wY8Z/f01E9K+c0UdEFC5n9BGxx+vUq/OpuOz4/bp+jJzRR0QULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4vv8KhF5+dDkioh907Yxe0vGSHpS0QdKybh0nIiJ2rytBL2kv4CvACcChwGmSDu3GsSIiYve6dUZ/JLDB9iO2/wkYAxZ36VgREbEbst35nUofAo63/bFq/SPAe21/vK7PEmBJtfpO4MEWD3MQ8HQHyp0JShoLlDWejGVmylhqDrF9cKNO3fpjrCZpe9VvFNsrgBVTPoC0xvbwVLefSUoaC5Q1noxlZspYWtOtqZtNwLy69bnAli4dKyIidqNbQf9jYL6kt0t6PXAqcFOXjhUREbvRlakb2zskfRz4NrAXcInt+zt8mClP+8xAJY0FyhpPxjIzZSwt6MofYyMiYubIVyBERBQuQR8RUbi+DXpJe0n6iaSbe11LOyRtlLRO0j2S1vS6nnZImiXpOkk/k7Re0m/3uqapkPTO6v9j5+05SX/S67qmStJ/lnS/pPskXS3pjb2uaaokfaIax/39+H8i6RJJT0q6r67tQEm3SXqouj+g08ft26AHPgGs73URHTJqe2EB7wv+MvAt278FvIc+/f+x/WD1/7EQWAS8CNzQ47KmRNIc4D8Bw7YPo/bmiFN7W9XUSDoM+GNqn7x/D/B7kub3tqqWXQYcv0vbMmC17fnA6mq9o/oy6CXNBT4IfK3XtUSNpH8FHAOsBLD9T7af7W1VHXEc8LDtn/e6kDbsDewraW/gTfTvZ1reBdxp+0XbO4DvAyf3uKaW2P4B8MwuzYuBVdXyKuCkTh+3L4Me+AvgvwK/6HUhHWDgO5LWVl8L0a/+NfAUcGk1pfY1Sfv1uqgOOBW4utdFTJXtzcAXgceAJ4Dttr/T26qm7D7gGElvkfQm4ERe/cHMfjVo+wmA6v6tnT5A3wW9pN8DnrS9tte1dMjRto+g9k2f50k6ptcFTdHewBHAxbYPB16gCy9Bp1P1Yb/fB77e61qmqprvXQy8HXgbsJ+kD/e2qqmxvR74AnAb8C3gp8COnhbVJ/ou6IGjgd+XtJHat2IeK+mK3pY0dba3VPdPUpsHPrK3FU3ZJmCT7R9V69dRC/5+dgJwt+2tvS6kDe8DHrX9lO1/Br4B/Jse1zRltlfaPsL2MdSmQB7qdU0dsFXSbIDq/slOH6Dvgt72+bbn2h6i9rL6e7b78gxF0n6S3rxzGXg/tZenfcf2PwCPS3pn1XQc8EAPS+qE0+jjaZvKY8BRkt4kSdT+X/ryj+QAkt5a3f868Af0//8P1L4e5oxq+Qzgxk4foO8vJdjnBoEbas8/9gausv2t3pbUlv8IXFlNeTwCnNXjeqasmgP+t8C5va6lHbZ/JOk64G5q0xw/ob+/PuB6SW8B/hk4z/a2XhfUCklXAyPAQZI2AZ8BlgPXSjqH2i/mUzp+3HwFQkRE2fpu6iYiIlqToI+IKFyCPiKicAn6iIjCJegjIgqXoI+IKFyCPiKicP8fz6XpQmHS4+EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFGJJREFUeJzt3X+0XWV95/H3p6StSCw/RO9gSIntUKdI1lC4izLDlHUzdJQfnSJdMgNF5Yc2/IEzOM10Glyzls6qzOAaqbXLGTpRVByQK4IsGEAUmabWtlQJ0gaIjoARSGjAAoEgqzb4nT/OTr3EJPfm/Mi5efJ+rXXX2Wef/Tz7++Te+7n7PGfvnVQVkqR2/cS4C5AkjZZBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINe816S1UnetYf2VUn+8U5eOz/JV0e475H2r32XQS9JjTPoJalxBr32qCTrk1ya5MEkzyT5ZJJXdK+dkeS+JM8leTjJKTOaHpHkz5I8n+RLSQ6d0ecJSf48ybNJ/irJ1IzXVif5QPf6liT/J8mrk1zb7efrSZZsV+ZpSR5J8r0k/z3JnH9PkvxRkg9tt+7mJL/dLa/sxvZ8929w5k76WdJNIy3YbizvmvH8wiTrun/HLyY5Yq51at9i0GsczgXeDPw88AvAf05yPPBp4HeAg4CTgPUz2vwmcAHwWuCngP8IkGQRcBvwAeCQbv2NSV4zo+3ZwNuBRd0+/wL4ZLf9OuB929V3JjAJHAucAVy4G2P7DPBvk6Sr72DgTcB09/rDwK8ABwL/BbgmyWG70T9dv28B3gv8BvAa4E+B63a3H+0bDHqNw0er6rGqehq4DDgHeCfwiaq6s6p+WFUbquqbM9p8sqr+X1W9CFwPHNOtfxtwe1Xd3rW7E7gHOG27tg9X1WbgC8DDVfXlqtoKfA74pe3q+2BVPV1VjwJ/0NU3V38KFL0wB3gr8BdVtRGgqj5XVRu7Wj8LfBs4fjf63+Yi4L9V1bpuHP8VOMajeu2IQa9xeGzG8neB1wGL6R3t7szfzFj+PrCwWz4COKubtnk2ybPAvwBmHiVvmrH84g6eL+TldlTfnFTvLoHT/OiPw28C1257Pck7uumpbbUeDRz64z3N6gjgIzP6eRoIvXct0ssY9BqHxTOWfxbYSC9cf76Pvh4D/ndVHTTj64CqunzI9e2O64C3dkfXvwzcCNA9/xjwbuDVVXUQcD+9gN7eC93jK2es+0czlh8DLtpu3PtX1Z/vZq3aBxj0GoeLkxye5BB688yfBa4CLkhycpKfSLIoyT+ZQ1/XAP86yZuT7JfkFUmmkhw+QH2/k+TgJIuBS7r65qyqvgE8BXwc+GJVPdu9dAC9aZ2nAJJcQO+Ifkd9PAVsAN7WjetCXv6H8I+AS5O8sevrwCRn7U6d2ncY9BqHzwBfAh7pvj5QVV+j92Hrh4HNwJ/Qm57Ypap6jN4Hpu+lF6CP0ftAd5Cf7ZuBNcB99D7ovaqPPq4DfpXeWLfV+iBwBb0PgzcBS4E/20Ufv0VvLH8LvBH4h6P1qroJ+CAwneQ5eu8MTu2jTu0D4n88oj0pyXrgXVX15XHXIu0rPKKXpMYtmH0TSTMl+RV6p2n+mKra/gweaeycupGkxjl1I0mNmxdTN4ceemgtWbKkr7YvvPACBxxwwHALmgdaHFeLY4I2x+WY9g5r1qz5XlW9Zrbt5kXQL1myhHvuuaevtqtXr2Zqamq4Bc0DLY6rxTFBm+NyTHuHJN+dy3ZO3UhS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuPmxZWx0myWrLxtbPtef/npY9u3NAwe0UtS4wx6SWqcQS9JjXOOXprFbJ8PrFi6lfNH8BmCnw1oWDyil6TGGfSS1DiDXpIaZ9BLUuNmDfoki5P8cZJ1SR5Ickm3/v1JNiS5r/s6bUabS5M8lORbSd48ygFIknZtLmfdbAVWVNW9SV4FrElyZ/fah6vqQzM3TnIUcDbwRuB1wJeT/EJVvTTMwiVJczPrEX1VPVFV93bLzwPrgEW7aHIGMF1Vf1dV3wEeAo4fRrGSpN2Xqpr7xskS4CvA0cBvA+cDzwH30DvqfybJR4G7q+qars1VwBeq6obt+loOLAeYmJg4bnp6uq8BbNmyhYULF/bVdj5rcVyDjGnths1DrmZ4JvaHTS8Ov9+liw4cfqdz5M/f3mHZsmVrqmpytu3mfMFUkoXAjcB7quq5JFcCvwdU93gFcCGQHTT/sb8mVbUKWAUwOTlZU1NTcy3lZVavXk2/beezFsc1yJhGcUHSsKxYupUr1g7/2sP1504Nvc+58uevLXM66ybJT9IL+Wur6vMAVbWpql6qqh8CH+NH0zOPA4tnND8c2Di8kiVJu2MuZ90EuApYV1W/P2P9YTM2OxO4v1u+BTg7yU8neT1wJPC14ZUsSdodc3m/eSLwdmBtkvu6de8FzklyDL1pmfXARQBV9UCS64EH6Z2xc7Fn3EjS+Mwa9FX1VXY87377LtpcBlw2QF2SpCHxylhJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJatyCcRegvcuSlbf13XbF0q2cP0B7Sf3xiF6SGmfQS1LjDHpJapxBL0mNmzXokyxO8sdJ1iV5IMkl3fpDktyZ5Nvd48Hd+iT5wyQPJfnrJMeOehCSpJ2byxH9VmBFVf0icAJwcZKjgJXAXVV1JHBX9xzgVODI7ms5cOXQq5YkzdmsQV9VT1TVvd3y88A6YBFwBnB1t9nVwFu65TOAT1fP3cBBSQ4beuWSpDlJVc1942QJ8BXgaODRqjpoxmvPVNXBSW4FLq+qr3br7wJ+t6ru2a6v5fSO+JmYmDhuenq6rwFs2bKFhQsX9tV2Ppuv41q7YXPfbSf2h00vDrGYeWJU41q66MDhdzpH8/XnbxAtjmnZsmVrqmpytu3mfMFUkoXAjcB7quq5JDvddAfrfuyvSVWtAlYBTE5O1tTU1FxLeZnVq1fTb9v5bL6Oa5ALnlYs3coVa9u7Rm9U41p/7tTQ+5yr+frzN4gWxzRXczrrJslP0gv5a6vq893qTdumZLrHJ7v1jwOLZzQ/HNg4nHIlSbtrLmfdBLgKWFdVvz/jpVuA87rl84CbZ6x/R3f2zQnA5qp6Yog1S5J2w1zeb54IvB1Ym+S+bt17gcuB65O8E3gUOKt77XbgNOAh4PvABUOtWJK0W2YN+u5D1Z1NyJ+8g+0LuHjAuiRJQ9LeJ2NSIwa5U+gg1l9++lj2q9HxFgiS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxnnBlKSXWbLyNlYs3TrQnUr75cVao+ERvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGjdr0Cf5RJInk9w/Y937k2xIcl/3ddqM1y5N8lCSbyV586gKlyTNzVyO6D8FnLKD9R+uqmO6r9sBkhwFnA28sWvzP5PsN6xiJUm7b9agr6qvAE/Psb8zgOmq+ruq+g7wEHD8APVJkgaUqpp9o2QJcGtVHd09fz9wPvAccA+woqqeSfJR4O6quqbb7irgC1V1ww76XA4sB5iYmDhuenq6rwFs2bKFhQsX9tV2Ppuv41q7YXPfbSf2h00vDrGYeaLFcY1rTEsXHTiyvufr79Qgli1btqaqJmfbbkGf/V8J/B5Q3eMVwIVAdrDtDv+SVNUqYBXA5ORkTU1N9VXI6tWr6bftfDZfx3X+ytv6brti6VauWNvvj9z81eK4xjWm9edOjazv+fo7tSf0ddZNVW2qqpeq6ofAx/jR9MzjwOIZmx4ObBysREnSIPoK+iSHzXh6JrDtjJxbgLOT/HSS1wNHAl8brERJ0iBmfW+W5DpgCjg0yePA+4CpJMfQm5ZZD1wEUFUPJLkeeBDYClxcVS+NpnRJ0lzMGvRVdc4OVl+1i+0vAy4bpChJ0vB4ZawkNc6gl6TGGfSS1DiDXpIaZ9BLUuPaupxvH7FkgKtTJe17PKKXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1btagT/KJJE8muX/GukOS3Jnk293jwd36JPnDJA8l+eskx46yeEnS7OZyRP8p4JTt1q0E7qqqI4G7uucApwJHdl/LgSuHU6YkqV+zBn1VfQV4ervVZwBXd8tXA2+Zsf7T1XM3cFCSw4ZVrCRp96WqZt8oWQLcWlVHd8+fraqDZrz+TFUdnORW4PKq+mq3/i7gd6vqnh30uZzeUT8TExPHTU9P9zWALVu2sHDhwr7azme7GtfaDZv3cDXDMbE/bHpx3FUMX4vjGteYli46cGR9t5gVy5YtW1NVk7Ntt2DI+80O1u3wL0lVrQJWAUxOTtbU1FRfO1y9ejX9tp3PdjWu81fetmeLGZIVS7dyxdph/8iNX4vjGteY1p87NbK+W82Kuej3rJtN26Zkuscnu/WPA4tnbHc4sLH/8iRJg+o36G8BzuuWzwNunrH+Hd3ZNycAm6vqiQFrlCQNYNb3ZkmuA6aAQ5M8DrwPuBy4Psk7gUeBs7rNbwdOAx4Cvg9cMIKaJUm7Ydagr6pzdvLSyTvYtoCLBy1KkjQ8XhkrSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGtfWdduS9mpLRnh7jxVLt+709iHrLz99ZPudDzyil6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxCwZpnGQ98DzwErC1qiaTHAJ8FlgCrAf+TVU9M1iZkqR+DeOIfllVHVNVk93zlcBdVXUkcFf3XJI0JqOYujkDuLpbvhp4ywj2IUmao1RV/42T7wDPAAX8r6paleTZqjpoxjbPVNXBO2i7HFgOMDExcdz09HRfNWzZsoWFCxf21XY+29W41m7YvIerGY6J/WHTi+OuYvhaHNe+Nqaliw7cs8UMybJly9bMmE3ZqYHm6IETq2pjktcCdyb55lwbVtUqYBXA5ORkTU1N9VXA6tWr6bftfLarcZ2/8rY9W8yQrFi6lSvWDvojN/+0OK59bUzrz53as8XsYQNN3VTVxu7xSeAm4HhgU5LDALrHJwctUpLUv76DPskBSV61bRl4E3A/cAtwXrfZecDNgxYpSerfIO/NJoCbkmzr5zNVdUeSrwPXJ3kn8Chw1uBlSpL61XfQV9UjwD/dwfq/BU4epChJ0vB4ZawkNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcW3dtWgPWzLCm4utWLp1r715maT5xSN6SWqcQS9JjTPoJalxBr0kNc4PYyXt80Z5YsVs1l9++sj34RG9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxI/uPR5KcAnwE2A/4eFVdPor9rN2wmfPH+J8GSNJ8N5Ij+iT7Af8DOBU4CjgnyVGj2JckaddGNXVzPPBQVT1SVT8ApoEzRrQvSdIupKqG32nyVuCUqnpX9/ztwC9X1btnbLMcWN49fQPwrT53dyjwvQHKna9aHFeLY4I2x+WY9g5HVNVrZttoVHP02cG6l/1FqapVwKqBd5TcU1WTg/Yz37Q4rhbHBG2OyzG1ZVRTN48Di2c8PxzYOKJ9SZJ2YVRB/3XgyCSvT/JTwNnALSPalyRpF0YydVNVW5O8G/givdMrP1FVD4xiXwxh+meeanFcLY4J2hyXY2rISD6MlSTNH14ZK0mNM+glqXF7fdAn2S/JN5LcOu5ahiHJ+iRrk9yX5J5x1zMsSQ5KckOSbyZZl+SfjbumQSR5Q/c92vb1XJL3jLuuYUjyH5I8kOT+JNclecW4axpUkku68TzQyvdpd4zsXjd70CXAOuBnxl3IEC2rqtYu7PgIcEdVvbU7E+uV4y5oEFX1LeAY+IdbfmwAbhprUUOQZBHw74GjqurFJNfTO2vuU2MtbABJjgZ+i94V+z8A7khyW1V9e7yV7Tl79RF9ksOB04GPj7sW7VySnwFOAq4CqKofVNWz461qqE4GHq6q7467kCFZAOyfZAG9P8h7+zUwvwjcXVXfr6qtwJ8AZ465pj1qrw564A+A/wT8cNyFDFEBX0qyprtNRAt+DngK+GQ3zfbxJAeMu6ghOhu4btxFDENVbQA+BDwKPAFsrqovjbeqgd0PnJTk1UleCZzGyy/obN5eG/RJfg14sqrWjLuWITuxqo6ld+fPi5OcNO6ChmABcCxwZVX9EvACsHK8JQ1HNw3168Dnxl3LMCQ5mN4NCF8PvA44IMnbxlvVYKpqHfBB4E7gDuCvgK1jLWoP22uDHjgR+PUk6+ndHfNfJrlmvCUNrqo2do9P0pvzPX68FQ3F48DjVfWX3fMb6AV/C04F7q2qTeMuZEh+FfhOVT1VVX8PfB7452OuaWBVdVVVHVtVJwFPA/vM/DzsxUFfVZdW1eFVtYTeW+f/W1V79ZFHkgOSvGrbMvAmem8792pV9TfAY0ne0K06GXhwjCUN0zk0Mm3TeRQ4Ickrk4Te92rdmGsaWJLXdo8/C/wGbX3PZtXCWTctmQBu6v1+sQD4TFXdMd6ShubfAdd2Ux2PABeMuZ6BdfO9/wq4aNy1DEtV/WWSG4B76U1vfIM2bh1wY5JXA38PXFxVz4y7oD3JWyBIUuP22qkbSdLcGPSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcf8fN7TfOfuIR74AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAF75JREFUeJzt3X+UXGV9x/H3p6QqZNsECE7TgKy0kSrZmpI5SGvlzBargG0DHqlQxIQfLp6Dp7amPUbaU2yVNlajtccWGwXBqllQRBCokuZ0xV+0bjS6wUBJcIUkuOFHSFjIoV389o+5W4dlsjOZe29m8+TzOmfO3HnuM8/9PtnNZ+/cuTNXEYGZmaXr57pdgJmZlctBb2aWOAe9mVniHPRmZolz0JuZJc5Bb2aWOAe9zXiShiRdeoC2FZJ+dR/rlkv6RonbLnV8O3Q56M3MEuegNzNLnIPeDihJo5LeI+mHknZJ+pSkF2XrlkraKGmPpK2Szmh46vGSvinpSUl3SprXMOapkr4l6QlJ35dUa1g3JOn92fpxSV+WdLSkz2bb+Y6k3illniXpAUmPSvqgpLb/n0j6uKQPTWm7RdK7suWV2dyezP4NztnHOL3ZYaRZU+ZyacPjiyVtzv4dvyrp+HbrtEOLg9664QLg9cCvAC8D/lLSKcCngT8H5gKnAaMNz/kj4CLgxcALgD8DkLQAuB14P3BU1n6TpGMannsecCGwINvmt4FPZf03A1dOqe8coAqcDCwFLt6PuX0OeLMkZfUdCbwOGMzWbwVeA8wB/hr4jKT5+zE+2bhnA1cAbwSOAb4OrN3fcezQ4KC3bvhYRDwUEY8DVwHnA5cA10bEuoj4aURsj4h7G57zqYj474jYC9wILM7a3wLcERF3ZM9bBwwDZ0157taI2A38G7A1Iv49IiaAzwO/MaW+D0TE4xHxIPAPWX3t+joQ1MMc4E3AtyNiB0BEfD4idmS13gDcD5yyH+NPugz4u4jYnM3jb4HF3qu3Zhz01g0PNSz/GPhl4Djqe7v78pOG5aeBnmz5eODc7LDNE5KeAH4baNxLHmtY3tvkcQ/P1ay+tkT9WwIH+dkfhz8CPju5XtJbs8NTk7UuAuY9f6SWjgc+2jDO44Cov2oxew4HvXXDcQ3LLwF2UA/XX+lgrIeAf42IuQ232RGxquD69sda4E3Z3vWrgJsAssefAN4BHB0Rc4FN1AN6qqey+yMa2n6pYfkh4LIp8z48Ir61n7XaIcBBb91wuaRjJR1F/TjzDcA1wEWSTpf0c5IWSPq1Nsb6DPD7kl4v6TBJL5JUk3Rsjvr+XNKRko4D3pnV17aI+B7wCPBJ4KsR8US2ajb1wzqPAEi6iPoefbMxHgG2A2/J5nUxz/1D+HHgPZJOysaaI+nc/anTDh0OeuuGzwF3Ag9kt/dHxH9Rf7P1I8Bu4GvUD09MKyIeov6G6RXUA/Qh6m/o5vndvgXYAGyk/kbvNR2MsRZ4LfW5Ttb6Q2A19TeDx4A+4JvTjPE26nN5DDgJ+P+99Yi4GfgAMChpD/VXBmd2UKcdAuQLj9iBJGkUuDQi/r3btZgdKrxHb2aWuFmtu5hZI0mvoX6a5vNExNQzeMy6zoduzMwS50M3ZmaJmxGHbubNmxe9vb3dLuM5nnrqKWbPnt3tMkrlOabBc0xDJ3PcsGHDoxFxTKt+MyLoe3t7GR4e7nYZzzE0NEStVut2GaXyHNPgOaahkzlK+nE7/XzoxswscQ56M7PEOejNzBLnoDczS5yD3swscQ56M7PEtQx6ScdJ+o/s2pT3SHpn1n6UpHWS7s/uj8zaJekfJW2R9ANJJ5c9CTMz27d29ugngBUR8XLgVOrfJf4KYCWwPiIWAuuzx1D/qtSF2W0AuLrwqs3MrG0tgz4iHo6I72bLT1K/mPIC6t8Bfn3W7Xrg7Gx5KfDpqLsbmNvJxY/NzKwY+/WlZpJ6gbuoXxXnwexSaJPrdkXEkZJuA1ZFxDey9vXAuyNieMpYA9T3+KlUKksGBwdzTqVY4+Pj9PSk/UWEncxxZPvukqqZXt+COR09zz/HNHiOzfX392+IiGqrfm1/BYKkHurXvvyTiNgjNbvMZb1rk7bn/TWJiDXAGoBqtRoz7ePN/sh1c8tX3l5OMS2MXlDr6Hn+OabBc8ynrbNuJP089ZD/bER8MWsemzwkk93vzNq38dyLKx/L/l9c2czMCtLOWTeifs3MzRHx4YZVtwLLsuVl1K+zOdn+1uzsm1OB3RHxcIE1m5nZfmjn0M2rgQuBEUkbs7YrgFXAjZIuAR4EJq9AfwdwFrAFeJr6BZ/NzKxLWgZ99qbqvg7In96kfwCX56zLzMwK4k/GmpklzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeIc9GZmiWvnUoLXStopaVND2w2SNma30ckrT0nqlbS3Yd3HyyzezMxaa+dSgtcBHwM+PdkQEW+eXJa0Gtjd0H9rRCwuqkAzM8unnUsJ3iWpt9m67MLhfwj8TrFlmZlZUfIeo38NMBYR9ze0vVTS9yR9TdJrco5vZmY5qX4t7xad6nv0t0XEointVwNbImJ19viFQE9EPCZpCfAl4KSI2NNkzAFgAKBSqSwZHBzMOZVijY+P09PT0+0yStXJHEe2727dqQR9C+Z09Dz/HNPgOTbX39+/ISKqrfq1c4y+KUmzgDcCSybbIuIZ4JlseYOkrcDLgOGpz4+INcAagGq1GrVardNSSjE0NMRMq6loncxx+crbyymmhdELah09zz/HNHiO+eQ5dPNa4N6I2DbZIOkYSYdlyycAC4EH8pVoZmZ5tHN65Vrg28CJkrZJuiRbdR6wdkr304AfSPo+8AXg7RHxeJEFm5nZ/mnnrJvz99G+vEnbTcBN+csyM7Oi+JOxZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJ6/gDU9Y9vQV9aGlF30TXPgBlZgeO9+jNzBLnoDczS5yD3swscT5GbweFTt+XKOJ9iNFVb8j1fLNu8x69mVniHPRmZolz0JuZJc5Bb2aWOAe9mVniHPRmZolz0JuZJa6dSwleK2mnpE0Nbe+VtF3Sxux2VsO690jaIuk+Sa8vq3AzM2tPO3v01wFnNGn/SEQszm53AEh6BfVryZ6UPeefJy8WbmZm3dEy6CPiLqDdC3wvBQYj4pmI+BGwBTglR31mZpaTIqJ1J6kXuC0iFmWP3wssB/YAw8CKiNgl6WPA3RHxmazfNcC/RcQXmow5AAwAVCqVJYODgwVMpzjj4+P09PR0u4ymRrbvLmScyuEwtreQoWasIubYt2BOMcWUZCb/rhbFc2yuv79/Q0RUW/Xr9LturgbeB0R2vxq4GFCTvk3/kkTEGmANQLVajVqt1mEp5RgaGmKm1TSpqO+QX9E3weqRtL/uqIg5jl5QK6aYkszk39WieI75dHTWTUSMRcSzEfFT4BP87PDMNuC4hq7HAjvylWhmZnl0FPSS5jc8PAeYPCPnVuA8SS+U9FJgIfBf+Uo0M7M8Wr6mlbQWqAHzJG0DrgRqkhZTPywzClwGEBH3SLoR+CEwAVweEc+WU7qZmbWjZdBHxPlNmq+Zpv9VwFV5ijIzs+L4k7FmZolz0JuZJc5Bb2aWOAe9mVniHPRmZolz0JuZJc5Bb2aWOAe9mVniHPRmZolz0JuZJc5Bb2aWOAe9mVniHPRmZolz0JuZJc5Bb2aWOAe9mVniHPRmZolrGfSSrpW0U9KmhrYPSrpX0g8k3SxpbtbeK2mvpI3Z7eNlFm9mZq21s0d/HXDGlLZ1wKKI+HXgv4H3NKzbGhGLs9vbiynTzMw61TLoI+Iu4PEpbXdGxET28G7g2BJqMzOzAigiWneSeoHbImJRk3VfBm6IiM9k/e6hvpe/B/jLiPj6PsYcAAYAKpXKksHBwc5mUJLx8XF6enq6XUZTI9t3FzJO5XAY21vIUDNWEXPsWzCnmGJKMpN/V4viOTbX39+/ISKqrfrlCnpJfwFUgTdGREh6IdATEY9JWgJ8CTgpIvZMN361Wo3h4eGWdRxIQ0ND1Gq1bpfRVO/K2wsZZ0XfBKtHZhUy1kx1MM9xdNUb2uo3k39Xi+I5NiepraDv+KwbScuA3wMuiOyvRUQ8ExGPZcsbgK3AyzrdhpmZ5ddR0Es6A3g38AcR8XRD+zGSDsuWTwAWAg8UUaiZmXWm5WtaSWuBGjBP0jbgSupn2bwQWCcJ4O7sDJvTgL+RNAE8C7w9Ih5vOrCZmR0QLYM+Is5v0nzNPvreBNyUtygzMyuOPxlrZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpa4toJe0rWSdkra1NB2lKR1ku7P7o/M2iXpHyVtkfQDSSeXVbyZmbXW7h79dcAZU9pWAusjYiGwPnsMcCb1i4IvBAaAq/OXaWZmnWor6CPiLmDqRb6XAtdny9cDZze0fzrq7gbmSppfRLFmZrb/FBHtdZR6gdsiYlH2+ImImNuwfldEHCnpNmBVRHwja18PvDsihqeMN0B9j59KpbJkcHCwgOkUZ3x8nJ6enm6X0dTI9t2FjFM5HMb2FjLUjHUwz7FvwZy2+s3k39WieI7N9ff3b4iIaqt+szquat/UpO15f00iYg2wBqBarUatViuhlM4NDQ0x02qatHzl7YWMs6JvgtUjZfwKzBwH8xxHL6i11W8m/64WxXPMJ89ZN2OTh2Sy+51Z+zbguIZ+xwI7cmzHzMxyyBP0twLLsuVlwC0N7W/Nzr45FdgdEQ/n2I6ZmeXQ1mtaSWuBGjBP0jbgSmAVcKOkS4AHgXOz7ncAZwFbgKeBiwqu2czM9kNbQR8R5+9j1elN+gZweZ6izMysOP5krJlZ4hz0ZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJc9CbmSXu4LyYptkhoLfNawOv6Jso7DrCk0ZXvaHQ8ay7vEdvZpa4jvfoJZ0I3NDQdALwV8Bc4G3AI1n7FRFxR8cVmplZLh0HfUTcBywGkHQYsB24mfo1Yj8SER8qpMIZrN2X1mZm3VTUoZvTga0R8eOCxjMzs4Kofi3vnINI1wLfjYiPSXovsBzYAwwDKyJiV5PnDAADAJVKZcng4GDuOoo0Pj5OT0/PtH1Gtu8+QNWUo3I4jO3tdhXl8hw707dgTrED5tTO/8eDXSdz7O/v3xAR1Vb9cge9pBcAO4CTImJMUgV4FAjgfcD8iLh4ujGq1WoMDw/nqqNoQ0ND1Gq1afsc7IduVvRNsHok7ROvPMfOzLSzbtr5/3iw62SOktoK+iIO3ZxJfW9+DCAixiLi2Yj4KfAJ4JQCtmFmZh0qIujPB9ZOPpA0v2HdOcCmArZhZmYdyvV6T9IRwO8ClzU0/72kxdQP3YxOWWdmZgdYrqCPiKeBo6e0XZirIjMzK5Q/GWtmljgHvZlZ4hz0ZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJc9CbmSXOQW9mlrjcl46XNAo8CTwLTEREVdJRwA1AL/XLCf5hROzKuy0zM9t/Re3R90fE4oioZo9XAusjYiGwPntsZmZdUNahm6XA9dny9cDZJW3HzMxaUETkG0D6EbALCOBfImKNpCciYm5Dn10RceSU5w0AAwCVSmXJ4OBgrjqKNj4+Tk9Pz7R9RrbvPkDVlKNyOIzt7XYV5fIcO9O3YE6xA+bUzv/Hg10nc+zv79/QcCRln3IfowdeHRE7JL0YWCfp3naeFBFrgDUA1Wo1arVaAaUUZ2hoiFY1LV95+4EppiQr+iZYPVLEr8DM5Tl2ZvSCWqHj5dXO/8eDXZlzzH3oJiJ2ZPc7gZuBU4AxSfMBsvudebdjZmadyRX0kmZL+oXJZeB1wCbgVmBZ1m0ZcEue7ZiZWefyvt6rADdLmhzrcxHxFUnfAW6UdAnwIHBuzu2YmVmHcgV9RDwAvLJJ+2PA6XnGNjOzYviTsWZmiUv7dAQz60hvl84oG131hq5sN3XeozczS5yD3swscQ56M7PEOejNzBLnoDczS5yD3swscQ56M7PEOejNzBLnoDczS5yD3swscQ56M7PEOejNzBLnoDczS5yD3swscQ56M7PEdRz0ko6T9B+SNku6R9I7s/b3StouaWN2O6u4cs3MbH/lufDIBLAiIr6bXSB8g6R12bqPRMSH8pdnZmZ5dRz0EfEw8HC2/KSkzcCCogozM7NiKCLyDyL1AncBi4B3AcuBPcAw9b3+XU2eMwAMAFQqlSWDg4O56yjS+Pg4PT090/YZ2b77AFVTjsrhMLa321WUy3M8uPQtmNO0vZ3/jwe7TubY39+/ISKqrfrlDnpJPcDXgKsi4ouSKsCjQADvA+ZHxMXTjVGtVmN4eDhXHUUbGhqiVqtN26db19Usyoq+CVaPpH3ZYM/x4LKva8a28//xYNfJHCW1FfS5zrqR9PPATcBnI+KLABExFhHPRsRPgU8Ap+TZhpmZ5ZPnrBsB1wCbI+LDDe3zG7qdA2zqvDwzM8srz+u9VwMXAiOSNmZtVwDnS1pM/dDNKHBZrgrNzCyXPGfdfANQk1V3dF6OmZkVzZ+MNTNLnIPezCxxaZyTZWZJ2Ncpyyv6Jlhe4unM+zqtMxXeozczS5yD3swscUkcuinjE6plv1Q0MztQvEdvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeKS+AoEM7M8yvgalXYdiG/OLG2PXtIZku6TtEXSyrK2Y2Zm0ysl6CUdBvwTcCbwCurXkX1FGdsyM7PplbVHfwqwJSIeiIj/AQaBpSVty8zMpqGIKH5Q6U3AGRFxafb4QuBVEfGOhj4DwED28ETgvsILyWce8Gi3iyiZ55gGzzENnczx+Ig4plWnst6MVZO25/xFiYg1wJqStp+bpOGIqHa7jjJ5jmnwHNNQ5hzLOnSzDTiu4fGxwI6StmVmZtMoK+i/AyyU9FJJLwDOA24taVtmZjaNUg7dRMSEpHcAXwUOA66NiHvK2FaJZuxhpQJ5jmnwHNNQ2hxLeTPWzMxmDn8FgplZ4hz0ZmaJc9BPIWlU0oikjZKGu11PGSTNlfQFSfdK2izpN7tdU9EknZj9DCdveyT9SbfrKpKkP5V0j6RNktZKelG3ayqDpHdmc7wnlZ+hpGsl7ZS0qaHtKEnrJN2f3R9Z1PYc9M31R8TihM/b/SjwlYj4NeCVwOYu11O4iLgv+xkuBpYATwM3d7mswkhaAPwxUI2IRdRPejivu1UVT9Ii4G3UP23/SuD3JC3sblWFuA44Y0rbSmB9RCwE1mePC+GgP8RI+kXgNOAagIj4n4h4ortVle50YGtE/LjbhRRsFnC4pFnAEaT5WZWXA3dHxNMRMQF8DTinyzXlFhF3AY9PaV4KXJ8tXw+cXdT2HPTPF8CdkjZkX9OQmhOAR4BPSfqepE9Kmt3tokp2HrC220UUKSK2Ax8CHgQeBnZHxJ3draoUm4DTJB0t6QjgLJ77YcyUVCLiYYDs/sVFDeygf75XR8TJ1L9583JJp3W7oILNAk4Gro6I3wCeosCXiDNN9oG9PwA+3+1aipQdv10KvBT4ZWC2pLd0t6riRcRm4APAOuArwPeBia4WdRBy0E8RETuy+53Uj+me0t2KCrcN2BYR/5k9/gL14E/VmcB3I2Ks24UU7LXAjyLikYj4X+CLwG91uaZSRMQ1EXFyRJxG/XDH/d2uqSRjkuYDZPc7ixrYQd9A0mxJvzC5DLyO+kvHZETET4CHJJ2YNZ0O/LCLJZXtfBI7bJN5EDhV0hGSRP3nmNyb6gCSXpzdvwR4I2n+PKH+NTHLsuVlwC1FDexPxjaQdAI/OzNjFvC5iLiqiyWVQtJi4JPAC4AHgIsiYld3qypedkz3IeCEiNjd7XqKJumvgTdTP5TxPeDSiHimu1UVT9LXgaOB/wXeFRHru1xSbpLWAjXqX008BlwJfAm4EXgJ9T/k50bE1DdsO9ueg97MLG0+dGNmljgHvZlZ4hz0ZmaJc9CbmSXOQW9mljgHvZlZ4hz0ZmaJ+z+wvZdt5q0FTAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for v,k in tpls:\n",
    "    if v>900:\n",
    "        serotonin_d.loc[serotonin_d['target_chembl_id']==k].hist('pchembl_value');\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_serotonin_assay(assay_id,d,assay_lookup,thresholds=None,archive=None):\n",
    "    if archive is None:\n",
    "        archive=defaultdict(list)\n",
    "    # these need to be converted into a classification problem.\n",
    "    # start with a pretty strict cutoff for active/inactive:\n",
    "    assay = d.loc[d['target_chembl_id']==assay_id]\n",
    "    acts = assay.loc[assay['pchembl_value']>9.0]\n",
    "    if len(acts)>=50:\n",
    "        inacts = assay.loc[assay['pchembl_value']<8.5]\n",
    "    else:\n",
    "        # but relax that if we don't end up with a reasonable number of actives:\n",
    "        acts = assay.loc[assay['pchembl_value']>8.0]\n",
    "        inacts = assay.loc[assay['pchembl_value']<7.5]\n",
    "    print('*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*')\n",
    "    print(f'assay_id {assay_id}, organism: {assay_lookup[assay_id][0]}, target: {assay_lookup[assay_id][1]}')\n",
    "    run_it_(assay_id,acts,inacts,thresholds,archive)\n",
    "    print('*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\\n\\n')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL3371, organism: Homo sapiens, target: Serotonin 6 (5-HT6) receptor\n",
      "--------- Default -----------\n",
      "ratio: 0.189 kappa: 0.654, AUC: 0.968, OOB_AUC: 0.949\n",
      "[[418  10]\n",
      " [ 32  49]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.98      0.95       428\n",
      "           1       0.83      0.60      0.70        81\n",
      "\n",
      "   micro avg       0.92      0.92      0.92       509\n",
      "   macro avg       0.88      0.79      0.83       509\n",
      "weighted avg       0.91      0.92      0.91       509\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.350 kappa: 0.803\n",
      "[[414  14]\n",
      " [ 13  68]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL224, organism: Homo sapiens, target: Serotonin 2a (5-HT2a) receptor\n",
      "--------- Default -----------\n",
      "ratio: 0.085 kappa: 0.714, AUC: 0.988, OOB_AUC: 0.958\n",
      "[[494   2]\n",
      " [ 17  26]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98       496\n",
      "           1       0.93      0.60      0.73        43\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       539\n",
      "   macro avg       0.95      0.80      0.86       539\n",
      "weighted avg       0.96      0.96      0.96       539\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.400 kappa: 0.692\n",
      "[[492   4]\n",
      " [ 17  26]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL214, organism: Homo sapiens, target: Serotonin 1a (5-HT1a) receptor\n",
      "--------- Default -----------\n",
      "ratio: 0.108 kappa: 0.471, AUC: 0.951, OOB_AUC: 0.945\n",
      "[[393   2]\n",
      " [ 28  15]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.99      0.96       395\n",
      "           1       0.88      0.35      0.50        43\n",
      "\n",
      "   micro avg       0.93      0.93      0.93       438\n",
      "   macro avg       0.91      0.67      0.73       438\n",
      "weighted avg       0.93      0.93      0.92       438\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.300 kappa: 0.672\n",
      "[[381  14]\n",
      " [ 12  31]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL3155, organism: Homo sapiens, target: Serotonin 7 (5-HT7) receptor\n",
      "--------- Default -----------\n",
      "ratio: 0.053 kappa: 0.408, AUC: 0.953, OOB_AUC: 0.918\n",
      "[[270   0]\n",
      " [ 11   4]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       270\n",
      "           1       1.00      0.27      0.42        15\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       285\n",
      "   macro avg       0.98      0.63      0.70       285\n",
      "weighted avg       0.96      0.96      0.95       285\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.350 kappa: 0.384\n",
      "[[269   1]\n",
      " [ 11   4]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL225, organism: Homo sapiens, target: Serotonin 2c (5-HT2c) receptor\n",
      "--------- Default -----------\n",
      "ratio: 0.288 kappa: 0.564, AUC: 0.914, OOB_AUC: 0.912\n",
      "[[182   4]\n",
      " [ 27  27]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      0.98      0.92       186\n",
      "           1       0.87      0.50      0.64        54\n",
      "\n",
      "   micro avg       0.87      0.87      0.87       240\n",
      "   macro avg       0.87      0.74      0.78       240\n",
      "weighted avg       0.87      0.87      0.86       240\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.671\n",
      "[[161  25]\n",
      " [  6  48]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1833, organism: Homo sapiens, target: Serotonin 2b (5-HT2b) receptor\n",
      "--------- Default -----------\n",
      "ratio: 0.168 kappa: 0.545, AUC: 0.965, OOB_AUC: 0.893\n",
      "[[132   1]\n",
      " [ 13  10]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.99      0.95       133\n",
      "           1       0.91      0.43      0.59        23\n",
      "\n",
      "   micro avg       0.91      0.91      0.91       156\n",
      "   macro avg       0.91      0.71      0.77       156\n",
      "weighted avg       0.91      0.91      0.90       156\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.692\n",
      "[[130   3]\n",
      " [  8  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "serotonin_archive = defaultdict(list)\n",
    "\n",
    "for v,k in tpls:\n",
    "    if v>900:\n",
    "        run_serotonin_assay(k,serotonin_d,assay_lookup,archive=serotonin_archive)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The \"Dataset 1\" datasets\n",
    "\n",
    "Now we'll work with the \"Dataset 1\" datasets from the benchmarking set. These are collections of diverse actives from various ChEMBL targets together with assumed inactives pulled from an older version of ZINC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "80\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['../../Code/benchmarking_platform/compounds/ChEMBL/cmp_list_ChEMBL_10198_actives.dat.gz',\n",
       " '../../Code/benchmarking_platform/compounds/ChEMBL/cmp_list_ChEMBL_10980_actives.dat.gz',\n",
       " '../../Code/benchmarking_platform/compounds/ChEMBL/cmp_list_ChEMBL_11279_actives.dat.gz',\n",
       " '../../Code/benchmarking_platform/compounds/ChEMBL/cmp_list_ChEMBL_87_actives.dat.gz',\n",
       " '../../Code/benchmarking_platform/compounds/ChEMBL/cmp_list_ChEMBL_12252_actives.dat.gz']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import glob\n",
    "datasets = glob.glob('../../Code/benchmarking_platform/compounds/ChEMBL/cmp_list*_actives.dat.gz')\n",
    "print(len(datasets))\n",
    "random.seed(0xf00d)\n",
    "random.shuffle(datasets)\n",
    "keep = datasets[:20]\n",
    "keep[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "with gzip.open('../../Code/benchmarking_platform/compounds/ChEMBL/cmp_list_ChEMBL_zinc_decoys.dat.gz') as inf:\n",
    "    inactive_df = pd.read_csv(inf,sep='\\t')\n",
    "\n",
    "chembl_active_sets = {}\n",
    "for fn in datasets:\n",
    "    nm = re.search(r'cmp_list_(ChEMBL_[0-9]*)_actives',fn).groups()[0]\n",
    "    with gzip.open(fn) as inf:\n",
    "        chembl_active_sets[nm] = pd.read_csv(inf,sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_dataset1_assay(assay_id,chembl_active_sets,inactive_df,factor=20,thresholds=None,archive=None):\n",
    "    if archive is None:\n",
    "        archive=defaultdict(list)\n",
    "    assay = chembl_active_sets[assay_id]\n",
    "    acts = assay.rename(index=str,columns={'SMILES':'canonical_smiles','# _Name':'compound_chembl_id'})\n",
    "    inacts = inactive_df.sample(n=factor*len(acts),random_state=hash(assay_id)%0xf00d).rename(index=str,columns={'SMILES':'canonical_smiles','# _Name':'compound_chembl_id'})\n",
    "\n",
    "    print('*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*')\n",
    "    print(f'assay_id {assay_id}')\n",
    "\n",
    "    run_it_(assay_id,acts,inacts,thresholds,archive)    \n",
    "    print('*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\\n\\n')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10198\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.700, AUC: 1.000, OOB_AUC: 0.981\n",
      "[[400   0]\n",
      " [  9  11]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99       400\n",
      "           1       1.00      0.55      0.71        20\n",
      "\n",
      "   micro avg       0.98      0.98      0.98       420\n",
      "   macro avg       0.99      0.78      0.85       420\n",
      "weighted avg       0.98      0.98      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.945\n",
      "[[400   0]\n",
      " [  2  18]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10980\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.994, OOB_AUC: 0.966\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.802\n",
      "[[398   2]\n",
      " [  5  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11279\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.741, AUC: 0.965, OOB_AUC: 0.977\n",
      "[[400   0]\n",
      " [  8  12]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99       400\n",
      "           1       1.00      0.60      0.75        20\n",
      "\n",
      "   micro avg       0.98      0.98      0.98       420\n",
      "   macro avg       0.99      0.80      0.87       420\n",
      "weighted avg       0.98      0.98      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.973\n",
      "[[400   0]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_87\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.908, OOB_AUC: 0.910\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.307\n",
      "[[399   1]\n",
      " [ 16   4]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_12252\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.252, AUC: 0.983, OOB_AUC: 0.978\n",
      "[[400   0]\n",
      " [ 17   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.15      0.26        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.57      0.62       420\n",
      "weighted avg       0.96      0.96      0.94       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.802\n",
      "[[398   2]\n",
      " [  5  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10280\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.323, AUC: 0.986, OOB_AUC: 0.976\n",
      "[[400   0]\n",
      " [ 16   4]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.20      0.33        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.60      0.66       420\n",
      "weighted avg       0.96      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.843\n",
      "[[397   3]\n",
      " [  3  17]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_17045\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.913, OOB_AUC: 0.886\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.100 kappa: 0.473\n",
      "[[387  13]\n",
      " [  9  11]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10188\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.000, AUC: 0.945, OOB_AUC: 0.926\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.746\n",
      "[[397   3]\n",
      " [  6  14]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10193\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.388, AUC: 0.972, OOB_AUC: 0.979\n",
      "[[400   0]\n",
      " [ 15   5]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.25      0.40        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.62      0.69       420\n",
      "weighted avg       0.97      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.780\n",
      "[[397   3]\n",
      " [  5  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11365\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.945, OOB_AUC: 0.898\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.100 kappa: 0.505\n",
      "[[379  21]\n",
      " [  5  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_93\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.252, AUC: 0.986, OOB_AUC: 0.977\n",
      "[[400   0]\n",
      " [ 17   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.15      0.26        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.57      0.62       420\n",
      "weighted avg       0.96      0.96      0.94       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.100 kappa: 0.659\n",
      "[[385  15]\n",
      " [  2  18]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10773\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.741, AUC: 1.000, OOB_AUC: 0.998\n",
      "[[400   0]\n",
      " [  8  12]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99       400\n",
      "           1       1.00      0.60      0.75        20\n",
      "\n",
      "   micro avg       0.98      0.98      0.98       420\n",
      "   macro avg       0.99      0.80      0.87       420\n",
      "weighted avg       0.98      0.98      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 1.000\n",
      "[[400   0]\n",
      " [  0  20]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10417\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.915, AUC: 1.000, OOB_AUC: 1.000\n",
      "[[400   0]\n",
      " [  3  17]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00       400\n",
      "           1       1.00      0.85      0.92        20\n",
      "\n",
      "   micro avg       0.99      0.99      0.99       420\n",
      "   macro avg       1.00      0.93      0.96       420\n",
      "weighted avg       0.99      0.99      0.99       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 1.000\n",
      "[[400   0]\n",
      " [  0  20]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_36\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.388, AUC: 0.993, OOB_AUC: 0.994\n",
      "[[400   0]\n",
      " [ 15   5]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.25      0.40        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.62      0.69       420\n",
      "weighted avg       0.97      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.802\n",
      "[[398   2]\n",
      " [  5  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_134\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.388, AUC: 1.000, OOB_AUC: 0.994\n",
      "[[400   0]\n",
      " [ 15   5]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.25      0.40        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.62      0.69       420\n",
      "weighted avg       0.97      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.950\n",
      "[[398   2]\n",
      " [  0  20]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_20014\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.559, AUC: 0.993, OOB_AUC: 0.994\n",
      "[[400   0]\n",
      " [ 12   8]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.99       400\n",
      "           1       1.00      0.40      0.57        20\n",
      "\n",
      "   micro avg       0.97      0.97      0.97       420\n",
      "   macro avg       0.99      0.70      0.78       420\n",
      "weighted avg       0.97      0.97      0.97       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.895\n",
      "[[398   2]\n",
      " [  2  18]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_114\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.091, AUC: 0.957, OOB_AUC: 0.959\n",
      "[[400   0]\n",
      " [ 19   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       1.00      0.05      0.10        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.98      0.53      0.54       420\n",
      "weighted avg       0.96      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.489\n",
      "[[391   9]\n",
      " [ 10  10]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_12209\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.323, AUC: 0.969, OOB_AUC: 0.976\n",
      "[[400   0]\n",
      " [ 16   4]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.20      0.33        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.60      0.66       420\n",
      "weighted avg       0.96      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.755\n",
      "[[399   1]\n",
      " [  7  13]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_25\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.323, AUC: 0.984, OOB_AUC: 0.974\n",
      "[[400   0]\n",
      " [ 16   4]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.20      0.33        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.60      0.66       420\n",
      "weighted avg       0.96      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.755\n",
      "[[399   1]\n",
      " [  7  13]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11359\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.700, AUC: 0.991, OOB_AUC: 0.983\n",
      "[[400   0]\n",
      " [  9  11]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99       400\n",
      "           1       1.00      0.55      0.71        20\n",
      "\n",
      "   micro avg       0.98      0.98      0.98       420\n",
      "   macro avg       0.99      0.78      0.85       420\n",
      "weighted avg       0.98      0.98      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.350 kappa: 0.851\n",
      "[[400   0]\n",
      " [  5  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_51\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.175, AUC: 0.937, OOB_AUC: 0.980\n",
      "[[400   0]\n",
      " [ 18   2]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.10      0.18        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.55      0.58       420\n",
      "weighted avg       0.96      0.96      0.94       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.704\n",
      "[[395   5]\n",
      " [  6  14]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10927\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.816, AUC: 0.999, OOB_AUC: 0.999\n",
      "[[400   0]\n",
      " [  6  14]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99       400\n",
      "           1       1.00      0.70      0.82        20\n",
      "\n",
      "   micro avg       0.99      0.99      0.99       420\n",
      "   macro avg       0.99      0.85      0.91       420\n",
      "weighted avg       0.99      0.99      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.973\n",
      "[[400   0]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10475\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.884, AUC: 1.000, OOB_AUC: 1.000\n",
      "[[400   0]\n",
      " [  4  16]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00       400\n",
      "           1       1.00      0.80      0.89        20\n",
      "\n",
      "   micro avg       0.99      0.99      0.99       420\n",
      "   macro avg       1.00      0.90      0.94       420\n",
      "weighted avg       0.99      0.99      0.99       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 1.000\n",
      "[[400   0]\n",
      " [  0  20]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11631\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.700, AUC: 0.995, OOB_AUC: 0.980\n",
      "[[400   0]\n",
      " [  9  11]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99       400\n",
      "           1       1.00      0.55      0.71        20\n",
      "\n",
      "   micro avg       0.98      0.98      0.98       420\n",
      "   macro avg       0.99      0.78      0.85       420\n",
      "weighted avg       0.98      0.98      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.828\n",
      "[[395   5]\n",
      " [  2  18]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10434\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.091, AUC: 0.989, OOB_AUC: 0.962\n",
      "[[400   0]\n",
      " [ 19   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       1.00      0.05      0.10        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.98      0.53      0.54       420\n",
      "weighted avg       0.96      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.835\n",
      "[[398   2]\n",
      " [  4  16]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10378\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.252, AUC: 0.988, OOB_AUC: 0.976\n",
      "[[400   0]\n",
      " [ 17   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.15      0.26        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.57      0.62       420\n",
      "weighted avg       0.96      0.96      0.94       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.802\n",
      "[[398   2]\n",
      " [  5  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_116\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.700, AUC: 0.998, OOB_AUC: 0.998\n",
      "[[400   0]\n",
      " [  9  11]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99       400\n",
      "           1       1.00      0.55      0.71        20\n",
      "\n",
      "   micro avg       0.98      0.98      0.98       420\n",
      "   macro avg       0.99      0.78      0.85       420\n",
      "weighted avg       0.98      0.98      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.948\n",
      "[[399   1]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_61\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.091, AUC: 0.958, OOB_AUC: 0.960\n",
      "[[400   0]\n",
      " [ 19   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       1.00      0.05      0.10        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.98      0.53      0.54       420\n",
      "weighted avg       0.96      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.467\n",
      "[[398   2]\n",
      " [ 13   7]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_12261\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.388, AUC: 0.986, OOB_AUC: 0.974\n",
      "[[400   0]\n",
      " [ 15   5]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.25      0.40        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.62      0.69       420\n",
      "weighted avg       0.97      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.884\n",
      "[[400   0]\n",
      " [  4  16]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11085\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.388, AUC: 0.991, OOB_AUC: 0.992\n",
      "[[400   0]\n",
      " [ 15   5]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.25      0.40        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.62      0.69       420\n",
      "weighted avg       0.97      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.973\n",
      "[[400   0]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_20174\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.559, AUC: 0.999, OOB_AUC: 1.000\n",
      "[[400   0]\n",
      " [ 12   8]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.99       400\n",
      "           1       1.00      0.40      0.57        20\n",
      "\n",
      "   micro avg       0.97      0.97      0.97       420\n",
      "   macro avg       0.99      0.70      0.78       420\n",
      "weighted avg       0.97      0.97      0.97       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.915\n",
      "[[400   0]\n",
      " [  3  17]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_19905\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.986, OOB_AUC: 0.973\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.835\n",
      "[[398   2]\n",
      " [  4  16]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11575\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.609, AUC: 0.949, OOB_AUC: 0.971\n",
      "[[400   0]\n",
      " [ 11   9]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.99       400\n",
      "           1       1.00      0.45      0.62        20\n",
      "\n",
      "   micro avg       0.97      0.97      0.97       420\n",
      "   macro avg       0.99      0.72      0.80       420\n",
      "weighted avg       0.97      0.97      0.97       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.872\n",
      "[[397   3]\n",
      " [  2  18]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11336\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.741, AUC: 0.999, OOB_AUC: 0.999\n",
      "[[400   0]\n",
      " [  8  12]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99       400\n",
      "           1       1.00      0.60      0.75        20\n",
      "\n",
      "   micro avg       0.98      0.98      0.98       420\n",
      "   macro avg       0.99      0.80      0.87       420\n",
      "weighted avg       0.98      0.98      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.900\n",
      "[[397   3]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_52\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.960, OOB_AUC: 0.946\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.611\n",
      "[[398   2]\n",
      " [ 10  10]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11536\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.816, AUC: 1.000, OOB_AUC: 0.994\n",
      "[[400   0]\n",
      " [  6  14]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99       400\n",
      "           1       1.00      0.70      0.82        20\n",
      "\n",
      "   micro avg       0.99      0.99      0.99       420\n",
      "   macro avg       0.99      0.85      0.91       420\n",
      "weighted avg       0.99      0.99      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.974\n",
      "[[399   1]\n",
      " [  0  20]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_108\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.961, OOB_AUC: 0.938\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.564\n",
      "[[391   9]\n",
      " [  8  12]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_107\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.000, AUC: 0.949, OOB_AUC: 0.950\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.633\n",
      "[[399   1]\n",
      " [ 10  10]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_130\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.091, AUC: 0.934, OOB_AUC: 0.966\n",
      "[[400   0]\n",
      " [ 19   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       1.00      0.05      0.10        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.98      0.53      0.54       420\n",
      "weighted avg       0.96      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.694\n",
      "[[398   2]\n",
      " [  8  12]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_100126\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.780, AUC: 0.976, OOB_AUC: 0.981\n",
      "[[400   0]\n",
      " [  7  13]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99       400\n",
      "           1       1.00      0.65      0.79        20\n",
      "\n",
      "   micro avg       0.98      0.98      0.98       420\n",
      "   macro avg       0.99      0.82      0.89       420\n",
      "weighted avg       0.98      0.98      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.835\n",
      "[[398   2]\n",
      " [  4  16]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11534\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.091, AUC: 0.999, OOB_AUC: 0.987\n",
      "[[400   0]\n",
      " [ 19   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       1.00      0.05      0.10        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.98      0.53      0.54       420\n",
      "weighted avg       0.96      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.895\n",
      "[[398   2]\n",
      " [  2  18]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_15\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.323, AUC: 0.976, OOB_AUC: 0.959\n",
      "[[400   0]\n",
      " [ 16   4]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.20      0.33        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.60      0.66       420\n",
      "weighted avg       0.96      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.724\n",
      "[[396   4]\n",
      " [  6  14]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11265\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.973, AUC: 1.000, OOB_AUC: 0.989\n",
      "[[400   0]\n",
      " [  1  19]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00       400\n",
      "           1       1.00      0.95      0.97        20\n",
      "\n",
      "   micro avg       1.00      1.00      1.00       420\n",
      "   macro avg       1.00      0.97      0.99       420\n",
      "weighted avg       1.00      1.00      1.00       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.948\n",
      "[[399   1]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_12911\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.946, OOB_AUC: 0.884\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.100 kappa: 0.563\n",
      "[[384  16]\n",
      " [  5  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_276\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.816, AUC: 0.999, OOB_AUC: 0.987\n",
      "[[400   0]\n",
      " [  6  14]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99       400\n",
      "           1       1.00      0.70      0.82        20\n",
      "\n",
      "   micro avg       0.99      0.99      0.99       420\n",
      "   macro avg       0.99      0.85      0.91       420\n",
      "weighted avg       0.99      0.99      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.973\n",
      "[[400   0]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_12679\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.945, AUC: 1.000, OOB_AUC: 1.000\n",
      "[[400   0]\n",
      " [  2  18]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00       400\n",
      "           1       1.00      0.90      0.95        20\n",
      "\n",
      "   micro avg       1.00      1.00      1.00       420\n",
      "   macro avg       1.00      0.95      0.97       420\n",
      "weighted avg       1.00      1.00      1.00       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.300 kappa: 0.973\n",
      "[[400   0]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11489\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.852, OOB_AUC: 0.912\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.417\n",
      "[[395   5]\n",
      " [ 13   7]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_12670\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.091, AUC: 0.990, OOB_AUC: 0.980\n",
      "[[400   0]\n",
      " [ 19   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       1.00      0.05      0.10        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.98      0.53      0.54       420\n",
      "weighted avg       0.96      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.859\n",
      "[[399   1]\n",
      " [  4  16]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_13001\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.091, AUC: 0.913, OOB_AUC: 0.962\n",
      "[[400   0]\n",
      " [ 19   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       1.00      0.05      0.10        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.98      0.53      0.54       420\n",
      "weighted avg       0.96      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.611\n",
      "[[398   2]\n",
      " [ 10  10]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10579\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.851, AUC: 1.000, OOB_AUC: 1.000\n",
      "[[400   0]\n",
      " [  5  15]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99       400\n",
      "           1       1.00      0.75      0.86        20\n",
      "\n",
      "   micro avg       0.99      0.99      0.99       420\n",
      "   macro avg       0.99      0.88      0.93       420\n",
      "weighted avg       0.99      0.99      0.99       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 1.000\n",
      "[[400   0]\n",
      " [  0  20]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11682\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.559, AUC: 0.996, OOB_AUC: 0.997\n",
      "[[400   0]\n",
      " [ 12   8]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.99       400\n",
      "           1       1.00      0.40      0.57        20\n",
      "\n",
      "   micro avg       0.97      0.97      0.97       420\n",
      "   macro avg       0.99      0.70      0.78       420\n",
      "weighted avg       0.97      0.97      0.97       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.973\n",
      "[[400   0]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_8\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.323, AUC: 0.996, OOB_AUC: 0.979\n",
      "[[400   0]\n",
      " [ 16   4]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.20      0.33        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.60      0.66       420\n",
      "weighted avg       0.96      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.812\n",
      "[[397   3]\n",
      " [  4  16]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_43\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.175, AUC: 0.875, OOB_AUC: 0.868\n",
      "[[400   0]\n",
      " [ 18   2]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.10      0.18        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.55      0.58       420\n",
      "weighted avg       0.96      0.96      0.94       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.518\n",
      "[[398   2]\n",
      " [ 12   8]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_90\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.252, AUC: 0.986, OOB_AUC: 0.978\n",
      "[[400   0]\n",
      " [ 17   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.15      0.26        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.57      0.62       420\n",
      "weighted avg       0.96      0.96      0.94       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.710\n",
      "[[397   3]\n",
      " [  7  13]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_126\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.091, AUC: 0.964, OOB_AUC: 0.966\n",
      "[[400   0]\n",
      " [ 19   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       1.00      0.05      0.10        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.98      0.53      0.54       420\n",
      "weighted avg       0.96      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.672\n",
      "[[397   3]\n",
      " [  8  12]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_219\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.956, OOB_AUC: 0.980\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.591\n",
      "[[397   3]\n",
      " [ 10  10]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_104\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.000, AUC: 0.996, OOB_AUC: 0.962\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.816\n",
      "[[400   0]\n",
      " [  6  14]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_121\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.000, AUC: 0.969, OOB_AUC: 0.965\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.527\n",
      "[[391   9]\n",
      " [  9  11]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11488\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.915, AUC: 1.000, OOB_AUC: 0.991\n",
      "[[400   0]\n",
      " [  3  17]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00       400\n",
      "           1       1.00      0.85      0.92        20\n",
      "\n",
      "   micro avg       0.99      0.99      0.99       420\n",
      "   macro avg       1.00      0.93      0.96       420\n",
      "weighted avg       0.99      0.99      0.99       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.973\n",
      "[[400   0]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_18061\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.449, AUC: 0.972, OOB_AUC: 0.989\n",
      "[[400   0]\n",
      " [ 14   6]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98       400\n",
      "           1       1.00      0.30      0.46        20\n",
      "\n",
      "   micro avg       0.97      0.97      0.97       420\n",
      "   macro avg       0.98      0.65      0.72       420\n",
      "weighted avg       0.97      0.97      0.96       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.973\n",
      "[[400   0]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_28\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.816, AUC: 1.000, OOB_AUC: 0.998\n",
      "[[400   0]\n",
      " [  6  14]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99       400\n",
      "           1       1.00      0.70      0.82        20\n",
      "\n",
      "   micro avg       0.99      0.99      0.99       420\n",
      "   macro avg       0.99      0.85      0.91       420\n",
      "weighted avg       0.99      0.99      0.98       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.974\n",
      "[[399   1]\n",
      " [  0  20]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_12968\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 1.000, AUC: 1.000, OOB_AUC: 1.000\n",
      "[[400   0]\n",
      " [  0  20]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00       400\n",
      "           1       1.00      1.00      1.00        20\n",
      "\n",
      "   micro avg       1.00      1.00      1.00       420\n",
      "   macro avg       1.00      1.00      1.00       420\n",
      "weighted avg       1.00      1.00      1.00       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.300 kappa: 1.000\n",
      "[[400   0]\n",
      " [  0  20]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_12952\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.175, AUC: 0.977, OOB_AUC: 0.985\n",
      "[[400   0]\n",
      " [ 18   2]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.10      0.18        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.55      0.58       420\n",
      "weighted avg       0.96      0.96      0.94       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.758\n",
      "[[396   4]\n",
      " [  5  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10498\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.323, AUC: 0.987, OOB_AUC: 0.996\n",
      "[[400   0]\n",
      " [ 16   4]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.20      0.33        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.60      0.66       420\n",
      "weighted avg       0.96      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.835\n",
      "[[398   2]\n",
      " [  4  16]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_72\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.323, AUC: 0.926, OOB_AUC: 0.969\n",
      "[[400   0]\n",
      " [ 16   4]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.20      0.33        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.60      0.66       420\n",
      "weighted avg       0.96      0.96      0.95       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.791\n",
      "[[399   1]\n",
      " [  6  14]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_237\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.851, AUC: 0.988, OOB_AUC: 0.994\n",
      "[[400   0]\n",
      " [  5  15]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99       400\n",
      "           1       1.00      0.75      0.86        20\n",
      "\n",
      "   micro avg       0.99      0.99      0.99       420\n",
      "   macro avg       0.99      0.88      0.93       420\n",
      "weighted avg       0.99      0.99      0.99       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.300 kappa: 0.895\n",
      "[[398   2]\n",
      " [  2  18]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_259\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.968, OOB_AUC: 0.914\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.611\n",
      "[[398   2]\n",
      " [ 10  10]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11140\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.252, AUC: 0.981, OOB_AUC: 0.985\n",
      "[[400   0]\n",
      " [ 17   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.15      0.26        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.57      0.62       420\n",
      "weighted avg       0.96      0.96      0.94       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.758\n",
      "[[396   4]\n",
      " [  5  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11442\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.915, AUC: 1.000, OOB_AUC: 0.999\n",
      "[[400   0]\n",
      " [  3  17]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00       400\n",
      "           1       1.00      0.85      0.92        20\n",
      "\n",
      "   micro avg       0.99      0.99      0.99       420\n",
      "   macro avg       1.00      0.93      0.96       420\n",
      "weighted avg       0.99      0.99      0.99       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 1.000\n",
      "[[400   0]\n",
      " [  0  20]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_12840\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.609, AUC: 0.997, OOB_AUC: 0.992\n",
      "[[400   0]\n",
      " [ 11   9]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.99       400\n",
      "           1       1.00      0.45      0.62        20\n",
      "\n",
      "   micro avg       0.97      0.97      0.97       420\n",
      "   macro avg       0.99      0.72      0.80       420\n",
      "weighted avg       0.97      0.97      0.97       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.851\n",
      "[[400   0]\n",
      " [  5  15]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10752\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.449, AUC: 0.995, OOB_AUC: 0.986\n",
      "[[400   0]\n",
      " [ 14   6]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98       400\n",
      "           1       1.00      0.30      0.46        20\n",
      "\n",
      "   micro avg       0.97      0.97      0.97       420\n",
      "   macro avg       0.98      0.65      0.72       420\n",
      "weighted avg       0.97      0.97      0.96       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.973\n",
      "[[400   0]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_100579\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.506, AUC: 1.000, OOB_AUC: 0.998\n",
      "[[400   0]\n",
      " [ 13   7]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98       400\n",
      "           1       1.00      0.35      0.52        20\n",
      "\n",
      "   micro avg       0.97      0.97      0.97       420\n",
      "   macro avg       0.98      0.68      0.75       420\n",
      "weighted avg       0.97      0.97      0.96       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.900\n",
      "[[397   3]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_234\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.559, AUC: 0.993, OOB_AUC: 0.983\n",
      "[[400   0]\n",
      " [ 12   8]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.99       400\n",
      "           1       1.00      0.40      0.57        20\n",
      "\n",
      "   micro avg       0.97      0.97      0.97       420\n",
      "   macro avg       0.99      0.70      0.78       420\n",
      "weighted avg       0.97      0.97      0.97       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.915\n",
      "[[400   0]\n",
      " [  3  17]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_105\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.449, AUC: 1.000, OOB_AUC: 0.992\n",
      "[[400   0]\n",
      " [ 14   6]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98       400\n",
      "           1       1.00      0.30      0.46        20\n",
      "\n",
      "   micro avg       0.97      0.97      0.97       420\n",
      "   macro avg       0.98      0.65      0.72       420\n",
      "weighted avg       0.97      0.97      0.96       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.973\n",
      "[[400   0]\n",
      " [  1  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_11225\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.609, AUC: 1.000, OOB_AUC: 0.988\n",
      "[[400   0]\n",
      " [ 11   9]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.99       400\n",
      "           1       1.00      0.45      0.62        20\n",
      "\n",
      "   micro avg       0.97      0.97      0.97       420\n",
      "   macro avg       0.99      0.72      0.80       420\n",
      "weighted avg       0.97      0.97      0.97       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.919\n",
      "[[399   1]\n",
      " [  2  18]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_65\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.252, AUC: 0.995, OOB_AUC: 0.981\n",
      "[[400   0]\n",
      " [ 17   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.15      0.26        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.57      0.62       420\n",
      "weighted avg       0.96      0.96      0.94       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.755\n",
      "[[399   1]\n",
      " [  7  13]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_165\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.910, OOB_AUC: 0.933\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.100 kappa: 0.370\n",
      "[[377  23]\n",
      " [  9  11]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_100166\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.656, AUC: 1.000, OOB_AUC: 0.996\n",
      "[[400   0]\n",
      " [ 10  10]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99       400\n",
      "           1       1.00      0.50      0.67        20\n",
      "\n",
      "   micro avg       0.98      0.98      0.98       420\n",
      "   macro avg       0.99      0.75      0.83       420\n",
      "weighted avg       0.98      0.98      0.97       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.915\n",
      "[[400   0]\n",
      " [  3  17]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_10260\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.252, AUC: 0.991, OOB_AUC: 0.969\n",
      "[[400   0]\n",
      " [ 17   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       400\n",
      "           1       1.00      0.15      0.26        20\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       420\n",
      "   macro avg       0.98      0.57      0.62       420\n",
      "weighted avg       0.96      0.96      0.94       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.821\n",
      "[[396   4]\n",
      " [  3  17]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id ChEMBL_100\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.960, OOB_AUC: 0.934\n",
      "[[400   0]\n",
      " [ 20   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       400\n",
      "           1       0.00      0.00      0.00        20\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       420\n",
      "   macro avg       0.48      0.50      0.49       420\n",
      "weighted avg       0.91      0.95      0.93       420\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.669\n",
      "[[395   5]\n",
      " [  7  13]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "ds1_archive = defaultdict(list)\n",
    "for k in chembl_active_sets:\n",
    "    run_dataset1_assay(k,chembl_active_sets,inactive_df,archive=ds1_archive)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the distribution of AUROC values here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADupJREFUeJzt3X+M5HV9x/HnS+4QFS1HWckVxFWLDdekPez2YmtUisVSTItWG4uRHJbktEojqTW92iYa0ya09Ufa2GjPglyNSqw/qimgXgmEaJB2EYSDq/KjVwUu3BJqlTapgu/+Md/T9dy9mZ1fu/u55yOZ7He+38/MvHZ25rXf/cx3ZlNVSJLWvyesdgBJ0nhY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGbOg3IMlxwI3AE7vxn6iqtyd5FnAVcCLwFeDCqvruka7rpJNOqtnZ2ZFDS9LR5JZbbnm4qmb6jetb6MD/AWdX1aNJNgJfTHIt8AfAe6vqqiQfAC4G3n+kK5qdnWV+fn6Am5QkHZLkPwcZ13fKpXoe7c5u7E4FnA18olu/G3j5EDklSWMy0Bx6kmOS3AYcBPYA9wLfqqrHuiH3A6dMJqIkaRADFXpVPV5VW4FTgW3AGUsNW+qySXYkmU8yv7CwMHxSSdIRregol6r6FnAD8HzghCSH5uBPBR5c5jK7qmququZmZvrO6UuShtS30JPMJDmhW34S8KvAPuB64FXdsO3AZyYVUpLU3yBHuWwGdic5ht4vgI9X1T8nuQu4KsmfAbcCl08wpySpj76FXlW3A2cusf4+evPpkqQ1wHeKSlIjLHRJasQgc+iS1ITZnVev2m3vv+xlE78N99AlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmN6FvoSZ6R5Pok+5LcmeTN3fp3JHkgyW3d6bzJx5UkLWfDAGMeA95SVV9J8lTgliR7um3vrap3TS6eJGlQfQu9qg4AB7rl7yTZB5wy6WCSpJVZ0Rx6klngTODmbtUlSW5PckWSTctcZkeS+STzCwsLI4WVJC1v4EJPcjzwSeDSqvo28H7gOcBWenvw717qclW1q6rmqmpuZmZmDJElSUsZqNCTbKRX5h+pqk8BVNVDVfV4VX0f+CCwbXIxJUn9DHKUS4DLgX1V9Z5F6zcvGvYKYO/440mSBjXIUS4vAC4E7khyW7fubcAFSbYCBewHXj+RhJKkgQxylMsXgSyx6Zrxx5EkDct3ikpSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIaYaFLUiMGeeu/JI3V7M6rVztCk9xDl6RGWOiS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1om+hJ3lGkuuT7EtyZ5I3d+tPTLInyd3d102TjytJWs4ge+iPAW+pqjOA5wNvSrIF2AlcV1WnA9d15yVJq6RvoVfVgar6Srf8HWAfcApwPrC7G7YbePmkQkqS+lvRHHqSWeBM4Gbg5Ko6AL3SB54+7nCSpMENXOhJjgc+CVxaVd9eweV2JJlPMr+wsDBMRknSAAYq9CQb6ZX5R6rqU93qh5Js7rZvBg4uddmq2lVVc1U1NzMzM47MkqQlDHKUS4DLgX1V9Z5Fmz4LbO+WtwOfGX88SdKgNgww5gXAhcAdSW7r1r0NuAz4eJKLgW8Avz2ZiJKkQfQt9Kr6IpBlNr9kvHEkScPynaKS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIaYaFLUiP6FnqSK5IcTLJ30bp3JHkgyW3d6bzJxpQk9TPIHvqVwLlLrH9vVW3tTteMN5YkaaX6FnpV3Qg8MoUskqQRjDKHfkmS27spmU1jSyRJGsqwhf5+4DnAVuAA8O7lBibZkWQ+yfzCwsKQNydJ6meoQq+qh6rq8ar6PvBBYNsRxu6qqrmqmpuZmRk2pySpj6EKPcnmRWdfAexdbqwkaTo29BuQ5GPAWcBJSe4H3g6clWQrUMB+4PUTzChJGkDfQq+qC5ZYffkEskiSRuA7RSWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5Jjej7Dy4ktWl259WrHUFj5h66JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEb0LfQkVyQ5mGTvonUnJtmT5O7u66bJxpQk9TPIHvqVwLmHrdsJXFdVpwPXdeclSauob6FX1Y3AI4etPh/Y3S3vBl4+5lySpBUadg795Ko6ANB9ffpyA5PsSDKfZH5hYWHIm5Mk9TPxF0WraldVzVXV3MzMzKRvTpKOWsMW+kNJNgN0Xw+OL5IkaRjDFvpnge3d8nbgM+OJI0ka1iCHLX4MuAn4mST3J7kYuAw4J8ndwDndeUnSKur7L+iq6oJlNr1kzFkkSSPwnaKS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEZY6JLUCAtdkhphoUtSIyx0SWqEhS5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIasWGUCyfZD3wHeBx4rKrmxhFKkrRyIxV651eq6uExXI8kaQROuUhSI0Yt9AK+kOSWJDvGEUiSNJxRp1xeUFUPJnk6sCfJv1fVjYsHdEW/A+C0004b8eYkScsZaQ+9qh7svh4EPg1sW2LMrqqaq6q5mZmZUW5OknQEQxd6kqckeeqhZeClwN5xBZMkrcwoUy4nA59Ocuh6PlpVnxtLKknSig1d6FV1H/DzY8wiSRqBhy1KUiMsdElqxDjeKSppBLM7r17tCGqEe+iS1AgLXZIaYaFLUiMsdElqhIUuSY2w0CWpER62qB9zNB5Gt/+yl612BGlk7qFLUiMsdElqhIUuSY2w0CWpERa6JDXCQpekRljoktQIj0MfwGodl+2x0dNzNB57r/a4hy5JjbDQJakRFrokNcJCl6RGWOiS1AgLXZIasW4OWzwaDys7Gr9nScNzD12SGmGhS1IjRir0JOcm+VqSe5LsHFcoSdLKDV3oSY4B/hb4dWALcEGSLeMKJklamVH20LcB91TVfVX1XeAq4PzxxJIkrdQohX4K8M1F5+/v1kmSVsEohy1miXX1Y4OSHcCO7uyjSb42wm1Oy0nAw6sdYgjrNTes3+zmnq71mpv8xUjZnznIoFEK/X7gGYvOnwo8ePigqtoF7BrhdqYuyXxVza12jpVar7lh/WY393St19wwneyjTLn8G3B6kmclORb4HeCz44klSVqpoffQq+qxJJcAnweOAa6oqjvHlkyStCIjvfW/qq4BrhlTlrVkXU0RLbJec8P6zW7u6VqvuWEK2VP1Y69jSpLWId/6L0mNOOoKvd/HFSQ5Lcn1SW5NcnuS87r1G5PsTnJHkn1J/niN5X5mkuu6zDckOXXRtu1J7u5O29dD7iRbk9yU5M5u26vXQ+5F25+W5IEk75te6h/c9iiPldOSfKF7jN+VZHad5P7L7rGyL8nfJFnqsOpJ5b4iycEke5fZni7TPV325y3aNt7nZlUdNSd6L97eCzwbOBb4KrDlsDG7gN/rlrcA+7vl1wBXdctPBvYDs2so9z8C27vls4EPd8snAvd1Xzd1y5vWQe7nAqd3yz8FHABOWOu5F23/a+CjwPum9fgeR3bgBuCcbvl44MlrPTfwy8CXuus4BrgJOGuK9/mLgOcBe5fZfh5wLb337jwfuLlbP/bn5tG2hz7IxxUU8LRu+Sf44bH1BTwlyQbgScB3gW9PPjIwWO4twHXd8vWLtv8asKeqHqmq/wL2AOdOITOMkLuqvl5Vd3fLDwIHgZmppB7t/ibJLwAnA1+YQtbDDZ29+yymDVW1B6CqHq2q/51O7JHu8wKOo/eL4InARuChiSc+dONVNwKPHGHI+cA/VM+XgROSbGYCz82jrdAH+biCdwCvTXI/vSN4fr9b/wngf+jtKX4DeFdVHemHOE6D5P4q8Mpu+RXAU5P85ICXnZRRcv9Akm30nqz3Tijn4YbOneQJwLuBt0485dJGuc+fC3wryae6Kce/6j6EbxqGzl1VN9Er+APd6fNVtW/CeVdiue9t7M/No63QB/m4gguAK6vqVHp/Kn24e5JuAx6n9+f/s4C3JHn2JMMuMkjuPwRenORW4MXAA8BjA152UkbJ3buC3p7Mh4HXVdX3JxX0MKPkfiNwTVV9k9UxSvYNwAu77b9Ib/rjookl/VFD507y08AZ9N6tfgpwdpIXTTLsCi33vY39ublu/gXdmAzycQUX0/3ZU1U3JTmO3udHvAb4XFV9DziY5EvAHL15r0nrm7ublvgtgCTHA6+sqv/u/tI467DL3jDJsIsMnbs7/zTgauBPuz9Vp2WU+/uXgBcmeSO9OehjkzxaVdP6fwGjPlZurar7um3/RG/O9/I1nnsH8OWqerTbdm2X+8Yp5B7Ect/b+J+b03rhYC2c6P0Cu4/eHvahF15+9rAx1wIXdctndHd8gD8CPtQtPwW4C/i5NZT7JOAJ3fKfA++sH77w8h/0XnTZ1C2fuA5yH0tvvvTSNfo4WTL3YWMuYvovio5ynx/TjZ/pzn8IeNM6yP1q4F+669jYPW5+Y8r3+yzLvyj6Mn70RdF/7daP/bk5tW94rZzoTaN8nd587J90694J/Ga3vIXeK+ZfBW4DXtqtP57eq+x30ivzt66x3K8C7u7G/D3wxEWX/V3gnu70uvWQG3gt8L3uZ3DotHWt5z7sOi5iyoU+hsfKOcDtwB3AlcCxaz03vV9Efwfs656b75ny/f0xenP336O3130x8AbgDd320PtnQPd29+vcosuO9bnpO0UlqRFH24uiktQsC12SGmGhS1IjLHRJaoSFLkmNsNAlqREWuiQ1wkKXpEb8P9m2lIl1Ya8gAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist([x[0][-1] for x in ds1_archive.values()]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('../data/ds1_archive.pkl','wb+') as outf:\n",
    "#    pickle.dump(ds1_archive,outf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PubChem HTS Validation assays found in ChEMBL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>canonical_smiles</th>\n",
       "      <th>compound_chembl_id</th>\n",
       "      <th>assay_chembl_id</th>\n",
       "      <th>standard_relation</th>\n",
       "      <th>Mean(standard_value)</th>\n",
       "      <th>activity_comment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Br.Br.C(c1ccncc1)c2cnc[nH]2</td>\n",
       "      <td>CHEMBL1316355</td>\n",
       "      <td>CHEMBL1614421</td>\n",
       "      <td>=</td>\n",
       "      <td>44668.4</td>\n",
       "      <td>Inconclusive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Br.Br.NCCSC(=N)N</td>\n",
       "      <td>CHEMBL1256182</td>\n",
       "      <td>CHEMBL1614249</td>\n",
       "      <td>=</td>\n",
       "      <td>31622.8</td>\n",
       "      <td>Not Active</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Br.Br.NCCSC(=N)N</td>\n",
       "      <td>CHEMBL1256182</td>\n",
       "      <td>CHEMBL1614364</td>\n",
       "      <td>=</td>\n",
       "      <td>446.7</td>\n",
       "      <td>Not Active</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Br.Br.NCCSC(=N)N</td>\n",
       "      <td>CHEMBL1256182</td>\n",
       "      <td>CHEMBL1614421</td>\n",
       "      <td>=</td>\n",
       "      <td>17782.8</td>\n",
       "      <td>Inconclusive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Br.Br.NCCSC(=N)N</td>\n",
       "      <td>CHEMBL1256182</td>\n",
       "      <td>CHEMBL1794375</td>\n",
       "      <td>NaN</td>\n",
       "      <td>35481.3</td>\n",
       "      <td>active</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              canonical_smiles compound_chembl_id assay_chembl_id  \\\n",
       "0  Br.Br.C(c1ccncc1)c2cnc[nH]2      CHEMBL1316355   CHEMBL1614421   \n",
       "1             Br.Br.NCCSC(=N)N      CHEMBL1256182   CHEMBL1614249   \n",
       "2             Br.Br.NCCSC(=N)N      CHEMBL1256182   CHEMBL1614364   \n",
       "3             Br.Br.NCCSC(=N)N      CHEMBL1256182   CHEMBL1614421   \n",
       "4             Br.Br.NCCSC(=N)N      CHEMBL1256182   CHEMBL1794375   \n",
       "\n",
       "  standard_relation  Mean(standard_value) activity_comment  \n",
       "0                 =               44668.4     Inconclusive  \n",
       "1                 =               31622.8       Not Active  \n",
       "2                 =                 446.7       Not Active  \n",
       "3                 =               17782.8     Inconclusive  \n",
       "4               NaN               35481.3           active  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with gzip.open('../data/ChEMBL_PubChem_HTS.csv.gz') as inf:\n",
    "    pubchem_d = pd.read_csv(inf)\n",
    "pubchem_d.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CHEMBL1794375 83178 PUBCHEM_BIOASSAY: qHTS for inhibitors of binding or entry into cells for Marburg Virus. (Class of assay: confirmatory) [Related pubchem assays (depositor defined):AID463114, AID540249, AID540278]\n",
      "CHEMBL1614421 49035 PUBCHEM_BIOASSAY: qHTS for Inhibitors of Tau Fibril Formation, Thioflavin T Binding. (Class of assay: confirmatory) [Related pubchem assays: 596 ]\n",
      "CHEMBL1614249 42030 PUBCHEM_BIOASSAY: qHTS Assay for Identification of Novel General Anesthetics. In this assay, a GABAergic mimetic model system, apoferritin and a profluorescent 1-aminoanthracene ligand (1-AMA), was used to construct a competitive binding assay for identification of novel general anesthetics (Class of assay: confirmatory) [Related pubchem assays: 2385 (Probe Development Summary for Identification of Novel General Anesthetics), 2323 (Validation apoferritin assay run on SigmaAldrich LOPAC1280 collection)]\n",
      "CHEMBL1614166 34201 PubChem BioAssay. qHTS Assay for Inhibitors of MBNL1-poly(CUG) RNA binding.   (Class of assay: confirmatory) \n",
      "CHEMBL1614364 11842 PUBCHEM_BIOASSAY: qHTS Assay for Inhibitors of Tyrosyl-DNA Phosphodiesterase (TDP1). (Class of assay: confirmatory) \n",
      "CHEMBL1613933 7149 PUBCHEM_BIOASSAY: qHTS Assay for Compounds Blocking the Interaction Between CBF-beta and RUNX1 for the Treatment of Acute Myeloid Leukemia. (Class of assay: confirmatory) [Related pubchem assays (depositor defined):AID1484, AID504370, AID504374, AID504375]\n",
      "CHEMBL3214913 4394 PubChem BioAssay. qHTS for Inhibitors of binding or entry into cells for Marburg Virus.   (Class of assay: confirmatory) \n",
      "CHEMBL3215169 3248 PubChem BioAssay. qHTS for Inhibitors of binding or entry into cells for Lassa Virus.   (Class of assay: confirmatory) \n"
     ]
    }
   ],
   "source": [
    "import requests,json\n",
    "pubchem_assay_lookup = {}\n",
    "tpls = sorted([(len(v),k) for k,v in pubchem_d.groupby('assay_chembl_id').groups.items()],reverse=True)\n",
    "for v,k in tpls:\n",
    "    resp=requests.get(f'https://www.ebi.ac.uk/chembl/api/data/assay/{k}?format=json')\n",
    "    info = json.loads(resp.text)\n",
    "    pubchem_assay_lookup[k] = info['description']\n",
    "    print(k,v,info['description'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../data/pubchem_data.pkl','wb+') as outf:\n",
    "    pickle.dump((pubchem_d,pubchem_assay_lookup),outf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../data/pubchem_data.pkl','rb') as inf:\n",
    "    pubchem_d,pubchem_assay_lookup = pickle.load(inf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_pubchem_assay(assay_id,d,assay_lookup,thresholds=None,archive=None):\n",
    "    if archive is None:\n",
    "        archive=defaultdict(list)\n",
    "    assay = d.loc[d['assay_chembl_id']==assay_id]\n",
    "    acts = pd.concat((assay.loc[assay['activity_comment'] == 'Active'], \n",
    "                      assay.loc[assay['activity_comment'] == 'active']))\n",
    "    inacts = pd.concat((assay.loc[assay['activity_comment'] == 'inactive'],\n",
    "                        assay.loc[assay['activity_comment'] == 'inconclusive'], \n",
    "                        assay.loc[assay['activity_comment'] == 'Inconclusive'], \n",
    "                        assay.loc[assay['activity_comment'] == 'Not Active']))\n",
    "    print('*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*')\n",
    "    print(f'assay_id {assay_id}, description: {assay_lookup[assay_id]}')\n",
    "    run_it_(assay_id,acts,inacts,thresholds,archive)       \n",
    "    print('*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\\n\\n')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1794375, description: PUBCHEM_BIOASSAY: qHTS for inhibitors of binding or entry into cells for Marburg Virus. (Class of assay: confirmatory) [Related pubchem assays (depositor defined):AID463114, AID540249, AID540278]\n",
      "--------- Default -----------\n",
      "ratio: 0.057 kappa: 0.002, AUC: 0.704, OOB_AUC: 0.694\n",
      "[[15737     1]\n",
      " [  897     1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97     15738\n",
      "           1       0.50      0.00      0.00       898\n",
      "\n",
      "   micro avg       0.95      0.95      0.95     16636\n",
      "   macro avg       0.72      0.50      0.49     16636\n",
      "weighted avg       0.92      0.95      0.92     16636\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.100 kappa: 0.129\n",
      "[[15542   196]\n",
      " [  808    90]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1614421, description: PUBCHEM_BIOASSAY: qHTS for Inhibitors of Tau Fibril Formation, Thioflavin T Binding. (Class of assay: confirmatory) [Related pubchem assays: 596 ]\n",
      "--------- Default -----------\n",
      "ratio: 0.129 kappa: 0.030, AUC: 0.861, OOB_AUC: 0.848\n",
      "[[8680    5]\n",
      " [1103   20]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      1.00      0.94      8685\n",
      "           1       0.80      0.02      0.03      1123\n",
      "\n",
      "   micro avg       0.89      0.89      0.89      9808\n",
      "   macro avg       0.84      0.51      0.49      9808\n",
      "weighted avg       0.88      0.89      0.84      9808\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.443\n",
      "[[8255  430]\n",
      " [ 603  520]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1614249, description: PUBCHEM_BIOASSAY: qHTS Assay for Identification of Novel General Anesthetics. In this assay, a GABAergic mimetic model system, apoferritin and a profluorescent 1-aminoanthracene ligand (1-AMA), was used to construct a competitive binding assay for identification of novel general anesthetics (Class of assay: confirmatory) [Related pubchem assays: 2385 (Probe Development Summary for Identification of Novel General Anesthetics), 2323 (Validation apoferritin assay run on SigmaAldrich LOPAC1280 collection)]\n",
      "--------- Default -----------\n",
      "ratio: 0.006 kappa: 0.000, AUC: 0.702, OOB_AUC: 0.728\n",
      "[[8356    0]\n",
      " [  51    0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      8356\n",
      "           1       0.00      0.00      0.00        51\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      8407\n",
      "   macro avg       0.50      0.50      0.50      8407\n",
      "weighted avg       0.99      0.99      0.99      8407\n",
      "\n",
      "--------- Balanced -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "thresh: 0.050 kappa: 0.099\n",
      "[[8299   57]\n",
      " [  45    6]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1614166, description: PubChem BioAssay. qHTS Assay for Inhibitors of MBNL1-poly(CUG) RNA binding.   (Class of assay: confirmatory) \n",
      "--------- Default -----------\n",
      "ratio: 0.003 kappa: 0.181, AUC: 0.822, OOB_AUC: 0.781\n",
      "[[6821    0]\n",
      " [  18    2]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6821\n",
      "           1       1.00      0.10      0.18        20\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      6841\n",
      "   macro avg       1.00      0.55      0.59      6841\n",
      "weighted avg       1.00      1.00      1.00      6841\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.100 kappa: 0.456\n",
      "[[6814    7]\n",
      " [  12    8]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1614364, description: PUBCHEM_BIOASSAY: qHTS Assay for Inhibitors of Tyrosyl-DNA Phosphodiesterase (TDP1). (Class of assay: confirmatory) \n",
      "--------- Default -----------\n",
      "ratio: 0.085 kappa: 0.009, AUC: 0.748, OOB_AUC: 0.763\n",
      "[[2183    1]\n",
      " [ 184    1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      1.00      0.96      2184\n",
      "           1       0.50      0.01      0.01       185\n",
      "\n",
      "   micro avg       0.92      0.92      0.92      2369\n",
      "   macro avg       0.71      0.50      0.49      2369\n",
      "weighted avg       0.89      0.92      0.89      2369\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.232\n",
      "[[2100   84]\n",
      " [ 141   44]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1613933, description: PUBCHEM_BIOASSAY: qHTS Assay for Compounds Blocking the Interaction Between CBF-beta and RUNX1 for the Treatment of Acute Myeloid Leukemia. (Class of assay: confirmatory) [Related pubchem assays (depositor defined):AID1484, AID504370, AID504374, AID504375]\n",
      "--------- Default -----------\n",
      "ratio: 0.000 kappa: 0.000, AUC: 1.000, OOB_AUC: 1.000\n",
      "[[1430    0]\n",
      " [   1    0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      1430\n",
      "           1       0.00      0.00      0.00         1\n",
      "\n",
      "   micro avg       1.00      1.00      1.00      1431\n",
      "   macro avg       0.50      0.50      0.50      1431\n",
      "weighted avg       1.00      1.00      1.00      1431\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.050 kappa: 1.000\n",
      "[[1430    0]\n",
      " [   0    1]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL3214913, description: PubChem BioAssay. qHTS for Inhibitors of binding or entry into cells for Marburg Virus.   (Class of assay: confirmatory) \n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.257 kappa: 0.106, AUC: 0.812, OOB_AUC: 0.799\n",
      "[[695   5]\n",
      " [166  14]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.81      0.99      0.89       700\n",
      "           1       0.74      0.08      0.14       180\n",
      "\n",
      "   micro avg       0.81      0.81      0.81       880\n",
      "   macro avg       0.77      0.54      0.52       880\n",
      "weighted avg       0.79      0.81      0.74       880\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.423\n",
      "[[618  82]\n",
      " [ 83  97]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL3215169, description: PubChem BioAssay. qHTS for Inhibitors of binding or entry into cells for Lassa Virus.   (Class of assay: confirmatory) \n",
      "--------- Default -----------\n",
      "ratio: 0.215 kappa: 0.230, AUC: 0.745, OOB_AUC: 0.769\n",
      "[[531   4]\n",
      " [ 96  19]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.85      0.99      0.91       535\n",
      "           1       0.83      0.17      0.28       115\n",
      "\n",
      "   micro avg       0.85      0.85      0.85       650\n",
      "   macro avg       0.84      0.58      0.59       650\n",
      "weighted avg       0.84      0.85      0.80       650\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.347\n",
      "[[491  44]\n",
      " [ 69  46]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pubchem_archive = defaultdict(list)\n",
    "for k in pubchem_assay_lookup:\n",
    "    run_pubchem_assay(k,pubchem_d,pubchem_assay_lookup,archive=pubchem_archive)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An assay that didn't work particularly well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "251 41779\n",
      "ratio: 0.006 kappa: 0.000, AUC: 0.733, OOB_AUC: 0.730\n",
      "[[8356    0]\n",
      " [  51    0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      8356\n",
      "           1       0.00      0.00      0.00        51\n",
      "\n",
      "   micro avg       0.99      0.99      0.99      8407\n",
      "   macro avg       0.50      0.50      0.50      8407\n",
      "weighted avg       0.99      0.99      0.99      8407\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XucjnX++PHX24yZcT7G15JD0SBEO6GUQ3albOfadFCplmkcQoUWkZBUihwGG61U2vwqtl3KltJWYioUoZM0kho5H8Yc3r8/rmu43Ttzzz1j7vu675n38/G4H+Y6v+/LzP2+P4fr8xFVxRhjjClIOa8DMMYYE9ksURhjjAnIEoUxxpiALFEYY4wJyBKFMcaYgCxRGGOMCcgShTHGmIAsUZQhInKLiKSJyCER2SUiy0XkYnfbOBFZlM8xKiJNffbJco/Pe+3z23e3iMT6rIsVkV9ERH3WvScix9zj94vIahFp7bM931jcbQPd95ApIs/ns72iiMwSkYy8c/ts6yYiq9z12wPcpy7ue5ngs+4OEflURA6ISLqITPF9nz77NXPf2yKfdSIio0Rkh3v8YhGpWtD1A8T1nojsFZH4fNbf47euq4ik+8UwWES+FJHD7nt41fe+BxmDiMjjIrLHfU0RESlg37/6/a4cFZFcEantbp8iIj+69+QHERnld3xb954fcf9tW5RYTcmxRFFGiMgw4BlgElAXaAjMAq4u4qleUdXKPq/qftv3AZf7LF8B7M3nPANVtTJQC3gPeCHI6/8ETADmF7B9LlATaOH+O9Rn22H3uAcLOrmIlAemAZ/4baoIDAFqAx2A7sAD+ZxiJrDOb93tQB+gE/A7oALwbEExFBBXY+ASQIGrinKsaxpwHzAY576cA7wB9CriefoB1wDnAW2APwH989tRVSf5/q4AjwPvqWqGu8tzQHNVrQpcBNwiItcBiEgcsBRYBNQA/g4sddebMLNEUQaISDVgPDBAVV9T1cOqmqWq/1TVAj80i+kFnA/GPLcDCwvaWVWzgcVAy2BO7sb/BrDHf5uIJOJ8iPZT1V9VNUdVP/U5dq2qvgB8F+AS9wNvA1v8rjtbVT9Q1eOquhN4EeeD3/f6vXES5Tt+57wSeE5Vf1TVQzgfmDeJSMVg3rPrdmAN8DxwRxGOQ0SaAQOAm1X1XVXNVNUjqvqiqk4uyrncaz+lqunufXgKuDOIGAQnWf49b52qblXVwz675QJN3Z+7ArHAM2680wEBLi1ivKYEWKIoGy4EEoDXw3CtN4DOIlJdRKrjfAteWtDO7jfEW3E+BE9XB+AH4BG36ukLEbk+2INFpBFwF05SLUxnYJPPsVXd4+7P79Tuy3c5HmjmHjtSRN4s5Hq34ySnF4HLRKRuEDHm6Q6kq+ragnZwY9hX0Mtn13OBDT7LG9x1hbkEpyT7//K57iEgHagEvORznY166hhDG4O8lilhlijKhlpAhvvtPZA/B/iAKGifVX7bjwH/BG4CegPL3HX+prvnPwQMBB4p0jvKXwOgFbAfp4pnIPB3EWkR5PHTgTHut/4CiUhfIAl40mf1o7ilhnwOWQ7cIyKN3dLdCHd9RQBVnayqfwpwvYuBRsA/3BLSt8AtQb4ncP7/dwXawY2hekEvn10r49zfPPuBygW1U/i4A1jif2/dEk0V4Hyc0mjeuf2vk3etKoVcx4SAJYqyYQ9QO7/GVz//CPABUdA+3fLZZyHON+BA1U6D3fMn4NRzLxGRNkG+n4IcBbKACW4V0fvAKqBHYQeKyJVAFVV9pZD9rgEmA5fn1bW7jax/AJ4u4LD5wMs4bTGb3JjA+RYdjDuAt33q9l/i1OqnbKC83zHlce4FOP//9YK8VmEOAb4N8VWBQ37f/E8hIhWAG/GpdvKljs9x/v/yvjD4XyfvWgeLGbc5DZYoyoaPcb7VXxOm632A88FUF/hvoB1VNVdVPwC+IYgP9EJsPI1juwNJIvKziPyMUyIaIiInqs1EpCcwD7hSVb/wObYr0BjY4R77AHC9iHwGJ97jWFVtrKoNcJLFTvcVkPsh+2egi09sQ4HzROQ8d7cd7vV9NcGphgOnzaSBiCQFuI5/D6VTXj67bsJpyM5zHj5VcAW4DvgNJ1EGEguc7XOdNn4llTZBXMuEgCWKMkBV9wMPAzNF5BpxupCWF5HLRWRKCK6nOA24VwX6pplHRC7Eacz2/RAoJyIJPq94d99YEUkAYoAYd1teSWk1zofmQ+5+nXA+xN9yjy3nHlveWZQEn140Y3B6ArV1X8twkkJf99hLcdoHrs+nrn8uzgdc3rGpwL+Ay9xja4rI2eJoCUwFxqtqbmH3Bie557j3J+/8LXCScV6ngVeAviLS3r3GOTjJZDGAqn6N08PtZXG6zca57723iIx09zmlh5L/yyeehcAwEakvIr/DaZN5vpD3cAew0Pd3wf2/6C8iNdyY2+M0uOd1BHjPfd+DRSReRAa6698N4p6Zkqaq9iojL5xG4zScbqI/43yYXeRuGwcsyucYBZr67JOFUy3g+6rjv6/fOZri5g93+T2cEk7e8d8AQ322j3PP5ftKD7BtnM+x5+KUoA4Dm4FrfbZ1zefY9wq4V8/jVGHlLa/CqeLxfd/LCzj2lHuJk4C2AkdwvuUP89v/rwHOtQKnl5H/+j+7/4ex7vJdOIn2gHs/RwLlfPYXnO6xm9w4duIkmHOL+DskwBScEsJv7s/is/0QcInPcn33vjX1O08597395h6zzb0PvudqB3yKUyX1GdDO67+hsvoS9z/EGGOMyZdVPRljjAnIEoUxxpiALFEYY4wJyBKFMcaYgAp7ACvi1K5dWxs3bux1GMYYE1U+/fTTDFU9ozjHRl2iaNy4MWlpaV6HYYwxUUVEfih8r/xZ1ZMxxpiALFEYY4wJyBKFMcaYgCxRGGOMCcgShTHGmIAsURhjjAkoZIlCROaLyC8i8mUB20VEpovINyKyUUTOD1Usxhhjii+UJYrngZ4Btl+OM2dwM6AfMDuEsRhjTJl1/Pjx0zo+ZIlCVVfjjDVfkKtxJzNR1TVAdREpqekajTHGAA8++CDVzml/Wufw8sns+oDvRPTp7rr/mQReRPrhlDpo2LBhWIIzxphI0XfBWlZt/bVYxx7anMOx9NObQdbLRCH5rMt3FiVVnYsz3SRJSUk205IxptQ5nWTg63jGDo7v/pbK53YDoFKrS0lo2IqdqXcX+5xeJop04Eyf5QbATx7FYowxYVeU5NAt8QwW9C24CunIkSNMmDCBJ/7+BDExMax5OpmmTZue2C5RmiiWAQNFZDHQAdivqv9T7WSMMZGupEoDhSWDgixfvpwBAwbw/fffA3D33XdTq1at044nT8gShYi8jDOZfW0RSQfGAuUBVDUV+DdwBc5E8EeAvqGKxRhjQul0kkRxkwPAzp07GTJkCEuWLAGgTZs2pKamcuGFFxY7nvyELFGo6s2FbFdgQKiub4wxwSqpEsH2yb1KIJrgDRgwgKVLl1KxYkXGjx/PfffdR2xsyX+sR918FMYYc7pKKjH46pZYrDmBiiw7O/tEMnj88ccpX748Tz31VEh7hFqiMMaUCYUlh9OpAgqH/fv3M3r0aLZt28aKFSsQERITE3n11VdDfm1LFMaYqHS6pYJITwx5VJVXX32VIUOGsGvXLmJiYli/fj3t2rULWwyWKIwxUaEkqouiJTnk+fbbbxk4cCArVqwA4MILLyQ1NZU2bdqENQ5LFMaYiFKSzxZEsyeffJIxY8Zw7NgxqlevzuOPP84999xDuXLhH/TbEoUxJqJEcztCSTpy5AjHjh2jT58+PPnkk9SpU8ezWCxRGGMign9JItxdTb3266+/snXrVi6++GIARowYQdeuXencubPHkVmiMMZ4LL+qpnB1NY0Eubm5zJ8/n+HDhxMbG8uWLVuoWbMm8fHxEZEkwBKFMcYjBSWIslK1BPDll1+SnJzMhx9+CMAf//hHjhw5Qs2aNT2O7FSWKIwxYeefJMpagjh8+DDjx49n6tSpZGdnU7duXZ555hluuukmRPIbWNtbliiMMWGXlyTKWoLIc8MNN5x4aC4lJYWJEydSvXp1r8MqkCUKY0xY9V2w9sTPZTFJgNNQvXv3bmbPnk2HDh28DqdQliiMMWHlW5ooC7Kzs3n22WfZvn0706ZNA6Br166kpaV58kxEcViiMMaUiKI+OV0WShNr166lf//+rF+/HoB+/fpx7rnnAkRNkgBLFMaY01ScoTVKe2li3759/PWvfyU1NRVVpVGjRsyYMeNEkog2liiMMaelLPdeys/ixYsZMmQIu3fvJjY2lvvvv58xY8ZQqVIlr0MrNksUxpgSUdaepC7I22+/ze7du+nUqROzZ8+mdevWXod02ixRGGPMacjMzGTnzp2cddZZAEyZMoVLLrmEO+64I6raIQIpHe/CGOMJ366uZdG7775LmzZt6NWrF8ePHwegdu3a9O3bt9QkCbAShTHGjzVOF2737t088MADLFq0CIDmzZuTnp5+olRR2liiMKaMKyszxZWE3Nxc5s2bx8iRI9m3bx8JCQmMHj2aBx98kLi4OK/DCxlLFMaUQdE+f7RXrr32WpYtWwbAZZddxsyZMzn77LM9jir0LFEYU4YUlCAsMQTnuuuuY+3atUybNo0bb7wxIgfwCwVRVa9jKJKkpCRNS0vzOgxjooYlh+JbtmwZ6enppKSkAKCqHDp0iCpVqngcWdGJyKeqmlScY61EYUwpFKhqyRJE4Xbs2MHgwYNZunQp8fHx9OzZk7POOgsRicokcbosURgTgU63gdmfJYfgZGVlMX36dMaOHcvhw4epUqUKEyZMoFGjRl6H5ilLFMaEWUkngYJYciiaNWvW0L9/fzZu3AjAjTfeyNNPP039+vU9jsx7liiMCZOiJgj7oA+vMWPGsHHjRpo0acKMGTO44oorvA4pYliiMCZMbPC8yKKqHDx4kKpVqwIwY8YMFi5cyKhRo6hYsaLH0UUWSxTGhIHvUBc2eJ73tm7dSkpKCiLCypUrERESExOZOHGi16FFpNIzGIkxEcq3yqmsDXURaY4dO8bYsWNp06YN7777LuvXr2f79u1ehxXxLFEYE0L+ScKqm7yzcuVKWrduzfjx4zl+/Dh33XUXW7dupUmTJl6HFvFCmihEpKeIbBWRb0RkZD7bG4rIKhH5XEQ2ioi1HplSxZKE91SVu+66ix49evDNN9/QsmVLVq9ezXPPPUetWrW8Di8qhCxRiEgMMBO4HGgJ3CwiLf12Gw38Q1XbAb2BWaGKxxgvWZLwjojQuHFjKlSowGOPPcbnn3/OJZdc4nVYUSWUjdntgW9U9TsAEVkMXA1s9tlHgaruz9WAn0IYjzFhE65nJUz+1q9fz65du7j88ssBGDFiBH369LFqpmIKZaKoD/zos5wOdPDbZxzwtogMAioBf8jvRCLSD+gH0LBhwxIP1JiSkl+CsAbs8Dl48CBjx45l2rRp1KpViy1btlCzZk3i4+MtSZyGUCaK/IZV9B+B8GbgeVV9SkQuBF4QkVaqmnvKQapzgbngDAoYkmiNKYJgSgzWLhE+qsobb7zB4MGDSU9Pp1y5ctxyyy2UL1/e69BKhVAminTgTJ/lBvxv1dLdQE8AVf1YRBKA2sAvIYzLmGKzBBF5fvjhBwYOHMibb74JQFJSEnPmzOH888/3OLLSI5SJYh3QTESaADtxGqtv8dtnB9AdeF5EWgAJgFXsmohiw3RHLlXl+uuv59NPP6Vq1apMmjSJ5ORkYmJivA6tVAlZolDVbBEZCLwFxADzVXWTiIwH0lR1GXA/ME9EhuJUS92p0TZBhinVCmpzsAThrdzcXMqVK4eI8OSTT5KamsrTTz9NvXr1vA6tVLKJi4wpgD0sF3n27NnDyJHOI1nz5s3zOJroYhMXGVOC/EsRliS8p6osXLiQBx54gIyMDOLi4hg7diwNGjTwOrQywYbwMMaHJYnI89VXX9GtWzfuvPNOMjIy6Nq1Kxs2bLAkEUZWojDGZVVNkUVVefjhh3n88cfJysqidu3aPPXUU/Tp0weR/Hrfm1CxEoUxLksSkUVE2LlzJ1lZWfzlL39h69at3H777ZYkPGAlCmP8WJLwzk8//URGRgZt2rQBYMqUKdx999106tTJ48jKNitRGMOpEwuZ8MvJyWHGjBm0aNGC3r17c/z4cQBq165tSSICWInClEmBHqIz4fXZZ5/Rv39/8rq9d+7cmQMHDlC7dm2PIzN5gipRiEiciDQNdTDGhIs9ae29AwcOcN9993HBBReQlpZGgwYNeO2111i2bJkliQhTaIlCRHoBU4E4oImItAXGquq1oQ7OmFCw+au9p6p07tyZDRs2EBMTw7Bhwxg3bhxVqlTxOjSTj2BKFONxhgffB6Cq6wErXZioZfNXe09EGDp0KO3btyctLY2nnnrKkkQEC6aNIktV9/l1SYuucT9MmVZQe4RVM4XP8ePHmTp1KjExMTz44IMA3H777dx22202gF8UCCZRfCUifwbKuSPB3gesCW1YxpQMa7T23gcffEBycjKbN28mPj6e22+/nbp16yIiliSiRDCJYiDwMJALvIYzGuxDoQzKmJJgT1p7KyMjg+HDh7NgwQIAmjVrxqxZs6hbt67HkZmiCqaN4jJVHaGq7dzXSODyUAdmzOmyJOENVWXBggU0b96cBQsWnBjAb+PGjfzhD/nOdmwiXDCJYnQ+60aVdCDGhIolifBbtGgRe/bs4dJLL2Xjxo2MGzeOhIQEr8MyxVRg1ZOIXIYzTWl9EZnqs6kqTjWUMRHLnrQOryNHjrB//37q1auHiDBr1izWrVvHrbfeamMzlQKB2ih+Ab4EjgGbfNYfBEaGMihjiiLQPNbWaB16y5cvZ8CAAZx11lmsXLkSESExMZHExESvQzMlpMBEoaqfA5+LyIuqeiyMMRmTr0AJIT/WNhFaO3fuZMiQISxZsgSAKlWqsGfPHnuquhQKptdTfRGZCLQETlQyquo5IYvKGD+FJQlLCuGTk5PDzJkzGT16NAcPHqRSpUqMHz+ewYMHExtrw8eVRsH8rz4PTACexOnt1BdrozBhZN1cI0dubi5dunThww8/BOCaa65h2rRpNGzY0OPITCgF0+upoqq+BaCq36rqaKBbaMMy5iRLEpGjXLly9OjRgzPPPJOlS5fy+uuvW5IoA4IpUWSK023hWxFJBnYCdUIbljH/y5JE+Kkq//jHP4iNjeX6668HYMSIEQwbNozKlSt7HJ0Jl2ASxVCgMjAYmAhUA+4KZVDGFLXh2pS8b7/9lpSUFN5++23OOOMMLr30UmrUqEF8fDzx8fFeh2fCqNBEoaqfuD8eBPoAiEiDUAZlyrb8koR1cw2fzMxMnnjiCSZOnMixY8eoUaMGEydOpFq1al6HZjwSMFGIyAVAfeC/qpohIucCI4BLAUsWpsRZw7W33nvvPe699162bNkCQJ8+fXjyySepU8dqm8uyQE9mPwZcD2wARovI6zgjxz4OJIcnPFPaBRrd1ZJEeOXk5JCSksKWLVtITExk9uzZdOtm/VZM4BLF1cB5qnpURGoCP7nLW8MTmikLLEl4Kzc3l2PHjlGxYkViYmKYPXs2q1evZvjw4dYOYU4IlCiOqepRAFX9TUS2WJIwoWJTkobfF198QXJyMs2bN+e5554DoEuXLnTp0sXjyEykCZQozhKR19yfBWjss4yqXhfSyEypYr2YIsfhw4cZP348U6dOJTs7m++//569e/dSo0YNr0MzESpQorjeb3lGKAMxpU+wycF6NIXPP//5TwYOHMiOHTsQEVJSUpg4cSLVq1f3OjQTwQINCvhOOAMxpYc1UEee7OxsbrrpJl57zakUaNu2LXPmzKF9e/v/MIWzEbxMifNNEpYcIkNsbCzVqlWjcuXKPProowwcONAG8DNBE1UN3clFegLTgBjgb6o6OZ99/gyMAxTYoKq3BDpnUlKSpqWlhSBaUxJ8SxPWQO2tTz5xnpXt0KEDAHv27OHo0aM0aGCPQJVFIvKpqiYV59igv1KISLyqZhZh/xhgJvBHIB1YJyLLVHWzzz7NgIeATqq6V0TsqZ4o5/uwnPHGvn37eOihh5gzZw7Nmzdn/fr1xMXFUatWLa9DM1Gq0NFjRaS9iHwBfO0unycizwZx7vbAN6r6naoeBxbjPJvh6y/ATFXdC6CqvxQpehOxrLop/FSVl156iebNm5OamkpMTAxXXXUVOTk5XodmolwwJYrpwJ+ANwBUdYOIBPO4Zn3gR5/ldKCD3z7nAIjIhzjVU+NUdUUQ5zYRxrq/euvrr78mJSWF//znPwB06tSJ1NRUWrVq5XFkpjQIJlGUU9Uf/CZID+YrSn4zqvs3iMQCzYCuOGNHfSAirVR13yknEukH9ANs7PsI5J8krNopvLKysrj00ktJT0+nZs2aTJkyhb59+1KuXDDTzRhTuGASxY8i0h5Qt91hELAtiOPSgTN9lhvgDAPiv88aVc0CvheRrTiJY53vTqo6F5gLTmN2ENc2YWSD+HlDVRERypcvz8SJE1m1ahVTpkzhjDMsUZuSVWivJ7eBeTrwB3fVf4CBqppRyHGxOAmlO85kR+uAW1R1k88+PYGbVfUOEakNfA60VdU9BZ3Xej1FhvyqmqyXU3js3r2bBx54gHPOOYcxY8Z4HY6JEqHu9ZStqr2LemJVzRaRgcBbOO0P81V1k4iMB9JUdZm7rYeIbMapznowUJIw3gv0MJ0JrdzcXObNm8fIkSPZt28f1atXZ8iQIVSpUsXr0EwpF0yiWOdWCb0CvKaqB4M9uar+G/i337qHfX5WYJj7MlHAHqbzxoYNG0hOTmbNmjUA9OzZk5kzZ1qSMGERzAx3Z4vIRUBv4BERWQ8sVtXFIY/ORJS+C9ae+NmqmcIjKyuLhx56iGeeeYacnBzq1avHtGnTuOGGG/DrYGJMyATVLUJVP1LVwcD5wAHgxZBGZSJK3wVraTzyX/YwnQdiY2P5/PPPyc3NZdCgQXz11VfceOONliRMWBVaohCRyjgPyvUGWgBLgYtCHJeJEPl1fbXqptDasWMHOTk5NGnSBBEhNTWV/fv3k5RUrHZIY05bMG0UXwL/BKao6gchjsdEGOv6Gj5ZWVlMmzaNsWPHcuGFF7Jy5UpEhGbNmnkdminjgkkUZ6lqbsgjMRHHt03CkkRoffzxxyQnJ7Nx40YAatasyZEjR6hUqZLHkRkTIFGIyFOqej/w/0Tkfx62sBnuSj9rkwi9vXv3MnLkSObOnQtAkyZNmDlzJpdffrnHkRlzUqASxSvuvzazXSlUlLGZrDQRGpmZmbRt25YdO3ZQvnx5HnzwQUaNGkXFihW9Ds2YUwSa4S6v3qGFqp6SLNwH6WwGvChTnIH7rDQROvHx8dx999288847zJ49m5YtW3odkjH5CmYIj89U9Xy/dZ+raruQRlYAG8Kj6Gxq0shw7NgxHnvsMRITE7nlFmd+ruzsbGJiYqy7qwm5kAzhISI34XSJbSIir/lsqgLsy/8oEwkClRwsOXhj5cqVpKSk8M0331CnTh2uvfZaKlSoYNORmqgQ6Ld0LbAHZ9TXmT7rD+IM3mcijCWIyPPzzz8zbNgwXn75ZQDOPfdcUlNTqVChgseRGRO8QG0U3wPf44wWayKcPRgXWXJycpgzZw5//etf2b9/PxUqVGDs2LEMHTqUuLg4r8MzpkgCVT29r6pdRGQvp044JDjj+dUMeXTmfxTWIG0JIjLk5OTw7LPPsn//fq644gpmzJhBkyZNvA7LmGIJVPWUN91p7XAEYoJjSSJyHTx4kJycHKpXr05cXBzz5s1j9+7dXHfdddZYbaJaoKqnvKexzwR+UtXjInIx0AZYhDM4oAkT/5KEjd4aOVSV119/ncGDB3PZZZfx3HPPAXDxxRd7HJkxJSOYLhdvABeIyNnAQuBfwEvAn0IZmHHkV9VkzzZEju3btzNo0CDefPNNAL788kuOHTtGQkKCx5EZU3KCGWY8153T+jrgGVUdBNQPbVgmj38D9fbJvax6KQJkZWXx+OOP07JlS958802qVq3KjBkz+OijjyxJmFInqKlQReRGoA9wjbuufOhCMmBVTZHsyJEjdOzYkS+++AKA3r17M3XqVOrVq+dxZMaERjCJ4i4gBWeY8e9EpAnwcmjDKptsPuroULFiRZKSkjhy5AizZs2iR48eXodkTEgVOoQHgIjEAk3dxW9UNTukUQVQWofwKKgtwqqZvKeqLFy4kLPPPvtEA/X+/fuJi4uzB+dM1AjJEB4+J78EeAHYifMMxf+JSB9V/bA4FzSnsgflIttXX33Fvffey/vvv0+LFi1Yv349cXFxVKtWzevQjAmbYKqengauUNXNACLSAidx2LyMJcCSRGQ6evQoEydOZMqUKWRlZXHGGWfw0EMPUb68Nc+ZsieYRBGXlyQAVPUrEbExCEqA7wxy1lgdOVasWMGAAQP47rvvAPjLX/7C5MmTqVnTBiMwZVMwieIzEZmDU4oAuBUbFLBYrLE68h06dIg+ffqQkZFBq1atSE1NpVOnTl6HZYyngkkUycBgYDhOG8Vq4NlQBlUa2ZwQkSsnJ4fc3FzKly9P5cqVmTZtGunp6QwdOtSqmoyhkEQhIq2Bs4HXVXVKeEIqPawnU+T79NNP6d+/P1dffTVjxowBODGpkDHGEWj02L8CdwOf4QzhMV5V54ctsihkI7tGjwMHDjBmzBhmzJhBbm4uBw4cYOTIkVaCMCYfgUoUtwJtVPWwiJwB/BuwRJEPmzAoeqgqS5Ys4b777mPXrl3ExMQwbNgwHnnkEUsSxhQgUKLIVNXDAKr6q4gEMy5UmWJVS9Hl4MGD3HTTTSxfvhyADh06kJqaStu2bT2OzJjIFihRnOUzV7YAZ/vOna2q14U0sihgz0BEl8qVK5OZmUm1atWYPHky/fr1o1w5+/5jTGECJYrr/ZZnhDKQaGbPQESu1atXU69ePZo1a4aIMH/+fBISEqhbt67XoRkTNQJNXPROOAOJJoU1WhvvZWRkMHz4cBYsWED37t1ZuXIlIkKjRo28Ds2YqGPl7iLKb2wmEzlyc3OZP38+iYmJLFiwgLi4OC7jDUdxAAAWe0lEQVS55BJycnK8Ds2YqBXMA3fFJiI9gWlADPA3VZ1cwH43AK8CF6hqRA0Naw/KRY9NmzZx77338sEHHwDQvXt3Zs2axTnnnONxZMZEt6AThYjEq2pmEfaPAWYCfwTSgXUissx33Ch3vyo4T35/Euy5w8mSRHTYv38/HTt25NChQ9SpU4epU6dyyy23ICJeh2ZM1AtmmPH2wHNANaChiJwH3ONOiRpIe5y5K75zz7MYuBrY7Lffo8AU4IEixh4SBZUgrME6MqkqIkK1atUYMWIEO3fuZNKkSdSoUcPr0IwpNYJpo5gO/AnYA6CqG4BuQRxXH/jRZzkdv7m2RaQdcKaqvhnoRCLST0TSRCTt119D24hsg/ZFh507d3LDDTewaNGiE+tGjRrF7NmzLUkYU8KCqXoqp6o/+BXhg2kZzK/Mf2I6PfcBvqeBOws7karOBeaCM8NdENc+bVaCiEzZ2dnMnDmT0aNHc+jQIT777DNuueUWYmJirJrJmBAJpkTxo1v9pCISIyJDgG1BHJcOnOmz3AD4yWe5CtAKeE9EtgMdgWUi4tmESL7zQ5jIs27dOjp06MCQIUM4dOgQ11xzDe+//z4xMTFeh2ZMqRZMorgXGAY0BHbjfKDfG8Rx64BmItLEneioN7Asb6Oq7lfV2qraWFUbA2uAq7zq9eTbNmFVTZHl8OHDDBw4kA4dOvDZZ5/RsGFDli5dyuuvv86ZZ55Z+AmMMael0KonVf0F50O+SFQ1W0QGAm/hdI+dr6qbRGQ8kKaqywKfIXz8k4T1aIossbGx/Oc//6FcuXIMGzaMsWPHUqlSJa/DMqbMENXAVf4iMg+ftoU8qtovVEEFkpSUpGlpJVfosCQRmb799luqV69OrVq1AKfaKSEhgdatW3scmTHRSUQ+VdViVe0HU/X0H+Ad9/UhUAcI+nmKSGdJIrJkZmYyYcIEWrVqxYgRI06sv+CCCyxJGOORYKqeXvFdFpEXgJUhiyiMfBuvLUl477333uPee+9ly5YtgNPDKScnxxqrjfFYcYbwaAJE9chqNl5TZPnll1948MEHWbhwIQCJiYnMnj2bbt2CeVzHGBNqwTyZvZeTbRTlgN+AkaEMKtRsHonIkZGRQYsWLfjtt9+Ij49n1KhRDB8+nPj4eK9DM8a4AiYKcZ5gOg/Y6a7K1cJavyOcb3WTPVTnvdq1a3P11VeTnp7OrFmzaNq0qdchGWP8BEwUqqoi8rqq/j5cAYWSPSvhvcOHDzN+/Hh69epF586dAZg1axbx8fH2ZLUxESqYXk9rReT8kEcSYtYN1nv//Oc/admyJVOmTCElJYXc3FwAEhISLEkYE8EKLFGISKyqZgMXA38RkW+BwzhjOKmqRlXysCThnR9//JH77ruP119/HYB27doxZ84cm6/amCgRqOppLXA+cE2YYgkLSxLhk52dzfTp03n44Yc5fPgwlStXZsKECQwYMIDY2JDOmWWMKUGB/loFQFW/DVMsppQ5cOAAjz32GIcPH+b666/nmWeeoUGDBl6HZYwpokCJ4gwRGVbQRlWdGoJ4SlRBkxCZ0Nm3bx8VKlQgPj6emjVrMmfOHOLj4+nVy3qYGROtAlUSxwCVcYYDz+8V8fyThPV0Ch1V5aWXXiIxMZEpU6acWH/ddddZkjAmygUqUexS1fFhiySE7HmJ0Nq2bRspKSm88847AKxevfrEFKXGmOgXqERhf+UmoGPHjvHII4/QunVr3nnnHWrWrMlzzz3HW2+9ZUnCmFIkUImie9iiMFHn559/pnPnznz99dcA3HnnnTzxxBPUrl3b48iMMSWtwEShqr+FMxATXerWrcuZZ55JbGwss2fPpkuXLl6HZIwJkVLZmd16O5W83Nxc5s2bR7du3TjnnHMQEV566SVq1KhBXFyc1+EZY0Ko1D0aa0OIl7wNGzbQqVMnkpOTSUlJIW9cyLp161qSMKYMKFUlChvPqWQdOnSIcePG8cwzz5CTk8Pvfvc7kpOTvQ7LGBNmpSpRWJIoOW+88QaDBg0iPT2dcuXKMWjQICZMmEDVqlW9Ds0YE2alKlHksSRxenbu3Env3r3JzMzk97//PampqSQlFWtOdmNMKVBqEoXvhESm6LKysoiNjUVEqF+/PhMnTiQuLo6UlBSbs9qYMq7UNGbbhETF99FHH/H73/+eRYsWnVh3//33M2jQIEsSxpjSkSh8SxNW7RS83377jf79+9OpUye++OILZs2aRZTPdGuMCYFSkSisNFE0qsoLL7xA8+bNmTt3LuXLl2fUqFG8++67NvSGMeZ/lJo2CrDSRDB2797NzTffzKpVqwDo0qULs2fPpkWLFh5HZoyJVKWiRGGCV716dXbt2kXt2rV5/vnnWbVqlSUJY0xAUV+isN5OhVu5ciXnn38+tWrVIj4+nldffZV69epRq1Ytr0MzxkSBqC9RWPtEwXbt2sXNN99Mjx49GDFixIn1rVq1siRhjAla1CeKPNY+cVJOTg6zZs2iefPmLF68mAoVKpCYmGg9mowxxRL1VU/mVJ999hnJycmsW7cOgF69ejFjxgwaN27sbWDGmKgV1YnC2idOtX37dtq3b09OTg7169dn+vTpXHvttdbl1RhzWkKaKESkJzANiAH+pqqT/bYPA+4BsoFfgbtU9Ydgz2/tE6dq3Lgxffv2pUqVKjzyyCNUqVLF65CMMaVAyNooRCQGmAlcDrQEbhaRln67fQ4kqWobYAkwpTjXKqvtE9u3b+fKK6/k/fffP7Fu7ty5TJ061ZKEMabEhLJE0R74RlW/AxCRxcDVwOa8HVR1lc/+a4DbQhhPqZGVlcXUqVN55JFHOHr0KBkZGXz88ccAVs1kjClxoez1VB/40Wc53V1XkLuB5fltEJF+IpImImm//lq2pzj973//S7t27Rg5ciRHjx6ld+/evPbaa16HZYwpxUKZKPL7aptv/0wRuQ1IAp7Ib7uqzlXVJFVNOuMMpz2irDVk7927l3vuuYdLLrmETZs2cfbZZ/PWW2/x8ssvU69ePa/DM8aUYqFMFOnAmT7LDYCf/HcSkT8Ao4CrVDUz2JOXtYbs3Nxcli5dSvny5RkzZgxffPEFPXr08DosY0wZEMo2inVAMxFpAuwEegO3+O4gIu2AOUBPVf2lOBcpzQ3ZW7ZsoUmTJsTHx1OrVi1efPFFGjZsSPPmzb0OzRhThoSsRKGq2cBA4C3gK+AfqrpJRMaLyFXubk8AlYFXRWS9iCwLVTzR5MiRI4waNYo2bdowZcrJjmA9evSwJGGMCbuQPkehqv8G/u237mGfn/8QyutHoxUrVpCSksL3338PQEZGhscRGWPKuqh+Mrs0+emnnxgyZAivvvoqAK1btyY1NZWLLrrI48iMMWWdJYoIsG3bNpKSkjh48CAVK1Zk3LhxDBkyhPLly3sdmjHGRGeiKG1dY5s1a8YFF1xApUqVePbZZ2nUqJHXIRljzAlROcx4tHeNPXDgAEOGDGHbtm2A8zT1smXLWLZsmSUJY0zEicoSRZ5o6xqrqixZsoT77ruPXbt2sWXLFlasWAFApUqVPI7OGGPyF9WJIpp89913DBw4kOXLnVFKOnbsyOOPP+5xVMYYU7iorHqKJsePH2fSpEmce+65LF++nOrVq5OamsqHH37Ieeed53V4xhhTKCtRhNiPP/7I+PHjyczM5NZbb+Wpp56ibt26XodljDFBi7pEsT3jMJW9DqIQe/fupXr16ogIZ599NtOmTaNp06Z0797d69CMMabIoq7q6WBmNhCZPZ5yc3OZP38+TZs2ZdGiRSfW9+/f35KEMSZqRV2iyBNpPZ42bdpE165dufvuu/ntt99ONFobY0y0i9pEESmOHDnCQw89RNu2bfnggw+oU6cOL774Ii+++KLXoRljTImIujaKSLJt2zYuu+wytm/fjoiQnJzMpEmTqFGjhtehGWNMibFEcRoaNWpEQkIC5513HqmpqXTs2NHrkIwxpsRZ1VMRZGdnM2PGDPbs2QNAfHw8K1asIC0tzZKEMabUskQRpLVr19K+fXsGDRrEiBEjTqxv1KgRsbFWMDPGlF6WKAqxf/9+Bg4cSMeOHfn8889p2LAhV199tddhGWNM2FiiKICqsnjxYpo3b87MmTOJiYlh+PDhbN68mSuvvNLr8IwxJmyszqQAGzZs4OabbwbgoosuIjU1ldatW3sclTHGhJ8lCh85OTnExMQA0LZtW4YOHUrLli256667KFfOCl/GmLLJPv1cq1atolWrVqxevfrEuqlTp3LPPfdYkjDGlGll/hPwl19+4Y477uDSSy9ly5YtTJ061euQjDEmopTZRJGbm8u8efNo3rw5CxcuJD4+nkcffZRXXnnF69CMMSailMk2iu+//57bbruNjz76CIAePXowc+ZMmjZt6nFkxhgTecpkiaJq1aps27aN//u//2Px4sWsWLHCkoQxxhSgzJQo3nrrLbp27Up8fDy1atVi2bJltGzZkmrVqnkdmjHGRLRSX6L48ccfufbaa+nZsydPPPHEifUXXnihJQljjAlCqU0U2dnZTJ06lRYtWvDGG29QuXJlatas6XVYxhgTdUpl1dOaNWtITk5mw4YNAFx//fVMmzaN+vXrexyZMcZEn1KXKD755BMuuugiVJXGjRszY8YMevXq5XVYxhgTtUpdomjfvj2XXXYZ7dq1Y/To0VSsWNHrkIwxJqpFZRtFt8QzTvz89ddf86c//Ylt27YBICL861//YtKkSZYkjDGmBERliWJB3/ZkZmYyefJkHnvsMTIzM0lISGDJkiUANjaTMcaUoJB+oopITxHZKiLfiMjIfLbHi8gr7vZPRKRxMOd95513aNOmDePGjSMzM5O+ffuSmppa0uEbY4wBRFVDc2KRGGAb8EcgHVgH3Kyqm332SQHaqGqyiPQGrlXVmwKdN6ZCFc09dgiAFi1akJqaSufOnUPyHowxprQQkU9VNak4x4ayRNEe+EZVv1PV48BiwH8O0auBv7s/LwG6i4gEOmnuscMkJCQwadIk1q9fb0nCGGNCLJQlihuAnqp6j7vcB+igqgN99vnS3SfdXf7W3SfD71z9gH7uYivgy5AEHX1qAxmF7lU22L04ye7FSXYvTkpU1SrFOTCUjdn5lQz8s1Iw+6Cqc4G5ACKSVtziU2lj9+Ikuxcn2b04ye7FSSKSVtxjQ1n1lA6c6bPcAPipoH1EJBaoBvwWwpiMMcYUUSgTxTqgmYg0EZE4oDewzG+fZcAd7s83AO9qqOrCjDHGFEvIqp5UNVtEBgJvATHAfFXdJCLjgTRVXQY8B7wgIt/glCR6B3HquaGKOQrZvTjJ7sVJdi9OsntxUrHvRcgas40xxpQO9gizMcaYgCxRGGOMCShiE0Wohv+IRkHci2EisllENorIOyLSyIs4w6Gwe+Gz3w0ioiJSartGBnMvROTP7u/GJhF5KdwxhksQfyMNRWSViHzu/p1c4UWcoSYi80XkF/cZtfy2i4hMd+/TRhE5P6gTq2rEvXAav78FzgLigA1AS799UoBU9+fewCtex+3hvegGVHR/vrcs3wt3vyrAamANkOR13B7+XjQDPgdquMt1vI7bw3sxF7jX/bklsN3ruEN0LzoD5wNfFrD9CmA5zjNsHYFPgjlvpJYoQjL8R5Qq9F6o6ipVPeIursF5ZqU0Cub3AuBRYApwLJzBhVkw9+IvwExV3Qugqr+EOcZwCeZeKFDV/bka//tMV6mgqqsJ/Cza1cBCdawBqotIvcLOG6mJoj7wo89yursu331UNRvYD9QKS3ThFcy98HU3zjeG0qjQeyEi7YAzVfXNcAbmgWB+L84BzhGRD0VkjYj0DFt04RXMvRgH3CYi6cC/gUHhCS3iFPXzBIjc+ShKbPiPUiDo9ykitwFJQJeQRuSdgPdCRMoBTwN3hisgDwXzexGLU/3UFaeU+YGItFLVfSGOLdyCuRc3A8+r6lMiciHO81utVDU39OFFlGJ9bkZqicKG/zgpmHuBiPwBGAVcpaqZYYot3Aq7F1VwBo18T0S249TBLiulDdrB/o0sVdUsVf0e2IqTOEqbYO7F3cA/AFT1YyABZ8DAsiaozxN/kZoobPiPkwq9F251yxycJFFa66GhkHuhqvtVtbaqNlbVxjjtNVeparEHQ4tgwfyNvIHT0QERqY1TFfVdWKMMj2DuxQ6gO4CItMBJFL+GNcrIsAy43e391BHYr6q7CjsoIqueNHTDf0SdIO/FE0Bl4FW3PX+Hql7lWdAhEuS9KBOCvBdvAT1EZDOQAzyoqnu8izo0grwX9wPzRGQoTlXLnaXxi6WIvIxT1VjbbY8ZC5QHUNVUnPaZK4BvgCNA36DOWwrvlTHGmBIUqVVPxhhjIoQlCmOMMQFZojDGGBOQJQpjjDEBWaIwxhgTkCUKE3FEJEdE1vu8GgfYt3FBI2UW8ZrvuaOPbnCHvEgsxjmSReR29+c7ReR3Ptv+JiItSzjOdSLSNohjhohIxdO9tim7LFGYSHRUVdv6vLaH6bq3qup5OINNPlHUg1U1VVUXuot3Ar/z2XaPqm4ukShPxjmL4OIcAliiMMVmicJEBbfk8IGIfOa+Lspnn3NFZK1bCtkoIs3c9bf5rJ8jIjGFXG410NQ9trs7h8EX7lj/8e76yXJyDpAn3XXjROQBEbkBZ8ytF91rVnBLAkkicq+ITPGJ+U4RebaYcX6Mz4BuIjJbRNLEmXviEXfdYJyEtUpEVrnreojIx+59fFVEKhdyHVPGWaIwkaiCT7XT6+66X4A/qur5wE3A9HyOSwamqWpbnA/qdHe4hpuATu76HODWQq5/JfCFiCQAzwM3qWprnJEM7hWRmsC1wLmq2gaY4Huwqi4B0nC++bdV1aM+m5cA1/ks3wS8Usw4e+IM05FnlKomAW2ALiLSRlWn44zl001Vu7lDeYwG/uDeyzRgWCHXMWVcRA7hYcq8o+6Hpa/ywAy3Tj4HZ9wifx8Do0SkAfCaqn4tIt2B3wPr3OFNKuAknfy8KCJHge04w1AnAt+r6jZ3+9+BAcAMnLku/iYi/wKCHtJcVX8Vke/ccXa+dq/xoXveosRZCWe4Ct8Zyv4sIv1w/q7r4UzQs9Hv2I7u+g/d68Th3DdjCmSJwkSLocBu4DyckvD/TEqkqi+JyCdAL+AtEbkHZ1jlv6vqQ0Fc41bfAQRFJN/5TdyxhdrjDDLXGxgIXFqE9/IK8GdgC/C6qqo4n9pBx4kzi9tkYCZwnYg0AR4ALlDVvSLyPM7Ad/4EWKmqNxchXlPGWdWTiRbVgF3u/AF9cL5Nn0JEzgK+c6tbluFUwbwD3CAiddx9akrwc4pvARqLSFN3uQ/wvlunX01V/43TUJxfz6ODOMOe5+c14BqcORJecdcVKU5VzcKpQuroVltVBQ4D+0WkLnB5AbGsATrlvScRqSgi+ZXOjDnBEoWJFrOAO0RkDU610+F89rkJ+FJE1gPNcaZ83Izzgfq2iGwEVuJUyxRKVY/hjK75qoh8AeQCqTgfum+653sfp7Tj73kgNa8x2++8e4HNQCNVXeuuK3KcbtvHU8ADqroBZ37sTcB8nOqsPHOB5SKySlV/xemR9bJ7nTU498qYAtnoscYYYwKyEoUxxpiALFEYY4wJyBKFMcaYgCxRGGOMCcgShTHGmIAsURhjjAnIEoUxxpiA/j8y50Dt0Sq/AQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "aid = 'CHEMBL1614249'\n",
    "assay = pubchem_d.loc[pubchem_d['assay_chembl_id']==aid]\n",
    "acts = pd.concat((assay.loc[assay['activity_comment'] == 'Active'], \n",
    "                  assay.loc[assay['activity_comment'] == 'active']))\n",
    "inacts = pd.concat((assay.loc[assay['activity_comment'] == 'inactive'],\n",
    "                    assay.loc[assay['activity_comment'] == 'inconclusive'], \n",
    "                    assay.loc[assay['activity_comment'] == 'Inconclusive'], \n",
    "                    assay.loc[assay['activity_comment'] == 'Not Active']))\n",
    "print(len(acts),len(inacts))\n",
    "ps,tgt,oob_tpl = run_it2(acts,inacts)\n",
    "oob_tgt,oob_probs = oob_tpl\n",
    "oob_probs = [x[1] for x in oob_probs]\n",
    "auroc = metrics.roc_auc_score(oob_tgt, oob_probs)\n",
    "scores = [1 if x>=0.5 else 0 for x in ps]\n",
    "kappa = metrics.cohen_kappa_score(tgt,scores)\n",
    "confusion = metrics.confusion_matrix(tgt,scores)\n",
    "fpr,tpr,_ = metrics.roc_curve(oob_tgt,oob_probs);\n",
    "plt.figure()\n",
    "lw = 2\n",
    "plt.plot(fpr, tpr, lw=lw)\n",
    "plt.plot([0, 1], [0, 1], color='k', lw=lw, linestyle='--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title(\"%s: AUC=%.3f\"%(aid,auroc))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An assay that did work well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5614 43421\n",
      "ratio: 0.129 kappa: 0.032, AUC: 0.859, OOB_AUC: 0.850\n",
      "[[8680    5]\n",
      " [1102   21]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      1.00      0.94      8685\n",
      "           1       0.81      0.02      0.04      1123\n",
      "\n",
      "   micro avg       0.89      0.89      0.89      9808\n",
      "   macro avg       0.85      0.51      0.49      9808\n",
      "weighted avg       0.88      0.89      0.84      9808\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xd4VGX2wPHvSQcSEkhogjQpoQqCgKKADVHXiq4FsWFBREUsoIgoC4qoKEoJ8LMs9rIqWEBdBXVVkCKIQESaEEAglIQkpM2c3x93CCGmDJDJzCTn8zzzMLefXJI585b7vqKqGGOMMSUJ8XcAxhhjApslCmOMMaWyRGGMMaZUliiMMcaUyhKFMcaYUlmiMMYYUypLFMYYY0pliaIKEZHrRGSpiGSIyA4RmSciZ3i2PS4ibxRzjIpIi0L75HmOP/TaX2TfnSISVmhdmIjsEhEttG6hiGR7jk8Tke9EpEOh7cXG4tk21PMz5IjIa8Vsry4i00Qk9dC5C207S0QWeNZvLuU+9fb8LONK2P6NZ3uYZ7muiLwtIts95/5BRLoX2r+BiMz1bFcRaVrStUvjuW/7RCSymPW3FlnXR0RSCi2LiNwjIr+JSKaIpIjI+4Xvu5cxiIg8LSJ7PK+JIiKl7H+3iGwSkXTP/9sZhbYV9/vUvND2TiKyTESyPP92OppYTfmxRFFFiMhw4AXgSaAe0BiYBlx6lKd6V1WjC73iimzfD1xQaPlCYF8x5xmqqtFAPLAQeN3L628HxgGvlLB9JlAbaOP5975C2zI9xz1Y0slFJByYDCwuYfsAIKzI6mhgCdDFc81/A5+JSLRnuxuYD/Qv6bpl8SSXMwEFLjmGU0wG7gXu8cTYCvgYuOgoz3M7cBlwMtAR+AdwRwkxdwcmAFcCscDLwEciElpot6K/Txs9x0YAc4A3gFo493SOZ72pYJYoqgARiQXGAnep6oeqmqmqear6iaqW+KF5jF4Hbii0fAMwu6SdVTUfeAdo683JPfF/DOwpuk1EWuN8iN6uqrtV1aWqywod+7Oqvg5sLOUS9wNfAsnFnD8WGAM8VCSmjao6SVV3eK45E4gAWnu271TVaTjJ5FjdACwCXgNuPJoDRaQlcBdwrap+o6o5qpqlqm+q6oSjjONG4DlVTVHVbcBzwE0l7NsUWK2qy9QZAmI2kADU9eI6fXAS8gueeF8EBDj7KOM15cASRdVwGhAFfFQB1/oY6CUicSISh/MteE5JO3u+IQ7A+RA8Xt2BP4EnPFVPq0TE62/xItIEuAUnqRbnSWA68FcZ5+mEkyjWe3ndkSLyaRm73QC86XmdLyL1vDm3xzlAiqr+XEYM+0t6Fdq1HbCy0PJKz7rizANCRaS7pxRxC7CCI+/fxSKyV0RWi8idRa7zqx45xtCvpVzL+FDRIrSpnOKBVM+399L8U0T+cZT7/KKqZxVazgY+Aa7G+QY417OuqBdF5FmgOnAQuKKM63qjEdAe+A9wAk6C/ExE1qjqWi+OfxEYraoZRavdRaQr0BOn+qZRSScQkZo4paonVDXNm6DL+lbvqddvArynqqkisgG4Dnjem/Pj/P/v8CIGb0oX0UDhnysNiBYRKfKhDnAA5//ifzi/C/uBCwrt9x5OVeFOnCT/HxHZr6pvF3OdQ9eK8SJGU86sRFE17AESpFAjcwneU9W4wi8v9jmrmH1m43wDLq3a6R7P+aNw6rk/EJGOXv48JTkI5AHjVDVXVb8FFgB9yzpQRC4GYlT13WK2heC059xbWrIVkWo4SXKRqj51jD9DcW4EvlTVVM/yWxxZ/ZQPhBc5JhznXoDz/9+gnGLJAGoWWq4JZBSTJABuxSlFtMMpYV0PfCoiJwCo6hpV3e6prvsRpx3lyhKuc+haB8rp5zBHwRJF1fATzrf6yyroet/jfDDVw/k2WSJVdavq9zjVNGV+oJfh1+M49hygq4j8JSJ/4ZSIhonIHJwPqK7Au55th9oaUkTkTABPT6SPgW2U0Lh7LDzJ559A70Kx3QecLCIne3bbgtMeUFgznGo4gK+BRp5SUUnXeaRI76MjXoV2XY3TkH3IyZ51xTkZ+ERV13n+n+fjlGxOL2F/xSl5HLpOxyI9qjqWci3jQ5YoqgBPFchjwFQRuUycLqThInKBiEz0wfUUuBi4pIRvmkcQkdNwGrMLfwiEiEhUoVekZ98wEYkCQnHqv6MKlZS+w/nQfNizX0+cRtEvPMeGeI4NdxYlqlAvmtE4PYE6eV5zgVnAzThVHicU2nah55guwGJPT6kPcEo0N6iqu5ifMQo41K010rPsjcsAl+f+HLp+G5xkfKjTwLvAzSLSzem9Kq1wksk7AKr6B06J6G1xus1GeH72a0RkpGefJ4v0PjriVSie2cBwEWnoKRncj9PAXpwlwEUi0twT13k49/g3zz25VERqebZ1w+mRdag9a6Hn575HRCJFZKhn/Tde3jdTnlTVXlXkhdNovBSnm+hfwGfA6Z5tjwNvFHOMAi0K7ZOHUy1Q+FW36L5FztECT/7wLC/EKeEcOn49cF+h7Y97zlX4lVLKtscLHdsOpwSVCawBLi+0rU8xxy4s4V69hlOFVdy2pp5jwzzLvT3LWUXuy5lF7uMRr0LbHgHmlXCt+Ti9jIqu/6fn//BQDLfgJNp0z/0cCYQU2l9w2ldWe+LchpNg2h3l75AAE4G9ntdEQAptL/i5PfuOxUneB4C1wMBC+76NUy2WgdPL7J4i1+oMLMNJwMuBzv7+G6qqL/H8hxhjjDHFsqonY4wxpbJEYYwxplSWKIwxxpTKEoUxxphSBd2T2QkJCdq0aVN/h2GMMUFl2bJlqapa51iODbpE0bRpU5YuXervMIwxJqiIyJ9l71U8q3oyxhhTKksUxhhjSmWJwhhjTKksURhjjCmVJQpjjDGlskRhjDGmVD5LFCLyiojsEpHfStguIvKiiKwXkV9F5BRfxWKMMebY+fI5iteAKZQ8w9kFQEvPqzvOXMTdfRiPMcYEFVXFreBWxeVWVMGlyoHsPPJdSp7LTb5b2Z+Vh1ud7aqKQsGyW5Xc3NzjisNniUJVvxORpqXscikwW51xzheJSJyINFDVUuf2NcYYf1HPB3ZmjovM3Hxy893sP5hHbr6bPJeb3Hw3W/ZmERoirN6ejqqyMTWT5B3p1K0Z5Znfwfmwd6vidh9OAm5VUjNyiQgLwe1ZdpfDLBD7FrxC7s4Nx3UOfz6Z3RDYWmg5xbPub4lCRG4Hbgdo3LhxhQRnjKmccvJdpB/MJ9/tJi9fSc/O42Ceix1p2WxOzWTD7gx2peeQ/Fc68dGR7Nh/kMxcFxFhIeS53BzrFD7p2Rll7wTk5h85QaIIhIoQIkJICISIkJXrIiYqjPgaEYSHhhAaIvyxK4OuTWoh4uxz6N8Nu9qxePknxxa0hz8ThRSzrtj/AlWdCcwE6Nq1q820ZEwVoqpk5brYvCeTlH0H2XUgh93p2azZkU5c9QjyXW7yXErKviwycvKJrRZOvts5Zv2uDOrERJLvcpPvUrLyXLiO4mv6vqy8gveHPsBDQ6TgHI1rVyciLIT1uzI4/aR4wkNDCA8Vdmfk0vnEOHJdblrUiaZ+bBR1YiKJqxZOSIjzoR/q+TAPDTmcBAQhJiqM0JDD24+cNrxsa9asYfny5Vx//fXO/bulG38+dgvNmjU7qvMU5s9EkQKcWGi5EbDdT7EYY3xEVTmY52JfVh4Hc11OvbpL2Z2RzZY9WbgVVm1LI7ZaOBt2Z5CT72btjnSqR4SS51L2Zh5f/fruAznFrq9fM4rwMCE8JISNqZmc0SKBfVm5tKgbTat6MXRoGEtCdCTRkWFEhocQWy284Nt7IMrKymLcuHE888wzhIaG0qNHD1q0aIGIcLwDqfozUcwFhorIOziN2GnWPmFM8MjJd7FhVyY5+S7+2JlBamYOq1LSWLsjne1p2dSJjmTb/oPHfP4D2flHLMdEhXEgO5/TmsdTOzqCRrWqERYitKgbTViI800+360kREdSLTy04EM9JiqMyLAQwjzf9sNCQogIq1xPBsybN4+77rqLTZs2ATBo0CDi4+PL7fw+SxQi8jbOZPYJIpICjAHCAVQ1CfgcuBBnIvgs4GZfxWKM8V6ey01qRg77MvNYt/MA837bQc2ocLLz3WzYlYHLrWzYnUF+GVU4RZNE9YhQsnJdtK4XQ1ioEBYaQuqBHBrERtGtWW3SDubRvmEsblUaxlUjvkYkDeKiiAoPpUZE6FFXwVQF27ZtY9iwYXzwwQcAdOzYkaSkJE477bRyvY4vez1dW8Z2Be7y1fWNMUdSVVL2HWT9rgy27M0i+a8DpB/M4/PfdlC/ZhQ5+e5jquZp37Am1SPCSM3I4bTm8cREhdO+YU1a1I0mvoZTdVMtItQHP5G56667mDNnDtWrV2fs2LHce++9hIWV/8d60M1HYYz5O1VlX1YeP23Yw8zvNxIWIgVtAfluNxt2Z5baiLsjLbvY9ac0jiM7z81JdaM5J7EuUeEhuBVOiKtGfA2n+se+6Ves/Pz8gmTw9NNPEx4eznPPPefTHqGWKIwJMqrK1r0H2bI3i/+tT+WNRX9y8Ch680SFh9AsIZomtavTIC6Kc9vUo35sFLHVwokICyEqLLTS1eFXBmlpaTz66KOsW7eO+fPnIyK0bt2a999/3+fXtkRhTABSVbLz3PywPpUd6dl8t2432XkuftuWdkSXzeL0blWHni3i6dKktqcR12nAjQoP4YTYaoQEaK8dUzxV5f3332fYsGHs2LGD0NBQVqxYQefOnSssBksUxgSA9Ow8FiTv4qs1O/n0V+86/7WoGw1A1ya1GHRGM1rWi/FliMYPNmzYwNChQ5k/fz4Ap512GklJSXTs2LFC47BEYUwFUlU2pWayflcG3yTvYn9WHvNX/1Xi/oce6Ordqg4ut3JKk1o0ja9Oq3oxRIVbA3Fl9uyzzzJ69Giys7OJi4vj6aef5tZbbyUkpOKrBS1RGOMjGTn5fP7rDn7dtp/lf+5nzY70Mo85tWktujSpzYUd6tP+hFirJqrCsrKyyM7OZuDAgTz77LPUrVvXb7FYojCmHKVn5/HS13/w8v82lTqgW/WIUM5KrEtuvpuzWtelQ8NY2jesaT2IqrDdu3fz+++/c8YZZwAwYsQI+vTpQ69evfwcmSUKY46ZqvJrShpL/9zH9IUbSM34+1ARYSFCg7goBnRvQvOEGrRpUJMTa1f3Q7QmULndbl555RUeeughwsLCSE5Opnbt2kRGRgZEkgBLFMZ4bfv+g6zens6363ax8PfdpOwrfniKE2KjuK1Xc67s0oiYqPAKjtIEk99++43Bgwfzww8/AHDeeeeRlZVF7dq1/RzZkSxRGFPEoZLCDxtS+WnDHlZtS2N/KV1SoyPD6NkinnMS63FZ54b2DIIpU2ZmJmPHjmXSpEnk5+dTr149XnjhBa6++uqArH60RGEMTmnhk5Xb+Tp5Fz9v2lvqvm0a1OTcNnW5tNMJtKhrXVLN0bvyyisLHpobMmQI48ePJy4uzt9hlcgShamSNqVmMm3Ber5au7PE0kLXJrWoFhFKx0axXN65Ic0SogN2iGkTXEaMGMHOnTuZPn063bsH/gzQlihMlaCqfLF6J//+cTM/bdxT7D41o8IYdEZzzkqsY11TTbnJz8/npZdeYvPmzUyePBmAPn36sHTpUr88E3EsLFGYSmv7/oP8+6fNLEjexbqdxU9DOaB7Yy7q2IAezeItMZhy9/PPP3PHHXewYsUKAG6//XbatWsHEDRJAixRmEpo94Ecbnr1Z1Zv//sDbhe0r8+FHRpwfrv61uhsfGb//v088sgjJCUloao0adKEKVOmFCSJYGOJwlQK+7NyeWfJVt5avIUte7OO2HZum3oMPbsFJzeKDcgeJaZyeeeddxg2bBg7d+4kLCyM+++/n9GjR1OjRg1/h3bMLFGYoOVyK1O+Wc/bP2/hr/S/z6fw2D/acssZxz6hvDHH4ssvv2Tnzp307NmT6dOn06FDB3+HdNwsUZigs27nAV78+o+/jbJ6QmwUV5/amAs61KeVjaRqKkhOTg7btm2jefPmAEycOJEzzzyTG2+8MajaIUpjicIEvB83pDL7xz9Z+udeUjP+PlVnr1Z1GNkvkTYNYqxqyVSob775hjvvvJOQkBBWrlxJREQECQkJ3Hzzzf4OrVxZojABKTffzZi5q3n75y3Fbo+ODOOBvq24rnsTa5Q2FW7nzp088MADvPHGGwAkJiaSkpJSUKqobCxRmICydW8Wg99Y9rceS20b1KRLk1pc1vkETm4UR1ioJQdT8dxuN7NmzWLkyJHs37+fqKgoHn30UR588EEiIiL8HZ7PWKIwfqWqfLlmJ28u3sLWvVlsSs08Yvs1p57Iwxe2IbaaDa5n/O/yyy9n7ty5AJx//vlMnTqVk046yc9R+Z4lCuMXeS43r/xvE0/NS/7btgaxUYz+R1su7NDAD5EZU7IrrriCn3/+mcmTJ3PVVVdVmTYxUS1ldpUA1LVrV126dKm/wzBHSVXZdSCH1dvTePWHzXz/R+oR2xPrx3Bzz6Zc0KEBNW1obhMg5s6dS0pKCkOGDAGc3+OMjAxiYoKvV52ILFPVrsdyrJUojE+t3ZHOzO828tOGPcU+69CndR3+74au1uZgAsqWLVu45557mDNnDpGRkfTr14/mzZsjIkGZJI6XJQpT7txu5dt1u5m+cAM/bz5yyO4Qgeu6N6ZPq7qclVjXRmM1ASUvL48XX3yRMWPGkJmZSUxMDOPGjaNJkyb+Ds2vLFGYcpOWlceDH6zkyzU7/7ZtzMVtua57YyLDQv0QmTFlW7RoEXfccQe//vorAFdddRXPP/88DRs29HNk/meJwhy3A9l5vL7oTybO//2I9c3r1GBEv0TOb1ffT5EZ473Ro0fz66+/0qxZM6ZMmcKFF17o75AChiUKc1w+WJbCA++vLFhuGFeN89rW4+ELE630YAKaqnLgwAFq1qwJwJQpU5g9ezajRo2ievXqfo4usFivJ3PUsvNczF2xnYf+8+sR60dd2IZbzmhm7Q4m4P3+++8MGTIEEeGrr76qEt1crdeTqRA5+S6e/Gwt//7pzyPWX9G5IROv7Gg9l0zAy87O5qmnnmLChAnk5uYSHx/P5s2badbMRhkujSUK45UFv+/i5leXHLHu3nNacl33xtSrGeWnqIzx3ldffcWQIUNYv349ALfccgsTJ04kPj7ez5EFPp8mChHpB0wGQoH/U9UJRbY3Bv4NxHn2Gamqn/syJnN0/tyTyRXTfmRP5uFRW+87txX3ntvSj1EZ4z1VZdCgQbz66qsAtG3blqSkJM4880w/RxY8fJYoRCQUmAqcB6QAS0RkrqquKbTbo8B7qjpdRNoCnwNNfRWT8V52nosJ85J57cfNBev6tavPpKtPpnqEFURN8BARmjZtSrVq1XjssccYPnx4pR7Azxd8+RffDVivqhsBROQd4FKgcKJQoKbnfSyw3YfxGC+oKg9+8CsfLEs5Yv3X9/fmpDrRforKmKOzYsUKduzYwQUXXADAiBEjGDhwoLVFHCNfJoqGwNZCyylA9yL7PA58KSJ3AzWAc4s7kYjcDtwO0Lhx43IP1MDBXBcTv0jm/aUpZOTkF6x/un8H/tn1xCrRK8QEvwMHDjBmzBgmT55MfHw8ycnJ1K5dm8jISEsSx8GXiaK4T5aifXGvBV5T1edE5DTgdRFpr6ruIw5SnQnMBKd7rE+ircJe/2kzo+esLliOCA3hoo4N+Ndl7YmOtGomE/hUlY8//ph77rmHlJQUQkJCuO666wgPtwEmy4MvPwVSgBMLLTfi71VLg4B+AKr6k4hEAQnALh/GZTzW7TzABZO/x+U+nHvvP68VN5/RzBKECRp//vknQ4cO5dNPPwWga9euzJgxg1NOOcXPkVUevvw0WAK0FJFmwDbgGuC6IvtsAc4BXhORNkAUsNuHMRkg7WAe0xauZ8a3GwvW9WldhynXnWIJwgQVVaV///4sW7aMmjVr8uSTTzJ48GBCQ21UgPLks08FVc0XkaHAFzhdX19R1dUiMhZYqqpzgfuBWSJyH0611E0abI+KB5m3Fm/hiU9Wk5Pv1O61aVCTf99yKnVj7FkIEzzcbjchISGICM8++yxJSUk8//zzNGhgk135gg3hUUVs3ZvFNTMXsW3/QQCaJ9TgljOacW23xjbkhgkae/bsYeTIkQDMmjXLz9EEFxvCw5QoO8/FuM/W8MaiLQXrBnRvzLjL2ltPJhM0VJXZs2fzwAMPkJqaSkREBGPGjKFRo0b+Dq1KsERRSakqI/+zineXbj1i/Tu396BHcxuywASPtWvXcuedd/Ltt98C0KdPH6ZPn25JogJZoqiEcvPdDPr3koJ5qSPDQji/XX1G/6MtdWIi/RydMd5RVR577DGefvpp8vLySEhI4LnnnmPgwIFWGq5gligqmV3p2XR78uuC5ZEXJHJHr+b2h2WCjoiwbds28vLyuO2225gwYQK1a9f2d1hVkjVmVxK5+W5mfb+RZ744PMvciH6J3NnnJD9GZczR2b59O6mpqXTs2BGA1NRUfv/9d3r27OnnyIKfNWZXcdMWrmfKN+vJynUBEBMVxqwbulpbhAkaLpeL6dOnM2rUKBo2bMiKFSuIiIggISGBhIQEf4dX5VmiCHKf/bqjYK7qhOhIRvRrzZVdGllVkwkay5cv54477uBQTUGvXr1IT0+3BBFAvEoUIhIBNFbV9T6Ox3hJVZn89R+88N8/AGhVL5ovhvWyBGGCRnp6OqNHj2bKlCm43W4aNWrEiy++yGWXXWa/xwGmzEQhIhcBk4AIoJmIdALGqOrlvg7OFC89O4+73/qFb9c5o51UCw/lwyE97Y/LBA1VpVevXqxcuZLQ0FCGDx/O448/TkxMjL9DM8XwpkQxFmd48AUAqrpCRFr4NCpTouw8F70nLmBfVh6hIcKjF7XhptObWpIwQUVEuO+++5g2bRozZsygU6dO/g7JlMKbRJGnqvuLfBAFV1epSuKvtGx6P7OgYJymWTd04ezEen6Oypiy5ebmMmnSJEJDQ3nwwQcBuOGGG7j++uttAL8g4E2iWCsi/wRCPCPB3gss8m1Ypqilm/dyZdJPBcv2hLUJFt9//z2DBw9mzZo1REZGcsMNN1CvXj1ExJJEkAjxYp+hQBfADXwIZOMkC1NBRn20qiBJVI8I5c1bu1uSMAEvNTWVW265hV69erFmzRpatmzJp59+Sr16VgoONt6UKM5X1RHAiEMrROQKnKRhfMjtVu59dwWfrHTme2pRN5r/DD6d2Oo2a5cJXKrKa6+9xoMPPsiePXuIiIjg4YcfZuTIkURF2XD2wcibRPEof08Ko4pZZ8rR/qxcbnzlZ1ampAEwsX9H/nnqiWUcZUxgeOONN9izZw9nn30206ZNo3Xr1v4OyRyHEhOFiJyPM01pQxGZVGhTTZxqKOMj7y3Zyug5vxU0Ws8c2IW+7er7OSpjSpaVlUVaWhoNGjRARJg2bRpLlixhwIAB1iOvEiitRLEL+A2nTWJ1ofUHgJG+DKqqcruVq2b8xLI/9wFQv2YUb9zajRZ1rW+5CVzz5s3jrrvuonnz5nz11VeICK1bt7ZSRCVSYqJQ1V+AX0TkTVXNrsCYqqSd6dlcPeMnNu/JAqBH89q8eWsPm33OBKxt27YxbNgwPvjgAwBiYmLYs2ePDb1RCXnTRtFQRMYDbYGClihVbeWzqKqYD5enMPy9lQXLz111Mv272KQsJjC5XC6mTp3Ko48+yoEDB6hRowZjx47lnnvuISzMho+rjLz5X30NGAc8C1wA3Iy1UZQLt1u56KX/sXZHesE6SxImkLndbnr37s0PP/wAwGWXXcbkyZNp3LixnyMzvuRNoqiuql+IyLOqugF4VES+93VglV3Kvix6TVyA2/OMe+fGcfzfDV2Jj7YZ6EzgCgkJoW/fvmzZsoUpU6ZwySWX+DskUwG8SRQ54nRb2CAig4FtQF3fhlW5pWXlcfFL/ytIEuMvb8+A7k38G5QxxVBV3nvvPcLCwujfvz8AI0aMYPjw4URHR/s5OlNRvEkU9wHRwD3AeCAWuMWXQVVm2XkuLp/+Q8Ggft8+2IdGtar7Oyxj/mbDhg0MGTKEL7/8kjp16nD22WdTq1YtIiMjiYy0km9VUmaiUNXFnrcHgIEAImKV6Mcg3+VmwP8tZuPuTAA+v+dMSxIm4OTk5PDMM88wfvx4srOzqVWrFuPHjyc2NtbfoRk/KTVRiMipQEPgf6qaKiLtcIbyOBuwZHEUtu0/yDnPLSQ7z+kH8J87T6d1fXs+wgSWhQsXcuedd5KcnAzAwIEDefbZZ6lb12qbq7ISBwUUkaeAN4EBwHwRGYUzJ8VKwLrGHoXf/zrAeZO+LUgSE67oQJcmtfwclTFHcrlcDBkyhOTkZFq3bs0333zD7NmzLUmYUksUlwInq+pBEakNbPcs/14xoVUOW/dmcdnUHziY5wLgs3vOoN0JVoQ3gcHtdpOdnU316tUJDQ1l+vTpfPfddzz00EPWDmEKlJYoslX1IICq7hWRZEsSR2dnejZnPbuQfLdyQmwUr93SjVb1rLrJBIZVq1YxePBgEhMTefnllwHo3bs3vXv39nNkJtCUliiai8ihEWIFaFpoGVW9wqeRBbnsPBcXTv6efE8f2A+H9KR+rA2xbPwvMzOTsWPHMmnSJPLz89m0aRP79u2jVi2rDjXFKy1R9C+yPMWXgVQmK7fuZ+DLi0nPzgdgzl2WJExg+OSTTxg6dChbtmxBRBgyZAjjx48nLi7O36GZAFbaoIBfV2QglUV2notLp/5QsPzeHadx8on2R2j8Kz8/n6uvvpoPP3QqBTp16sSMGTPo1q2bnyMzwcBG8Cpnj3y4quD9jyPP5oS4an6MxhhHWFgYsbGxREdH869//YuhQ4faAH7Ga97MmX3MRKSfiPwuIutFpNg5LETknyKyRkRWi8hbvozH195bupUPf9kGwPDzWlmSMH61ePFiFi9eXLD8zDPPsHbtWoYNG2ZJwhwVr39bRCRSVXOOYv9QYCpwHpACLBGRuaq6ptA+LYGHgZ6quk9EgrbDdsq+LB7By/opAAAgAElEQVT64FcALu/ckHvOaenniExVtX//fh5++GFmzJhBYmIiK1asICIigvj4eH+HZoJUmSUKEekmIquAPzzLJ4vIS16cuxuwXlU3qmou8A7OsxmF3QZMVdV9AKq666iiDxBpWXmc8fSCguWJV3b0YzSmqlJV3nrrLRITE0lKSiI0NJRLLrkEl8vl79BMkPOm6ulF4B/AHgBVXQmc5cVxDYGthZZTPOsKawW0EpEfRGSRiPTz4rwBRVXpn/RjwfKsG7oSHurTGj1j/uaPP/6gb9++DBgwgJ07d9KzZ09++eUXJkyYQLVqVgVqjo83VU8hqvpnkQnSvfmKUtwcnlrM9VsCfXDGjvpeRNqr6v4jTiRyO3A7EHATpEyYn8z6XRkAvHN7D3o0t+K9qVh5eXmcffbZpKSkULt2bSZOnMjNN99MSIh9YTHlw5vfpK0i0g1QEQkVkWHAOi+OSwFOLLTcCGcYkKL7zFHVPFXdBPyOkziOoKozVbWrqnatU6eOF5euGGkH85jx7UYAXri6kyUJU6FUne9d4eHhjB8/nptuuonk5GQGDRpkScKUK29+m+4EhgONgZ1AD8+6siwBWopIMxGJAK4B5hbZ52M81VgikoBTFbXRu9D9747XlwIQFR7CpZ1O8HM0pqrYuXMnAwcOZNy4cQXrbrjhBl599VUC6YuUqTy8qXrKV9VrjvbEqpovIkOBL4BQ4BVVXS0iY4GlqjrXs62viKzBqc56UFX3HO21/GH+bztYtHEvAG/f1oMiVXPGlDu3282sWbMYOXIk+/fvJy4ujmHDhhETY+OHGd/yJlEsEZHfgXeBD1X1gLcnV9XPgc+LrHus0HvFKa0M9/acgWBXejYPvO90hb3p9KZ0bmxj5BjfWrlyJYMHD2bRokUA9OvXj6lTp1qSMBWizKonVT0JGAd0AVaJyMcictQljMrkqXnJZOTk0yS+OqMuauPvcEwllpeXxwMPPECXLl1YtGgRDRo04L333uPzzz+nefPm/g7PVBFetXip6o+qeg9wCpCOM6FRlXT77KV85Hn6+pkrT7ausManwsLC+OWXX3C73dx9992sXbuWq666yqo6TYUqs+pJRKJxHpS7BmgDzAFO93FcAWlzaiZfrtkJwDmJdenWrLafIzKV0ZYtW3C5XDRr1gwRISkpibS0NLp27erv0EwV5c3X4d9wejpNVNUWqnq/qi4u66DKxuVW/jnjJwDq14zi5ZtO9XNEprLJy8vj2WefpU2bNtx2220F3V9btmxpScL4lTeN2c1V1e3zSALctAXr2XXAGerq5Zvsj9aUr59++onBgwfz669OJ4natWuTlZVFjRo1/ByZMaUkChF5TlXvB/4jIkWfqK5SM9yt33WA575ynjF84pJ2Nue1KTf79u1j5MiRzJw5E4BmzZoxdepULrjgAj9HZsxhpZUo3vX8W6VntnO7lXMnfQfA6SfFc+PpTf0bkKk0cnJy6NSpE1u2bCE8PJwHH3yQUaNGUb16dX+HZswRSpvh7mfP2zaqekSy8DxIVyVmwLv77V8K3r9wTSc/RmIqm8jISAYNGsTXX3/N9OnTadu2rb9DMqZY3jRm31LMukHlHUgg+nL1X3y2agcA4y9vT90Ym/faHLvs7GzGjBnDW28dnp/rkUceYeHChZYkTEArrY3iapwusc1E5MNCm2KA/cUfVXlk5ORz++vLAOjWrDYDujfxc0QmmH311VcMGTKE9evXU7duXS6//HKqVatmM82ZoFDab+nPOHNQNMKZqe6QA8AvxR5RSagqPSd8U7A89bpT/BiNCWZ//fUXw4cP5+233wagXbt2JCUl2RwRJqiU1kaxCdgE/LfiwgkMT81LJu1gHgCTr+lEnZhIP0dkgo3L5WLGjBk88sgjpKWlUa1aNcaMGcN9991HRESEv8Mz5qiUVvX0rar2FpF9HDnhkOCM51cpH0vOys1n5nfOSOddm9Ti0k5FJ+Uzpmwul4uXXnqJtLQ0LrzwQqZMmUKzZs38HZYxx6S0qqdD050mVEQggWLEf1YBEFc9nPfuOM3P0ZhgcuDAAVwuF3FxcURERDBr1ix27tzJFVdcYWMzmaBWYq+nQk9jnwiEqqoLOA24A6iUj4uO/2wNn6x0JuGbObArISH2x23Kpqp8+OGHtGnThvvvv79g/RlnnEH//v0tSZig50332I9xpkE9CZiNMzDgW6UfEnwycvJ57cfNAPRuVccG/DNe2bx5M5dccgn9+/dn27Zt/Pbbb2RnZ/s7LGPKlTeJwq2qecAVwAuqejdQ6SruR320ijyXUj0ilFdtwD9Thry8PJ5++mnatm3Lp59+Ss2aNZkyZQo//vgjUVH2vI2pXLyaClVErgIGApd51oX7LqSKt3LrfuascKqcXru5m1U5mVJlZWXRo0cPVq1y2rOuueYaJk2aRIMGDfwcmTG+4U2iuAUYgjPM+EYRaQa87duwKlbStxsAuKB9fatyMmWqXr06Xbt2JSsri2nTptG3b19/h2SMT5WZKFT1NxG5B2ghIonAelUd7/vQKsa7S7Yw77e/ALjrrBZ+jsYEIlVl9uzZnHTSSZxxxhkAPP/880RERNiDc6ZK8GaGuzOB14FtOM9Q1BeRgar6g6+D8zVVZeoCpzTRr1192je04cPNkdauXcudd97Jt99+S5s2bVixYgURERHExtrviqk6vKl6eh64UFXXAIhIG5zEEfSz98xZsZ0te7MIDxUmX2sjw5rDDh48yPjx45k4cSJ5eXnUqVOHhx9+mPDwStU8Z4xXvEkUEYeSBICqrhWRoB+DYOveLIa9uwKAs1rXJTIs1M8RmUAxf/587rrrLjZudJ7Qv+2225gwYQK1a1v7lamavEkUy0VkBk4pAmAAlWBQwBtfdabbaJ5QgxkDu/g5GhMoMjIyGDhwIKmpqbRv356kpCR69uzp77CM8StvEsVg4B7gIZw2iu+Al3wZlK/9vGkvG3dnAvDUFR3sydkqzuVy4Xa7CQ8PJzo6msmTJ5OSksJ9991nVU3GUEaiEJEOwEnAR6o6sWJC8i1VZczc1QDceFoTujeP93NExp+WLVvGHXfcwaWXXsro0aMBuO666/wclTGBpcQns0XkEZzhOwYAX4lIcTPdBZ2PV2xj7Y50oiPDuO+8Vv4Ox/hJeno69957L926dWPZsmW8/vrr5OXl+TssYwJSaUN4DAA6qupVwKnAnRUTku/k5LsY/9laAIad25K46kHfJm+Okqry/vvvk5iYyIsvvoiIMHz4cJYvX27VTMaUoLSqpxxVzQRQ1d0i4s24UAFt4e+7Sc3IpWZUGANPs6lNq5oDBw5w9dVXM2/ePAC6d+9OUlISnTpZ12hjSlNaomheaK5sAU4qPHe2ql7h08h84KPl2wC49czm1h22CoqOjiYnJ4fY2FgmTJjA7bffTkhI0H//McbnSksU/YssT/FlIL6WdjCP/67dCcCFHer7ORpTUb777jsaNGhAy5YtERFeeeUVoqKiqFevnr9DMyZolDZn9tcVGYivTVuwnny30rZBTVrUjfF3OMbHUlNTeeihh3j11Vc555xz+OqrrxARmjSxKkdjjlaVKHfnudy8vywFgOHW06lSc7vdvPLKK7Ru3ZpXX32ViIgIzjzzTFwul79DMyZo+TRRiEg/EfldRNaLyMhS9rtSRFREfDJ+1Ccrt7M3M5eGcdU4K7GuLy5hAsDq1avp06cPgwYNYu/evZxzzjmsWrWKMWPGEBbmzbOlxpjieP3XIyKRqppzFPuHAlOB84AUYImIzC08bpRnvxicJ78Xe3vuo5HvcvPIR84EMwNPa0KoTUpUKaWlpdGjRw8yMjKoW7cukyZN4rrrrrOn7o0pB2WWKESkm4isAv7wLJ8sIt4M4dENZ+6KjaqaC7wDXFrMfv8CJgI+mWj4mS9/JzvPDUD/Uxr54hLGj1QVgNjYWEaMGMHgwYNJTk5mwIABliSMKSfeVD29CPwD2AOgqiuBs7w4riGwtdByCkXm2haRzsCJqvppaScSkdtFZKmILN29e7cXlz4seccBwOnpVCcm8qiONYFr27ZtXHnllbzxxhsF60aNGsX06dOpVauWHyMzpvLxJlGEqOqfRdZ50zJY3Nc5LdjoPMD3PHB/WSdS1Zmq2lVVu9apU8eLSzt2pWfz7TonsYz+R1uvjzOBKz8/n8mTJ5OYmMh//vMfxowZU9BQbSUIY3zDm0SxVUS6ASoioSIyDFjnxXEpwImFlhsB2wstxwDtgYUishnoAcwtzwbtWd878wnUiYmkQaxNWRnslixZQvfu3Rk2bBgZGRlcdtllfPvtt4SG2sOTxviSN4niTmA40BjYifOB7s24T0uAliLSzDPR0TXA3EMbVTVNVRNUtamqNgUWAZeo6tKj/BmKle9yM2eFk5fGX9a+PE5p/CQzM5OhQ4fSvXt3li9fTuPGjZkzZw4fffQRJ554YtknMMYclzJ7PanqLpwP+aOiqvkiMhT4AggFXlHV1SIyFliqqnNLP8Px+e/anew6kMMJsVGcbV1ig1pYWBj//e9/CQkJYfjw4YwZM4YaNWr4OyxjqowyE4WIzKJQ28Ihqnp7Wceq6ufA50XWPVbCvn3KOt/RmP/bXwBc260xYaFV4rnCSmXDhg3ExcURHx9PZGQkr7/+OlFRUXTo0MHfoRlT5XjzCfpf4GvP6wegLuD18xT+8rGn2qlXK+8bv43/5eTkMG7cONq3b8+IESMK1p966qmWJIzxE2+qnt4tvCwirwNf+SyicrDrwOFHMhIb2LhOwWLhwoXceeedJCcnA04PJ5fLZY3VxvjZsdTJNAMCemS1//2RCkDDuGo2nHgQ2LVrFzfeeCNnnXUWycnJtG7dmm+++YbXXnvNkoQxAcCbNop9HG6jCAH2AiWO2xQI3ljkPPZxXlsbSjrQpaam0qZNG/bu3UtkZCSjRo3ioYceIjLSHo40JlCUmijEeYLpZGCbZ5VbD42ZEMD2ZuYCcG4bSxSBLiEhgUsvvZSUlBSmTZtGixYt/B2SMaaIUhOFqqqIfKSqXSoqoON1IDuPzXuyEIGuTW0oh0CTmZnJ2LFjueiii+jVqxcA06ZNIzIy0p6sNiZAedNG8bOInOLzSMrJ4o17AUisX5OocKvfDiSffPIJbdu2ZeLEiQwZMgS32xmsMSoqypKEMQGsxBKFiISpaj5wBnCbiGwAMnHGcFJVDcjksXjTHgC6N6vt50jMIVu3buXee+/lo48+AqBz587MmDHD5qs2JkiUVvX0M3AKcFkFxXLcVJW5K53nJ3rb8xN+l5+fz4svvshjjz1GZmYm0dHRjBs3jrvuussmEjImiJT21yoAqrqhgmI5bp+v+oud6TnE14jgjJYJ/g6nyktPT+epp54iMzOT/v3788ILL9Cokc0JYkywKS1R1BGR4SVtVNVJPojnuCzZ7LRPnNEygXAbtsMv9u/fT7Vq1YiMjKR27drMmDGDyMhILrroIn+HZow5RqV9moYC0TjDgRf3CjjvL3XmSbq44wl+jqTqUVXeeustWrduzcSJEwvWX3HFFZYkjAlypZUodqjq2AqL5Dhl5OSTledMYNOhUayfo6la1q1bx5AhQ/j6668B+O6771BV68lkTCVRWokiqP7Kf960B1VoXS+GejWj/B1OlZCdnc0TTzxBhw4d+Prrr6lduzYvv/wyX3zxhSUJYyqR0koU51RYFOXg0CRF3Ztbt9iK8Ndff9GrVy/++OMPAG666SaeeeYZEhKsE4ExlU2JiUJV91ZkIMcrZd9BANqdUNPPkVQN9erV48QTTyQsLIzp06fTu3dvf4dkjPGRStOZfdmf+wA4pbEN2+ELbrebWbNmcdZZZ9GqVStEhLfeeotatWoRERHh7/CMMT5UKfqQ/rkns+B9swSbIrO8rVy5kp49ezJ48GCGDBnCoXEh69WrZ0nCmCqgUiSKj35xBrc9s2WCTXtajjIyMnjggQfo0qULixYt4oQTTmDw4MH+DssYU8EqRdXTy99vAmzYjvL08ccfc/fdd5OSkkJISAh3330348aNo2ZNawMypqoJ+kShqgXPT5xj80+Ui23btnHNNdeQk5NDly5dSEpKomvXrv4OyxjjJ0GfKDbszsTldurMm9Su7udogldeXh5hYWGICA0bNmT8+PFEREQwZMgQm47UmCou6Cv0l3t6O/VqVYeQEHvI61j8+OOPdOnShTfeeKNg3f3338/dd99tScIYE/yJ4j/LUwA4u7W1TxytvXv3cscdd9CzZ09WrVrFtGnTCIKZbo0xFSyoE4XLrSze5DwXeGHHBn6OJnioKq+//jqJiYnMnDmT8PBwRo0axTfffGNDbxhj/iao2yi273eexo6vEUHdGBvfyRs7d+7k2muvZcGCBQD07t2b6dOn06ZNGz9HZowJVEFdoli+xWmfiAwL6h+jQsXFxbFjxw4SEhJ47bXXWLBggSUJY0ypgrpEsWLrfgD+eeqJfo4ksH311VeccsopxMfHExkZyfvvv0+DBg2Ij4/3d2jGmCAQ1F/Fk3ccAKBtA3sIrDg7duzg2muvpW/fvowYMaJgffv27S1JGGO8FtSJYsPuDABa1w/ICff8xuVyMW3aNBITE3nnnXeoVq0arVu3th5NxphjErRVT9l5LnYdyCE0RDghrpq/wwkYy5cvZ/DgwSxZsgSAiy66iClTptC0aVP/BmaMCVpBmyi27s0CoH7NKMJtIEAANm/eTLdu3XC5XDRs2JAXX3yRyy+/3Lq8GmOOi08ThYj0AyYDocD/qeqEItuHA7cC+cBu4BZV/dObc29PywagsQ3bUaBp06bcfPPNxMTE8MQTTxATY1Vyxpjj57Ov4iISCkwFLgDaAteKSNsiu/0CdFXVjsAHwERvz596IAeAujUjyyXeYLR582Yuvvhivv3224J1M2fOZNKkSZYkjDHlxpclim7AelXdCCAi7wCXAmsO7aCqCwrtvwi43tuT70hzHrarV7PqPWiXl5fHpEmTeOKJJzh48CCpqan89NNPAFbNZIwpd76s3G8IbC20nOJZV5JBwLziNojI7SKyVESW7t692zmZZ47s+lUsUfzvf/+jc+fOjBw5koMHD3LNNdfw4Ycf+jssY0wl5stEUdxX22L7Z4rI9UBX4JnitqvqTFXtqqpd69RxBv/b5al6aliravR42rdvH7feeitnnnkmq1ev5qSTTuKLL77g7bffpkEDG+fKGOM7vkwUKUDhR6YbAduL7iQi5wKjgEtUNcfbkx8a5ykhumq0UbjdbubMmUN4eDijR49m1apV9O3b199hGWOqAF+2USwBWopIM2AbcA1wXeEdRKQzMAPop6q7jubkf6U7vZ5OrF15SxTJyck0a9aMyMhI4uPjefPNN2ncuDGJiYn+Ds0YU4X4rEShqvnAUOALYC3wnqquFpGxInKJZ7dngGjgfRFZISJzvTl3Vm4++7PyiAgNoU4lLFFkZWUxatQoOnbsyMSJhzuC9e3b15KEMabC+fQ5ClX9HPi8yLrHCr0/91jOu2FXJuC0T1S2Xj7z589nyJAhbNq0CYDU1FQ/R2SMqeqC8snstX+lA9CybrSfIyk/27dvZ9iwYbz//vsAdOjQgaSkJE4//XQ/R2aMqeqCMlHs9DyV3TShhp8jKR/r1q2ja9euHDhwgOrVq/P4448zbNgwwsPD/R2aMcYEZ6I41DW2bkzlaJ9o2bIlp556KjVq1OCll16iSZMm/g7JGGMKBOVoersLhu8Izoft0tPTGTZsGOvWrQOcp6nnzp3L3LlzLUkYYwJOUJYoftq4B4Da1SP8HMnRUVU++OAD7r33Xnbs2EFycjLz588HoEaNylGNZoypfIIyUWTl5gNQIzLUz5F4b+PGjQwdOpR585xRSnr06MHTTz/t56iMMaZsQVn1dGiitmCYsCg3N5cnn3ySdu3aMW/ePOLi4khKSuKHH37g5JNP9nd4xhhTpqAsUeS7nUxRu0bgVz1t3bqVsWPHkpOTw4ABA3juueeoV6+ev8MyxhivBV2icLkPjysYqDPb7du3j7i4OESEk046icmTJ9OiRQvOOeccf4dmjDFHLTA/aUuRm+8GoFEAjhrrdrt55ZVXaNGiBW+88UbB+jvuuMOShDEmaAVfonA5iSLQnspevXo1ffr0YdCgQezdu7eg0doYY4Jd0CWKPE+iCJS5srOysnj44Yfp1KkT33//PXXr1uXNN9/kzTff9HdoxhhTLoKujSI3300E0CAAejytW7eO888/n82bNyMiDB48mCeffJJatWr5OzRjjCk3QZcoMnNdRAD1avp/+I4mTZoQFRXFySefTFJSEj169PB3SMYYU+6CruopxDOqeGhIxYeen5/PlClT2LPHeTI8MjKS+fPns3TpUksSxphKK+gSRVauC4ATK7jX088//0y3bt24++67GTFiRMH6Jk2aEBYWdAUzY4zxWtAlikPia1RM1VNaWhpDhw6lR48e/PLLLzRu3JhLL720Qq5tjDGBIGgTRYM4344cq6q88847JCYmMnXqVEJDQ3nooYdYs2YNF198sU+vbYwxgSQo60yiwkN8/lT2ypUrufbaawE4/fTTSUpKokOHDj69pjHGBKKgTBS+Gl7c5XIRGuqMSNupUyfuu+8+2rZtyy233EKIHxrPjTEmEATlp18tHwwGuGDBAtq3b893331XsG7SpEnceuutliSMMVVaUH4CxkeXX0P2rl27uPHGGzn77LNJTk5m0qRJ5XZuY4ypDIIyUaRl5R73OdxuN7NmzSIxMZHZs2cTGRnJv/71L959991yiNAYYyqPoGyjaBB7fM9QbNq0ieuvv54ff/wRgL59+zJ16lRatGhRHuEZY0ylEpQlitrRx9dGUbNmTdatW0f9+vV55513mD9/viUJY4wpQVCWKMIOjeNxFL744gv69OlDZGQk8fHxzJ07l7Zt2xIbG+uDCI0xpvIIyhLFoWE8vLF161Yuv/xy+vXrxzPPPFOw/rTTTrMkYYwxXgjKROHNpEX5+flMmjSJNm3a8PHHHxMdHU3t2rUrIDpjjKlcgrLqScqoeVq0aBGDBw9m5cqVAPTv35/JkyfTsGHDCojOGGMql6BMFE3ja5S4bfHixZx++umoKk2bNmXKlClcdNFFFRidMcZULkGZKKKjSg67W7dunH/++XTu3JlHH32U6tUDY8pUY4wJVkHZRhEVHlrw/o8//uAf//gH69atA0BE+Oyzz3jyySctSRhjTDkIyhJFVFgoOTk5TJgwgaeeeoqcnByioqL44IMPAGxsJmOMKUc+/UQVkX4i8ruIrBeRkcVsjxSRdz3bF4tIU2/Ou2Lxd3Ts2JHHH3+cnJwcbr75ZpKSkso7fGOMMYCoqm9OLBIKrAPOA1KAJcC1qrqm0D5DgI6qOlhErgEuV9WrSztvaLUYdWdnANCmTRuSkpLo1auXT34GY4ypLERkmap2PZZjfVmi6AasV9WNqpoLvAMUnUP0UuDfnvcfAOeIlN751Z2dSVRUFE8++SQrVqywJGGMMT7myxLFlUA/Vb3VszwQ6K6qQwvt85tnnxTP8gbPPqlFznU7cLtnsT3wm0+CDj4JQGqZe1UNdi8Os3txmN2Lw1qrasyxHOjLxuziSgZFs5I3+6CqM4GZACKy9FiLT5WN3YvD7F4cZvfiMLsXh4nI0mM91pdVTynAiYWWGwHbS9pHRMKAWGCvD2MyxhhzlHyZKJYALUWkmYhEANcAc4vsMxe40fP+SuAb9VVdmDHGmGPis6onVc0XkaHAF0Ao8IqqrhaRscBSVZ0LvAy8LiLrcUoS13hx6pm+ijkI2b04zO7FYXYvDrN7cdgx3wufNWYbY4ypHOwRZmOMMaWyRGGMMaZUAZsofDX8RzDy4l4MF5E1IvKriHwtIk38EWdFKOteFNrvShFREam0XSO9uRci8k/P78ZqEXmromOsKF78jTQWkQUi8ovn7+RCf8TpayLyiojs8jyjVtx2EZEXPffpVxE5xasTq2rAvXAavzcAzYEIYCXQtsg+Q4Akz/trgHf9Hbcf78VZQHXP+zur8r3w7BcDfAcsArr6O24//l60BH4BanmW6/o7bj/ei5nAnZ73bYHN/o7bR/eiF3AK8FsJ2y8E5uE8w9YDWOzNeQO1ROGT4T+CVJn3QlUXqGqWZ3ERzjMrlZE3vxcA/wImAtkVGVwF8+Ze3AZMVdV9AKq6q4JjrCje3AsFanrex/L3Z7oqBVX9jtKfRbsUmK2ORUCciDQo67yBmigaAlsLLad41hW7j6rmA2lAfIVEV7G8uReFDcL5xlAZlXkvRKQzcKKqflqRgfmBN78XrYBWIvKDiCwSkX4VFl3F8uZePA5cLyIpwOfA3RUTWsA52s8TIHDnoyi34T8qAa9/ThG5HugK9PZpRP5T6r0QkRDgeeCmigrIj7z5vQjDqX7qg1PK/F5E2qvqfh/HVtG8uRfXAq+p6nMichrO81vtVdXt+/ACyjF9bgZqicKG/zjMm3uBiJwLjAIuUdWcCoqtopV1L2JwBo1cKCKbcepg51bSBm1v/0bmqGqeqm4CfsdJHJWNN/diEPAegKr+BEThDBhY1Xj1eVJUoCYKG/7jsDLvhae6ZQZOkqis9dBQxr1Q1TRVTVDVpqraFKe95hJVPebB0AKYN38jH+N0dEBEEnCqojZWaJQVw5t7sQU4B0BE2uAkit0VGmVgmAvc4On91ANIU9UdZR0UkFVP6rvhP4KOl/fiGSAaeN/Tnr9FVS/xW9A+4uW9qBK8vBdfAH1FZA3gAh5U1T3+i9o3vLwX9wOzROQ+nKqWmyrjF0sReRunqjHB0x4zBggHUNUknPaZC4H1QBZws1fnrYT3yhhjTDkK1KonY4wxAcIShTHGmFJZojDGGFMqSxTGGGNKZYnCGGNMqSxRmIAjIi4RWVHo1bSUfZuWNFLmUV5zoWf00RzydOUAAAPdSURBVJWeIS9aH8M5BovIDZ73N4nICYW2/Z+ItC3nOJeISCcvjhkmItWP99qm6rJEYQLRQVXtVOi1uYKuO0BVT8YZbPKZoz1YVZNUdbZn8SbghELbblXVNeUS5eE4p+FdnMMASxTmmFmiMEHBU3L4XkSWe16nF7NPOxH52VMK+VVEWnrWX19o/QwRCS3jct8BLTzHnuOZw2CVZ6z/SM/6CXJ4DpBnPeseF/n/9u4nxKoyjOP497comooEF0UQ9IegIJqkLAQXYf8oIigRbyFii4gkCYrZhG2CFhG0yKwkIsaFxaAoRH8oicGFzFSzMC2RBGsRRLmQCJkg7NfieSdutzt35rpymN9nd997z3nfe+Ce557nHJ5HY5I2UDW39rQ5R9qVwGpJWyW91rXmJyW9eZ7rnKKroJukdyTNqHpPvNzGnqMC1qSkyTb2gKSpdhz3Srp8gXlimUugiAvRSFfa6UAb+w243/btQAfY0We7Z4A3bK+iTtQ/t3INHWBtGz8HbFpg/keAY5IuAcaBju1bqUoGWyWtBB4DbrE9CrzSvbHtfcAM9c9/le3Zrrf3Aeu7XneAifNc54NUmY45222vBkaBuyWN2t5B1fJZZ3tdK+XxEnBfO5YzwAsLzBPL3AVZwiOWvdl2sux2EbCz5eTPUXWLek0B2yVdA+y3fVLSvcAdwDetvMkIFXT62SNpFviJKkN9E/Cj7R/a+7uBZ4GdVK+L9yR9Aiy6pLnt05JOtTo7J9sch9t+h1nnZVS5iu4OZRslPU39rq+mGvQc7dl2TRs/3Oa5mDpuEfNKoIil4nngV+A26kr4f02JbH8g6SvgYeBzSU9RZZV3235xEXNs6i4gKKlvf5NWW+guqsjc48A24J4hvssEsBE4ARywbdVZe9HrpLq4vQq8BayXdD0wBtxp+4ykcarwXS8BB20/McR6Y5lL6imWihXAL61/wGbq3/R/SLoBONXSLR9RKZgvgQ2SrmyfWanF9xQ/AVwn6cb2ejNwqOX0V9j+lLpR3O/Joz+osuf97AcepXokTLSxodZp+y8qhbSmpa2uAM4Cv0u6CnhonrVMA2vnvpOkSyX1uzqL+FcCRSwVbwNbJE1TaaezfT7TAb6TdAS4mWr5eJw6oX4h6ShwkErLLMj2n1R1zb2SjgF/A7uok+7HbX+HqKudXuPArrmb2T37PQMcB661/XUbG3qd7d7H68CY7W+p/tjfA+9T6aw57wKfSZq0fZp6IuvDNs80dawi5pXqsRERMVCuKCIiYqAEioiIGCiBIiIiBkqgiIiIgRIoIiJioASKiIgYKIEiIiIG+gdoRJ7eYdTivwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "aid = 'CHEMBL1614421'\n",
    "assay = pubchem_d.loc[pubchem_d['assay_chembl_id']==aid]\n",
    "acts = pd.concat((assay.loc[assay['activity_comment'] == 'Active'], \n",
    "                  assay.loc[assay['activity_comment'] == 'active']))\n",
    "inacts = pd.concat((assay.loc[assay['activity_comment'] == 'inactive'],\n",
    "                    assay.loc[assay['activity_comment'] == 'inconclusive'], \n",
    "                    assay.loc[assay['activity_comment'] == 'Inconclusive'], \n",
    "                    assay.loc[assay['activity_comment'] == 'Not Active']))\n",
    "print(len(acts),len(inacts))\n",
    "ps,tgt,oob_tpl = run_it2(acts,inacts)\n",
    "oob_tgt,oob_probs = oob_tpl\n",
    "oob_probs = [x[1] for x in oob_probs]\n",
    "auroc = metrics.roc_auc_score(oob_tgt, oob_probs)\n",
    "scores = [1 if x>=0.5 else 0 for x in ps]\n",
    "kappa = metrics.cohen_kappa_score(tgt,scores)\n",
    "confusion = metrics.confusion_matrix(tgt,scores)\n",
    "fpr,tpr,_ = metrics.roc_curve(oob_tgt,oob_probs);\n",
    "plt.figure()\n",
    "lw = 2\n",
    "plt.plot(fpr, tpr, lw=lw)\n",
    "plt.plot([0, 1], [0, 1], color='k', lw=lw, linestyle='--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title(\"%s: AUC=%.3f\"%(aid,auroc))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ChEMBL DrugMatrix assays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>canonical_smiles</th>\n",
       "      <th>compound_chembl_id</th>\n",
       "      <th>assay_chembl_id</th>\n",
       "      <th>standard_relation</th>\n",
       "      <th>Mean(standard_value)</th>\n",
       "      <th>activity_comment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Br.CN1[C@@H]2CC[C@H]1C[C@H](C2)OC(=O)C(O)c3ccccc3</td>\n",
       "      <td>CHEMBL1319362</td>\n",
       "      <td>CHEMBL1909097</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not Active (inhibition &lt; 50% @ 10 uM and thus ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Br.CN1[C@@H]2CC[C@H]1C[C@H](C2)OC(=O)C(O)c3ccccc3</td>\n",
       "      <td>CHEMBL1319362</td>\n",
       "      <td>CHEMBL1909212</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not Active (inhibition &lt; 50% @ 10 uM and thus ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Br.CN1[C@@H]2CC[C@H]1C[C@H](C2)OC(=O)C(O)c3ccccc3</td>\n",
       "      <td>CHEMBL1319362</td>\n",
       "      <td>CHEMBL1909213</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not Active (inhibition &lt; 50% @ 10 uM and thus ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Br.CN1[C@@H]2CC[C@H]1C[C@H](C2)OC(=O)C(O)c3ccccc3</td>\n",
       "      <td>CHEMBL1319362</td>\n",
       "      <td>CHEMBL1909214</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not Active (inhibition &lt; 50% @ 10 uM and thus ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Br.CN1[C@@H]2CC[C@H]1C[C@H](C2)OC(=O)C(O)c3ccccc3</td>\n",
       "      <td>CHEMBL1319362</td>\n",
       "      <td>CHEMBL1909215</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not Active (inhibition &lt; 50% @ 10 uM and thus ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    canonical_smiles compound_chembl_id  \\\n",
       "0  Br.CN1[C@@H]2CC[C@H]1C[C@H](C2)OC(=O)C(O)c3ccccc3      CHEMBL1319362   \n",
       "1  Br.CN1[C@@H]2CC[C@H]1C[C@H](C2)OC(=O)C(O)c3ccccc3      CHEMBL1319362   \n",
       "2  Br.CN1[C@@H]2CC[C@H]1C[C@H](C2)OC(=O)C(O)c3ccccc3      CHEMBL1319362   \n",
       "3  Br.CN1[C@@H]2CC[C@H]1C[C@H](C2)OC(=O)C(O)c3ccccc3      CHEMBL1319362   \n",
       "4  Br.CN1[C@@H]2CC[C@H]1C[C@H](C2)OC(=O)C(O)c3ccccc3      CHEMBL1319362   \n",
       "\n",
       "  assay_chembl_id standard_relation  Mean(standard_value)  \\\n",
       "0   CHEMBL1909097               NaN                   NaN   \n",
       "1   CHEMBL1909212               NaN                   NaN   \n",
       "2   CHEMBL1909213               NaN                   NaN   \n",
       "3   CHEMBL1909214               NaN                   NaN   \n",
       "4   CHEMBL1909215               NaN                   NaN   \n",
       "\n",
       "                                    activity_comment  \n",
       "0  Not Active (inhibition < 50% @ 10 uM and thus ...  \n",
       "1  Not Active (inhibition < 50% @ 10 uM and thus ...  \n",
       "2  Not Active (inhibition < 50% @ 10 uM and thus ...  \n",
       "3  Not Active (inhibition < 50% @ 10 uM and thus ...  \n",
       "4  Not Active (inhibition < 50% @ 10 uM and thus ...  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with gzip.open('../data/ChEMBL_DrugMatrix.csv.gz') as inf:\n",
    "    drugmatrix_d = pd.read_csv(inf)\n",
    "drugmatrix_d.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CHEMBL1909215 842 DRUGMATRIX: Adenosine A3 radioligand binding (ligand: AB-MECA)\n",
      "CHEMBL1909211 842 DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT2A radioligand binding (ligand: [3H] Ketanserin)\n",
      "CHEMBL1909210 842 DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT1B radioligand binding (ligand: [125I] Cyanopindolol)\n",
      "CHEMBL1909209 842 DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT1A radioligand binding (ligand: [3H] 8-OH-DPAT)\n",
      "CHEMBL1909204 842 DRUGMATRIX: Protein Tyrosine Kinase, Fyn enzyme inhibition (substrate: Poly(Glu:Tyr))\n",
      "CHEMBL1909203 842 DRUGMATRIX: Protein Tyrosine Kinase, EGF Receptor enzyme inhibition (substrate: Poly(Glu:Tyr))\n",
      "CHEMBL1909191 842 DRUGMATRIX: Progesterone radioligand binding (ligand: [3H] R-5020)\n",
      "CHEMBL1909174 842 DRUGMATRIX: Muscarinic M5 radioligand binding (ligand: [3H] N-Methylscopolamine)\n",
      "CHEMBL1909173 842 DRUGMATRIX: Muscarinic M4 radioligand binding (ligand: [3H] N-Methylscopolamine)\n",
      "CHEMBL1909172 842 DRUGMATRIX: Muscarinic M3 radioligand binding (ligand: [3H] N-Methylscopolamine)\n",
      "CHEMBL1909171 842 DRUGMATRIX: Muscarinic M2 radioligand binding (ligand: [3H] N-Methylscopolamine)\n",
      "CHEMBL1909170 842 DRUGMATRIX: Muscarinic M1 radioligand binding (ligand: [3H] N-Methylscopolamine)\n",
      "CHEMBL1909165 842 DRUGMATRIX: Lipoxygenase 15-LO enzyme inhibition (substrate: Linoleic acid)\n",
      "CHEMBL1909159 842 DRUGMATRIX: Imidazoline I2, Central radioligand binding (ligand: [3H] Idazoxan)\n",
      "CHEMBL1909157 842 DRUGMATRIX: Histamine H2 radioligand binding (ligand: [125I] Aminopotentidine)\n",
      "CHEMBL1909156 842 DRUGMATRIX: Histamine H1, Central radioligand binding (ligand: [3H] Pyrilamine)\n",
      "CHEMBL1909150 842 DRUGMATRIX: Glucocorticoid radioligand binding (ligand: [3H] Dexamethasone)\n",
      "CHEMBL1909143 842 DRUGMATRIX: Dopamine Transporter radioligand binding (ligand: [125I] RTI-55)\n",
      "CHEMBL1909141 842 DRUGMATRIX: Dopamine D3 radioligand binding (ligand: [3H] Spiperone)\n",
      "CHEMBL1909140 842 DRUGMATRIX: Dopamine D2L radioligand binding (ligand: [3H] Spiperone)\n",
      "CHEMBL1909139 842 DRUGMATRIX: Dopamine D1 radioligand binding (ligand: [3H] SCH-23390)\n",
      "CHEMBL1909136 842 DRUGMATRIX: CYP450, 2D6 enzyme inhibition (substrate: 3-Cyano-7-ethoxycoumarin)\n",
      "CHEMBL1909135 842 DRUGMATRIX: CYP450, 2C9 enzyme inhibition (substrate: 3-Cyano-7-ethoxycoumarin)\n",
      "CHEMBL1909134 842 DRUGMATRIX: CYP450, 2C19 enzyme inhibition (substrate: 3-Cyano-7-ethoxycoumarin)\n",
      "CHEMBL1909132 842 DRUGMATRIX: CYP450, 1A2 enzyme inhibition (substrate: 3-Cyano-7-ethoxycoumarin)\n",
      "CHEMBL1909130 842 DRUGMATRIX: Cyclooxygenase COX-1 enzyme inhibition (substrate: Arachidonic acid)\n",
      "CHEMBL1909121 842 DRUGMATRIX: Calcium Channel Type L, Phenylalkylamine radioligand binding (ligand: [3H] (-)-Desmethoxyverapamil (D-888))\n",
      "CHEMBL1909115 842 DRUGMATRIX: Androgen (Testosterone) AR radioligand binding (ligand: [3H] Mibolerone)\n",
      "CHEMBL1909114 842 DRUGMATRIX: Tachykinin NK2 radioligand binding (ligand: [3H] SR-48968)\n",
      "CHEMBL1909112 842 DRUGMATRIX: Sodium Channel, Site 2 radioligand binding (ligand: [3H] Batrachotoxin)\n",
      "CHEMBL1909111 842 DRUGMATRIX: Sigma2 radioligand binding (ligand: [3H] Ifenprodil)\n",
      "CHEMBL1909110 842 DRUGMATRIX: Sigma1 radioligand binding (ligand: [3H] Haloperidol)\n",
      "CHEMBL1909109 842 DRUGMATRIX: Transporter, Serotonin (5-Hydroxytryptamine) (SERT) radioligand binding (ligand: [3H] Paroxetine)\n",
      "CHEMBL1909108 842 DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT6 radioligand binding (ligand: [3H] Lysergic acid diethylamide)\n",
      "CHEMBL1909105 842 DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT2C radioligand binding (ligand: [3H] Mesulergine)\n",
      "CHEMBL1909104 842 DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT2B radioligand binding (ligand: [3H] Lysergic acid diethylamide)\n",
      "CHEMBL1909102 842 DRUGMATRIX: Calcium Channel Type L, Benzothiazepine radioligand binding (ligand: [3H] Diltiazem)\n",
      "CHEMBL1909094 842 DRUGMATRIX: Norepinephrine Transporter radioligand binding (ligand: [125I] RTI-55)\n",
      "CHEMBL1909090 842 DRUGMATRIX: Adrenergic Alpha-2C radioligand binding (ligand: [3H] MK-912)\n",
      "CHEMBL1909089 842 DRUGMATRIX: Alpha-2B adrenergic receptor radioligand binding (ligand: Rauwolscine)\n",
      "CHEMBL1909088 842 DRUGMATRIX: Alpha-2A adrenergic receptor radioligand binding (ligand: MK-912)\n",
      "CHEMBL1909087 842 DRUGMATRIX: Alpha-1D adrenergic receptor radioligand binding (ligand: prazosin)\n",
      "CHEMBL1909086 842 DRUGMATRIX: Alpha-1B adrenergic receptor radioligand binding (ligand: prazosin)\n",
      "CHEMBL1909085 842 DRUGMATRIX: Alpha-1A adrenergic receptor radioligand binding (ligand: prazosin)\n"
     ]
    }
   ],
   "source": [
    "import requests,json\n",
    "drugmatrix_assay_lookup = {}\n",
    "tpls = sorted([(len(v),k) for k,v in drugmatrix_d.groupby('assay_chembl_id').groups.items()],reverse=True)\n",
    "for v,k in tpls:\n",
    "    # we will only keep the subset of assays that have enough actives,\n",
    "    assay = drugmatrix_d.loc[drugmatrix_d['assay_chembl_id']==k]\n",
    "    # skip the assay if it doesn't have at least 40 actives\n",
    "    if(len([1 for y in assay['activity_comment'] if y.find('Active')==0])<40):\n",
    "        #print(f'Skipping assay: {k}')\n",
    "        continue\n",
    "    resp=requests.get(f'https://www.ebi.ac.uk/chembl/api/data/assay/{k}?format=json')\n",
    "    info = json.loads(resp.text)\n",
    "    drugmatrix_assay_lookup[k] = info['description']\n",
    "    print(k,v,info['description'])\n",
    "#     drugmatrix_assay_lookup[k] = k\n",
    "#     print(k,v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../data/drugmatrix_data.pkl','wb+') as outf:\n",
    "    pickle.dump((drugmatrix_d,drugmatrix_assay_lookup),outf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../data/drugmatrix_data.pkl','rb') as inf:\n",
    "    drugmatrix_d,drugmatrix_assay_lookup = pickle.load(inf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "def run_drugmatrix_assay(assay_id,d,assay_lookup,thresholds=None,archive=None):\n",
    "    if archive is None:\n",
    "        archive=defaultdict(list)\n",
    "    assay = d.loc[d['assay_chembl_id']==assay_id]\n",
    "    inact_indices = [x for x,y in enumerate(assay['activity_comment']) if y.find('Not Active')==0]\n",
    "    act_indices = [x for x,y in enumerate(assay['activity_comment']) if y.find('Active')==0]\n",
    "    acts = assay.iloc[act_indices]\n",
    "    inacts = assay.iloc[inact_indices]\n",
    "    print('*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*')\n",
    "\n",
    "    if(len(act_indices)<40):\n",
    "        print(f'>>>>> SKIPPING {assay_id} DUE TO INSUFFICIENT ACTIVES <<<<<< ')\n",
    "        print('*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\\n\\n')\n",
    "        return\n",
    "\n",
    "    print(f'assay_id {assay_id}, description: {assay_lookup[assay_id]}')\n",
    "    run_it_(assay_id,acts,inacts,thresholds,archive)       \n",
    "\n",
    "    print('*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\\n\\n')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909215, description: DRUGMATRIX: Adenosine A3 radioligand binding (ligand: AB-MECA)\n",
      "--------- Default -----------\n",
      "ratio: 0.073 kappa: 0.000, AUC: 0.839, OOB_AUC: 0.830\n",
      "[[157   0]\n",
      " [ 12   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      1.00      0.96       157\n",
      "           1       0.00      0.00      0.00        12\n",
      "\n",
      "   micro avg       0.93      0.93      0.93       169\n",
      "   macro avg       0.46      0.50      0.48       169\n",
      "weighted avg       0.86      0.93      0.89       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.417\n",
      "[[152   5]\n",
      " [  7   5]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909211, description: DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT2A radioligand binding (ligand: [3H] Ketanserin)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.127 kappa: 0.366, AUC: 0.924, OOB_AUC: 0.935\n",
      "[[149   1]\n",
      " [ 14   5]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.99      0.95       150\n",
      "           1       0.83      0.26      0.40        19\n",
      "\n",
      "   micro avg       0.91      0.91      0.91       169\n",
      "   macro avg       0.87      0.63      0.68       169\n",
      "weighted avg       0.91      0.91      0.89       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.751\n",
      "[[147   3]\n",
      " [  5  14]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909210, description: DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT1B radioligand binding (ligand: [125I] Cyanopindolol)\n",
      "--------- Default -----------\n",
      "ratio: 0.050 kappa: 0.000, AUC: 0.777, OOB_AUC: 0.903\n",
      "[[161   0]\n",
      " [  8   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       161\n",
      "           1       0.00      0.00      0.00         8\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       169\n",
      "   macro avg       0.48      0.50      0.49       169\n",
      "weighted avg       0.91      0.95      0.93       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.282\n",
      "[[158   3]\n",
      " [  6   2]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909209, description: DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT1A radioligand binding (ligand: [3H] 8-OH-DPAT)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.089 kappa: 0.308, AUC: 0.914, OOB_AUC: 0.921\n",
      "[[154   1]\n",
      " [ 11   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.99      0.96       155\n",
      "           1       0.75      0.21      0.33        14\n",
      "\n",
      "   micro avg       0.93      0.93      0.93       169\n",
      "   macro avg       0.84      0.60      0.65       169\n",
      "weighted avg       0.92      0.93      0.91       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.611\n",
      "[[150   5]\n",
      " [  5   9]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909204, description: DRUGMATRIX: Protein Tyrosine Kinase, Fyn enzyme inhibition (substrate: Poly(Glu:Tyr))\n",
      "--------- Default -----------\n",
      "ratio: 0.066 kappa: 0.000, AUC: 0.783, OOB_AUC: 0.891\n",
      "[[158   0]\n",
      " [ 11   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      1.00      0.97       158\n",
      "           1       0.00      0.00      0.00        11\n",
      "\n",
      "   micro avg       0.93      0.93      0.93       169\n",
      "   macro avg       0.47      0.50      0.48       169\n",
      "weighted avg       0.87      0.93      0.90       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.100 kappa: 0.385\n",
      "[[144  14]\n",
      " [  4   7]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909203, description: DRUGMATRIX: Protein Tyrosine Kinase, EGF Receptor enzyme inhibition (substrate: Poly(Glu:Tyr))\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.000, AUC: 0.917, OOB_AUC: 0.865\n",
      "[[161   0]\n",
      " [  8   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       161\n",
      "           1       0.00      0.00      0.00         8\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       169\n",
      "   macro avg       0.48      0.50      0.49       169\n",
      "weighted avg       0.91      0.95      0.93       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.100 kappa: 0.288\n",
      "[[149  12]\n",
      " [  4   4]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909191, description: DRUGMATRIX: Progesterone radioligand binding (ligand: [3H] R-5020)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.060 kappa: 0.556, AUC: 0.975, OOB_AUC: 0.936\n",
      "[[159   0]\n",
      " [  6   4]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98       159\n",
      "           1       1.00      0.40      0.57        10\n",
      "\n",
      "   micro avg       0.96      0.96      0.96       169\n",
      "   macro avg       0.98      0.70      0.78       169\n",
      "weighted avg       0.97      0.96      0.96       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.883\n",
      "[[159   0]\n",
      " [  2   8]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909174, description: DRUGMATRIX: Muscarinic M5 radioligand binding (ligand: [3H] N-Methylscopolamine)\n",
      "--------- Default -----------\n",
      "ratio: 0.084 kappa: 0.228, AUC: 0.946, OOB_AUC: 0.918\n",
      "[[155   1]\n",
      " [ 11   2]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.99      0.96       156\n",
      "           1       0.67      0.15      0.25        13\n",
      "\n",
      "   micro avg       0.93      0.93      0.93       169\n",
      "   macro avg       0.80      0.57      0.61       169\n",
      "weighted avg       0.91      0.93      0.91       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.487\n",
      "[[152   4]\n",
      " [  7   6]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909173, description: DRUGMATRIX: Muscarinic M4 radioligand binding (ligand: [3H] N-Methylscopolamine)\n",
      "--------- Default -----------\n",
      "ratio: 0.082 kappa: -0.011, AUC: 0.915, OOB_AUC: 0.918\n",
      "[[155   1]\n",
      " [ 13   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.99      0.96       156\n",
      "           1       0.00      0.00      0.00        13\n",
      "\n",
      "   micro avg       0.92      0.92      0.92       169\n",
      "   macro avg       0.46      0.50      0.48       169\n",
      "weighted avg       0.85      0.92      0.88       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.304\n",
      "[[154   2]\n",
      " [ 10   3]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909172, description: DRUGMATRIX: Muscarinic M3 radioligand binding (ligand: [3H] N-Methylscopolamine)\n",
      "--------- Default -----------\n",
      "ratio: 0.082 kappa: 0.000, AUC: 0.916, OOB_AUC: 0.906\n",
      "[[156   0]\n",
      " [ 13   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      1.00      0.96       156\n",
      "           1       0.00      0.00      0.00        13\n",
      "\n",
      "   micro avg       0.92      0.92      0.92       169\n",
      "   macro avg       0.46      0.50      0.48       169\n",
      "weighted avg       0.85      0.92      0.89       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.301\n",
      "[[150   6]\n",
      " [  9   4]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909171, description: DRUGMATRIX: Muscarinic M2 radioligand binding (ligand: [3H] N-Methylscopolamine)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.078 kappa: 0.115, AUC: 0.929, OOB_AUC: 0.896\n",
      "[[156   1]\n",
      " [ 12   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.99      0.96       157\n",
      "           1       0.50      0.08      0.13        13\n",
      "\n",
      "   micro avg       0.92      0.92      0.92       170\n",
      "   macro avg       0.71      0.54      0.55       170\n",
      "weighted avg       0.90      0.92      0.90       170\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.500\n",
      "[[151   6]\n",
      " [  6   7]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909170, description: DRUGMATRIX: Muscarinic M1 radioligand binding (ligand: [3H] N-Methylscopolamine)\n",
      "--------- Default -----------\n",
      "ratio: 0.086 kappa: 0.106, AUC: 0.947, OOB_AUC: 0.903\n",
      "[[154   1]\n",
      " [ 13   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.99      0.96       155\n",
      "           1       0.50      0.07      0.12        14\n",
      "\n",
      "   micro avg       0.92      0.92      0.92       169\n",
      "   macro avg       0.71      0.53      0.54       169\n",
      "weighted avg       0.89      0.92      0.89       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.434\n",
      "[[147   8]\n",
      " [  7   7]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909165, description: DRUGMATRIX: Lipoxygenase 15-LO enzyme inhibition (substrate: Linoleic acid)\n",
      "--------- Default -----------\n",
      "ratio: 0.054 kappa: 0.000, AUC: 0.693, OOB_AUC: 0.808\n",
      "[[160   0]\n",
      " [  9   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97       160\n",
      "           1       0.00      0.00      0.00         9\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       169\n",
      "   macro avg       0.47      0.50      0.49       169\n",
      "weighted avg       0.90      0.95      0.92       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.296\n",
      "[[154   6]\n",
      " [  6   3]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909159, description: DRUGMATRIX: Imidazoline I2, Central radioligand binding (ligand: [3H] Idazoxan)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.078 kappa: 0.133, AUC: 0.852, OOB_AUC: 0.903\n",
      "[[157   0]\n",
      " [ 12   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      1.00      0.96       157\n",
      "           1       1.00      0.08      0.14        13\n",
      "\n",
      "   micro avg       0.93      0.93      0.93       170\n",
      "   macro avg       0.96      0.54      0.55       170\n",
      "weighted avg       0.93      0.93      0.90       170\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.611\n",
      "[[153   4]\n",
      " [  5   8]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909157, description: DRUGMATRIX: Histamine H2 radioligand binding (ligand: [125I] Aminopotentidine)\n",
      "--------- Default -----------\n",
      "ratio: 0.054 kappa: 0.000, AUC: 0.905, OOB_AUC: 0.824\n",
      "[[160   0]\n",
      " [  9   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97       160\n",
      "           1       0.00      0.00      0.00         9\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       169\n",
      "   macro avg       0.47      0.50      0.49       169\n",
      "weighted avg       0.90      0.95      0.92       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.443\n",
      "[[156   4]\n",
      " [  5   4]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909156, description: DRUGMATRIX: Histamine H1, Central radioligand binding (ligand: [3H] Pyrilamine)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.062 kappa: 0.288, AUC: 0.963, OOB_AUC: 0.939\n",
      "[[158   1]\n",
      " [  8   2]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.99      0.97       159\n",
      "           1       0.67      0.20      0.31        10\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       169\n",
      "   macro avg       0.81      0.60      0.64       169\n",
      "weighted avg       0.93      0.95      0.93       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.787\n",
      "[[157   2]\n",
      " [  2   8]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909150, description: DRUGMATRIX: Glucocorticoid radioligand binding (ligand: [3H] Dexamethasone)\n",
      "--------- Default -----------\n",
      "ratio: 0.059 kappa: 0.814, AUC: 0.939, OOB_AUC: 0.953\n",
      "[[159   0]\n",
      " [  3   7]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99       159\n",
      "           1       1.00      0.70      0.82        10\n",
      "\n",
      "   micro avg       0.98      0.98      0.98       169\n",
      "   macro avg       0.99      0.85      0.91       169\n",
      "weighted avg       0.98      0.98      0.98       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.400 kappa: 0.814\n",
      "[[159   0]\n",
      " [  3   7]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909143, description: DRUGMATRIX: Dopamine Transporter radioligand binding (ligand: [125I] RTI-55)\n",
      "--------- Default -----------\n",
      "ratio: 0.109 kappa: 0.000, AUC: 0.819, OOB_AUC: 0.871\n",
      "[[152   0]\n",
      " [ 17   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.90      1.00      0.95       152\n",
      "           1       0.00      0.00      0.00        17\n",
      "\n",
      "   micro avg       0.90      0.90      0.90       169\n",
      "   macro avg       0.45      0.50      0.47       169\n",
      "weighted avg       0.81      0.90      0.85       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.393\n",
      "[[139  13]\n",
      " [  8   9]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909141, description: DRUGMATRIX: Dopamine D3 radioligand binding (ligand: [3H] Spiperone)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.120 kappa: 0.095, AUC: 0.946, OOB_AUC: 0.896\n",
      "[[151   0]\n",
      " [ 17   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.90      1.00      0.95       151\n",
      "           1       1.00      0.06      0.11        18\n",
      "\n",
      "   micro avg       0.90      0.90      0.90       169\n",
      "   macro avg       0.95      0.53      0.53       169\n",
      "weighted avg       0.91      0.90      0.86       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.713\n",
      "[[147   4]\n",
      " [  5  13]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909140, description: DRUGMATRIX: Dopamine D2L radioligand binding (ligand: [3H] Spiperone)\n",
      "--------- Default -----------\n",
      "ratio: 0.075 kappa: 0.000, AUC: 0.924, OOB_AUC: 0.933\n",
      "[[157   0]\n",
      " [ 12   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      1.00      0.96       157\n",
      "           1       0.00      0.00      0.00        12\n",
      "\n",
      "   micro avg       0.93      0.93      0.93       169\n",
      "   macro avg       0.46      0.50      0.48       169\n",
      "weighted avg       0.86      0.93      0.89       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.641\n",
      "[[153   4]\n",
      " [  4   8]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909139, description: DRUGMATRIX: Dopamine D1 radioligand binding (ligand: [3H] SCH-23390)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.081 kappa: -0.011, AUC: 0.917, OOB_AUC: 0.876\n",
      "[[155   1]\n",
      " [ 13   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.99      0.96       156\n",
      "           1       0.00      0.00      0.00        13\n",
      "\n",
      "   micro avg       0.92      0.92      0.92       169\n",
      "   macro avg       0.46      0.50      0.48       169\n",
      "weighted avg       0.85      0.92      0.88       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.394\n",
      "[[151   5]\n",
      " [  8   5]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909136, description: DRUGMATRIX: CYP450, 2D6 enzyme inhibition (substrate: 3-Cyano-7-ethoxycoumarin)\n",
      "--------- Default -----------\n",
      "ratio: 0.095 kappa: 0.000, AUC: 0.809, OOB_AUC: 0.809\n",
      "[[154   0]\n",
      " [ 15   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      1.00      0.95       154\n",
      "           1       0.00      0.00      0.00        15\n",
      "\n",
      "   micro avg       0.91      0.91      0.91       169\n",
      "   macro avg       0.46      0.50      0.48       169\n",
      "weighted avg       0.83      0.91      0.87       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.247\n",
      "[[134  20]\n",
      " [  8   7]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909135, description: DRUGMATRIX: CYP450, 2C9 enzyme inhibition (substrate: 3-Cyano-7-ethoxycoumarin)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.063 kappa: 0.320, AUC: 0.819, OOB_AUC: 0.778\n",
      "[[159   0]\n",
      " [  8   2]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       159\n",
      "           1       1.00      0.20      0.33        10\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       169\n",
      "   macro avg       0.98      0.60      0.65       169\n",
      "weighted avg       0.95      0.95      0.94       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.414\n",
      "[[155   4]\n",
      " [  6   4]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909134, description: DRUGMATRIX: CYP450, 2C19 enzyme inhibition (substrate: 3-Cyano-7-ethoxycoumarin)\n",
      "--------- Default -----------\n",
      "ratio: 0.060 kappa: 0.000, AUC: 0.762, OOB_AUC: 0.718\n",
      "[[159   0]\n",
      " [ 10   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      1.00      0.97       159\n",
      "           1       0.00      0.00      0.00        10\n",
      "\n",
      "   micro avg       0.94      0.94      0.94       169\n",
      "   macro avg       0.47      0.50      0.48       169\n",
      "weighted avg       0.89      0.94      0.91       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.300 kappa: 0.409\n",
      "[[158   1]\n",
      " [  7   3]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909132, description: DRUGMATRIX: CYP450, 1A2 enzyme inhibition (substrate: 3-Cyano-7-ethoxycoumarin)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.050 kappa: 0.000, AUC: 0.793, OOB_AUC: 0.676\n",
      "[[161   0]\n",
      " [  8   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.98       161\n",
      "           1       0.00      0.00      0.00         8\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       169\n",
      "   macro avg       0.48      0.50      0.49       169\n",
      "weighted avg       0.91      0.95      0.93       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.050 kappa: 0.117\n",
      "[[103  58]\n",
      " [  1   7]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909130, description: DRUGMATRIX: Cyclooxygenase COX-1 enzyme inhibition (substrate: Arachidonic acid)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.056 kappa: -0.011, AUC: 0.894, OOB_AUC: 0.833\n",
      "[[159   1]\n",
      " [  9   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.99      0.97       160\n",
      "           1       0.00      0.00      0.00         9\n",
      "\n",
      "   micro avg       0.94      0.94      0.94       169\n",
      "   macro avg       0.47      0.50      0.48       169\n",
      "weighted avg       0.90      0.94      0.92       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.406\n",
      "[[158   2]\n",
      " [  6   3]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909121, description: DRUGMATRIX: Calcium Channel Type L, Phenylalkylamine radioligand binding (ligand: [3H] (-)-Desmethoxyverapamil (D-888))\n",
      "--------- Default -----------\n",
      "ratio: 0.105 kappa: 0.108, AUC: 0.870, OOB_AUC: 0.912\n",
      "[[153   0]\n",
      " [ 15   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      1.00      0.95       153\n",
      "           1       1.00      0.06      0.12        16\n",
      "\n",
      "   micro avg       0.91      0.91      0.91       169\n",
      "   macro avg       0.96      0.53      0.54       169\n",
      "weighted avg       0.92      0.91      0.87       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.488\n",
      "[[147   6]\n",
      " [  8   8]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909115, description: DRUGMATRIX: Androgen (Testosterone) AR radioligand binding (ligand: [3H] Mibolerone)\n",
      "--------- Default -----------\n",
      "ratio: 0.067 kappa: 0.158, AUC: 0.739, OOB_AUC: 0.851\n",
      "[[158   0]\n",
      " [ 10   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      1.00      0.97       158\n",
      "           1       1.00      0.09      0.17        11\n",
      "\n",
      "   micro avg       0.94      0.94      0.94       169\n",
      "   macro avg       0.97      0.55      0.57       169\n",
      "weighted avg       0.94      0.94      0.92       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.300 kappa: 0.240\n",
      "[[156   2]\n",
      " [  9   2]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909114, description: DRUGMATRIX: Tachykinin NK2 radioligand binding (ligand: [3H] SR-48968)\n",
      "--------- Default -----------\n",
      "ratio: 0.051 kappa: 0.000, AUC: 0.850, OOB_AUC: 0.906\n",
      "[[161   0]\n",
      " [  9   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97       161\n",
      "           1       0.00      0.00      0.00         9\n",
      "\n",
      "   micro avg       0.95      0.95      0.95       170\n",
      "   macro avg       0.47      0.50      0.49       170\n",
      "weighted avg       0.90      0.95      0.92       170\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.345\n",
      "[[157   4]\n",
      " [  6   3]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909112, description: DRUGMATRIX: Sodium Channel, Site 2 radioligand binding (ligand: [3H] Batrachotoxin)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.138 kappa: 0.192, AUC: 0.940, OOB_AUC: 0.868\n",
      "[[146   2]\n",
      " [ 18   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.99      0.94       148\n",
      "           1       0.60      0.14      0.23        21\n",
      "\n",
      "   micro avg       0.88      0.88      0.88       169\n",
      "   macro avg       0.75      0.56      0.58       169\n",
      "weighted avg       0.85      0.88      0.85       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.619\n",
      "[[132  16]\n",
      " [  2  19]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909111, description: DRUGMATRIX: Sigma2 radioligand binding (ligand: [3H] Ifenprodil)\n",
      "--------- Default -----------\n",
      "ratio: 0.069 kappa: 0.000, AUC: 0.909, OOB_AUC: 0.919\n",
      "[[158   0]\n",
      " [ 11   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      1.00      0.97       158\n",
      "           1       0.00      0.00      0.00        11\n",
      "\n",
      "   micro avg       0.93      0.93      0.93       169\n",
      "   macro avg       0.47      0.50      0.48       169\n",
      "weighted avg       0.87      0.93      0.90       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.543\n",
      "[[154   4]\n",
      " [  5   6]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909110, description: DRUGMATRIX: Sigma1 radioligand binding (ligand: [3H] Haloperidol)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/other_linux/home/glandrum/anaconda3/envs/py37_dash/lib/python3.7/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.085 kappa: -0.011, AUC: 0.886, OOB_AUC: 0.899\n",
      "[[155   1]\n",
      " [ 14   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.99      0.95       156\n",
      "           1       0.00      0.00      0.00        14\n",
      "\n",
      "   micro avg       0.91      0.91      0.91       170\n",
      "   macro avg       0.46      0.50      0.48       170\n",
      "weighted avg       0.84      0.91      0.88       170\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.284\n",
      "[[142  14]\n",
      " [  8   6]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909109, description: DRUGMATRIX: Transporter, Serotonin (5-Hydroxytryptamine) (SERT) radioligand binding (ligand: [3H] Paroxetine)\n",
      "--------- Default -----------\n",
      "ratio: 0.121 kappa: 0.156, AUC: 0.924, OOB_AUC: 0.938\n",
      "[[150   1]\n",
      " [ 17   2]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.90      0.99      0.94       151\n",
      "           1       0.67      0.11      0.18        19\n",
      "\n",
      "   micro avg       0.89      0.89      0.89       170\n",
      "   macro avg       0.78      0.55      0.56       170\n",
      "weighted avg       0.87      0.89      0.86       170\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.400\n",
      "[[146   5]\n",
      " [ 12   7]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909108, description: DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT6 radioligand binding (ligand: [3H] Lysergic acid diethylamide)\n",
      "--------- Default -----------\n",
      "ratio: 0.084 kappa: 0.133, AUC: 0.893, OOB_AUC: 0.866\n",
      "[[156   0]\n",
      " [ 12   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      1.00      0.96       156\n",
      "           1       1.00      0.08      0.14        13\n",
      "\n",
      "   micro avg       0.93      0.93      0.93       169\n",
      "   macro avg       0.96      0.54      0.55       169\n",
      "weighted avg       0.93      0.93      0.90       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.500\n",
      "[[150   6]\n",
      " [  6   7]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909105, description: DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT2C radioligand binding (ligand: [3H] Mesulergine)\n",
      "--------- Default -----------\n",
      "ratio: 0.139 kappa: 0.226, AUC: 0.875, OOB_AUC: 0.902\n",
      "[[148   0]\n",
      " [ 18   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      1.00      0.94       148\n",
      "           1       1.00      0.14      0.25        21\n",
      "\n",
      "   micro avg       0.89      0.89      0.89       169\n",
      "   macro avg       0.95      0.57      0.60       169\n",
      "weighted avg       0.91      0.89      0.86       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.504\n",
      "[[143   5]\n",
      " [ 11  10]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909104, description: DRUGMATRIX: Serotonin (5-Hydroxytryptamine) 5-HT2B radioligand binding (ligand: [3H] Lysergic acid diethylamide)\n",
      "--------- Default -----------\n",
      "ratio: 0.169 kappa: 0.189, AUC: 0.943, OOB_AUC: 0.889\n",
      "[[144   0]\n",
      " [ 22   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      1.00      0.93       144\n",
      "           1       1.00      0.12      0.21        25\n",
      "\n",
      "   micro avg       0.87      0.87      0.87       169\n",
      "   macro avg       0.93      0.56      0.57       169\n",
      "weighted avg       0.89      0.87      0.82       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.300 kappa: 0.606\n",
      "[[142   2]\n",
      " [ 12  13]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909102, description: DRUGMATRIX: Calcium Channel Type L, Benzothiazepine radioligand binding (ligand: [3H] Diltiazem)\n",
      "--------- Default -----------\n",
      "ratio: 0.101 kappa: -0.011, AUC: 0.869, OOB_AUC: 0.915\n",
      "[[152   1]\n",
      " [ 16   0]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.90      0.99      0.95       153\n",
      "           1       0.00      0.00      0.00        16\n",
      "\n",
      "   micro avg       0.90      0.90      0.90       169\n",
      "   macro avg       0.45      0.50      0.47       169\n",
      "weighted avg       0.82      0.90      0.86       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.461\n",
      "[[138  15]\n",
      " [  5  11]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909094, description: DRUGMATRIX: Norepinephrine Transporter radioligand binding (ligand: [125I] RTI-55)\n",
      "--------- Default -----------\n",
      "ratio: 0.147 kappa: 0.119, AUC: 0.934, OOB_AUC: 0.861\n",
      "[[145   2]\n",
      " [ 20   2]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.99      0.93       147\n",
      "           1       0.50      0.09      0.15        22\n",
      "\n",
      "   micro avg       0.87      0.87      0.87       169\n",
      "   macro avg       0.69      0.54      0.54       169\n",
      "weighted avg       0.83      0.87      0.83       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.490\n",
      "[[140   7]\n",
      " [ 11  11]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909090, description: DRUGMATRIX: Adrenergic Alpha-2C radioligand binding (ligand: [3H] MK-912)\n",
      "--------- Default -----------\n",
      "ratio: 0.105 kappa: 0.078, AUC: 0.861, OOB_AUC: 0.904\n",
      "[[151   2]\n",
      " [ 15   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.99      0.95       153\n",
      "           1       0.33      0.06      0.11        16\n",
      "\n",
      "   micro avg       0.90      0.90      0.90       169\n",
      "   macro avg       0.62      0.52      0.53       169\n",
      "weighted avg       0.86      0.90      0.87       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.150 kappa: 0.392\n",
      "[[133  20]\n",
      " [  5  11]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909089, description: DRUGMATRIX: Alpha-2B adrenergic receptor radioligand binding (ligand: Rauwolscine)\n",
      "--------- Default -----------\n",
      "ratio: 0.141 kappa: 0.391, AUC: 0.873, OOB_AUC: 0.895\n",
      "[[147   1]\n",
      " [ 15   6]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.99      0.95       148\n",
      "           1       0.86      0.29      0.43        21\n",
      "\n",
      "   micro avg       0.91      0.91      0.91       169\n",
      "   macro avg       0.88      0.64      0.69       169\n",
      "weighted avg       0.90      0.91      0.88       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.497\n",
      "[[136  12]\n",
      " [  8  13]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909088, description: DRUGMATRIX: Alpha-2A adrenergic receptor radioligand binding (ligand: MK-912)\n",
      "--------- Default -----------\n",
      "ratio: 0.150 kappa: 0.183, AUC: 0.881, OOB_AUC: 0.917\n",
      "[[145   2]\n",
      " [ 19   3]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.99      0.93       147\n",
      "           1       0.60      0.14      0.22        22\n",
      "\n",
      "   micro avg       0.88      0.88      0.88       169\n",
      "   macro avg       0.74      0.56      0.58       169\n",
      "weighted avg       0.85      0.88      0.84       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.531\n",
      "[[136  11]\n",
      " [  8  14]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909087, description: DRUGMATRIX: Alpha-1D adrenergic receptor radioligand binding (ligand: prazosin)\n",
      "--------- Default -----------\n",
      "ratio: 0.096 kappa: 0.180, AUC: 0.878, OOB_AUC: 0.940\n",
      "[[152   2]\n",
      " [ 13   2]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.99      0.95       154\n",
      "           1       0.50      0.13      0.21        15\n",
      "\n",
      "   micro avg       0.91      0.91      0.91       169\n",
      "   macro avg       0.71      0.56      0.58       169\n",
      "weighted avg       0.88      0.91      0.89       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.442\n",
      "[[142  12]\n",
      " [  6   9]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909086, description: DRUGMATRIX: Alpha-1B adrenergic receptor radioligand binding (ligand: prazosin)\n",
      "--------- Default -----------\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ratio: 0.096 kappa: 0.099, AUC: 0.906, OOB_AUC: 0.926\n",
      "[[153   1]\n",
      " [ 14   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.99      0.95       154\n",
      "           1       0.50      0.07      0.12        15\n",
      "\n",
      "   micro avg       0.91      0.91      0.91       169\n",
      "   macro avg       0.71      0.53      0.54       169\n",
      "weighted avg       0.88      0.91      0.88       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.200 kappa: 0.510\n",
      "[[148   6]\n",
      " [  7   8]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "assay_id CHEMBL1909085, description: DRUGMATRIX: Alpha-1A adrenergic receptor radioligand binding (ligand: prazosin)\n",
      "--------- Default -----------\n",
      "ratio: 0.094 kappa: 0.099, AUC: 0.893, OOB_AUC: 0.921\n",
      "[[153   1]\n",
      " [ 14   1]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.99      0.95       154\n",
      "           1       0.50      0.07      0.12        15\n",
      "\n",
      "   micro avg       0.91      0.91      0.91       169\n",
      "   macro avg       0.71      0.53      0.54       169\n",
      "weighted avg       0.88      0.91      0.88       169\n",
      "\n",
      "--------- Balanced -----------\n",
      "thresh: 0.250 kappa: 0.398\n",
      "[[151   3]\n",
      " [ 10   5]]\n",
      "*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "drugmatrix_archive = defaultdict(list)\n",
    "for k in drugmatrix_assay_lookup:\n",
    "    run_drugmatrix_assay(k,drugmatrix_d,drugmatrix_assay_lookup,archive=drugmatrix_archive)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, look at the distribution of AUROC values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADlRJREFUeJzt3XuMZnddx/H3h66lFkEW98FL2+0UAo2FkNQMBiGItiJLixSVP7pas0CTTUiseCG4TRMhJCZViWCikWxgLYG6BCsqsaA0paXRlOq0tLSl9GJZYdvqTq2iqKEWvv4xp2E67M5zOWfm2fnt+5VM5jzn8pzvt7Pn05NzTVUhSdr6njbvAiRJwzDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY3Ytpkr27FjRy0sLGzmKiVpy7v11lsfrarRuPk2NdAXFhZYWlrazFVK0paX5J8nmc9DLpLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1IhNvVNUkgAW9l07l/UeuvLCuax3s7iHLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSI8YGepIDSY4kuWvN+MuS3Jvk7iS/u3ElSpImMcke+lXArtUjkvwkcBHwkqp6EfCe4UuTJE1jbKBX1U3AY2tGvxW4sqq+0c1zZANqkyRNYdZj6C8EXpnkliSfTfLSIYuSJE1v1me5bAO2Ay8DXgp8LMnzqqrWzphkL7AXYOfOnbPWKUkaY9Y99MPAx2vFPwDfAnYcbcaq2l9Vi1W1OBqNZq1TkjTGrIH+l8B5AEleCJwMPDpUUZKk6Y095JLkIPATwI4kh4F3AgeAA92ljI8De452uEWStHnGBnpV7T7GpEsGrkWS1IN3ikpSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGjE20JMcSHKkezvR2mlvT1JJjvo+UUnS5plkD/0qYNfakUnOAF4NfGXgmiRJMxgb6FV1E/DYUSa9F3gH4LtEJek4MNMx9CSvBx6qqjsmmHdvkqUkS8vLy7OsTpI0gakDPcmpwBXAb00yf1Xtr6rFqlocjUbTrk6SNKFZ9tCfD5wF3JHkEHA6cFuSHxiyMEnSdLZNu0BV3Qk898nPXagvVtWjA9YlSZrSJJctHgRuBs5OcjjJpRtfliRpWmP30Ktq95jpC4NVI0mamXeKSlIjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1IhJXnBxIMmRJHetGvd7Sb6U5AtJ/iLJsze2TEnSOJPsoV8F7Foz7jrgxVX1EuA+4PKB65IkTWlsoFfVTcBja8Z9uqqe6D5+jpUXRUuS5miIY+hvAT41wPdIknroFehJrgCeAK5eZ569SZaSLC0vL/dZnSRpHTMHepI9wOuAX6yqOtZ8VbW/qharanE0Gs26OknSGNtmWSjJLuA3gVdV1f8MW5IkaRaTXLZ4ELgZODvJ4SSXAn8IPBO4LsntSd6/wXVKksYYu4deVbuPMvqDG1CLJKkH7xSVpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEZO84OJAkiNJ7lo17jlJrktyf/d7+8aWKUkaZ5I99KuAXWvG7QOur6oXANd3nyVJczQ20KvqJuCxNaMvAj7UDX8IeMPAdUmSpjTrMfTvr6pHALrfzx2uJEnSLDb8pGiSvUmWkiwtLy9v9Ook6YQ1a6D/a5IfBOh+HznWjFW1v6oWq2pxNBrNuDpJ0jizBvongD3d8B7gr4YpR5I0q0kuWzwI3AycneRwkkuBK4FXJ7kfeHX3WZI0R9vGzVBVu48x6fyBa5Ek9eCdopLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRvQK9CS/luTuJHclOZjklKEKkyRNZ+ZAT3Ia8CvAYlW9GDgJuHiowiRJ0+l7yGUb8N1JtgGnAg/3L0mSNIuZA72qHgLeA3wFeAT4WlV9eu18SfYmWUqytLy8PHulkqR19Tnksh24CDgL+CHgGUkuWTtfVe2vqsWqWhyNRrNXKklaV59DLj8FfLmqlqvq/4CPAy8fpixJ0rT6BPpXgJclOTVJgPOBe4YpS5I0rT7H0G8BrgFuA+7svmv/QHVJkqa0rc/CVfVO4J0D1SJJ6sE7RSWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJakSv69AlbV0L+66ddwmbbp49H7rywg1fh3voktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1olegJ3l2kmuSfCnJPUl+bKjCJEnT6Xtj0R8Af1NVb0xyMnDqADVJkmYwc6AneRbw48CbAKrqceDxYcqSJE2rzyGX5wHLwJ8k+XySDyR5xkB1SZKm1CfQtwE/AvxxVZ0L/Dewb+1MSfYmWUqytLy83GN1kqT19An0w8Dhqrql+3wNKwH/FFW1v6oWq2pxNBr1WJ0kaT0zB3pV/Qvw1SRnd6POB744SFWSpKn1vcrlMuDq7gqXB4E39y9JkjSLXoFeVbcDiwPVIknqwTtFJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqRN8biyT1tLDv2nmXoEa4hy5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiN6B3qSk7qXRP/1EAVJkmYzxB7624B7BvgeSVIPvQI9yenAhcAHhilHkjSrvnvo7wPeAXxrgFokST3MHOhJXgccqapbx8y3N8lSkqXl5eVZVydJGqPPHvorgNcnOQR8FDgvyUfWzlRV+6tqsaoWR6NRj9VJktYzc6BX1eVVdXpVLQAXA5+pqksGq0ySNBWvQ5ekRgzyPPSquhG4cYjvkiTNxj10SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaMch16NJWt7Dv2nmXIPXmHrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEX1eEn1GkhuS3JPk7iRvG7IwSdJ0+twp+gTwG1V1W5JnArcmua6qvjhQbZKkKfR5SfQjVXVbN/xfwD3AaUMVJkmaziDPckmyAJwL3HKUaXuBvQA7d+4cYnVqmM9UkWbX+6Roku8B/hz41ar6z7XTq2p/VS1W1eJoNOq7OknSMfQK9CTfxUqYX11VHx+mJEnSLPpc5RLgg8A9VfX7w5UkSZpFnz30VwC/BJyX5Pbu54KB6pIkTWnmk6JV9XdABqxFktSDd4pKUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktSIQR7OtRlOxIc2Hbrywrms90T8by21wD10SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIa0fcVdLuS3JvkgST7hipKkjS9Pq+gOwn4I+C1wDnA7iTnDFWYJGk6ffbQfxR4oKoerKrHgY8CFw1TliRpWn0C/TTgq6s+H+7GSZLmoM+zXI72PtH6jpmSvcDe7uPXk9zbY52bZQfw6LyLyO9syNceF71tgFb7Anvbqp7SW8/t+cxJZuoT6IeBM1Z9Ph14eO1MVbUf2N9jPZsuyVJVLc67jo3Qam+t9gX2tlXNo7c+h1z+EXhBkrOSnAxcDHximLIkSdOaeQ+9qp5I8svA3wInAQeq6u7BKpMkTaXX89Cr6pPAJweq5XiypQ4RTanV3lrtC+xtq9r03lL1HecxJUlbkLf+S1IjTrhAH/e4giTvTXJ793Nfkv9YNW1Pkvu7nz2bW/n6evb1zVXTjrsT2xP0tjPJDUk+n+QLSS5YNe3ybrl7k7xmcysfb9bekiwk+d9Vf7f3b37165ugtzOTXN/1dWOS01dN28rb2np9bey2VlUnzA8rJ2//CXgecDJwB3DOOvNfxsrJXoDnAA92v7d3w9vn3VPfvrrPX593D316Y+VY5Vu74XOAQ6uG7wCeDpzVfc9J8+5poN4WgLvm3UPP3v4M2NMNnwd8uBve0tvasfrqPm/otnai7aFP+7iC3cDBbvg1wHVV9VhV/TtwHbBrQ6udXJ++jneT9FbAs7rh7+Xb90NcBHy0qr5RVV8GHui+73jRp7fj3SS9nQNc3w3fsGr6Vt/WjtXXhjvRAn3ixxUkOZOVvbrPTLvsHPTpC+CUJEtJPpfkDRtX5kwm6e1dwCVJDrNy1dVlUyw7T316AzirOxTz2SSv3NBKpzdJb3cAP98N/yzwzCTfN+Gy89KnL9jgbe1EC/SJHlfQuRi4pqq+OcOym61PXwA7a+WOtl8A3pfk+UMX2MMkve0Grqqq04ELgA8nedqEy85Tn94eYeXvdi7w68CfJnkWx49Jens78KoknwdeBTwEPDHhsvPSpy/Y4G3tRAv0iR5X0LmYpx6WmGbZzdanL6rq4e73g8CNwLnDlzizSXq7FPgYQFXdDJzCynM0jue/GfTorTuM9G/d+FtZOa77wg2veHJje6uqh6vq57r/KV3RjfvaJMvOUZ++Nn5bm/dJhs38YeVGqgdZOeTw5AmNFx1lvrOBQ3TX6de3T9R8mZWTNNu74efMu6cB+toOPL0b3gHczzonVI/H3oBPAW/qhn+YlQ0swIt46knRBzm+Tor26W30ZC+snKB76Hj59zhFbzuAp3XDvw28uxve0tvaOn1t+LY29/9Ac/iDXADcx8oezRXduHcDr181z7uAK4+y7FtYObH2APDmefcyRF/Ay4E7u3+YdwKXzruXaXtj5STU33c93A789Kplr+iWuxd47bx7Gao3Vo7R3t2Nvw34mXn3MkNvb+xC7T7gA0+GXTdty25rx+prM7Y17xSVpEacaMfQJalZBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY34fwY1+UQodq3nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist([x[0][-1] for x in drugmatrix_archive.values()]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bringing it all together\n",
    "\n",
    "Plot the balanced kappa value versus the original value for all of the datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump((drugmatrix_archive,pubchem_archive,ds1_archive,serotonin_archive),\n",
    "            open('../data/unbalanced_data_results.pkl','wb+'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XlcVOX+wPHPo2guueWKgqKCCqJoUi5pimWL3aJ7r92s1AL3Jf2Z3bIsSy2ztFxyyS21xbVyuWZaaoKKC64oKAqKgSi5IbkizPP7Y2BiGeAAc2Z93q+XL5kzZ2a+B2bme86zfB8hpURRFEVRAMrYOgBFURTFfqikoCiKopiopKAoiqKYqKSgKIqimKikoCiKopiopKAoiqKYqKSgKIqimKikoCiKopiopKAoiqKYuNk6gOKqVauW9PLysnUYiqIoDuXgwYOXpZS1i9rP4ZKCl5cXBw4csHUYiqIoDkUIcU7Lfqr5SFEURTFRSUFRFEUxUUlBURRFMVFJQVEURTFRSUFRFEUxUUlBURRFMVFJQVEURTHRLSkIIb4WQvwphDhewP1CCDFLCBEnhIgSQjyoVyyKoiiKNnpOXlsKzAa+KeD+pwGfrH/tgXlZ/1uVwWBgZ/RpYs4l49eoPl1a+lCmjLqAUhRLsofPmdYYtOyXkZHBV7+Ec/BUAu2aeTHk6Udxc7Ps16mtfme6JQUpZbgQwquQXYKBb6SUEtgrhKguhHCXUl7QK6a8DAYDI+YuZ/vRk6Zt3QNaMHvYyyoxKIqF2MPnTGsMWvbLyMig29vTuJx2A4A9J8+wcsd+dnz6psUSQ844pMGAKFPGar8zW37zNQASc9xOytpmNTujT+f64wNsP3qSndGnrRmGojg1e/icaY1By35f/RJuSgjZLqfd4Ktfwi0a72/7DnNhxwYu7vy5wHj1YMukIMxsk2Z3FGKQEOKAEOLApUuXLBZAzLlks9tP/GG1ixVFcXr28DnTGoOW/Q6eSjC7z6HTmkoLabJ85WrO/jiftPhoyt1fFWODinV+Z7ZMCkmAZ47bHoDZv4iUcoGUMlBKGVi7dpFF/jTza1Tf7Hbfhu4Wew1FcXX28DnTGoOW/do18zK7z4M+jUoWXA6XL1/mpZdeYsHk8bhVup9Gz4dQq11XhBBm49WDLZPCBqBf1iikDsB1a/YnAHRp6UP3gBa5tnUPaEGXlj7WDEOxAIPBQNixWOZt/J2wY7EYDIYSPU96ejpvLlpNj3c+581Fq0lPT7dwpEWz1LHYiy4tfQjK8zkLsuDn7M6du/SdtoiHR31E32mLuHPnrtkYtHzWtew35OlHqVX1/lz71Kp6P0OefjTXtuL+HdesWYOfnx8//vgjEyZM4NX3p1ChZr1C49WDbh3NQogVQDeglhAiCfgAKAcgpfwK2AT0BOKAW0CIXrEUpEyZMswe9jI7o09z4o8L+DZ0V6OPHJClOjLT09MJHPUx6RmZACRevsavB6M5MHMc5cuXt3jc5thDp6wupCz8dgnduXOXtiMnmZ4uMjaBtiMncXjW+1SocJ9pP62fdS37lSlTBn+vBuyIijVt8/dqUOwO62wpKSkMHz6cH3/8kXbt2rFt2zZatWplGn1k7e8mIS30x7GWwMBAqdZTUHIKOxbL4Fnf5ts+f2RfurZqrvl53ly0mo37ovJt/0f71kwb8J9SxaiVpY7Fnuh5TH2nLSIyNiHf9oeae/HtmwNK9dwF0XI8WvaRUrJ8+XJGjhzJjRs3mDBhAm++abkRTHkJIQ5KKQOL2s+BTz0UxchSHZlH4xPNbo86k1TsmErKHjplLU3PY4pNvGh2+6nElFI/d0G0HE9R+yQnJxMcHEyfPn1o1qwZR44cYezYsbolhOJQSUFxeJbqyAxo6ml2e+smHsWOqaTsoVPW0vQ8puae9cxub+ZZt9TPXRAtx1PQPi0867F06VJatmzJb7/9xrRp09i1axe+vr66xFoSKikoDs9SHZmT+z1PebeyubaVdyvL5H7PlzpGrRxl8ENxOlH1PKaFI/oi8gxuF8K4XS9ajsfcPg83rM2nb48mJCQEf39/Dh8+TOAT/2DBL+F2NaBA9SkoDs9gMDB8zvf8nqPjL6h1c+YMf6XYHXPp6em8+806os4k0bqJhzFRWKmTOZutOhi1MhgMDJ+7nN9zdKIGBbRgTiGd4Xoe0507dxk4+1tOJabQzLMuC0f0zdXJrActx5OzTMWZyF0smDGNjIwMpkyZwtChQxn51UqrDijQ2qegkoLi8Jyxc9ae/X70JENnf5dv+7wRffJdsbm6hIQEBg4cyNatWwkKCmLRokU0adLEJu9Z1dGsuAxn7Jy1Z78cOGZ2++aDZgsiuySDwcDcuXNp1aoVe/fuZd68eWzdupUmTZoA9v2etX1Xt6KUkjN2zjokx2p00E18fDwDBgxgx44d9OjRg4ULF9KoUe7Zzvb8nlVXCorDc5TOWWfx9EOtzG5/6iF/K0diXwwGAzNnzqR169YcOnSIRYsWsWXLlnwJAez7Pav6FBSnYO+ds8VRmlr9JanBX9zHWLJj31mcOnWK0NBQdu/eTc+ePZk/fz4eHoUPZbb2e1Z1NCuKA8pbqx+MdXW01OovSYmMkpbVcKYkXBqZmZlMnz6d999/nwoVKjBz5kz69u1rKmBnT1RHs6I4oNLU6i/JugUlXeugTJkydG3VnCHPdKNrq+YumRBiYmLo1KkT//3vf3nyySeJiYmhX79+dpkQisP1/pKKorPSVDktTa3+koxosdYoGGeq/JqRkcEnn3xC27ZtiY+PZ/ny5axduxZ3d9t3EluCGn1kJfawRq2iv9JWOW3XzIs9J8/k266lVn9JRrRYYxSMvVR+tcRn8NixY4SEhHDw4EF69erF7NmzqVtXv5IatqCSghXYy4dC0V9hzTFaJiUNerIzCzaFmcp3g7HUxqAnOxf52OwRLXnfZ4WNaCnJY4qrtL8TSyjtZ/DevXt88sknfPTRR1SvXp01a9bQq1cvPUO2GZUUrMAePhSKdRTWHKPlb70n9myuhACQnpHJntizRT6+JOuDWGNNkdL+TiyhNJ/Bw4cPExISwtGjR3nppZeYNWsWtWrV0jNcm1JJwQrs4UPh7Oylea60zTHRCefNbtf6XsnuAC7O+6okjykOe5ioVZLP4N27d5k0aRJTpkyhdu3arFu3juDgYD3DtAsqKViBPXwonJk9Nc+VpjnGYDCw49gps/c58nvFGk1URSnuZzAyMpKQkBCio6Pp168f06dP54EHHtAzRLuhkoIV2MOHwpnZU/NcaZpjdkafJups/gV9Ahp7aH6v2MsVU072sOyt1s/gnTt3+OCDD5g2bRru7u5s3LiRZ555xmpx2gOVFKzAHj4UzszemudK2hxT0HF0a61tHoA9XTHlpXcTlZbXL+ozGBERQWhoKLGxsfTv35/PP/+catWq2SReW1LfSlbmaDPIHYGzNM8VdBwFbc+rpBPRXEVBE+5u3brFG2+8QefOnbl9+zZbtmxh0aJFLpkQQF0pWIU9n8E5A2dpnivtcdjbFZMjCA8Pp3///sTFxTF06FA+/fRTqlSpYuuwbEolBSuwpzZvZ+QszXOlPQ5rXjHZY99Fcdy4cYN33nmH2bNn07hxY7Zv305QUJCtw7ILKilYgTqD05+t26wtpTTH8YhvU2pVvT9fMb1HfJtaMkSHv/Ldvn07AwYMICEhgZEjRzJ58mQqV65s67Dshv3/BZ2As7R5W5oz1cOxB7tPxJstprf7RLxFX8dR+y7S0tIYMmQIjz32GG5uboSHhzNz5kyVEPJw+SsFa1wGd2npQ1BAi3wLnTtam7clqZr8lmetK1JrvI6lP5dbtmxh4MCBnD9/njFjxjBx4kQqVapkkVidjUsnBateBucddeTio5DCjp/KlRAAfo+KJez4KYJaq8XfS8JaV6R6v44lP5epqamMGTOGr7/+mhYtWrB79246dOhgkTidlUufklnrMnhn9GmzX4D2frmtp18iC1j8PVIt/l5S1lriMbvvIidL9l1Y6nO5ceNGWrZsydKlSxk7diyHDx9WCUEDl75ScKbLbb1ZbbSJY69PYlPWGoVVWN+FJd7Ppf28XL16lVGjRvHdd9/h7+/P+vXrCQwscsExJYtLJwVnudzWmx7NbE8HtmLD3qP5tj/VzrUXfy8ta4zC0vskpzifl7wnK5dPHWf48OFcuXKF999/n3HjxnHfffeVOiZX4tJJwVpD+Bx9cpUe8yy6tmpGUOvm+Tqau7ZqVqpYFf3pfZKj9fOS82Ql4/ZN/tzzK3+diaFNmzZs3ryZNm3aWCQeV6NrUhBCPAXMBMoCi6SUU/Lc3xBYBlTP2meslHKTnjHlpPdlcDZHn1zlDM1fSsGK2zSo90mO1s9L9snKX2dOkBKxmcz0O9Rs9yjTFsyhTRt1xVlSuiUFIURZYA7QA0gCIoUQG6SUMTl2ew9YLaWcJ4TwAzYBXnrFlJc1v+wceXKVHmeGhXW+l+R35OgzbG2lJE2D1jjJ0fJ52XvkOOe3/siNhJNUqOWO56OvcN8DdTh94TKPWSwS16PnlcLDQJyU8gyAEGIlEAzkTAoSqJr1czXA/Le0Tlp4mv9Sa+5Rz5ph2L2OzRtT3q1sviUiOzZvXOLntGRCdvQZtrZU0qZBW57kSClZvnw5k18fzs0bN6j1UBAPtOqAyPpbO0pfnb3S8xPTAEjMcTspa1tOHwJ9hBBJGK8SXtcxnvwKGumiRsDksmDLLrNLRC7Yssvs/lpmKlvy6sNRZ9jag8KSsz1KTk4mODiYPn364Ofry4tjP6ZmQCdTQnCkvjp7peeVgrmv1rwztl4ClkopPxdCdAS+FUL4SylzfYsIIQYBgwAaNmxosQBPFvDGj028qCZQ5XDwVILZ7YdOn8u3TetZuyU7+VWfR8k5ysg4KSXLli1j9OjR3Llzh88//5xRo0YhhHDYvjp7pedvLwnwzHHbg/zNQ/2B1QBSyj1ABSDfithSygVSykApZWDt2rUtFqCjfCBsrV0zL7PbH/RplG+b1rN2S9bpUX/HkrPWhLfSSExM5JlnniEkJAR/f3+OHj3KG2+8QdmyZQtcI0EpOT1/g5GAjxCisRCiPNAb2JBnnz/A2CckhPDFmBQu6RhTLo7wgbAHQ55+1OwM1iFPP5pvX63NEZZsttB7hq0zy+40nj+yL//3/OPMH9nXbvpipJQsXLiQli1bEhYWxqxZswgLC6NZMzVsWU+6NR9JKTOEECOALRiHm34tpYwWQkwEDkgpNwBjgIVCiNEYm5Zek1ZcmszRh4pai5ubGzs+fZOvfgnn0OlzPOjTiCFPP4qbW/63j9azdkue3VtraLGzsseRcQkJCQwcOJCtW7cSFBTEokWLaNKkia3Dcgm6zlPImnOwKc+28Tl+jgEe0TOGotjjB8Ieubm5MeLZ7kXup3UMuyXHuqs+BedhMBj46quveOuttxBCMG/ePAYNGqRO1KzIpWc0Oxs9x+prfW6tV1+WvEpTfQrOIT4+ngEDBrBjxw569OjBwoULadQof7+Voi+VFJyEnmP1i7v2gdarL4PBwLGE8xw8lUCGwcAjvk2LjNVccnL0MiKuzmAw8OWXX/Luu+/i5ubGokWLCA0NRQg1NtwWVFJwEnquA63H2gcZGRl0e3uaqS9gz8kzrNyxnx2fvmm2rwIKT3yqb8gxnTp1itDQUHbv3k3Pnj2ZP38+Hh4etg7Lpbn8p8ZZloTUcxKSHmsffPVLuNnO4a9+CS/wMYUlPjU00bFkZmYybdo0AgICiI6OZtmyZWzcuFElBDvg0lcKzlQewSbt6qW4ui/OhLhsqkPZOcTExBASEsL+/fsJDg5m3rx5uLur/h974VjffBbmTOUR9Jxz8XRgK7PbS7P2QXEmxGVTHcqOLSMjg8mTJ9O2bVvi4+NZvnw5a9euVQnBzrj0lYIznXnqOeeia6tmBAW04PccCTQooEWp1j4Y8vSjrNyxP1+ZC3MT4rKpDmXHFRUVRUhICIcOHaJXr17Mnj2bunXr2josxQyXTgrOduap15yLMmXKMMfCCcfNzY3tn7zBu9+s4+iZJAKaeDC53/MFdjJnx6E6lB1Leno6n3zyCR9//DHVq1dnzZo19OrVy9ZhKYVw6aSgzjy1s3TCMRgM/N+C1abffeKlq9y6k15kf46abOg4Dh8+zGuvvUZUVBQvvfQSs2bNolatfKXNFDvj0knBFmeeUkriz6dw8Uoq9WpWp2mDug4xHtvSE+P0HEKr2Nbdu3eZNGkSU6ZMoXbt2qxbt47g4GBbh6VoVGRSEEKUAQKA+sBtIFpKmaJ3YNZizTNPKSVrtu/ldOLfw0R9PN15oXsHu04MeozScqb+HOVvkZGRhISEEB0dzauvvsr06dOpUaOGrcNSiqHAT7QQoqkQYgEQB0zBuPbBMOA3IcReIURIVsJQNIo/n5IrIQCcTrxA/HnL5Fi95lzoMUrL2fpzXN2dO3d4++236dChA6mpqfz8888sXbpUJQQHVNiVwkfAPGBw3sqlQog6wMtAX2CZfuE5l4tXUgvc7l3KJUD1nHOhx1m96s9xHhEREYSGhhIbG8uAAQOYNm0a1apVs3VYSgkVmBSklC8Vct+fwAxdInJSBoOBS3/dMHtfvZrVS/38erbR63FWr0YS/c1R+5lu3brFe++9x4wZM/D09GTz5s1UqO/F8p2HLF6QUbGeApOCEOJfhT1QSvmT5cNxTjnP4ru1aIJnjiTg4+lO0walH6+tZxu9Xmf1aiSR4/YzhYeH079/f+Li4hg6dCiffPIJ73z7P7b/sNO0j6NWB3B1hTUfPZv1fx2gE7A963YQsANQSUGjnGfxO06eoX6NqjxQuRKvPNaB5x9pZ5EPv55t9OqsXj+F9TOVtklRDzdu3OCdd95h9uzZNG7cmO3btxMUFETYsVg1msxJFPipllKGSClDMK6I5iel/LeU8t9AS6tF5yTynsUnX0vjeNJFUq7fsNjZoN5Li6qCc/oorJ/J3mzbto1WrVoxZ84cRo4cybFjxwgKCgL0LcioWJeWeQpeUsqcf9kUwGkWSdVzYZps1hhpo87m9afHe6Wg/qQ6NaoSdixW1/elVmlpabz11lvMnz8fHx8fwsPD6dy5c6591Ggy56ElKewQQmwBVmC8augN/K5rVFZirSqp2QvL563zY+mF5VUbvX70eq80bVAXH0/33H0KHvWYsWG7XVTv3bJlCwMHDuT8+fOMGTOGiRMnUqlSpXz7qdFkzqPIpCClHCGE+CeQXalsgZRyrb5hWYe1ZtU6w8Ly1riismd6vVeEELzQvUOu0UdJV1PZvirX0uZWb59PTU1lzJgxfP3117Ro0YLdu3fToUOHAvdXV6rOQ2uZi0PAX1LKrUKISkKIKlLKv/QMzBqsNavW0WfvOtO6EyWl599QCIG3Rz1Tx/JvR07o9lpabNy4kcGDB5OSksLYsWP54IMPqFChQpGPU1eqzqHIT7QQYiDwAzA/a1MDYJ2eQVmLtdpBHb291ZnWnSgpa/4NbfV+uXr1Kn379uXZZ5/lgQceYO/evXzyySeaEoLiPLSc5g0HHgHSAKSUpzEOU3V4eo/Ysfbr6EWNLLHu39AW75e1a9fi5+fHypUrGT9+PAcPHiQwMFC311Psl5bmo7tSyvTsoZNCCDeMHc4Or0yZMswa0puvfgnn4KkE2jXzYsjTj1q8ScTR21sd/UrHEqz5N7Tma126dInXX3+dVatW0aZNGzZv3kybNm0s/jqK4xB5yhrl30GIz4BUoB/wOsaieDFSynH6h5dfYGCgPHDggEWeS7WVa6N+T85HSsmaNWsYMWIEqampvP/++4wdO5Zy5crZOjRFJ0KIg1LKIi//tFwpjAX6A8eAwcAmYFHpwrMPqqa/No5+paPklpKSwrBhw/jpp58IDAw0TUpTFNCWFCoCX0spFwIIIcpmbbulZ2DWEJ1w3uz2mHPJKinkoUaWOD4pJcuXL2fkyJHcvHmTKVOmMGbMmEKXQFVcj5ZTvW0Yk0C2isBWfcKxroJWG8gsoknNkUgpiUu6yK6jJ4lLukhRzYWKc0pOTiY4OJg+ffrQrFkzDh8+zNtvv60SgpKPlndEBSmlaeaVlPKGECL/lEYHVFBGLGPH1SmLw1ErcCqWI6Vk2bJljB49mjt37vD5558zatQoypYta+vQFDulJSncFEI8KKU8BCCEaIdxWU6H19KrgfntBYy2KY2SzAgu7Sxie6vA6eqzoq0tMTGRwYMH88svv9ClSxcWL16Mj49jDINWbEdLUvg/YI0QInuwujvwopYnF0I8BcwEygKLpJRTzOzzH+BDjMNcj0opX9by3JbQsXljyruVJT0j07StvFtZOjZvbNHXKcnoHUuM+Llw+ZrZ7ZZY6a241Agm65FSsmjRIsaMGUNmZiazZs1i+PDh6vesaFLku0RKGQm0AIZiHI7qK6U8WNTjsjqk5wBPA37AS0IIvzz7+ADvAI9IKVtiTEBWs2DLrlwJASA9I5MFW3ZZ9HVKMiO4tLOIs/sSzLHESm/FpWZFW0dCQgJPPPEEgwYNIjAwkGPHjvH666+rhKBopvWd8hDQGmiL8cu9n4bHPAzESSnPSCnTgZVAcJ59BgJzpJTXwLTMp9UcPJVgdvuh0+cs+jolmRFc2lnE8edTSDZzpVC/Vg2LrPRWXGpWtL4MBgNz5szB39+fvXv3Mm/ePLZu3UqTJk1sHZriYIpsPhJCfAs0BY4A2afVEvimiIc2ABJz3E4C2ufZp1nWa+zG2MT0oZRyc9FhW0a7Zl7sOXkm3/YHfRpZ9HVKMiO4tLOIC1qkxcfT3SadzGpWtH7i4+Pp378/YWFh9OjRg4ULF9KokWXfw4rr0HKlEIixeWeYlPL1rH8jNTzO3DdP3vGQboAP0A14CVgkhMjXtiGEGCSEOCCEOHDp0iUNL63NoCc7U94t9yiM8m5lGfRk5wIeUTIlqWVT2vo3BTUR2aLpCBy//pM9yszMZMaMGbRq1YrDhw+zaNEitmzZohKCUipaOpqPA/WA4l7nJwGeOW57AHnbEJKAvVLKe8BZIUQsxiQRmXMnKeUCYAEYy1wUM44C7Yk9a7ZPYU/sWYtO0irJjODSziI2u3iLp7tNmo5AzYq2tNjYWEJDQ4mIiKBnz57Mnz8fDw8PW4elOAEtSaEWECOE2A/czd4opXyuiMdFAj5CiMbAeYwrtuUdWbQO4xXCUiFELYzNSfnbc3RizXUOSjIjuDSziM0t3tK0QV2bzk9Qs6JLLzMzk+nTp/P+++9ToUIFli1bRt++fdW8E8VitCSFD0vyxFLKDCHECGALxv6Cr6WU0UKIicABKeWGrPueEELEYOyv+K+U8kpJXq8knL2dO+/iLYpji4mJISQkhP379xMcHMy8efNwd3eO96piP7QsxxlW0ieXUm7CWEAv57bxOX6WwBtZ/6yuS0sfggJaEPtHMjUrV+LKzVs0b1hftXMrdiUjI4PPPvuMCRMmUKVKFVasWMGLL76org4UXWgZfdQB+BLwBcpjPOu/KaWsqnNsupNSUvO+8jzm523advtehqoPpNiNqKgoQkJCOHToEC+88AKzZ8+mTh2nWONKsVNaevlmY2z3P42xGN6ArG0Ob8Oew1QslzsvViznxoY9h20UkaIYpaenM2HCBAIDA0lMTGTNmjWsXr1aJQRFd5pKJEop44QQZaWUmcASIUSEznFZxdlk83PlziZbbtirohTXoUOHCAkJISoqipdeeolZs2ZRq1YtW4eluAgtVwq3hBDlgSNCiM+EEKOByjrHZRWN65s/62pcv7aVI1EUuHv3Lu+99x4PP/wwf/75J+vWrWP58uUqIShWpSUp9MXYjzACuIlx7sG/9QzKWv7RPoCUtBu5tqWk3eAf7QNsFJHiqvbv30+7du34+OOP6dOnDzExMQQH560Koyj60zL6KLsQ0G1ggr7hWFfEyTP8euwU9WtU5YHKlbh68xbJ19KIOHnG4cfSqzLVjuHOnTt88MEHTJs2DXd3d37++Wd69uxp67AUF1ZgUhBCHCN/WQoTKWVrXSKyouzJa8nX0ki+lmbarsfkNWtSZaodQ0REBKGhocTGxjJgwACmTZtGtWrVbB2W4uIKu1L4h9WisJEWBUxSa+7p2JO9CitT7cjJzlncunWL9957jxkzZuDp6cmvv/5Kjx49bB2WogCFJIUczUbOq6DrIAedpiClJP58CgdPxFO/RtVcVz/g+FdAziA8PJz+/fsTFxfH0KFD+fTTT6lSpYqtw1IUkyLbEoQQHYQQkUKIG0KIdCFEphAirajHOYKTieZr/MUWsDiNPctej3nV1ghu37rDY37edGuRu5a+s5TvcEQ3btzg9ddfp2vXrhgMBrZv387cuXNVQlDsjpZ5CrMxFrNbg7GMdj/Au9BHOIjs2kf1a1Q1lblIvpam+5enHp3A5tZj9qxZ3XTFoMpU2862bdsYMGAA586dY+TIkUyePJnKlZ1iVLfihFx68lpnP296tW9NRbe/fw23MzLo7KdfziuoE/jLoS9x9sKlXBVNAc1VTgtaVOcfga1o59ukyMSjRitZXlpaGv/9739ZsGABPj4+hIeH07mzZdfqUBRL05IUck1ew7iuglOc5py9cClXQgCo6ObG2QuXdKssWlAn8IIN27l6/S/TNm+PeggBpxP/bsry8XTnhe4dzCaGghbPea5T2yKPxVlGK2X3qdhDqfAtW7YwcOBAzp8/z5gxY5g4cSKVKlWySSyKUhxaJ6+VwQknrxV0dl3Qdkswt4ZD/RpVcyUEgLiki7kSAsDpxAvEn08x+7zZi+rkpHVRncJGKzmKnH0qYYdjWLU1gjXb91q9uGFqaiqhoaE89dRTVK5cmd27dzNt2jSVEBSHoWnyWtaVghfwExArpUzXOzBrsOaSldnNM2dTLue7r2Zl7V8YF6+kmj3zL82iOgUtNhR9LtlhRiuZ61PJTqLWWk9i48aNDB48mJSUFMaOHcsHH3xAhQoVrPLaimIpWkpnPwN8BcRjXHe5sRBisJTyF72D05u1lqw01zyT05WbtzQ/V2EJq6SL6hS02NCOqFiG9OzqEE1IhV316Z0Url4T2LNiAAAgAElEQVS9yqhRo/juu+/w9/dn/fr1BAYG6vqaiqIXLX0KnwNBUso4ACFEU+BnwOGTgrWWrDTXPJNT8rU0rt+6TbVKFQt9Hr3WWO7S0oeAxh4cPZuUa3vU2SSHmfBmzau+nNauXcvQoUO5cuUK48ePZ9y4cZQvX17X11QUPWk5BfwzOyFkOQOYrzmtmFVQ80xOBxLOm93epY0vXdv68eLjnQrsZC4t49rJzczed+IP83M57E1p+lRK4tKlS/Tu3Zt//etfuLu7ExkZyYQJE1RCUBxeYbWP/pX1Y7QQYhOwGuNc3xeASCvEprvszsm8zUeW/vItqHkmp+RraSReScUzx5mtj6c7XQJaWGUETUuvBma3O8qEN2td9UkpWbNmDSNGjCA1NZVJkybx9ttvU65cOYu+jqLYSmHNR8/m+DkF6Jr18yWghm4R6SznePx61atYpXPyEd+m1Kp6P5dzlOmuVfV+/BvVZ8exU6ZtZe4rzwvdO/DntTSrD6ns0tKH7gEt8g1LdaQJbyXtU9EqJSWFYcOG8dNPPxEYGMj27dvx9/fX5bUUxVYKq30UYs1ArCFvh28rj3q0MXMWb+nOyd0n4nMlBIDLaTd4sevDvBTUnhN/XMC3obtpwlizhkVfWRRGGgxc3xPJrZOnqdTCh2odH0IU0VlcpkwZZg97mZ3Rp/PF4+qklCxfvpyRI0dy8+ZNpkyZwpgxY3Bz0zT3U1EcSmHNR+8Bc6SU1wq4vztQSUq5Ua/gLC1vh29Bo34s3TlZUJ9CbNJFhjzTzaIdudJgIO6tD0gN/3vSefVHO+H92QRNiaFrq+YO0bFsLcnJyQwZMoT//e9/dOjQga+//hpfX19bh6UouinsVOcYsFEIcQc4hLHZqALgA7QBtgKTdY/QgvJ+ORfUjp+zc9ISs2QL6lPQo73++p7IXAkBIDU8gut7Iqn+SHuLv56zklKydOlSRo8ezd27d/n8888ZNWoUZcuWtXVoiqKrwpqP1gPrhRA+wCOAO5AGfAcMklLetk6IlmPuy3nHyTO82+tJygKedWvRuXVz05e+pTqirdlef+uk+VnIt2LjVFLQKDExkUGDBrF582a6dOnC4sWL8fFxnL4VRSkNLTOaTwOOU++gEOa+nHu1b03ihUsAJGQVpMv+0rfULNmC2usBwo7FWrQIXcXmTc1vb9bE7Hblb1JKFi1axJgxY8jMzGTWrFkMHz5c9asoLsWlesryfjnXrXY/MXF/5Non55e+JWfJ5m2v16sIXVSV6pyoWoN2aX93BR2sWoNbVaqbho8p+SUkJDBw4EC2bt1KUFAQixYtokkTlUgV1+Nyp0DZX85DnulG7Sr3m90nOxnoOUtWryJ0MYkXmenVnGmNW7CmnifTGrdgpldzTiSZL6Tn6gwGA3PmzMHf35+9e/cyb948tm7dqhKC4rJcLinkVNSXvp6zZGPOJSOkJCDtGsEpSQSkXUNIWeoZxH6N6iOF4GjVGmyo68HRqjWQQjjMJDRrio+Pp3v37owYMYJHHnmE6OhohgwZopqLFJempSBeBaA/0BLj6CMApJShOsZlFU0b1MXbox5xOZbf9PaoZ/rS13OWrJ9nPUYlxOZr5vH16FOq53WUSWglmUthKZmZmXz55Ze8++67lCtXjsWLFxMSEmKztRcUxZ5o6VP4FjgJPAlMBF4BTugZlLUYDAaOnU3EM/k8VS5d5q/atTh27x69DO1NQw/1miXb+q9UKqXlngLSLu0a3n+Vbi0HR5iEVpq5FKUVGxtLaGgoERER9OzZk/nz5+Ph4aHrayqKI9GSFLyllC8IIYKllMuEEMuBLVqeXAjxFDATKAssklJOKWC/XhjXgH5ISnlAY+wlknPewZnkFDr+9it1EhJM93t4ebGheWP++ejDeobB7dh489tPnaFG546lem57n4Rmi7kUmZmZfPHFF4wfP54KFSqwbNky+vbtq64OFCUPLUnhXtb/qUIIf+AixgV3CiWEKAvMAXoASUCkEGKDlDImz35VgJHAvmLEXSJ55x3UOpeQKyEA1ElI4HhEJOiYFAwGA/EVKmJu+ZVKzbWtD+3Iaypbey5FTEwMISEh7N+/n+DgYObNm4e7u+pjURRztCSFBUKIGsD7wAbg/qyfi/IwECelPAMghFgJBAMxefabBHwGvKk16JLKO++gyqX8q6ABeN6+aZHXM/fFDTBi7nJ+P3KCUXmGjlZ/tBPVOj6k6XkdeU3lSi3M929oTYhaZWRk8NlnnzFhwgSqVKnCihUrePHFF9XVgaIUQsvktUVZP4YBxRmn1wBIzHE7Cch1GiiEaAt4Sik3CiF0TwoXLuduw/+rdi2z+7V9rHOpX6ugL+4XugQatwnBTK/mtP4rlUa3b/Lsi8/i/dI/NbWpFzac1V6bjHKq1vEhqj/aKV+fgpaEqFVUVBQhISEcOnSIF154gdmzZ1OnTh2LPb+iOCsto49qAh9iLHUhgZ3AJCnllaIeamabaRV1IUQZYDrwmoYYBgGDABo2bFjU7gXKu4T75YaN+NPLK1cT0p9eXlT38qJmiV/FqKAv7vsr3vd3PFlDR49WrUGT6jXpovEsv6ACeyf+uOAQSUGUKYP3ZxOMo49i46jU3Ntio4/S09P55JNP+Pjjj6lRowarV6+mTovW/Lg/Gr9G1xyqmU1RbEFL89FKIBz4d9btV4BVwONFPC4J8Mxx2wPI+W1WBfAHdmRdztcDNgghnsvb2SylXAAsAAgMDMz73a5ZviwlBEee7kmtP86ZRh9dbtiIatfS8Cll+eqCvriF2VxZvOJ41iywpxdRpgzVH2lv0T6EQ4cOERISQlRUFC+//DLTp0/nw9W/sv3Xb037OFIzm6LYgpZPxgNSyklSyrNZ/z4CtEzpjQR8hBCNhRDlgd4Y+yQAkFJel1LWklJ6SSm9gL1AvoRgSe61zKwNJASXG3lxNjCQy428QAgMssR5x6SgL+6nHvKne0CLXNuKO48gey5CaZ7Dmdy9e5f33nuPhx9+mEuXLrF+/Xq+//57TqRc02XWuKI4My1XCr8LIXpjXI4ToBfwc1EPklJmCCFGYBy+Whb4WkoZLYSYCByQUm4o/BksL3uGct4id3lZohuyoElkXf2b0dW/WanmETjCXARr2b9/P6GhoURHR/Pqq68yffp0atQwJn9Hb2ZTFFsQsoizYiHEX0BlwJC1qQyQPTxHSimr6hdefoGBgfLAgZJfTOScp5CZmcmuqNh8+7zQvUOpVz+Dv0cfufoXtx5u377NBx98wOeff467uzsLFiygZ8+eufYJOxbL4Fnf5nvs/JF9VVJQXI4Q4qCUMrCo/Yr8hpJSVpFSlpFSumX9K5O1rYq1E4IlZM9Q7hzQAulWlsQ8lVATr6SSnJpmkdfKWXyva6vmKiFYSEREBG3btmXq1Kmmq4S8CQFUM5uilISm0tlCiH8BnckafSSlXKdrVFZy4o8L7Dh5hvo1qvJA5UpcvXmL5GtptGnRhG6tWxT9BIpV3bp1i3HjxjFz5kw8PT359ddf6dGjR4H7q2Y2RSk+LUNS5wLewIqsTUOEED2klMN1jcwKsjuDk6+lkXzt76sDRxrFY2m2LFRXmPDwcEJDQ4mPj2fo0KF8+umnVKlSpcjH2XvJD0WxN1quFLoC/jKr80EIsQzj+s0Or0tLH4ICWvB7js7gIBduXrBlobqC3Lhxg3feeYfZs2fTpEkTtm/fTlBQkE1iURRXoOWTHgvknDHmCUTpE44N5O1ot8BwVEdVWKE6W9i2bRutWrVizpw5jBo1iqioKJUQFEVnWpJCTeCEEGKHEGIHxtpFtYUQG4QQVh9Wakk7o0/ze57RR79HxbrsOPbCCtVZU1paGoMHD+bxxx+nXLlyhIeHM2PGDCpXrmzVOBTFFWlpPhqvexQ2osax52atQnWF2bJlCwMHDuT8+fO8+eabTJw4kYoVK1rt9RXF1WkpiBdmjUBswRnKRViSNQrVFSQ1NZU33niDJUuW4OvrS0REBO3b67O2gqIoBdMy+qgD8CXgC5THODv5piPOUcjLUZautBY9C9UVZuPGjQwePJiUlBTGjh3LBx98QIUK5labUBRFb1pmNB/AWLdoDRAI9AN8pJTv6h9efqWd0ZyXmnVsO1evXmXUqFF89913+Pv7s2TJEgIDi5xwqShKCVhsRjOAlDIOKCulzJRSLgG6lTI+u1NUclQsa+3atfj5+bFy5UrGjx/PwYMHVUJQFDugpaP5VlaV0yNCiM+ACxhrITmknKuhtWjozuqwyFwjkIICWjBHlVbWzaVLl3j99ddZtWoVbdq0YfPmzbRp08bWYSmKkkVLUuiL8YpiBDAa4zyFfxf6CDtlbjU0gPo1qlKzciWu3LzF70dPEnbsFEEBqsyFJUkpWbNmDSNGjCA1NZVJkybx9ttvU65cOVuHpihKDlqSQi0p5TngDjABQAjxLGDdwesWYG41tG4tmuBZ8+/lIRKvpLL54HGVFCwoJSWFYcOG8dNPPxEYGMj27dvx9/e3dViKopihpY1koRCiVfaNrLUV3tMvJP3knZdQv0bVXAkBwLNmddxU94JFSCn5/vvv8fPz4+eff2bKlCns2bNHJQRFsWNarhR6AT8IIV7BWCm1H/CErlHpJHteQnZzUZUK95ndr7lHXWuG5ZSSk5MZMmQI//vf/+jQoQNLliyhRQt19aUo9k7L5LUzWVcH64BE4Akp5W3dI9NBZz9verVvTUW3wg+7Ye0HrBSR85FSsnTpUkaPHs3du3f5/PPPGTVqFGXLlrV1aIqiaFDgt6MQ4hjG9ROyPYBx4to+IQRSytZ6B2dpZy9cKjIhACReumqFaJxPYmIigwYNYvPmzXTp0oXFixfj4+OaEwEVxVEV9g35D6tFYSUX86yyVpBrNx3yQshmpJQsXLiQN998k8zMTGbNmsXw4cPVsF5FcUAFJoWsEUdOpV6eTmVzEq+k0vtBPytE4xwSEhIYMGAA27ZtIygoiEWLFtGkSRNbh6UoSgm51Klc0wZ18fHMXezudkYG22LiOHwumW0xcZS5rzyP+jezUYSOw2AwMGfOHPz9/dm3bx/z5s1j69atKiEoioPTtEazsxBC8EL3DsSfT+HilVTq1axOY/fa7IqJU7WPiiE+Pp7+/fsTFhbGE088wcKFC2nYsGHRD1QUxe65VFIAY2Lw9qiHt0c9wNge3qBGNcoaJPVqVEMIYeMI7VdmZiZffvkl7777LuXLl2fx4sWEhISo35miOBGXSwo5SSlZs30vpxMvmLb5eLrzQvcOdv9FJ6XMdcXTtEFdXWOOjY0lNDSUiIgInnnmGebPn0+DBg10ez1FUWzDpZNC/PmUXAkB4HTiBeLPp5iuJOyRNZNZZmYmX3zxBePHj6dixYp888039OnTx+6TpqIoJePSjee5hqhKSa1zCTQ+cIA/d+xCGgy2C6wIhSUzS4qJiaFTp0689dZbPPXUU0RHR9O3b1+VEBTFibn0lYJpiKqUtPllE3USEoy39+8j7shRvD+boPuqYyVR0HyLi1dSLXKFc+/ePaZOncqECROoUqUKK1as4MUXX1TJQFFcgP1941lR9hDVWn+c+zshZEkNj+D6nkjbBFaEguZbaJmHUZSoqCg6dOjAuHHjCA4OJiYmht69e6uEoCguwqWTQvYQ1U5VK5q9/1asfVYHNzffwsfTnaYNSl7ILz09nQ8//JB27dqRlJTEDz/8wOrVq6lTp05pw1UUxYG4dPMRGBOD58MPcnrFD/nuq9Tc2wYRFc3cfIvSjD46dOgQISEhREVF8fLLLzNz5kxq1apl4agVRXEEuiYFIcRTwEyMhfQWSSmn5Ln/DWAAkAFcAkJtUV6jWseHqP5oJ1LDI0zbqj/aiWodH7J2KJrlnW9REnfv3mXSpElMmTKFOnXqsH79ep577jkLRqnYyr1790hKSuLOnTu2DkWxsgoVKuDh4VHiVQ2FXgvWCyHKAqeAHkASEAm8JKWMybFPELBPSnlLCDEU6CalfLGw5w0MDJQHDhyweLzSYOD6nkhuxcZRqbk31To+ZJedzJayf/9+QkNDiY6O5tVXX2X69OnUqFHD1mEpFnL27FmqVKlCzZo1VX+QC5FScuXKFf766y8aN26c6z4hxEEpZWBRz6Hnt97DQJyU8oyUMh1YCQTn3EFK+buU8lbWzb2Ah47xFEqUKUP1R9pTP/QVqj/S3mkTwu3bt3nrrbfo2LEj169fZ9OmTSxdulQlBCdz584dlRBckBCCmjVrluoKUc/mowYYF+XJlgS0L2T//sAvOsbj8iIiIggNDSU2NpaBAwcydepUqlWrZuuwFJ2ohOCaSvt31/N02FxkZtuqhBB9gEBgagH3DxJCHBBCHLh06ZIFQ3QNt27dYvTo0XTu3Jnbt2/z66+/smDBApUQFN2tXbsWIQQnT54EYMeOHfzjH7mXannttdf44QfjQI9u3brRvHlzAgICeOihhzhy5Ihpv+vXr9OvXz+aNm1K06ZN6devH9evXzfdf+rUKXr27Im3tze+vr785z//ISVF24TOzZs307x5c7y9vZkyZYrZfc6dO8djjz1G69at6datG0lJScX6XTgKPZNCEuCZ47YHkJx3JyHE48A44Dkp5V1zTySlXCClDJRSBtauXVuXYJ1VWFgYrVu3ZsaMGQwZMoTjx4/To0cPW4eluIgVK1bQuXNnVq5cqfkx33//PUePHmXYsGH897//NW3v378/TZo0IT4+nvj4eBo3bsyAAQMAY3PZM888w9ChQ4mLi+PEiRMMHTqUS5cucfPmTdLT0wt8vczMTIYPH84vv/xCTEwMK1asICYmJt9+b775Jv369SMqKorx48fzzjvvFOM34Tj0TAqRgI8QorEQojzQG9iQcwchRFtgPsaE8KeOsbicGzduMGLECLp164aUku3btzN37lyqVKli69AUO2QwGAg7Fsu8jb8TdiwWgwXKvNy4cYPdu3ezePHiYiWFbB07duT8+fMAxMXFcfDgQd5//33T/ePHj+fAgQPEx8ezfPlyOnbsyLPPPmu6PygoCH9/f06dOkXz5s0ZM2YMJ06cyPc6+/fvx9vbmyZNmlC+fHl69+7N+vXr8+0XExPDY489Znpuc/s4A92SgpQyAxgBbAFOAKullNFCiIlCiOxxj1OB+4E1QogjQogNBTydUgzbtm2jVatWzJ07l1GjRhEVFUVQUJCtw1LslMFgYMTc5Qye9S0z129j8KxvGTF3eakTw7p163jqqado1qwZDzzwAIcOHSrW4zdv3szzzz8PGL+Q27RpQ9myZU33ly1bljZt2hAdHc3x48dp166d2edp27YtUVFR+Pr6MmDAADp37sySJUu4efMmAOfPn8fT8+9GDQ8PD1MyyikgIIAff/wRMDaL/fXXX1y5cqVYx+QIdB1iI6XcJKVsJqVsKqX8OGvbeCnlhqyfH5dS1pVStsn6pwbJl0JaWhqDBw/m8ccfp1y5coSHhzNjxgwqV65s69AUO7Yz+jTbj57MtW370ZPsjD5dquddsWIFvXv3BqB3796sWLGiwE7QnNtfeeUVPDw8+PTTT3n99dcB41BLc48taHteVapUYcCAAezevZsFCxawcOFC3N3dTc9RWDzZpk2bRlhYGG3btiUsLIwGDRrg5uZ883+d74hc1JYtWxg4cCDnz5/nzTffZOLEiVSsaL58h6LkFHMuX1cfACf+uEDXVs1L9JxXrlxh+/btHD9+HCEEmZmZCCHo168f165dy7Xv1atXc82g//777wkICGDs2LEMHz6cn376iZYtW3L48GEMBoNpZUSDwcDRo0fx9fXlzz//JCwsrNCYzp07x9KlS1mxYgUBAQF8+OGHgPHKIDHx74GSSUlJ1K9fP9/j69evz08//QQYm8Z+/PFHpxys4ZyD8V1IamoqoaGhPPXUU9x///1EREQwdepUlRAUzfwa5f8CBPBt6G52uxY//PAD/fr149y5cyQkJJCYmEjjxo25evUqycnJprb9c+fOcfToUdq0aZPr8eXKleOjjz5i7969nDhxAm9vb9q2bctHH31k2uejjz7iwQcfxNvbm5dffpmIiAh+/vln0/2bN2/m2LFjJCQk8PjjjxMcHEz16tXZvXs3q1at4oknngDgoYce4vTp05w9e5b09HRWrlxpdmb/5cuXTU1qn3zyCaGhoSX+/dgzlRQc2MaNG2nZsiXffPMN77zzDocOHaJ9+8KmgihKfl1a+tA9oEWubd0DWtClpU+Jn3PFihX885//zLXt3//+NytXruS7774jJCSENm3a0KtXLxYtWmT2jLtixYqMGTOGadOmAbB48WJOnTqFt7c3TZs25dSpUyxevNi078aNG/nyyy/x8fHBz8+PpUuXUqdOHcqWLcvkyZM5cuQIo0aNombNmrlex83NjdmzZ/Pkk0+ahrK2bNkSMHZmb9hg7OrcsWMHzZs3p1mzZqSkpDBu3LgS/37smW5lLvSiV5kLR3L16lVGjRrFd999R6tWrViyZEmBnWyKazpx4gS+vr6a9zcYDOyMPs2JPy7g29CdLi19TM00iuMx9/fXWuZC9Sk4mLVr1zJ06FCuXLnC+PHjGTduHOXLl7d1WIqDK1OmDF1bNS9xH4LiPFRScBCXLl1ixIgRrF69mjZt2rB58+Z87bCKoiil5fLXh9JgIHX3PpIXf0fq7n12tzazlJJVq1bh5+fH2rVrmTRpEvv371cJQVEUXbj0lYI0GIh764N86yjYy9rMFy9eZNiwYaxdu5bAwECWLFmCv7+/rcNSFMWJ2f6bz4au74nMlRDAPtZmllLy/fff07JlSzZt2sSUKVPYs2ePSgiKoujOpZPCzROnzG8/aX67NSQnJxMcHEyfPn1o1qwZR44c4e2333bKmZOKotgfl04KFDQc12D9YbpSSpYsWYKfnx9bt27liy++YNeuXbRo0aLoByuKHSqqHPUff/xBUFAQbdu2pXXr1mzatKlYz5+3LDfYb2nuoly9epUePXrg4+NDjx498s36zvb222/j7++Pv78/q1atsshr5+XaSaGgmilW7k9ITEykZ8+ehIaG0rp1a44ePcro0aNzFf9SFEeipRz1Rx99xH/+8x8OHz7MypUrGTZsWKHPmfeLsiRlucHypbm1xluYKVOm8Nhjj3H69Gkee+wxs0n0559/5tChQxw5coR9+/YxdepU0tLSinHk2rh0Uqjs28z89hYln8lZHFJKFixYQMuWLdm5cydffvklO3bswMfHOq+vKNmklMQlXWTX0ZPEJV00WySuOLSUoxZCmL7Url+/brbeUHbZ95dffpnAwL/nXZW2LDdYrjS3lniLsn79el599VUAXn31VdatW5dvn5iYGLp27YqbmxuVK1cmICCAzZs3F+uYtXDphuqq7dvhVrMGGVf+zuhuNWtQtb3+s4MTEhIYMGAA27ZtIygoiMWLF+dbaFtRrEFKyZrtezmdeMG0zcfTnRe6dyjx0o7mylHv27cv1z4ffvghTzzxBF9++SU3b95k69atpvuSk5NZunQp3377LX5+foSGhvLtt9+a7jdXlvvBBx8sVoyWKs2tJd4uXbrw119/5XvctGnTePzxx0lJSTFVbXV3d+fPP/MvLxMQEMCECRN44403uHXrFr///jt+fn7FOmYtXDoppO07mCshAGRcuUbavoNUf0SfGkIGg4F58+bx9ttvI4Rg3rx5DBo0SJUUUGwm/nxKroQAcDrxAvHnU/D2qFei59RSjnrFihW89tprjBkzhj179tC3b1+OHz/OgQMH6NSpEwMGDGDnzp25KqjmfOz//d//AX+X5X7wwQc1l+a+efMmmZmZpjUeSlOae//+/UXGu3PnzkKfQ4snnniCyMhIOnXqRO3atenYsaMuA1Bc+pvo1knz9eJvxcbp8npxcXF0796dESNG8MgjjxAdHc2QIUNUQlBs6uKV1GJt10JLOerFixfzn//8BzA25dy5c4fLly/TunVrFi9eTExMDMHBwSxcuDBX23l2We4BAwbg5eXF1KlTWbVqFVJKatasqak099mzZ3n55ZcZPnw4QK7S3NlyluZu2bIlBw8eNHusRcULxiuFNm3a5PuXfXVUt25dLlwwJuYLFy5Qp04ds681btw4jhw5wm+//YaUUpemZpf+NqpUQN9BpebeFn2dzMxMZsyYQevWrTly5AiLFy9m8+bNNGzY0KKvoyglUa9m9WJt10JLOeqGDRuybds2wFjA7c6dO9SuXZsKFSrw6quvEh4eztKlS4mPj6dt27b07dsXKLgs965du/Dx8bF6ae6i4gXjlcKRI0fy/Xv88ccBeO6551i2bBkAy5YtIzg4ON/vNDMz07TSW1RUFFFRUaby3xYlpXSof+3atZOWYsjMlKfGvCf3P9Td9O/UmPekITPTYq9x8uRJ2alTJwnIZ555RiYlJVnsuRWlIDExMZr3NRgMctXWCPnRkh9N/1ZtjZAGg6FUMfz888/Sx8dHNmnSRH700UdSSinff/99uX79eimllNHR0bJTp06ydevWMiAgQG7ZsqXA57p3755ct26dlFLKrl27yl9++SXX/TNnzpRDhgyRUkq5a9cu2b59exkQECADAwPlr7/+atqva9euMjIy0nR72rRpMjQ0VEop5dWrV+Urr7wimzZtKps0aSJfeeUVee3aNdO+J06ckE8++aT09vaWvr6+8sUXX5QXL14sMl4tLl++LLt37y69vb1l9+7d5ZUrV6SUUkZGRsr+/ftLKaW8ffu29PX1lb6+vrJ9+/by8OHDBT6fub8/cEBq+I51+dLZ0mDg+p5IbsXGUam5N9U6PmSREheZmZl88cUXjB8/nooVKzJz5kz69OlT4o47RSmO4pbOllISfz6Fi1dSqVezOk0b1FXvVQemSmeXgihThuqPtLdox3JMTAwhISHs37+f559/nrlz55pGFiiKPaHAHkUAAAz1SURBVBJC4O1Rr8Qdy4rzcOk+BUu7d+8ekydPpm3btsTHx7NixQp++uknlRAURXEYLn+lYClRUVGEhIRw6NAhXnjhBWbPnl3gCAJFURR7pa4USik9PZ0PP/yQdu3akZSUxA8//MDq1atVQlAUxSGpK4VSOHToECEhIURFRfHyyy8zc+ZMsxNXFEVRHIW6UiiBu3fvMm7cOB5++GEuXbrE+vXr+f7771VCUBTF4amkUEz79+/nwQcfZPLkyfTp04fo6Oh8k3IURTHKW97a2Utbv/XWW7Rs2RJfX19GjhxZ6sKCtqCSgka3b9/mrbfeomPHjqSlpbFp0yaWLl1KjRo1bB2aotitkpS3dtTS1hEREezevZuoqCiOHz9OZGQkYWFhml/DXqikoEFERARt27Zl6tSp9O/fn+PHj/P000/bOixFsRhpMJC6ex/Ji78jdfc+ZI4aQCVV2vLWjlbaWgjBnTt3SE9P5+7du9y7d4+6desW65jtgepoLsStW7cYN24cM2fOpGHDhvz222+mWiWK4iykwUDcWx/kWq+8+qOd8P5sQqlm95srb10cjlbaumPHjgQFBeHu7o6UkhEjRhRrVrm9UEmhAGFhYfTv35/4+HiGDRvGlClTqFKliq3DUhSLu74nMldCAEgNj+D6nshSzfQ3V946b39CNmcobZ3dbJWUlARAjx49CA8P59FHHy31c1uTrs1HQoinhBCxQog4IcRYM/ffJ4RYlXX/PiGEl57xaHHjxg1GjBhBt27dkFLy+++/M2fOHJUQFKelRwn5gspbP/DAA05b2nrt2rV06NCB+++/n/vvv5+nn36avXv3luC3Z1u6JQUhRFlgDvA04Ae8JITIu0xQf+CalNIbmA58qlc8Wmzbto1WrVoxd+5cRo0aRVRUFN26dbNlSIqiOz1KyBdU3vrq1atOW9q6YcOGhIWFkZGRwb179wgLC3PI5iPdSlwDHYEtOW6/A7yTZ58tQMesn92Ay2Cs3FrQP0uWzs52/fp1OWjQIAlIHx8fuWvXLou/hqJYU7FKZ+tQQr6w8tbOWto6IyNDDho0SLZo0UL6+vrK0aNHa35+S7PL0tlCiF7AU1LKAVm3+wLtpZQjcuxzPGufpKzb8Vn7XC7oeS1dOvvo0aM8++yznD9/njfeeIOJEydSsWJFiz2/othCsUtn61RCXrENey2dba73J28G0rIPQohBwCDA4quVeXl54evry5o1a2jfXp91mRXF3ulRQl5xTHomhSTAM8dtDyC5gH2ShBBuQDXgat4nklIuABaA8UrBkkFWq1aNLVu2WPIpFUVRHJae14eRgI8QorEQojzQG9iQZ58NwKtZP/cCtku92rMURVGUIul2pSClzBBCjMDYmVwW+FpKGS2EmIixw2MDsBj4VggRh/EKobde8SiKq5EaxvArzqe059W6Tl6TUm4CNuXZNj7Hz3eAF/SMQVFcUYUKFbhy5Qo1a9ZUicGFSCm5cuUKFSpUKPFzqBnNiuKEPDw8SEpKKrQgnOKcKlSogIeHR4kfr5KCojihcuXK0bhxY1uHoTggNRBZURRFMVFJQVEURTFRSUFRFEUx0a3MhV6EEJeAczo8dS2MtZdcgTpW56SO1TlZ6lgbSSlrF7WTwyUFvQghDmipC+IM1LE6J3Wszsnax6qajxRFURQTlRQURVEUE5UU/rbA1gFYkTpW56SO1TlZ9VhVn4KiKIpioq4UFEVRFBOXSgpCiKeEELFCiDghxFgz998nhFiVdf8+IYSX9aO0DA3H+oYQIkYIESWE2CaEaGSLOC2hqGPNsV8vIYQUQjj0qBUtxyuE+E/W3zdaCLHc2jFaiob3cUMhxO9CiMNZ7+WetoiztIQQXwsh/sxajdLc/UIIMSvr9xAlhHhQt2C0rNnpDP8wlu+OB5oA5YGjgF+efYYBX2X93BtYZeu4dTzWIKBS1s9DnflYs/arAoQDe4FAW8et89/WBzgM1Mi6XcfWcet4rAuAoVk/+wEJto67hMf6KPAgcLyA+3sCv2BcrbIDsE+vWFzpSuFhIE5KeUZKmQ6sBILz7BMMLMv6+QfgMeGYdYeLPFYp5e9SyltZN/diXBnPEWn5uwJMAj4D7lgzOB1oOd6BwBwp5TUAKeWfVo7RUrQcqwSqZv1cjfyrOzoEKWU4ZladzCEY+EYa7QWqCyHc9YjFlZJCAyAxx+2krG1m95FSZgDXgZpWic6ytBxrTv0xnoU4oiKPVQjRFvCUUm60ZmA60fK3bQY0E0LsFkLsFUI8ZbXoLEvLsX4I9BFCJGFcu+V164RmdcX9TJeYK5XONnfGn3folZZ9HIHm4xBC9AECga66RqSfQo9VCFEGmA68Zq2AdKblb+uGsQmpG8YrwJ1CCH8pZarOsVmalmN9CVgqpfxcCNER40qO/lJKg/7hWZXVvptc6UohCfDMcduD/Jeapn2EEG4YL0cLu6SzV1qOFSHE48A44Dkp5V0rxWZpRR1rFcAf2CGESMDYHrvBgTubtb6P10sp70kpzwKxGJOEo9FyrP2B1QBSyj1ABYy1gpyNps+0JbhSUogEfIQQjYUQ5TF2JG/Is88G4NWsn3sB22VWL4+DKfJYs5pU5mNMCI7a5gxFHKuU8rqUspaU0ktK6YWx/+Q5KeUB24Rbalrex+swDiRACFELY3PSGatGaRlajvUP4DEAIYQvxqTgjMvNbQD6ZY1C6gBcl1Je0OOFXKb5SEqZIYQYAWzBOKrhaylltBBiInBASrkBWIzx8jMO4xVCb9tFXHIaj3UqcD+wJqsv/Q8p5XM2C7qENB6r09B4vFuAJ4QQMUAm8F8p5RXbRV0yGo91DLBQCDEaY3PKa454IieEWIGxua9WVv/IB0A5ACnlVxj7S3oCccAtIES3WBzw96coiqLoxJWajxRFUZQiqKSgKIqimKikoCiKopiopKAoiqKYqKSgKIqimKikoNgFIYRXQRUirU0I8bwQYnwpHp+QNT9A6/6vCSFml/T1SiM7ViFEeSFEeNakTcWFqaSgKPm9Bcy1dRDW9P/tnV2IVVUUx3//gVGopq9hkCiiTCwyUYsZKnswKSIwCq0kJa2HqAexggp6CA0qehGzJijqQagQwRm18CFzSEdBnKEptbBe7INEUiGquYTWtHrY61y2d84Ml+Zh7tzWDw53n3XW/rqXe9be+5773y441wcsn+y2BJNLBIWg4ZA00/XxO30GsV/SkB+3u88iH9lu930D3nGdIyQNS9rg/n2SOtz+hKRBSYcl9Ui6oKTu2cBZMzvj55slPZhdH87q3ytpm6RvJX1Uo6j7vKQBP2Z5nvuU9un4UtIeSTNK6i/1kbReSXN/r6TjktZmeVa5xv5hSR+4rcP7OOjHQre3S9rt5b/L+Zo6O4CV/+EjC5qICApBQyHpeqAHeNzMBoFTwN1mdjNpFPtm5t5F+kfrXOA6YKnbLwSGPM8+0r9DAXrNrNPM5gHHSLo5tSwEhups7gLgGZKO/0zPW/C7mXUB3cAbbjsA3GpmC0gy0C+UlDmezw3APd7vdZJaJc0h6Vct9n497b6bgI1m1gksA953+zrggJf/MXB1Vv7XQGedfQ+alFg/DBqJDmAnsMzMvnFbK9AtaT5JsmF25j9gZsehKhNwB2kfjH+Are7zIdDr6ZskvQJcSpL4+LSkDVdQv3bOgJn97PV/BVxDuqkDbMleN3r6KmCrkg7+NOD7kjLH89nlwoVnJZ0CZgCLgW3FzMbMCgHHu4Abs8nLxZLaSJu5LHXfXZJ+LRzMbETSOUltZvZHne9B0GTETCFoJH4jacbnI+5ngV+AeSSJ72nZtVqNlrE0Wwr7ZmCNmc0FXiaJp9XyZ439b/x74stDef25suwI5w+yrCT9FtDt9T85Rv3j+ZTVJ8r73QLcZmbz/bgyu9GPp20znam/EVEwASIoBI3EOeABkhrkCrddApx0ffxHScJoBV2uoNlCWloqRuktJJVbgBWZvQ04KamVsdfOjwGzsvMfgFs8fT8uUlYHy7PXg1lfTnh69agc9fvk9AEPS2oHkHS523cDawonn2lB2pJ0pdvuBS7LfNqB02b2Vx31Bk1KLB8FDYWZVSQtAT6TVCE9BdQj6SHgc6CSuR8EXif9ptAPbHd7BZgj6QvS7KO4Qb8EHAJ+BI6SgkQt/cAGSXK1zfeAnZIGSDfgSkmeMqZLOkQKUI+4bT1JlfYEScL72pJ89fhUcdXQV4F9kkZIezM/BqwF3pZ0hPQ97weeIs2QtkgaIv3e8lNW3J0kNc7gf0yopAZTEkmLgOfMbEnJtWEzu2gCZW8CPjGzPRNo4pRDUi/wopl9N9ltCSaPWD4KgtG8Box6XLWZUdrEZkcEhCBmCkEQBEGVmCkEQRAEVSIoBEEQBFUiKARBEARVIigEQRAEVSIoBEEQBFUiKARBEARV/gVbcHnoMs8zDAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "allInitVs = [x[0] for x in drugmatrix_archive.values()] + \\\n",
    "    [x[0] for x in pubchem_archive.values()] + \\\n",
    "    [x[0] for x in ds1_archive.values()] + \\\n",
    "    [x[0] for x in serotonin_archive.values()]\n",
    "allFinalVs = [x[1] for x in drugmatrix_archive.values()] + \\\n",
    "    [x[1] for x in pubchem_archive.values()] + \\\n",
    "    [x[1] for x in ds1_archive.values()] + \\\n",
    "    [x[1] for x in serotonin_archive.values()]\n",
    "  \n",
    "initVs = [x[1] for x in allInitVs if x[-1]>0.9]\n",
    "finalVs = [y[1] for x,y in zip(allInitVs,allFinalVs) if x[-1]>0.9]\n",
    "plt.scatter(initVs,finalVs,c='#27647bff',linewidth=0,label='AUROC>0.9');\n",
    "initVs = [x[1] for x in allInitVs if x[-1]<=0.9 and x[-1]>0.8]\n",
    "finalVs = [y[1] for x,y in zip(allInitVs,allFinalVs) if x[-1]<=0.9 and x[-1]>0.8]\n",
    "plt.scatter(initVs,finalVs,c='#849fadff',linewidth=0,label='0.8<AUROC<=0.9');\n",
    "initVs = [x[1] for x in allInitVs if x[-1]<=0.8]\n",
    "finalVs = [y[1] for x,y in zip(allInitVs,allFinalVs) if x[-1]<=0.8 ]\n",
    "plt.scatter(initVs,finalVs,c='#ca3542ff',linewidth=0,label='AUROC<=0.8');\n",
    "# # initVs = [x[0][1] for x in drugmatrix_archive.values() if x[0][-1]<=0.8]\n",
    "# # finalVs = [x[1][1] for x in drugmatrix_archive.values() if x[0][-1]<=0.8]\n",
    "# # plt.scatter(initVs,finalVs,linewidth=0);\n",
    "plt.plot([0,1],[0,1],'k-')\n",
    "plt.xlabel('kappa (unbalanced)')\n",
    "plt.legend()\n",
    "plt.ylabel('kappa (balanced)');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}