resampling_methods_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Permutation Tests\n",
    "Also referred to as randomization, permutation tests are a type of resampling wherein the distribution of the test statistic under the null hypothesis, $H_0$, is obtained by computing the test statistic for all possible permutations of data points and labels. Of course, in practice the distribution is estimated by sampling random permutations for scalability. Why do we need to know the distribution of the test statistic? \n",
    "\n",
    "To answer this question, we must first define the meaning of a p-value. A popular misconception about the p-value is that it is a probability for how likely it is that either $H_0$ or $H_A$ is true; however, it actually describes how likely it is to observe a test statistic _at least_ as extreme as the observed test statistic on a given distribution. When the p-value is very small, we reject $H_0$ and deem the observed data to be *significant*. Typically, $p<0.05$ is considered to be small.\n",
    "\n",
    "\n",
    "As an example, we might want to test whether the data for groups A and B come from the same underlying distribution or not. For each group, the observed means are $\\mu_A$ and $\\mu_B$, respectively. We can define the null hypothesis that the data come from the same distribution as: \n",
    "\n",
    "$$H_0:\\mu_A-\\mu_B=0$$\n",
    "\n",
    "and the alternative hypothesis that the data do not come from thes ame distribution as: \n",
    "\n",
    "$$H_A:\\mu_A-\\mu_B\\neq0$$\n",
    "\n",
    "The test statistic for the observed data can be described as $T_{obs}=\\mu_A-\\mu_B$. However, because we have no idea what the underlying distribution for this test statistic looks like, we have no way of computing the p-value for this observation! We also have no more observations for this data, so how can we estimate the sampling distribution? In the absence of additional observations, we will randomly relabel the observed data points (i.e. membership in group A vs. B), recompute new test statistics $T_i$ for each permutation $i\\in(1,...,n)$ for $n$ permutations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Efficacy of a new drug treatment\n",
    "Let's say a new drug has been developed to treat disease X and is in clinical trials. We want to determine the efficacy of this drug based on patient survival outcomes. We have the survival times, measured in days after diagnosis, for a group of patients that received the drug and another group of patients that did not receive the drug. Here, we will test the null hypothesis is that the drug treatment does not prolong survival time from the point of diagnoses, or:\n",
    "$$H_0:\\mu_{treated}-\\mu_{untreated}=0$$\n",
    "The alternative hypothesis states that the drug treatment results in a statistically significant increase in the post-diagnosis survival time:\n",
    "$$H_A:\\mu_{treated}-\\mu_{untreated}\\geq0$$\n",
    "\n",
    "We will start by loading the post-diagnosis survival times, in days, for patients in the untreated and treated groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "untreated = np.asarray([int(x) for x in open('./untreated_survival_times.txt').read().split(',')])\n",
    "treated = np.asarray([int(x) for x in open('./treated_survival_times.txt').read().split(',')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many patients are in each group?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50 patients in untreated group\n",
      "44 patients in treated group\n"
     ]
    }
   ],
   "source": [
    "print('%d patients in untreated group' % untreated.shape[0])\n",
    "print('%d patients in treated group' % treated.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the distribution of survival times for each of these groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYVOWZ/vHvLRJBRFRAB0UEE5e4pcUWYTBGoqJoYjaTiFk0E8WYZDQzEydiZhAd8wsOjvEyJipRohM3HCNuUQM6GNTEBUgLKCouOLYYQVRAEbTx+f1x3sairO4+NF0L9v25rrrqnPe8dc5TRdFPne15FRGYmZm1ZbNqB2BmZpsGJwwzM8vFCcPMzHJxwjAzs1ycMMzMLBcnDDMzy8UJw8zMcnHCMDOzXJwwzMwsl82rHUBH6tOnTwwcOLDaYZiZbTJmz579WkT0zdP3I5UwBg4cyKxZs6odhpnZJkPSi3n7+pCUmZnl4oRhZma5OGGYmVkuH6lzGGb20fHee+/R2NjI6tWrqx3KR0K3bt3o378/Xbt2bfc6nDDMrCY1NjbSs2dPBg4ciKRqh7NJiwiWLVtGY2MjgwYNavd6ynZIStLOkmZIWiDpCUlnpPbtJE2XtDA9b9vC609MfRZKOrFccZpZbVq9ejW9e/d2sugAkujdu/dG762V8xxGE/AvEfFJYCjwA0l7AWcB90XEbsB9aX49krYDzgEOAoYA57SUWMzso8vJouN0xGdZtoQREa9ExJw0vRJYAOwEfAG4JnW7BvhiiZcfCUyPiNcj4g1gOnBUuWI1M7O2VeQchqSBwP7AI8AOEfEKZElF0vYlXrIT8FLBfGNqM7NOauBZf+jQ9S2acEyHrg+goaGBxYsXc/TRR3fI+t58802uv/56vv/972/Q68aPH89WW23Fj3/84w6Jo1nZE4akrYDfAz+KiBU5d4tKdYoW1j8GGAMwYMCA9oa5wV/GcnzZzGzT1tDQwKxZs0omjKamJjbffMP+5L755pv8+te/3uCEUS5lvQ9DUleyZHFdRNySml+V1C8t7wcsKfHSRmDngvn+wOJS24iISRFRHxH1ffvmKodiZpbLokWL2GeffdbNX3jhhYwfP55DDz2Un/zkJwwZMoTdd9+dBx54gHfffZdx48YxZcoU6urqmDJlCuPHj2fMmDGMHDmSb3/726xdu5YzzzyTAw88kP32248rrrgCgLfeeovDDjuMwYMHs++++3LbbbcBcNZZZ/Hcc89RV1fHmWeeCcDEiRPXvf6cc85ZF9vPfvYz9thjDw4//HCefvrpsnweZdvDULYrcRWwICIuKlh0O3AiMCE931bi5X8E/l/Bie6RwNhyxWpmtqGampp49NFHueuuuzj33HO59957Oe+885g1axaXXnopkB0amj17Ng8++CDdu3dn0qRJ9OrVi8cee4w1a9YwfPhwRo4cyc4778zUqVPZeuutee211xg6dCjHHnssEyZMYP78+TQ0NAAwbdo0Fi5cyKOPPkpEcOyxxzJz5kx69OjBjTfeyF//+leampoYPHgwBxxwQIe/53IekhoOfAuYJ6khtZ1NlihukvRd4P+ArwJIqge+FxEnR8Trkv4DeCy97ryIeL2MsZqZbZAvf/nLABxwwAEsWrSoxX7HHnss3bt3B7I/+HPnzuXmm28GYPny5SxcuJD+/ftz9tlnM3PmTDbbbDNefvllXn311Q+ta9q0aUybNo39998fyPZMFi5cyMqVK/nSl77ElltuuW6b5VC2hBERD1L6XATAYSX6zwJOLpifDEwuT3RmZm3bfPPNef/999fNF97HsMUWWwDQpUsXmpqaWlxHjx491k1HBL/85S858sgj1+tz9dVXs3TpUmbPnk3Xrl0ZOHBgyXsmIoKxY8dy6qmnrtd+8cUXV+QSZNeSMjNrwQ477MCSJUtYtmwZa9as4c4772y1f8+ePVm5cmWLy4888kguu+wy3nvvPQCeeeYZ3n77bZYvX872229P165dmTFjBi+++GLJ9R155JFMnjyZt956C4CXX36ZJUuWcMghhzB16lTeeecdVq5cyR133LGxb70klwYxs01CNa5M7Nq1K+PGjeOggw5i0KBB7Lnnnq32HzFiBBMmTKCuro6xYz982vXkk09m0aJFDB48mIigb9++3HrrrXzjG9/g85//PPX19dTV1a3bTu/evRk+fDj77LMPo0aNYuLEiSxYsIBhw4YBsNVWW3HttdcyePBgvv71r1NXV8cuu+zCpz/96Y7/MABFlLxadZNUX18f7R1AyZfVmtWWBQsW8MlPfrLaYXyklPpMJc2OiPo8r/chKTMzy8UJw8zMcnHCMDOzXJwwzMwsFycMMzPLxQnDzMxy8X0YZrZpGN+rg9e3vNXF7S0t3pKLL76YMWPGrCvfkcf999/PhRde2OYNg5XiPQwzsxKaS4sXW7t2bbvWd/HFF7Nq1aqNDauqnDDMzEooLC1+4IEHMmLECE444QT23XdfAK699lqGDBlCXV0dp5566rpEctppp1FfX8/ee++9rvz4JZdcwuLFixkxYgQjRowAskKCw4YNY/DgwXz1q19dV+7jnnvuYc899+Tggw/mlltuKRFZ9ThhmJmVMGHCBD7+8Y/T0NDAxIkTefTRR/nZz37Gk08+yYIFC5gyZQoPPfQQDQ0NdOnSheuuuw7IxqWYNWsWc+fO5U9/+hNz587l9NNPZ8cdd2TGjBnMmDGD1157jfPPP597772XOXPmUF9fz0UXXcTq1as55ZRTuOOOO3jggQf429/+VuVPYX0+h2FmlsOQIUMYNGgQAPfddx+zZ8/mwAMPBOCdd95h++2z0aZvuukmJk2aRFNTE6+88gpPPvkk++2333rrevjhh3nyyScZPnw4AO+++y7Dhg3jqaeeYtCgQey2224AfPOb32TSpEmVeottcsIwM8uhuEz5iSeeyM9//vP1+rzwwgtceOGFPPbYY2y77bacdNJJLZYpP+KII7jhhhvWa29oaKhImfL28iEpM7MSWitVfthhh3HzzTezZEk2wvTrr7/Oiy++yIoVK+jRowe9evXi1Vdf5e677y65vqFDh/LQQw/x7LPPArBq1SqeeeYZ9txzT1544QWee+45gA8llGrzHoaZbRrauAy2oxWWFu/evTs77LDDumV77bUX559/PiNHjuT999+na9eu/OpXv2Lo0KHsv//+7L333uy6667rDjkBjBkzhlGjRtGvXz9mzJjB1VdfzejRo1mzZg0A559/PrvvvjuTJk3imGOOoU+fPhx88MHMnz+/ou+7NWUrby5pMvA5YElE7JPapgB7pC7bAG9GRF2J1y4CVgJrgaa8pXdd3tzso8PlzTvexpY3L+cextXApcB/NzdExNebpyX9F9DaT4YREfFa2aIzM7MNUs4xvWdKGlhqmbKzOl8DPluu7ZuZWceq1knvTwOvRsTCFpYHME3SbEljKhiXmdWQj9KIoNXWEZ9ltRLGaKC10//DI2IwMAr4gaRDWuooaYykWZJmLV26tKPjNLMq6datG8uWLXPS6AARwbJly+jWrdtGrafiV0lJ2hz4MnBAS30iYnF6XiJpKjAEmNlC30nAJMhOend4wGZWFf3796exsRH/EOwY3bp1o3///hu1jmpcVns48FRENJZaKKkHsFlErEzTI4HzKhmgmVVf165d191ZbbWhbIekJN0A/AXYQ1KjpO+mRcdTdDhK0o6S7kqzOwAPSnoceBT4Q0TcU644zcwsn3JeJTW6hfaTSrQtBo5O088DnypXXGZm1j4uDWJmZrk4YZiZWS5OGGZmlosThpmZ5eKEYWZmuThhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVkuHtO7Bnh4WDPbFHgPw8zMcnHCMDOzXJwwzMwsFycMMzPLxQnDzMxyccIwM7NcyjlE62RJSyTNL2gbL+llSQ3pcXQLrz1K0tOSnpV0VrliNDOz/Mq5h3E1cFSJ9l9ERF163FW8UFIX4FfAKGAvYLSkvcoYp5mZ5VC2hBERM4HX2/HSIcCzEfF8RLwL3Ah8oUODMzOzDVaNcxg/lDQ3HbLatsTynYCXCuYbU1tJksZImiVp1tKlSzs6VjMzS9pMGJL+U9LWkrpKuk/Sa5K+2c7tXQZ8HKgDXgH+q9QmS7RFSyuMiEkRUR8R9X379m1nWGZm1pY8exgjI2IF8DmyX/u7A2e2Z2MR8WpErI2I94HfkB1+KtYI7Fww3x9Y3J7tmZlZx8mTMLqm56OBGyKiPeclAJDUr2D2S8D8Et0eA3aTNEjSx4Djgdvbu00zM+sYearV3iHpKeAd4PuS+gKr23qRpBuAQ4E+khqBc4BDJdWRHWJaBJya+u4IXBkRR0dEk6QfAn8EugCTI+KJDX5nZmbWofIkjHOAC4AVEbFW0irg2LZeFBGjSzRf1ULfxWR7MM3zdwEfuuTWzMyqJ88hqb9ExBsRsRYgIt4G7i5vWGZmVmta3MOQ9Hdkl7N2l7Q/H1y9tDWwZQViMzOzGtLaIakjgZPIrlK6qKB9JXB2GWMyM7Ma1GLCiIhrgGskfSUifl/BmMzMrAblOel9p6QTgIGF/SPivHIFZWZmtSdPwrgNWA7MBtaUNxwzM6tVeRJG/4goVXXWzMw6kTyX1f5Z0r5lj8TMzGpanj2Mg4GTJL1AdkhKQETEfmWNzMzMakqehDGq7FGYmVnNa/OQVES8SFY99rNpelWe15mZ2UdLnvEwzgF+AoxNTV2Ba8sZlJmZ1Z48ewpfIis2+DasKxTYs5xBmZlZ7cmTMN6NiCCNeiepR3lDMjOzWpQnYdwk6QpgG0mnAPeSjZZnZmadSJtXSUXEhZKOAFYAewDjImJ62SOrdeN7teM1yzs+DjOzCslzWS0pQThJmJl1Yi0ekpL0YHpeKWlFwWOlpBVtrVjSZElLJM0vaJso6SlJcyVNlbRNC69dJGmepAZJs9rzxszMrGO1mDAi4uD03DMiti549IyIrXOs+2qguAbVdGCfdJf4M3xwqW4pIyKiLiLqc2zLzMzKLM99GL/L01YsImYCrxe1TYuIpjT7MNngTGZmtgnIc5XU3oUzkjYHDuiAbf8DLY8NHsA0SbMljWltJZLGSJoladbSpUs7ICwzMyultXMYYyWtBPYrPH8BvEo2Rka7Sfop0ARc10KX4RExmKyO1Q8kHdLSuiJiUkTUR0R93759NyYsMzNrRWvnMH4eET2BiUXnL3pHRGvnHlol6UTgc8A30g2Bpba9OD0vAaYCQ9q7PTMz6xh57sMYK2lbYDegW0H7zA3dmKSjyOpSfSYiVrXQpwewWUSsTNMjAQ8Ha2ZWZW0mDEknA2eQnaBuAIYCfwE+28brbgAOBfpIagTOIbsqagtguiSAhyPie5J2BK6MiKOBHYCpafnmwPURcU+73p2ZmXWYPDfunQEcSPbHfYSkPYFz23pRRIwu0XxVC30XA0en6eeBT+WIy8zMKijPVVKrI2I1gKQtIuIpshIhZmbWieTZw2hMd2TfSnYo6Q1gcXnDMjOzWpPnpPeX0uR4STOAXoDPKZiZdTK5ig9KGgwcTHZD3UMR8W5ZozIzs5qTpzTIOOAaoDfQB/itpH8rd2BmZlZb8uxhjAb2LzjxPQGYA5xfzsDMzKy25LlKahEFN+yR3UfxXFmiMTOzmpVnD2MN8ISk6WTnMI4AHpR0CUBEnF7G+MzMrEbkSRhT06PZ/eUJxczMalmey2qvqUQgZmZW2/KcwzAzM3PCMDOzfFobQOl36fmMyoVjZma1qrU9jAMk7QL8g6RtJW1X+KhUgGZmVhtaO+l9OVnNqF2B2YAKlkVqNzOzTqK1IVoviYhPApMjYteIGFTwcLIwM+tk8lxWe5qkTwGfTk0zI2JuecMyM7Nak6f44OnAdcD26XGdpH/Ms3JJkyUtkTS/oG07SdMlLUzP27bw2hNTn4WSTsz3dszMrFzyXFZ7MnBQRIyLiHFkY3qfknP9VwNHFbWdBdwXEbsB96X59aST6ucABwFDgHNaSixmZlYZeRKGgLUF82tZ/wR4iyJiJvB6UfMXyMqlk56/WOKlRwLTI+L1iHgDmM6HE4+ZmVVQnlpSvwUekdRcT+qLwFUbsc0dIuIVgIh4RdL2JfrsBLxUMN+Y2j5E0hhgDMCAAQM2IiwzM2tNm3sYEXER8B2yPYU3gO9ExMVljqvUHkyU6hgRkyKiPiLq+/btW+awzMw6r1xDtEbEHLJBkzrCq5L6pb2LfsCSEn0agUML5vvjKrlmZlVVjVpStwPNVz2dCNxWos8fgZHpDvNtgZGpzczMqqSsCUPSDcBfgD0kNUr6LjABOELSQrLBmCakvvWSrgSIiNeB/wAeS4/zUpuZmVVJq4ekJHUB/hgRh7dn5RExuoVFh5XoO4vsEt7m+cnA5PZs18zMOl6rexgRsRZYJalXheIxM7Maleek92pgXhrT++3mRo/lbWbWueRJGH9IDzMz68RyjektqTswICKerkBMZmZWg/IUH/w80EA2NgaS6iTdXu7AzMystuQ5JDWerADg/QAR0SBpUBljsraMb8c1COOXd3wcZtap5LkPoykiiv/alCzTYWZmH1159jDmSzoB6CJpN+B04M/lDcvMzGpNnj2MfwT2BtYANwArgB+VMygzM6s9ea6SWgX8VNIF2WysLH9YZmZWa/JcJXWgpHnAXLIb+B6XdED5QzMzs1qS5xzGVcD3I+IBAEkHkw2qtF85AzMzs9qS5xzGyuZkARARDwI+LGVm1sm0uIchaXCafFTSFWQnvAP4Oh7MyMys02ntkNR/Fc2fUzDt+zDMzDqZFhNGRIyoZCBmZlbb2jzpLWkb4NvAwML+7S1vLmkPYEpB067AuIi4uKDPoWRDt76Qmm6JiPPasz0zM+sYea6Sugt4GJgHvL+xG0wVb+tg3Yh+LwNTS3R9ICI+t7HbMzOzjpEnYXSLiH8u0/YPA56LiBfLtH4zM+sgeS6r/Z2kUyT1k7Rd86ODtn882dVXpQxLNwneLWnvDtqemZm1U549jHeBicBP+eDqqCA799Bukj4GHAuMLbF4DrBLRLwl6WjgVmC3FtYzBhgDMGDAgI0JyczMWpFnD+OfgU9ExMCIGJQeG5UsklHAnIh4tXhBRKyIiLfS9F1AV0l9Sq0kIiZFRH1E1Pft27cDwjIzs1LyJIwngFVl2PZoWjgcJenvJClNDyGLc1kZYjAzs5zyHJJaCzRImkFW4hxo/2W1AJK2BI4ATi1o+15a7+XAccBpkpqAd4DjI8I3C5qZVVGehHFrenSYVDK9d1Hb5QXTlwKXduQ2zcxs4+QZD+OaSgRiZma1Lc+d3i9QonZUB534NjOzTUSeQ1L1BdPdgK8CHXUfhpmZbSLavEoqIpYVPF5ONZ8+W4HYzMyshuQ5JDW4YHYzsj2OnmWLyMzMalKeQ1KF42I0AYuAr5UlGjMzq1l5rpLyuBhmZpbrkNQWwFf48HgYHp/CzKwTyXNI6jZgOTCbgju9zcysc8mTMPpHxFFlj8TMzGpanuKDf5a0b9kjMTOzmpZnD+Ng4KR0x/caQEBExH5ljcxq0/he7XjN8o6Pw8wqLk/CGFX2KMzMrObluazW422bmVmucxhmZmZOGGZmlo8ThpmZ5VK1hCFpkaR5khokzSqxXJIukfSspLlFRRDNzKzC8lwlVU4jIuK1FpaNAnZLj4OAy9KzmZlVQS0fkvoC8N+ReRjYRlK/agdlZtZZVTNhBDBN0mxJY0os3wl4qWC+MbWZmVkVVPOQ1PCIWCxpe2C6pKciYmbBcpV4zYfGFk/JZgzAgAEDyhPpR9jAs/6wQf0XdStTIGZW86q2hxERi9PzEmAqMKSoSyOwc8F8f2BxifVMioj6iKjv27dvucI1M+v0qpIwJPWQ1LN5GhgJzC/qdjvw7XS11FBgeUS8UuFQzcwsqdYhqR2AqZKaY7g+Iu6R9D2AiLgcuAs4GngWWAV8p0qxmpkZVUoYEfE88KkS7ZcXTAfwg0rGZWZmLavly2rNzKyGOGGYmVkuThhmZpaLE4aZmeXihGFmZrlUu/igdWIbfpf5CRu+kY4aT9xjmZt5D8PMzPJxwjAzs1ycMMzMLBcnDDMzy8UJw8zMcnHCMDOzXJwwzMwsFycMMzPLxQnDzMxyccIwM7NcXBrEbFOwoaVJPiplSTrr+65RFd/DkLSzpBmSFkh6QtIZJfocKmm5pIb0GFfpOM3MbH3V2MNoAv4lIuZI6gnMljQ9Ip4s6vdARHyuCvGZmVkJFd/DiIhXImJOml4JLAB2qnQcZma2Yap60lvSQGB/4JESi4dJelzS3ZL2rmhgZmb2IVU76S1pK+D3wI8iYkXR4jnALhHxlqSjgVuB3VpYzxhgDMCAAQPKGLGZWedWlT0MSV3JksV1EXFL8fKIWBERb6Xpu4CukvqUWldETIqI+oio79u3b1njNjPrzKpxlZSAq4AFEXFRC33+LvVD0hCyOJdVLkozMytWjUNSw4FvAfMkNaS2s4EBABFxOXAccJqkJuAd4PiIiCrEamZmScUTRkQ8CKiNPpcCl1YmIjMzy8N3ept1IgPP+sMGv2bRhGOqt+1uHbJp6yCuJWVmZrk4YZiZWS5OGGZmlosThpmZ5eKEYWZmuThhmJlZLk4YZmaWixOGmZnl4oRhZma5+E5vM2udx9XeKO27w/2EDXtBhT5z72GYmVkuThhmZpaLE4aZmeXihGFmZrk4YZiZWS5OGGZmlktVEoakoyQ9LelZSWeVWL6FpClp+SOSBlY+SjMzK1TxhCGpC/ArYBSwFzBa0l5F3b4LvBERnwB+AVxQ2SjNzKxYNfYwhgDPRsTzEfEucCPwhaI+XwCuSdM3A4dJanUccDMzK69qJIydgJcK5htTW8k+EdEELAd6VyQ6MzMrSRFR2Q1KXwWOjIiT0/y3gCER8Y8FfZ5IfRrT/HOpz7IS6xsDjEmzewBP5wylD/Bau99IZW1KscKmFa9jLZ9NKd7OHOsuEdE3T8dq1JJqBHYumO8PLG6hT6OkzYFewOulVhYRk4BJGxqEpFkRUb+hr6uGTSlW2LTidazlsynF61jzqcYhqceA3SQNkvQx4Hjg9qI+twMnpunjgP+NSu8KmZnZeiq+hxERTZJ+CPwR6AJMjognJJ0HzIqI24GrgN9JepZsz+L4SsdpZmbrq0p584i4C7irqG1cwfRq4KtlDmODD2NV0aYUK2xa8TrW8tmU4nWsOVT8pLeZmW2aXBrEzMxy6XQJo62yJNUgabKkJZLmF7RtJ2m6pIXpedvULkmXpPjnShpc4Vh3ljRD0gJJT0g6o1bjldRN0qOSHk+xnpvaB6WSMwtTCZqPpfaql6SR1EXSXyXduQnEukjSPEkNkmaltpr7HqTtbyPpZklPpe/usBqOdY/0mTY/Vkj6UU3EGxGd5kF2kv05YFfgY8DjwF41ENchwGBgfkHbfwJnpemzgAvS9NHA3YCAocAjFY61HzA4TfcEniEr8VJz8aZtbpWmuwKPpBhuAo5P7ZcDp6Xp7wOXp+njgSlV+C78M3A9cGear+VYFwF9itpq7nuQtn8NcHKa/hiwTa3GWhR3F+BvwC61EG9VPoQqfvjDgD8WzI8FxlY7rhTLwKKE8TTQL033A55O01cAo0v1q1LctwFH1Hq8wJbAHOAgspueNi/+TpBduTcsTW+e+qmCMfYH7gM+C9yZ/gDUZKxpu6USRs19D4CtgReKP59ajLVE7COBh2ol3s52SCpPWZJasUNEvAKQnrdP7TXzHtJhkP3JfrnXZLzpEE8DsASYTraH+WZkJWeK46l2SZqLgX8F3k/zvandWAECmCZptrKKC1Cb34NdgaXAb9Phvisl9ajRWIsdD9yQpqseb2dLGKUKGG5ql4nVxHuQtBXwe+BHEbGita4l2ioWb0SsjYg6sl/vQ4BPthJP1WKV9DlgSUTMLmxuJZ5a+B4Mj4jBZJWnfyDpkFb6VjPezckO+V4WEfsDb5Md0mlJLXy2pPNVxwL/01bXEm1libezJYw8ZUlqxauS+gGk5yWpvervQVJXsmRxXUTckpprNl6AiHgTuJ/sGO82ykrOFMezLla1UZKmDIYDx0paRFbB+bNkexy1GCsAEbE4PS8BppIl5Fr8HjQCjRHxSJq/mSyB1GKshUYBcyLi1TRf9Xg7W8LIU5akVhSWRzmR7FxBc/u305URQ4HlzbuplSBJZHfiL4iIi2o5Xkl9JW2TprsDhwMLgBlkJWdKxVqVkjQRMTYi+kfEQLLv5f9GxDdqMVYAST0k9WyeJjvWPp8a/B5ExN+AlyTtkZoOA56sxViLjOaDw1HNcVU33mqcyKnmg+yKgmfIjmX/tNrxpJhuAF4B3iP7tfBdsuPR9wEL0/N2qa/IBqB6DpgH1Fc41oPJdnfnAg3pcXQtxgvsB/w1xTofGJfadwUeBZ4l293fIrV3S/PPpuW7Vun7cCgfXCVVk7GmuB5Pjyea/y/V4vcgbb8OmJW+C7cC29ZqrCmGLYFlQK+CtqrH6zu9zcwsl852SMrMzNrJCcPMzHJxwjAzs1ycMMzMLBcnDDMzy8UJwzqUpC9K2itn3/GSfpymz5N0eHmjazOeWojhyryfX4nXLpLUp0T72UXzf25vfG1sf39JV25IbBuxrRsl7dZR67N8fFmtdShJV5PdQ3Bzjr7jgbci4sJyx1VLJHWJiLVlWO8ismvwXytqfysituro7ZXY/v8A50fE43lj24htfQb4ZkSc0hHrs3y8h2FAVkhQ2VgB16Sa+jdL2jItOywVbZunbOyOLVL7BElPpv4XSvp7sto3E1Md/4+X2M5PlY1Hci+wR0H71ZKOS9PjJD0mab6kSenuciQdmLb1F0kTlcYPkXSSpFsk3aNsrID/LFjv6BT3fEkXpLYuaXvz07J/KhHDeu+txPv4jD4Yr+CvknpKOlRpHIvU51JJJ6XpRel9PQj8q6RHiz77uWn6fkn1kk4reh8nSfplmr5VWcG/J/RB0b+W/l0nAN1TnNeltrfS86GS/iTpJknPpPf8DWVjiMxr/vdTdsf879O/yWOShpfYTk9gv+ZkIam3pGnps7mCgnpHpeKX9F1JvyjKkZprAAAFTElEQVToc4qki5TdUf4HZWOazJf09dTlAeBwfVA2xSqh0ncw+lGbD7Ly6kFWUA5gMvBjsjuKXwJ2T+3/DfwI2I6sjHLzXuo26flq4LgWtnEA2Z2oW5KVnH4W+HHx60h3sKbp3wGfT9Pzgb9P0xNI5eCBk4DnyeopdQNeJKutsyPwf0BfsgJ0/wt8McUxvWAb68Xe0nsrei93FHxWW6X1H0q6Qzu1XwqclKYXAf9asKyBdHc28BPg39L0/UB9ivnZgv53AwcXfj5A9/SZ9C7YRp8Ssb5Vaj7F+yZZqewtgJeBc9OyM4CL0/T1BdseQFYWpngbI4DfF8xfwgd31h9D9t3q01L8QA+yO5W7pmV/BvYFvgL8pmC9hXc+TwcOqPb/nc708B6GFXopIh5K09eSlQHZA3ghIp5J7deQDfi0AlgNXCnpy8CqHOv/NDA1IlZFVuG2pTpeI5SNIjePrAjf3spqQvWMiObj79cXvea+iFgeEavJ6gTtAhwI3B8RSyMrAX5div15YFdJv5R0VHovhfK8t4eAiySdTpZQmkr0KTalYPom4Gtp+utFy4iIpcDzkoZK6k3279D8b3O6pMeBh8kS48Ycy38sIl6JiDVkf7CnpfZ5ZD8iIKvBdamyMvG3A1unPYpC/chKiDc7hOw7RET8AXijYNmH4o+It8kS+uck7UmWOOalOA6XdIGkT0fE8oL1LCH7UWAV4oRhhYpPaAWlSyeT/kAOIata+0XgnuI+yoZzbT5s870WtlH8mm7Ar8n2NvYFfkO211AyjgJrCqbXkv3ibyn2N4BPkf2a/wFwZdHyNt9bREwATib7lfxw+iPXxPr/p7oVveztgukpwNck7Z6tLhaWCHUKWVL5ClmiDUmHkv0BHxYRnyKrlVW8nQ1R+Lm9XzD/PtlnCNl7GhYRdemxU0SsLFrPOyXi+NC/dRvxX0m2t/gd4LcA6YdK857pzyWNK1hdt7RdqxAnDCs0QNKwND0aeBB4Chgo6ROp/VvAn5SNh9ErIu4iO0RVl5avJBu6lYh4qeCPzOXATOBLkrqnX6ifLxFD8x+P19I2jkvregNYqawaJ2QVXdvyCPAZSX0kdUnv6U/KrtbZLCJ+D/w7WanrdVp5b4V9Ph4R8yLiArKidnuSHQrbS9l4273IqqKWFBHPkSW2f6do76LALWQJa3RBn17AGxGxKiWpoS28ttB7ykrSt9c04IfNM5I+9HmQVQH+RMH8TOAbqf8osmJ/0Er8kZUf3xk4gVSlVdKOwKqIuBa4kPX/rXYnK3xoFeITRlZoAXBiOkm5kGzAmdWSvgP8TzrB+BjZ2NLbAbelPQIB/5TWcSPwm3So5rj0hxGAiJgjaQrZ8fsXyU5crici3pT0G7JflIvS9pp9N637bbK9g+XFry9a1yuSxpKVCBdwV0TcJulTZKOvNf9gGlv00p4tvLdCP5I0guyP/pPA3RGxRtJNZBVRF5L9em7NFGAiMKiF+N+Q9CTZuPPNJ8nvAb6XTpI/TXZYpy2TgLmS5kRWMn1DnQ78Km1zc7Jk8L3CDhHxlKReknqmvY9zgRskzQH+RHYuKU/8NwF16QcCZOcxJkp6n6ya82kAknYA3onqlB3vtHxZrQHrhlu9MyL2qXIoLZK0VUQ0X+FzFtm4xWdUOSxLlF1ttjIiSt6LkXMddwK/iIj7cmxrRURc1d5t2YbzISnblByTzofMJzuBfn61A7L1XMb650Ryk7SNpGfI9hpaTRbJm2QXYFgFeQ/DzMxy8R6GmZnl4oRhZma5OGGYmVkuThhmZpaLE4aZmeXihGFmZrn8f22osNMgFvulAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist((untreated, treated))\n",
    "plt.legend(['untreated', 'treated'])\n",
    "plt.xlabel('post-diagnosis survival time (days)')\n",
    "plt.ylabel('number of patients')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute the test statistic $T_{obs}$\n",
    "$$T_{obs} = \\mu_{treated}-\\mu_{untreated}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t_Obs = 61\n"
     ]
    }
   ],
   "source": [
    "tObs = treated.mean() - untreated.mean()\n",
    "print('t_Obs = %.3g' % tObs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Permutation test\n",
    "In order to estimate the distribution of the test statistic $T_{obs}$, we will shuffle the data points and randomly relabel them (treated/untreated) $N$ times ($N$ permutations) and recompute the test statistic $T_i = \\mu_{treated}-\\mu_{untreated}$ for $i\\in\\{1,...,N\\}$. For this example, we will run 1000 permutations ($N=1000$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "permutation 1: t_i = -4.26\n",
      "permutation 2: t_i = 7.4\n",
      "permutation 3: t_i = -45.5\n",
      "permutation 4: t_i = -3.07\n",
      "permutation 5: t_i = -32.8\n",
      "permutation 6: t_i = 9.54\n",
      "permutation 7: t_i = -9.01\n",
      "permutation 8: t_i = 8.51\n",
      "permutation 9: t_i = 20.6\n",
      "permutation 10: t_i = 75.8\n",
      "permutation 11: t_i = 74.4\n",
      "permutation 12: t_i = 2.06\n",
      "permutation 13: t_i = 65.8\n",
      "permutation 14: t_i = -13.8\n",
      "permutation 15: t_i = 22.7\n",
      "permutation 16: t_i = -48.8\n",
      "permutation 17: t_i = -1.79\n",
      "permutation 18: t_i = 27.3\n",
      "permutation 19: t_i = -55.2\n",
      "permutation 20: t_i = 88.6\n",
      "permutation 21: t_i = -41.7\n",
      "permutation 22: t_i = 14.9\n",
      "permutation 23: t_i = -20.8\n",
      "permutation 24: t_i = -7.26\n",
      "permutation 25: t_i = -32.7\n",
      "permutation 26: t_i = -40.1\n",
      "permutation 27: t_i = 40.3\n",
      "permutation 28: t_i = 12.1\n",
      "permutation 29: t_i = 5.69\n",
      "permutation 30: t_i = 31.1\n",
      "permutation 31: t_i = 1.67\n",
      "permutation 32: t_i = -13.5\n",
      "permutation 33: t_i = -33.2\n",
      "permutation 34: t_i = 21.2\n",
      "permutation 35: t_i = -6.96\n",
      "permutation 36: t_i = -23.2\n",
      "permutation 37: t_i = -15.5\n",
      "permutation 38: t_i = 23.2\n",
      "permutation 39: t_i = -29.3\n",
      "permutation 40: t_i = -13\n",
      "permutation 41: t_i = -25.7\n",
      "permutation 42: t_i = -67.6\n",
      "permutation 43: t_i = -46.3\n",
      "permutation 44: t_i = -29.9\n",
      "permutation 45: t_i = -4.09\n",
      "permutation 46: t_i = 19.3\n",
      "permutation 47: t_i = -41.5\n",
      "permutation 48: t_i = -28.1\n",
      "permutation 49: t_i = 11.4\n",
      "permutation 50: t_i = -33\n",
      "permutation 51: t_i = -9.86\n",
      "permutation 52: t_i = -22.9\n",
      "permutation 53: t_i = -1.44\n",
      "permutation 54: t_i = -31.3\n",
      "permutation 55: t_i = -12.1\n",
      "permutation 56: t_i = -27.6\n",
      "permutation 57: t_i = 26.8\n",
      "permutation 58: t_i = 6.93\n",
      "permutation 59: t_i = -20.7\n",
      "permutation 60: t_i = 19.4\n",
      "permutation 61: t_i = 61.2\n",
      "permutation 62: t_i = 62.4\n",
      "permutation 63: t_i = 21.4\n",
      "permutation 64: t_i = -19.2\n",
      "permutation 65: t_i = -37.8\n",
      "permutation 66: t_i = 11.3\n",
      "permutation 67: t_i = 3.21\n",
      "permutation 68: t_i = 4.41\n",
      "permutation 69: t_i = -4.39\n",
      "permutation 70: t_i = -31.9\n",
      "permutation 71: t_i = 56.1\n",
      "permutation 72: t_i = -33\n",
      "permutation 73: t_i = 20.1\n",
      "permutation 74: t_i = -15.3\n",
      "permutation 75: t_i = -51.2\n",
      "permutation 76: t_i = 7.06\n",
      "permutation 77: t_i = 1.16\n",
      "permutation 78: t_i = -10.9\n",
      "permutation 79: t_i = 30.7\n",
      "permutation 80: t_i = 25.3\n",
      "permutation 81: t_i = 55.2\n",
      "permutation 82: t_i = -54.6\n",
      "permutation 83: t_i = -46.2\n",
      "permutation 84: t_i = -65.4\n",
      "permutation 85: t_i = 18.5\n",
      "permutation 86: t_i = -42.2\n",
      "permutation 87: t_i = 34.2\n",
      "permutation 88: t_i = -50.8\n",
      "permutation 89: t_i = -13\n",
      "permutation 90: t_i = -52.8\n",
      "permutation 91: t_i = -19.9\n",
      "permutation 92: t_i = 17\n",
      "permutation 93: t_i = -29.6\n",
      "permutation 94: t_i = -42.1\n",
      "permutation 95: t_i = -46.5\n",
      "permutation 96: t_i = -4.39\n",
      "permutation 97: t_i = 2.79\n",
      "permutation 98: t_i = 32.1\n",
      "permutation 99: t_i = -18.1\n",
      "permutation 100: t_i = -43.4\n",
      "permutation 101: t_i = 16.8\n",
      "permutation 102: t_i = -29.9\n",
      "permutation 103: t_i = 17.6\n",
      "permutation 104: t_i = 0.521\n",
      "permutation 105: t_i = 36.1\n",
      "permutation 106: t_i = 61.3\n",
      "permutation 107: t_i = 52.7\n",
      "permutation 108: t_i = -41.4\n",
      "permutation 109: t_i = -43.5\n",
      "permutation 110: t_i = -24\n",
      "permutation 111: t_i = -10.7\n",
      "permutation 112: t_i = 16.5\n",
      "permutation 113: t_i = 31.6\n",
      "permutation 114: t_i = 19.7\n",
      "permutation 115: t_i = -15.6\n",
      "permutation 116: t_i = -0.975\n",
      "permutation 117: t_i = 21.2\n",
      "permutation 118: t_i = -8.88\n",
      "permutation 119: t_i = 27.6\n",
      "permutation 120: t_i = -53.7\n",
      "permutation 121: t_i = -29.6\n",
      "permutation 122: t_i = -1.32\n",
      "permutation 123: t_i = 34.3\n",
      "permutation 124: t_i = -12.2\n",
      "permutation 125: t_i = -10.5\n",
      "permutation 126: t_i = -13.1\n",
      "permutation 127: t_i = -3.45\n",
      "permutation 128: t_i = 49.3\n",
      "permutation 129: t_i = 32.4\n",
      "permutation 130: t_i = -3.11\n",
      "permutation 131: t_i = 52\n",
      "permutation 132: t_i = -27\n",
      "permutation 133: t_i = 55.3\n",
      "permutation 134: t_i = -14.4\n",
      "permutation 135: t_i = -43.4\n",
      "permutation 136: t_i = 54.1\n",
      "permutation 137: t_i = -38.6\n",
      "permutation 138: t_i = -27\n",
      "permutation 139: t_i = 65.4\n",
      "permutation 140: t_i = -67.9\n",
      "permutation 141: t_i = 35.3\n",
      "permutation 142: t_i = -46.7\n",
      "permutation 143: t_i = 12.6\n",
      "permutation 144: t_i = -40.8\n",
      "permutation 145: t_i = 16.7\n",
      "permutation 146: t_i = -40.9\n",
      "permutation 147: t_i = -0.59\n",
      "permutation 148: t_i = 5.69\n",
      "permutation 149: t_i = 6.76\n",
      "permutation 150: t_i = -49.6\n",
      "permutation 151: t_i = 75.1\n",
      "permutation 152: t_i = -2.21\n",
      "permutation 153: t_i = 28.8\n",
      "permutation 154: t_i = 0.222\n",
      "permutation 155: t_i = 60.2\n",
      "permutation 156: t_i = -25.2\n",
      "permutation 157: t_i = 8.55\n",
      "permutation 158: t_i = -43.1\n",
      "permutation 159: t_i = 0.948\n",
      "permutation 160: t_i = 2.27\n",
      "permutation 161: t_i = 50.6\n",
      "permutation 162: t_i = -50.2\n",
      "permutation 163: t_i = -8.41\n",
      "permutation 164: t_i = 7.66\n",
      "permutation 165: t_i = 46.6\n",
      "permutation 166: t_i = -8.49\n",
      "permutation 167: t_i = -46.6\n",
      "permutation 168: t_i = 26.3\n",
      "permutation 169: t_i = -6.44\n",
      "permutation 170: t_i = 44.9\n",
      "permutation 171: t_i = -30.3\n",
      "permutation 172: t_i = 13.8\n",
      "permutation 173: t_i = 24.2\n",
      "permutation 174: t_i = 12.9\n",
      "permutation 175: t_i = 13.4\n",
      "permutation 176: t_i = 43.1\n",
      "permutation 177: t_i = -0.205\n",
      "permutation 178: t_i = 23\n",
      "permutation 179: t_i = 29.6\n",
      "permutation 180: t_i = -8.79\n",
      "permutation 181: t_i = -35.8\n",
      "permutation 182: t_i = 64\n",
      "permutation 183: t_i = 61.4\n",
      "permutation 184: t_i = -30.7\n",
      "permutation 185: t_i = 8.34\n",
      "permutation 186: t_i = -13.8\n",
      "permutation 187: t_i = 10.4\n",
      "permutation 188: t_i = 27.7\n",
      "permutation 189: t_i = -11.6\n",
      "permutation 190: t_i = -23.2\n",
      "permutation 191: t_i = -45\n",
      "permutation 192: t_i = -33\n",
      "permutation 193: t_i = 28.7\n",
      "permutation 194: t_i = 13.5\n",
      "permutation 195: t_i = -73.3\n",
      "permutation 196: t_i = 40.8\n",
      "permutation 197: t_i = 41.3\n",
      "permutation 198: t_i = 13.7\n",
      "permutation 199: t_i = 21.5\n",
      "permutation 200: t_i = -6.83\n",
      "permutation 201: t_i = -36.9\n",
      "permutation 202: t_i = 30.6\n",
      "permutation 203: t_i = -2.56\n",
      "permutation 204: t_i = 37.4\n",
      "permutation 205: t_i = -31.8\n",
      "permutation 206: t_i = -46.9\n",
      "permutation 207: t_i = -27.7\n",
      "permutation 208: t_i = -9.52\n",
      "permutation 209: t_i = 37.9\n",
      "permutation 210: t_i = 32\n",
      "permutation 211: t_i = -48.3\n",
      "permutation 212: t_i = -0.205\n",
      "permutation 213: t_i = 21.4\n",
      "permutation 214: t_i = -36.1\n",
      "permutation 215: t_i = 26.3\n",
      "permutation 216: t_i = -1.57\n",
      "permutation 217: t_i = -6.27\n",
      "permutation 218: t_i = 31.6\n",
      "permutation 219: t_i = -30.2\n",
      "permutation 220: t_i = 63.2\n",
      "permutation 221: t_i = -45.1\n",
      "permutation 222: t_i = -13\n",
      "permutation 223: t_i = 50.1\n",
      "permutation 224: t_i = -19.8\n",
      "permutation 225: t_i = -22.4\n",
      "permutation 226: t_i = 25.2\n",
      "permutation 227: t_i = 3.73\n",
      "permutation 228: t_i = 5.26\n",
      "permutation 229: t_i = -43.5\n",
      "permutation 230: t_i = 34.7\n",
      "permutation 231: t_i = 65.5\n",
      "permutation 232: t_i = 79.4\n",
      "permutation 233: t_i = -41.5\n",
      "permutation 234: t_i = -18.2\n",
      "permutation 235: t_i = 13.8\n",
      "permutation 236: t_i = 33.1\n",
      "permutation 237: t_i = 5.31\n",
      "permutation 238: t_i = 82.6\n",
      "permutation 239: t_i = -10.8\n",
      "permutation 240: t_i = -4.73\n",
      "permutation 241: t_i = 8.64\n",
      "permutation 242: t_i = -4.01\n",
      "permutation 243: t_i = -43.6\n",
      "permutation 244: t_i = 39.7\n",
      "permutation 245: t_i = 17.4\n",
      "permutation 246: t_i = 35.2\n",
      "permutation 247: t_i = -15.6\n",
      "permutation 248: t_i = -61\n",
      "permutation 249: t_i = 39.5\n",
      "permutation 250: t_i = 7.96\n",
      "permutation 251: t_i = 48.8\n",
      "permutation 252: t_i = 17.1\n",
      "permutation 253: t_i = -9.09\n",
      "permutation 254: t_i = -14.4\n",
      "permutation 255: t_i = 14.2\n",
      "permutation 256: t_i = -34\n",
      "permutation 257: t_i = -53.4\n",
      "permutation 258: t_i = -51.1\n",
      "permutation 259: t_i = -38\n",
      "permutation 260: t_i = -46.7\n",
      "permutation 261: t_i = -35.5\n",
      "permutation 262: t_i = -31.2\n",
      "permutation 263: t_i = -22.2\n",
      "permutation 264: t_i = 2.91\n",
      "permutation 265: t_i = -16.9\n",
      "permutation 266: t_i = -1.06\n",
      "permutation 267: t_i = 17.9\n",
      "permutation 268: t_i = -3.88\n",
      "permutation 269: t_i = 19.5\n",
      "permutation 270: t_i = 88.3\n",
      "permutation 271: t_i = -39.1\n",
      "permutation 272: t_i = 25.4\n",
      "permutation 273: t_i = -33.3\n",
      "permutation 274: t_i = 40.7\n",
      "permutation 275: t_i = 0.905\n",
      "permutation 276: t_i = -52.5\n",
      "permutation 277: t_i = -37.8\n",
      "permutation 278: t_i = 13.6\n",
      "permutation 279: t_i = 6.08\n",
      "permutation 280: t_i = -6.74\n",
      "permutation 281: t_i = -4.95\n",
      "permutation 282: t_i = -44.9\n",
      "permutation 283: t_i = 36.2\n",
      "permutation 284: t_i = -12.1\n",
      "permutation 285: t_i = -30.7\n",
      "permutation 286: t_i = -30.4\n",
      "permutation 287: t_i = -5.38\n",
      "permutation 288: t_i = 16.7\n",
      "permutation 289: t_i = -50.3\n",
      "permutation 290: t_i = -5.2\n",
      "permutation 291: t_i = -49\n",
      "permutation 292: t_i = -33.7\n",
      "permutation 293: t_i = -2.34\n",
      "permutation 294: t_i = -22.7\n",
      "permutation 295: t_i = 12.4\n",
      "permutation 296: t_i = -46.8\n",
      "permutation 297: t_i = 24\n",
      "permutation 298: t_i = 49.6\n",
      "permutation 299: t_i = -46.7\n",
      "permutation 300: t_i = -72.3\n",
      "permutation 301: t_i = -4.35\n",
      "permutation 302: t_i = 61.3\n",
      "permutation 303: t_i = -40.1\n",
      "permutation 304: t_i = 5.09\n",
      "permutation 305: t_i = 52.8\n",
      "permutation 306: t_i = -56.2\n",
      "permutation 307: t_i = 5.9\n",
      "permutation 308: t_i = -53.1\n",
      "permutation 309: t_i = 59.2\n",
      "permutation 310: t_i = 21.3\n",
      "permutation 311: t_i = -24.4\n",
      "permutation 312: t_i = 59.4\n",
      "permutation 313: t_i = -11.8\n",
      "permutation 314: t_i = -9.31\n",
      "permutation 315: t_i = -7.47\n",
      "permutation 316: t_i = 1.89\n",
      "permutation 317: t_i = -27.6\n",
      "permutation 318: t_i = -50\n",
      "permutation 319: t_i = -42.3\n",
      "permutation 320: t_i = -15.6\n",
      "permutation 321: t_i = -1.79\n",
      "permutation 322: t_i = 20.9\n",
      "permutation 323: t_i = -17.3\n",
      "permutation 324: t_i = -68.1\n",
      "permutation 325: t_i = -42.4\n",
      "permutation 326: t_i = -23.6\n",
      "permutation 327: t_i = 8.68\n",
      "permutation 328: t_i = 2.19\n",
      "permutation 329: t_i = 20.3\n",
      "permutation 330: t_i = 40\n",
      "permutation 331: t_i = -30.1\n",
      "permutation 332: t_i = 15.2\n",
      "permutation 333: t_i = -93\n",
      "permutation 334: t_i = -35.5\n",
      "permutation 335: t_i = 8.77\n",
      "permutation 336: t_i = 21.5\n",
      "permutation 337: t_i = 12\n",
      "permutation 338: t_i = 41.7\n",
      "permutation 339: t_i = 8.43\n",
      "permutation 340: t_i = -14.3\n",
      "permutation 341: t_i = 19.7\n",
      "permutation 342: t_i = -19.8\n",
      "permutation 343: t_i = -41.7\n",
      "permutation 344: t_i = 58.1\n",
      "permutation 345: t_i = 52.6\n",
      "permutation 346: t_i = 12\n",
      "permutation 347: t_i = -8.79\n",
      "permutation 348: t_i = -28.7\n",
      "permutation 349: t_i = 28\n",
      "permutation 350: t_i = 55.3\n",
      "permutation 351: t_i = -78.5\n",
      "permutation 352: t_i = -51.8\n",
      "permutation 353: t_i = 0.564\n",
      "permutation 354: t_i = 42\n",
      "permutation 355: t_i = 26.6\n",
      "permutation 356: t_i = 17.4\n",
      "permutation 357: t_i = -40.9\n",
      "permutation 358: t_i = 18.9\n",
      "permutation 359: t_i = -36.1\n",
      "permutation 360: t_i = 6.08\n",
      "permutation 361: t_i = -64.9\n",
      "permutation 362: t_i = 32.6\n",
      "permutation 363: t_i = 46.1\n",
      "permutation 364: t_i = -53.6\n",
      "permutation 365: t_i = 4.79\n",
      "permutation 366: t_i = -26.8\n",
      "permutation 367: t_i = -1.62\n",
      "permutation 368: t_i = -1.79\n",
      "permutation 369: t_i = -16.4\n",
      "permutation 370: t_i = 71.4\n",
      "permutation 371: t_i = -53.3\n",
      "permutation 372: t_i = -11.3\n",
      "permutation 373: t_i = -10.8\n",
      "permutation 374: t_i = -10.1\n",
      "permutation 375: t_i = 35.3\n",
      "permutation 376: t_i = 15.6\n",
      "permutation 377: t_i = 13.9\n",
      "permutation 378: t_i = -64.9\n",
      "permutation 379: t_i = 54.9\n",
      "permutation 380: t_i = 18.3\n",
      "permutation 381: t_i = -1.19\n",
      "permutation 382: t_i = -66.2\n",
      "permutation 383: t_i = 1.08\n",
      "permutation 384: t_i = -58\n",
      "permutation 385: t_i = -8.92\n",
      "permutation 386: t_i = -27.6\n",
      "permutation 387: t_i = 36.6\n",
      "permutation 388: t_i = 20\n",
      "permutation 389: t_i = -15.1\n",
      "permutation 390: t_i = -10.9\n",
      "permutation 391: t_i = 31.9\n",
      "permutation 392: t_i = -20\n",
      "permutation 393: t_i = -18\n",
      "permutation 394: t_i = 34.3\n",
      "permutation 395: t_i = -43.3\n",
      "permutation 396: t_i = -73.9\n",
      "permutation 397: t_i = 12\n",
      "permutation 398: t_i = -12.7\n",
      "permutation 399: t_i = -101\n",
      "permutation 400: t_i = 0.179\n",
      "permutation 401: t_i = 8\n",
      "permutation 402: t_i = 33.6\n",
      "permutation 403: t_i = -3.24\n",
      "permutation 404: t_i = 6.42\n",
      "permutation 405: t_i = 6.55\n",
      "permutation 406: t_i = 40.4\n",
      "permutation 407: t_i = 7.87\n",
      "permutation 408: t_i = -46.8\n",
      "permutation 409: t_i = -8.96\n",
      "permutation 410: t_i = -45.4\n",
      "permutation 411: t_i = 49.9\n",
      "permutation 412: t_i = 41.7\n",
      "permutation 413: t_i = -4.95\n",
      "permutation 414: t_i = 18.6\n",
      "permutation 415: t_i = 18\n",
      "permutation 416: t_i = -0.205\n",
      "permutation 417: t_i = -11.6\n",
      "permutation 418: t_i = 22.5\n",
      "permutation 419: t_i = 4.11\n",
      "permutation 420: t_i = -23.2\n",
      "permutation 421: t_i = -36.4\n",
      "permutation 422: t_i = -28.5\n",
      "permutation 423: t_i = -16.4\n",
      "permutation 424: t_i = -59.9\n",
      "permutation 425: t_i = -69.9\n",
      "permutation 426: t_i = 6.08\n",
      "permutation 427: t_i = 43.2\n",
      "permutation 428: t_i = 6.12\n",
      "permutation 429: t_i = 63\n",
      "permutation 430: t_i = 7.1\n",
      "permutation 431: t_i = 13.7\n",
      "permutation 432: t_i = 37.6\n",
      "permutation 433: t_i = 7.53\n",
      "permutation 434: t_i = -15.5\n",
      "permutation 435: t_i = -8.92\n",
      "permutation 436: t_i = 24.3\n",
      "permutation 437: t_i = 40.5\n",
      "permutation 438: t_i = 47.3\n",
      "permutation 439: t_i = 8.9\n",
      "permutation 440: t_i = -42.8\n",
      "permutation 441: t_i = -13.8\n",
      "permutation 442: t_i = 15\n",
      "permutation 443: t_i = 37.4\n",
      "permutation 444: t_i = -19.7\n",
      "permutation 445: t_i = -17.9\n",
      "permutation 446: t_i = -27.9\n",
      "permutation 447: t_i = 2.79\n",
      "permutation 448: t_i = 73.9\n",
      "permutation 449: t_i = -24.7\n",
      "permutation 450: t_i = 47.7\n",
      "permutation 451: t_i = -23.4\n",
      "permutation 452: t_i = 81.6\n",
      "permutation 453: t_i = 2.74\n",
      "permutation 454: t_i = -46.6\n",
      "permutation 455: t_i = 13.3\n",
      "permutation 456: t_i = -46.3\n",
      "permutation 457: t_i = -20\n",
      "permutation 458: t_i = 21.5\n",
      "permutation 459: t_i = 27.9\n",
      "permutation 460: t_i = 65.9\n",
      "permutation 461: t_i = -10.6\n",
      "permutation 462: t_i = -7.43\n",
      "permutation 463: t_i = -20.9\n",
      "permutation 464: t_i = 0.136\n",
      "permutation 465: t_i = -0.419\n",
      "permutation 466: t_i = -57.2\n",
      "permutation 467: t_i = -51.5\n",
      "permutation 468: t_i = -44.8\n",
      "permutation 469: t_i = -7.81\n",
      "permutation 470: t_i = 28.8\n",
      "permutation 471: t_i = 10.1\n",
      "permutation 472: t_i = 56.6\n",
      "permutation 473: t_i = 58.5\n",
      "permutation 474: t_i = 11.3\n",
      "permutation 475: t_i = 57.6\n",
      "permutation 476: t_i = 18.6\n",
      "permutation 477: t_i = 53.7\n",
      "permutation 478: t_i = -51.2\n",
      "permutation 479: t_i = -41.5\n",
      "permutation 480: t_i = -12.6\n",
      "permutation 481: t_i = 8.94\n",
      "permutation 482: t_i = 17.1\n",
      "permutation 483: t_i = -11.2\n",
      "permutation 484: t_i = -31.4\n",
      "permutation 485: t_i = -53.1\n",
      "permutation 486: t_i = -0.846\n",
      "permutation 487: t_i = 15.6\n",
      "permutation 488: t_i = 25.1\n",
      "permutation 489: t_i = -24.6\n",
      "permutation 490: t_i = -29.3\n",
      "permutation 491: t_i = 7.49\n",
      "permutation 492: t_i = -37.1\n",
      "permutation 493: t_i = 107\n",
      "permutation 494: t_i = 17.9\n",
      "permutation 495: t_i = 36.2\n",
      "permutation 496: t_i = -5.72\n",
      "permutation 497: t_i = -42.4\n",
      "permutation 498: t_i = -8.2\n",
      "permutation 499: t_i = -41.1\n",
      "permutation 500: t_i = -59.2\n",
      "permutation 501: t_i = -16.3\n",
      "permutation 502: t_i = 8.51\n",
      "permutation 503: t_i = -2.04\n",
      "permutation 504: t_i = 3.34\n",
      "permutation 505: t_i = -21.7\n",
      "permutation 506: t_i = -7.81\n",
      "permutation 507: t_i = -53.4\n",
      "permutation 508: t_i = 62.2\n",
      "permutation 509: t_i = 15.6\n",
      "permutation 510: t_i = -60.5\n",
      "permutation 511: t_i = -0.975\n",
      "permutation 512: t_i = -14.5\n",
      "permutation 513: t_i = -4.91\n",
      "permutation 514: t_i = 28.6\n",
      "permutation 515: t_i = 34.9\n",
      "permutation 516: t_i = 37.1\n",
      "permutation 517: t_i = 25.1\n",
      "permutation 518: t_i = 29.1\n",
      "permutation 519: t_i = 33.5\n",
      "permutation 520: t_i = -46\n",
      "permutation 521: t_i = -79.2\n",
      "permutation 522: t_i = -26.2\n",
      "permutation 523: t_i = 9.92\n",
      "permutation 524: t_i = -58.5\n",
      "permutation 525: t_i = 12.2\n",
      "permutation 526: t_i = 57.7\n",
      "permutation 527: t_i = 4.75\n",
      "permutation 528: t_i = 24.5\n",
      "permutation 529: t_i = 10.2\n",
      "permutation 530: t_i = 5.82\n",
      "permutation 531: t_i = -20.8\n",
      "permutation 532: t_i = 30.3\n",
      "permutation 533: t_i = 37.3\n",
      "permutation 534: t_i = -57.9\n",
      "permutation 535: t_i = 23.8\n",
      "permutation 536: t_i = 37.9\n",
      "permutation 537: t_i = -0.419\n",
      "permutation 538: t_i = -9.52\n",
      "permutation 539: t_i = 51.1\n",
      "permutation 540: t_i = 21.5\n",
      "permutation 541: t_i = 57.4\n",
      "permutation 542: t_i = 30.6\n",
      "permutation 543: t_i = 43\n",
      "permutation 544: t_i = -16.7\n",
      "permutation 545: t_i = -66\n",
      "permutation 546: t_i = 23.3\n",
      "permutation 547: t_i = -32.4\n",
      "permutation 548: t_i = 15\n",
      "permutation 549: t_i = -22\n",
      "permutation 550: t_i = 10.9\n",
      "permutation 551: t_i = -7.13\n",
      "permutation 552: t_i = 4.11\n",
      "permutation 553: t_i = 73.2\n",
      "permutation 554: t_i = 5.69\n",
      "permutation 555: t_i = 63.3\n",
      "permutation 556: t_i = -22.5\n",
      "permutation 557: t_i = 38.5\n",
      "permutation 558: t_i = -38.9\n",
      "permutation 559: t_i = 43.6\n",
      "permutation 560: t_i = -4.56\n",
      "permutation 561: t_i = -6.1\n",
      "permutation 562: t_i = 0.82\n",
      "permutation 563: t_i = 6.72\n",
      "permutation 564: t_i = 31.2\n",
      "permutation 565: t_i = 12.5\n",
      "permutation 566: t_i = -86.2\n",
      "permutation 567: t_i = 39.2\n",
      "permutation 568: t_i = 3.3\n",
      "permutation 569: t_i = -6.87\n",
      "permutation 570: t_i = 4.37\n",
      "permutation 571: t_i = -24.9\n",
      "permutation 572: t_i = 34.5\n",
      "permutation 573: t_i = -84.3\n",
      "permutation 574: t_i = -33.7\n",
      "permutation 575: t_i = -31.2\n",
      "permutation 576: t_i = 41.2\n",
      "permutation 577: t_i = -30.1\n",
      "permutation 578: t_i = 43.4\n",
      "permutation 579: t_i = -36.5\n",
      "permutation 580: t_i = -16.3\n",
      "permutation 581: t_i = 0.905\n",
      "permutation 582: t_i = -54\n",
      "permutation 583: t_i = 61.6\n",
      "permutation 584: t_i = 15.9\n",
      "permutation 585: t_i = 29.8\n",
      "permutation 586: t_i = 0.777\n",
      "permutation 587: t_i = 70.1\n",
      "permutation 588: t_i = 33\n",
      "permutation 589: t_i = 37.9\n",
      "permutation 590: t_i = -93.8\n",
      "permutation 591: t_i = -6.32\n",
      "permutation 592: t_i = 12.6\n",
      "permutation 593: t_i = -9.43\n",
      "permutation 594: t_i = -20.3\n",
      "permutation 595: t_i = 22.3\n",
      "permutation 596: t_i = 13.8\n",
      "permutation 597: t_i = 59.2\n",
      "permutation 598: t_i = -44.3\n",
      "permutation 599: t_i = 28.3\n",
      "permutation 600: t_i = -18.2\n",
      "permutation 601: t_i = 10.9\n",
      "permutation 602: t_i = 20\n",
      "permutation 603: t_i = -52\n",
      "permutation 604: t_i = -12.8\n",
      "permutation 605: t_i = 8.68\n",
      "permutation 606: t_i = 0.265\n",
      "permutation 607: t_i = -32.2\n",
      "permutation 608: t_i = 13.2\n",
      "permutation 609: t_i = -14.8\n",
      "permutation 610: t_i = 71.3\n",
      "permutation 611: t_i = -40.2\n",
      "permutation 612: t_i = 19.9\n",
      "permutation 613: t_i = -18.9\n",
      "permutation 614: t_i = -49.1\n",
      "permutation 615: t_i = -61.5\n",
      "permutation 616: t_i = 27.9\n",
      "permutation 617: t_i = 27.6\n",
      "permutation 618: t_i = -17.7\n",
      "permutation 619: t_i = -5.63\n",
      "permutation 620: t_i = -11.6\n",
      "permutation 621: t_i = -20.1\n",
      "permutation 622: t_i = -35.8\n",
      "permutation 623: t_i = 7.44\n",
      "permutation 624: t_i = -20.6\n",
      "permutation 625: t_i = -28.3\n",
      "permutation 626: t_i = -9.48\n",
      "permutation 627: t_i = 11.1\n",
      "permutation 628: t_i = -84\n",
      "permutation 629: t_i = 33.5\n",
      "permutation 630: t_i = -9.05\n",
      "permutation 631: t_i = -56\n",
      "permutation 632: t_i = 29.4\n",
      "permutation 633: t_i = 13.4\n",
      "permutation 634: t_i = -5.59\n",
      "permutation 635: t_i = 35.5\n",
      "permutation 636: t_i = -4.35\n",
      "permutation 637: t_i = 37.6\n",
      "permutation 638: t_i = 16.1\n",
      "permutation 639: t_i = 33\n",
      "permutation 640: t_i = -5.8\n",
      "permutation 641: t_i = 16\n",
      "permutation 642: t_i = 30.5\n",
      "permutation 643: t_i = -29.2\n",
      "permutation 644: t_i = 34.6\n",
      "permutation 645: t_i = -44.6\n",
      "permutation 646: t_i = 16.9\n",
      "permutation 647: t_i = 57.3\n",
      "permutation 648: t_i = -9.26\n",
      "permutation 649: t_i = 34.3\n",
      "permutation 650: t_i = -28.1\n",
      "permutation 651: t_i = -40.3\n",
      "permutation 652: t_i = 10.7\n",
      "permutation 653: t_i = -38\n",
      "permutation 654: t_i = 41.7\n",
      "permutation 655: t_i = -11.2\n",
      "permutation 656: t_i = 54.6\n",
      "permutation 657: t_i = -23.5\n",
      "permutation 658: t_i = -28.3\n",
      "permutation 659: t_i = -31.1\n",
      "permutation 660: t_i = 22.2\n",
      "permutation 661: t_i = 41.4\n",
      "permutation 662: t_i = 42.4\n",
      "permutation 663: t_i = 16.8\n",
      "permutation 664: t_i = -6.91\n",
      "permutation 665: t_i = 41.7\n",
      "permutation 666: t_i = 8.25\n",
      "permutation 667: t_i = -5.25\n",
      "permutation 668: t_i = -44.5\n",
      "permutation 669: t_i = -19.8\n",
      "permutation 670: t_i = -3.24\n",
      "permutation 671: t_i = 14.7\n",
      "permutation 672: t_i = -53.4\n",
      "permutation 673: t_i = -14.4\n",
      "permutation 674: t_i = 37.8\n",
      "permutation 675: t_i = -18.8\n",
      "permutation 676: t_i = -24.3\n",
      "permutation 677: t_i = -10.9\n",
      "permutation 678: t_i = 10\n",
      "permutation 679: t_i = -30.2\n",
      "permutation 680: t_i = 24.8\n",
      "permutation 681: t_i = -11.5\n",
      "permutation 682: t_i = -34.5\n",
      "permutation 683: t_i = -24.4\n",
      "permutation 684: t_i = 13.7\n",
      "permutation 685: t_i = 50.4\n",
      "permutation 686: t_i = 28.6\n",
      "permutation 687: t_i = 43.2\n",
      "permutation 688: t_i = 12.4\n",
      "permutation 689: t_i = 2.7\n",
      "permutation 690: t_i = -23\n",
      "permutation 691: t_i = -16.4\n",
      "permutation 692: t_i = -36.1\n",
      "permutation 693: t_i = -33.2\n",
      "permutation 694: t_i = 43.4\n",
      "permutation 695: t_i = 27.5\n",
      "permutation 696: t_i = -11.7\n",
      "permutation 697: t_i = -51.4\n",
      "permutation 698: t_i = 59.2\n",
      "permutation 699: t_i = -4.52\n",
      "permutation 700: t_i = 13.8\n",
      "permutation 701: t_i = -41.5\n",
      "permutation 702: t_i = -32.4\n",
      "permutation 703: t_i = -5.97\n",
      "permutation 704: t_i = 5.56\n",
      "permutation 705: t_i = 16.4\n",
      "permutation 706: t_i = -46.2\n",
      "permutation 707: t_i = -23\n",
      "permutation 708: t_i = 13.6\n",
      "permutation 709: t_i = -10.1\n",
      "permutation 710: t_i = -52.4\n",
      "permutation 711: t_i = 1.63\n",
      "permutation 712: t_i = -12.3\n",
      "permutation 713: t_i = -11.2\n",
      "permutation 714: t_i = 12.2\n",
      "permutation 715: t_i = 65.9\n",
      "permutation 716: t_i = -4.56\n",
      "permutation 717: t_i = 36\n",
      "permutation 718: t_i = 10.4\n",
      "permutation 719: t_i = -57.9\n",
      "permutation 720: t_i = 10.4\n",
      "permutation 721: t_i = 8.94\n",
      "permutation 722: t_i = 45.7\n",
      "permutation 723: t_i = 3.26\n",
      "permutation 724: t_i = -65.2\n",
      "permutation 725: t_i = 27.4\n",
      "permutation 726: t_i = -30.6\n",
      "permutation 727: t_i = -45.2\n",
      "permutation 728: t_i = 11\n",
      "permutation 729: t_i = -16.9\n",
      "permutation 730: t_i = -25.9\n",
      "permutation 731: t_i = 1.38\n",
      "permutation 732: t_i = -31.9\n",
      "permutation 733: t_i = 1.38\n",
      "permutation 734: t_i = -16.2\n",
      "permutation 735: t_i = 49.9\n",
      "permutation 736: t_i = 12.9\n",
      "permutation 737: t_i = -38.1\n",
      "permutation 738: t_i = 10.6\n",
      "permutation 739: t_i = 10.3\n",
      "permutation 740: t_i = 47.6\n",
      "permutation 741: t_i = -23.4\n",
      "permutation 742: t_i = -12.9\n",
      "permutation 743: t_i = 23.6\n",
      "permutation 744: t_i = -1.36\n",
      "permutation 745: t_i = 14.9\n",
      "permutation 746: t_i = 34\n",
      "permutation 747: t_i = -42.5\n",
      "permutation 748: t_i = -22.4\n",
      "permutation 749: t_i = -37.1\n",
      "permutation 750: t_i = -7.94\n",
      "permutation 751: t_i = -36.7\n",
      "permutation 752: t_i = 29.1\n",
      "permutation 753: t_i = 10.5\n",
      "permutation 754: t_i = -22\n",
      "permutation 755: t_i = -31.4\n",
      "permutation 756: t_i = 19.9\n",
      "permutation 757: t_i = 26.5\n",
      "permutation 758: t_i = 5.22\n",
      "permutation 759: t_i = -3.28\n",
      "permutation 760: t_i = 24.9\n",
      "permutation 761: t_i = -9.39\n",
      "permutation 762: t_i = -20.7\n",
      "permutation 763: t_i = -24.6\n",
      "permutation 764: t_i = -15.1\n",
      "permutation 765: t_i = -4.82\n",
      "permutation 766: t_i = -58.9\n",
      "permutation 767: t_i = 17.1\n",
      "permutation 768: t_i = 32.9\n",
      "permutation 769: t_i = 7.19\n",
      "permutation 770: t_i = 26.9\n",
      "permutation 771: t_i = -8.15\n",
      "permutation 772: t_i = -32\n",
      "permutation 773: t_i = -33.1\n",
      "permutation 774: t_i = -20.6\n",
      "permutation 775: t_i = -14.1\n",
      "permutation 776: t_i = 28\n",
      "permutation 777: t_i = 21.1\n",
      "permutation 778: t_i = 11\n",
      "permutation 779: t_i = 21.9\n",
      "permutation 780: t_i = -13.1\n",
      "permutation 781: t_i = 13.6\n",
      "permutation 782: t_i = 16.8\n",
      "permutation 783: t_i = -22.5\n",
      "permutation 784: t_i = -6.27\n",
      "permutation 785: t_i = 0.606\n",
      "permutation 786: t_i = 7.1\n",
      "permutation 787: t_i = 38.5\n",
      "permutation 788: t_i = -0.12\n",
      "permutation 789: t_i = 2.53\n",
      "permutation 790: t_i = 36.3\n",
      "permutation 791: t_i = 41.3\n",
      "permutation 792: t_i = -6.02\n",
      "permutation 793: t_i = -1.79\n",
      "permutation 794: t_i = -32.8\n",
      "permutation 795: t_i = 38.8\n",
      "permutation 796: t_i = -35.2\n",
      "permutation 797: t_i = 1.63\n",
      "permutation 798: t_i = 23.2\n",
      "permutation 799: t_i = 7.36\n",
      "permutation 800: t_i = -37.6\n",
      "permutation 801: t_i = -23.3\n",
      "permutation 802: t_i = -56.8\n",
      "permutation 803: t_i = 57\n",
      "permutation 804: t_i = 50.6\n",
      "permutation 805: t_i = 0.222\n",
      "permutation 806: t_i = 21.2\n",
      "permutation 807: t_i = -33.1\n",
      "permutation 808: t_i = -9.99\n",
      "permutation 809: t_i = -41.1\n",
      "permutation 810: t_i = -68.4\n",
      "permutation 811: t_i = 2.53\n",
      "permutation 812: t_i = -4.01\n",
      "permutation 813: t_i = -67.8\n",
      "permutation 814: t_i = -24.3\n",
      "permutation 815: t_i = -50.1\n",
      "permutation 816: t_i = 2.53\n",
      "permutation 817: t_i = 46.5\n",
      "permutation 818: t_i = -4.82\n",
      "permutation 819: t_i = 33.5\n",
      "permutation 820: t_i = 10.6\n",
      "permutation 821: t_i = 4.02\n",
      "permutation 822: t_i = 7.14\n",
      "permutation 823: t_i = -50.9\n",
      "permutation 824: t_i = 65.2\n",
      "permutation 825: t_i = 3.68\n",
      "permutation 826: t_i = -14.9\n",
      "permutation 827: t_i = -52.2\n",
      "permutation 828: t_i = 12.1\n",
      "permutation 829: t_i = -29.5\n",
      "permutation 830: t_i = 61.4\n",
      "permutation 831: t_i = 25\n",
      "permutation 832: t_i = -26\n",
      "permutation 833: t_i = -80.7\n",
      "permutation 834: t_i = -12.1\n",
      "permutation 835: t_i = 12.7\n",
      "permutation 836: t_i = 45.8\n",
      "permutation 837: t_i = 7.31\n",
      "permutation 838: t_i = -34.8\n",
      "permutation 839: t_i = -38\n",
      "permutation 840: t_i = 16.7\n",
      "permutation 841: t_i = 22.1\n",
      "permutation 842: t_i = -20.3\n",
      "permutation 843: t_i = -47.5\n",
      "permutation 844: t_i = -11.9\n",
      "permutation 845: t_i = -0.0773\n",
      "permutation 846: t_i = -117\n",
      "permutation 847: t_i = -3.03\n",
      "permutation 848: t_i = 30.5\n",
      "permutation 849: t_i = -21.1\n",
      "permutation 850: t_i = -9.99\n",
      "permutation 851: t_i = 7.83\n",
      "permutation 852: t_i = -13.8\n",
      "permutation 853: t_i = -13.1\n",
      "permutation 854: t_i = 3.81\n",
      "permutation 855: t_i = 4.54\n",
      "permutation 856: t_i = 16.1\n",
      "permutation 857: t_i = 41.8\n",
      "permutation 858: t_i = 20.9\n",
      "permutation 859: t_i = 18.5\n",
      "permutation 860: t_i = -41.9\n",
      "permutation 861: t_i = -3.32\n",
      "permutation 862: t_i = -32.6\n",
      "permutation 863: t_i = 73\n",
      "permutation 864: t_i = -2.21\n",
      "permutation 865: t_i = 0.478\n",
      "permutation 866: t_i = 16.9\n",
      "permutation 867: t_i = 35.6\n",
      "permutation 868: t_i = 24.2\n",
      "permutation 869: t_i = 41.2\n",
      "permutation 870: t_i = 5.39\n",
      "permutation 871: t_i = 7.61\n",
      "permutation 872: t_i = -72\n",
      "permutation 873: t_i = -43.5\n",
      "permutation 874: t_i = 21.4\n",
      "permutation 875: t_i = 34.3\n",
      "permutation 876: t_i = 37.1\n",
      "permutation 877: t_i = -19.5\n",
      "permutation 878: t_i = 2.06\n",
      "permutation 879: t_i = -50.2\n",
      "permutation 880: t_i = 16.3\n",
      "permutation 881: t_i = -56.8\n",
      "permutation 882: t_i = -1.79\n",
      "permutation 883: t_i = -66.7\n",
      "permutation 884: t_i = 48.7\n",
      "permutation 885: t_i = 41.4\n",
      "permutation 886: t_i = -12.2\n",
      "permutation 887: t_i = -6.49\n",
      "permutation 888: t_i = -21.4\n",
      "permutation 889: t_i = -22.6\n",
      "permutation 890: t_i = 18\n",
      "permutation 891: t_i = -32.4\n",
      "permutation 892: t_i = 23\n",
      "permutation 893: t_i = 23\n",
      "permutation 894: t_i = 13.3\n",
      "permutation 895: t_i = -14.8\n",
      "permutation 896: t_i = -68\n",
      "permutation 897: t_i = -20.9\n",
      "permutation 898: t_i = 33\n",
      "permutation 899: t_i = 33.1\n",
      "permutation 900: t_i = 3.6\n",
      "permutation 901: t_i = -30.8\n",
      "permutation 902: t_i = -35.4\n",
      "permutation 903: t_i = -54.9\n",
      "permutation 904: t_i = 36.4\n",
      "permutation 905: t_i = 12.4\n",
      "permutation 906: t_i = -22.3\n",
      "permutation 907: t_i = -56.1\n",
      "permutation 908: t_i = 26.2\n",
      "permutation 909: t_i = -13.8\n",
      "permutation 910: t_i = -32.3\n",
      "permutation 911: t_i = 28.8\n",
      "permutation 912: t_i = -8.32\n",
      "permutation 913: t_i = 14.4\n",
      "permutation 914: t_i = 34.3\n",
      "permutation 915: t_i = -55.1\n",
      "permutation 916: t_i = 29\n",
      "permutation 917: t_i = -13.7\n",
      "permutation 918: t_i = -15\n",
      "permutation 919: t_i = -22.7\n",
      "permutation 920: t_i = 40.2\n",
      "permutation 921: t_i = -3.71\n",
      "permutation 922: t_i = 29.1\n",
      "permutation 923: t_i = -32.9\n",
      "permutation 924: t_i = 33.9\n",
      "permutation 925: t_i = 32.5\n",
      "permutation 926: t_i = -28.7\n",
      "permutation 927: t_i = 0.435\n",
      "permutation 928: t_i = -20.3\n",
      "permutation 929: t_i = -23.9\n",
      "permutation 930: t_i = -57.4\n",
      "permutation 931: t_i = -15.4\n",
      "permutation 932: t_i = 13\n",
      "permutation 933: t_i = 21.6\n",
      "permutation 934: t_i = -20.5\n",
      "permutation 935: t_i = 16.4\n",
      "permutation 936: t_i = -13\n",
      "permutation 937: t_i = -28.9\n",
      "permutation 938: t_i = -18.6\n",
      "permutation 939: t_i = -52.7\n",
      "permutation 940: t_i = -25\n",
      "permutation 941: t_i = 12.1\n",
      "permutation 942: t_i = 6.42\n",
      "permutation 943: t_i = -31.8\n",
      "permutation 944: t_i = -16\n",
      "permutation 945: t_i = 6.29\n",
      "permutation 946: t_i = 10.2\n",
      "permutation 947: t_i = 6.12\n",
      "permutation 948: t_i = -6.36\n",
      "permutation 949: t_i = 14\n",
      "permutation 950: t_i = -20.9\n",
      "permutation 951: t_i = 35.4\n",
      "permutation 952: t_i = -67\n",
      "permutation 953: t_i = 27.5\n",
      "permutation 954: t_i = -79.3\n",
      "permutation 955: t_i = -11.8\n",
      "permutation 956: t_i = -29.4\n",
      "permutation 957: t_i = -49.5\n",
      "permutation 958: t_i = -33\n",
      "permutation 959: t_i = -28.1\n",
      "permutation 960: t_i = -85.4\n",
      "permutation 961: t_i = -38.1\n",
      "permutation 962: t_i = -25.6\n",
      "permutation 963: t_i = 6.8\n",
      "permutation 964: t_i = 39.3\n",
      "permutation 965: t_i = -17.7\n",
      "permutation 966: t_i = -45.5\n",
      "permutation 967: t_i = -0.718\n",
      "permutation 968: t_i = -30.5\n",
      "permutation 969: t_i = -8.54\n",
      "permutation 970: t_i = -10.3\n",
      "permutation 971: t_i = -4.01\n",
      "permutation 972: t_i = 8.6\n",
      "permutation 973: t_i = 29.5\n",
      "permutation 974: t_i = -20.2\n",
      "permutation 975: t_i = 4.37\n",
      "permutation 976: t_i = 39.7\n",
      "permutation 977: t_i = 19.4\n",
      "permutation 978: t_i = 8.04\n",
      "permutation 979: t_i = -38.5\n",
      "permutation 980: t_i = 32.4\n",
      "permutation 981: t_i = 32.4\n",
      "permutation 982: t_i = 68.1\n",
      "permutation 983: t_i = 25.9\n",
      "permutation 984: t_i = -74.6\n",
      "permutation 985: t_i = -8.45\n",
      "permutation 986: t_i = -21.8\n",
      "permutation 987: t_i = -2.09\n",
      "permutation 988: t_i = -22.4\n",
      "permutation 989: t_i = -26\n",
      "permutation 990: t_i = -52.6\n",
      "permutation 991: t_i = 25.7\n",
      "permutation 992: t_i = 77\n",
      "permutation 993: t_i = -0.846\n",
      "permutation 994: t_i = -13.1\n",
      "permutation 995: t_i = 23.6\n",
      "permutation 996: t_i = -23.7\n",
      "permutation 997: t_i = 31.6\n",
      "permutation 998: t_i = -20\n",
      "permutation 999: t_i = 43.1\n",
      "permutation 1000: t_i = -25.2\n"
     ]
    }
   ],
   "source": [
    "N = 1000\n",
    "testStats = []\n",
    "\n",
    "pooled_data = np.hstack([treated, untreated])\n",
    "sizeTreated = np.shape(treated)[0]\n",
    "sizeUntreated = np.shape(untreated)[0]\n",
    "\n",
    "for i in range(N):\n",
    "    np.random.shuffle(pooled_data)\n",
    "    permTreated = pooled_data[:sizeTreated]\n",
    "    permUntreated = pooled_data[-sizeUntreated:]\n",
    "    ti = permTreated.mean() - permUntreated.mean()\n",
    "    testStats.append(ti)\n",
    "    print('permutation %d: t_i = %.3g' % (i+1, ti))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at that distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEWCAYAAABollyxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAF3xJREFUeJzt3XvwZGV95/H3R1AwiOEyA3LT8TJuRFeJNbImZmuxwIC42YFNESFegBBHq3Bds+6a0Wwi0aiYimg00SwqAW8glXiZlYmGEJWlNgQGlyADuM7KCOMMzChy84IC3/3jnJ/0NL9L/27TM8+8X1Vd3f2cy/Oc59e/T59+zunTqSokSe16zLgbIElaXAa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDPqdVJIzklw18Pz+JE9boHW/NclH+8fLklSSPRdo3U/u27rHQqxvFvUenOTKJPclee+OrHtHmk//JvmrJH84x3rfneSNc1l2V5TkDUnOHXc7FopBP4MkG5P8uP/nmrj9xQjLVZJnLFQ7quoJVfXtGeo8JsmmEdb1rqr63YVoV98/xw2s+7a+rQ8txPpnYRXwPeCJVfWm4YlJLkzyJ/OtZD5vjHNZdq79O7yj0C/7uqp6xxzavRR4NfA/+ufH9Nvxl0PzXZXkjNmuf4a6X5zkK0nuSbJxkunL+uk/SnLLYF/1038vyR398hck2WvEZc8HXpnkoIXcnnEx6EfzG/0/18Tt9eNu0Fwt1J77TugpwE3lNwAXwxnA2qr68UDZD4FXJ1m2yHX/ELgA+G9TTL8Y+D/AgcAfAH/TvzGR5HhgNXAssAx4GvDHoyxbVT8B/o7uDW7XV1XeprkBG4Hjppj2DOBrwD10e5Of6cuvBIruRXo/8PIR6jkQWAPcC1wDvAO4amB6Ac/oH58I3ATcB3wX+K/APsCPgYf7Ou8HDgXOAf4G+GS/7t/tyz7Zr2tZv+5VwGZgC/CmgXovBP5k4PkxwKb+8Sf6+n7c1/fmgfXt2c9zaL9ddwEbgNcMrOsc4FLg4/22rAdWTNNHvwpc2/f3tcCvDrTxZ8BP+3YcN7TcqqHp/3OgbX8LbANuBd4wsMzRwLq+z+4EzuvLb+u3b6KPf2WSdo68LPB04B+B79O9hj4F7DeL/j0D+Hbff7cCrwCeBfwEeKhf7u4p/pYrgev7dv4/4IQp+v0fgVcOvwaADwJ/PVB+FXDGIv0fHgdsHCp7JvAAsO9A2f8CXtc//jTwroFpxwJ3jLJs//wVwFfGnUEL0n/jbsDOfmP6oL+Ybk/gMcDewK8NTPt5MI9YzyV0obcP8By6AJ8q6LcA/7Z/vD/w/P7xMfQhPLDcOXQhd1LfzsczedBf3Nf9r+mC77h++nA4bFfHcP/w6CD6GvChvn+O6td97EDbfkL3xrUH8G7g6in65wDgB8CrgD2B0/rnB07WzkmWH96OxwDXAX8EPI5ub+/bwPH99H8CXtU/fgLwwsm2b4q6Rl6WbmfhJcBewFK6nYT3j9K//d/rXuBf9dMOAZ7dPz6DgdfPcB/QvRnd09f9GOAw4Jem2J5twAuGXwPAk4bqnzLogd8G7p7m9uQZ/j8mC/qTgZuHyv4C+GD/+F8Y2MkClvR9d+BMy/bPnw/ctVjZsiNvDt2M5vNJ7h64vaYv/xndkMGhVfWTqrpqmnVMqT+w9pvAH1XVD6vqRuCiaRb5GXBkkidW1Q+q6uszVPFPVfX5qnq4tv/4PeiP+7q/Afw1XZDOS5IjgF8Dfr/vn+uBj9KF9YSrqmptdWPOnwCeN8XqXgZ8q6o+UVUPVtXFwC3Ab8yxeS8AllbV26vqp9Ud//gIcGo//WfAM5Isqar7q+rqWax75GWrakNVXV5VD1TVNuA84N/Noq6HgeckeXxVbamq9SMudxZwQV/3w1X13aq6ZYp596P7xDDc9juAvwLePlNlVfXpqtpvmtttI7Z70BPo3qwG3QPsO8X0icf7jrAsdNv8i3No107HoB/NSUMvyo/05W8GAlyTZH2S35nj+pfS7aHdPlD2nWnm/026veDvJPlakl+ZYf23zzB9eJ7v0A1rzNehdHtEgyHxHbq9xwl3DDz+EbD3FMcRDuXRfTK8rtl4CnDo4Bs48Fbg4H76WXQf729Jcm2Sfz+LdY+8bJKDklyS5LtJ7qUbYlsySiVV9UPg5cDrgC1JLkvySyO28Qi64ZpR/IDtA3DQe4Djk0z1Br2Y7geeOFT2RB55UxqePvH4vhGWhW6bh98MdkkG/TxU1R1V9ZqqOhR4LfChOZ5psw14kO6fb8KTp6n32qpaCRwEfJ5uyAe6j6WTLjJCG4br3tw//iHwCwPTnjSLdW8GDkgyGBJPphuWmq3NdOE8aDbrGm7n7cCtQ2/g+1bViQBV9a2qOo2uj99Dd6Bun0nW8+iKZrfsu/vy51bVE4FX0u08TNXu4bq+XFUvoRu2uYXuU8mMy9Ft/9Nn2pbeDXRvXJPV/33g/XTHlKaU5BVDZ64N36Z8vU9jPfC0odfX8/ryienPG5p2Z9/mmZaF7ljHv8yhXTsdg34ekpyS5PD+6Q/o/rkmTnu7k27cd0b9sMVngXOS/EKSI4HTp6jzcf0/zS9W1c/oxkgH6zwwyVw+bv5hX/ezgTOBz/Tl1wMnJjkgyZOA4XOpp9zOqrod+N/Au5PsneS5dHu7n5pD+9YCz0zy20n2TPJy4EjgiyMuP9zOa4B7k/x+kscn2SPJc5K8ACDJK5MsraqH6caQoevnbXTDJVP+bWe57L70B0yTHMajzy6Zsn/TfXfgP/RvIg/06xl8LRye5HFTNPNjwJlJjk3ymCSHTfNpYC3TDyedR3eg/FlTzVBVn6rtz1wbvk06dNO3bW/gsd3T7D2xTVX1f+len2/ry08Gnkt3gB26g/xnJTkyyf7Af6c7TjHKsvTb/HfTbPeuY9wHCXb2G93BsImzHiZun+un/SndHuX9dB+DVw0s9zq6g6Z3A79Ft/d5P1McdKIbvvkiM5x1Q3fg8Et0byz30p19MngQ+AK6Mzju5pGzbj45VNfPy3j0WTd3AG8emHdvutC/l27P7vfY/mDsSrqzSe6mO/tnYn0TB2MP77frrr6PXjdZO4baMumBTrrx/uvoPk5fN7TdFzL9wdjldP/YdwOf78sOpTsIfUffn1fzyEHoTwJb+7/Zerrhu4l1vZ0utO+mP9A6VNfIywLP7rfl/r59bxq1f+n24ifO+rob+CpwZL/c44DL+n7/3mR9RHdA8ga64YoN9AeiJ9meJXQHXx/fPz+GRx/0f3PfrjMW+P/vmH69g7evDr1mvkr3P/pNHn3G1X+he9O7l+7Y016jLEv3ut8EHDzuDFqIW/qNkqQpJXkXsLWq3j/utuwISf4TcERVvXncbVkIBr0kNc4xeklqnEEvSY0z6CWpcTvFBa6WLFlSy5YtG3czJGmXct11132vqpbONN9OEfTLli1j3bp1426GJO1Skkz3Dfqfc+hGkhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaN+M3Y/sfeP443U/IPQycX1V/nuQc4DV0P6IA8NaqWtsv8xa6XxJ6CHhDVX15Edou7RDLVl82lno3nvuysdSr9oxyCYQHgTdV1df731e8Lsnl/bT3VdWfDc7c/wzeqXS/nHMo8A9Jnlndz+VJknawGYduqmpLVX29f3wfcDNw2DSLrAQuqaoHqupWup8oO3ohGitJmr1ZjdEnWQb8MvDPfdHrk9yQ5IL+x3ehexO4fWCxTUzyxpBkVZJ1SdZt27ZteLIkaYGMHPRJnkD3C+lvrKp7gQ8DTweOovsR7PdOzDrJ4o/6vcKqOr+qVlTViqVLZ7zKpiRpjkYK+iSPpQv5T1XVZwGq6s6qeqiqHgY+wiPDM5uAIwYWPxzYvHBNliTNxoxBnyTAx4Cbq+q8gfJDBmY7Gbixf7wGODXJXkmeCiwHrlm4JkuSZmOUs25eBLwK+EaS6/uytwKnJTmKblhmI/BagKpan+RS4Ca6M3bO9owbSRqfGYO+qq5i8nH3tdMs807gnfNolyRpgfjNWElqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWrcKD8lKI3dstWXjbsJ0i7LPXpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1LgZgz7JEUm+kuTmJOuT/Oe+/IAklyf5Vn+/f1+eJB9IsiHJDUmev9gbIUma2ih79A8Cb6qqZwEvBM5OciSwGriiqpYDV/TPAV4KLO9vq4APL3irJUkjmzHoq2pLVX29f3wfcDNwGLASuKif7SLgpP7xSuDj1bka2C/JIQvecknSSGY1Rp9kGfDLwD8DB1fVFujeDICD+tkOA24fWGxTXza8rlVJ1iVZt23bttm3XJI0kpGDPskTgL8F3lhV90436yRl9aiCqvOrakVVrVi6dOmozZAkzdJIQZ/ksXQh/6mq+mxffOfEkEx/v7Uv3wQcMbD44cDmhWmuJGm2RjnrJsDHgJur6ryBSWuA0/vHpwNfGCh/dX/2zQuBeyaGeCRJO96eI8zzIuBVwDeSXN+XvRU4F7g0yVnAbcAp/bS1wInABuBHwJkL2mJJ0qzMGPRVdRWTj7sDHDvJ/AWcPc92SZIWiN+MlaTGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNG+V69JLGYNnqy8ZS78ZzXzaWerV43KOXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIa57VuNCvjuv6KpLlzj16SGmfQS1LjDHpJapxBL0mNmzHok1yQZGuSGwfKzkny3STX97cTB6a9JcmGJN9McvxiNVySNJpR9ugvBE6YpPx9VXVUf1sLkORI4FTg2f0yH0qyx0I1VpI0ezMGfVVdCdw14vpWApdU1QNVdSuwATh6Hu2TJM3TfMboX5/khn5oZ/++7DDg9oF5NvVlkqQxmWvQfxh4OnAUsAV4b1+eSeatyVaQZFWSdUnWbdu2bY7NkCTNZE5BX1V3VtVDVfUw8BEeGZ7ZBBwxMOvhwOYp1nF+Va2oqhVLly6dSzMkSSOYU9AnOWTg6cnAxBk5a4BTk+yV5KnAcuCa+TVRkjQfM17rJsnFwDHAkiSbgLcBxyQ5im5YZiPwWoCqWp/kUuAm4EHg7Kp6aHGaLkkaxYxBX1WnTVL8sWnmfyfwzvk0SpK0cPxmrCQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxMwZ9kguSbE1y40DZAUkuT/Kt/n7/vjxJPpBkQ5Ibkjx/MRsvSZrZKHv0FwInDJWtBq6oquXAFf1zgJcCy/vbKuDDC9NMSdJczRj0VXUlcNdQ8Urgov7xRcBJA+Ufr87VwH5JDlmoxkqSZm+uY/QHV9UWgP7+oL78MOD2gfk29WWPkmRVknVJ1m3btm2OzZAkzWShD8ZmkrKabMaqOr+qVlTViqVLly5wMyRJE+Ya9HdODMn091v78k3AEQPzHQ5snnvzJEnzNdegXwOc3j8+HfjCQPmr+7NvXgjcMzHEI0kajz1nmiHJxcAxwJIkm4C3AecClyY5C7gNOKWffS1wIrAB+BFw5iK0WZI0CzMGfVWdNsWkYyeZt4Cz59soSdLC8ZuxktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS42a81o2k3cuy1ZeNre6N575sbHW3zD16SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY3zEgi7oHF+RV3Srsc9eklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJaty8vhmbZCNwH/AQ8GBVrUhyAPAZYBmwEfitqvrB/JopSZqrhdijf3FVHVVVK/rnq4Erqmo5cEX/XJI0JosxdLMSuKh/fBFw0iLUIUka0XyDvoC/T3JdklV92cFVtQWgvz9osgWTrEqyLsm6bdu2zbMZkqSpzPfqlS+qqs1JDgIuT3LLqAtW1fnA+QArVqyoebZDkjSFee3RV9Xm/n4r8DngaODOJIcA9Pdb59tISdLczTnok+yTZN+Jx8CvAzcCa4DT+9lOB74w30ZKkuZuPkM3BwOfSzKxnk9X1ZeSXAtcmuQs4DbglPk3U5I0V3MO+qr6NvC8Scq/Dxw7n0ZJkhaO34yVpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcfP9zVhJWjDLVl82lno3nvuysdS7o7hHL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4zyPfh7Gdc6vJM2Ge/SS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxi3aF6aSnAD8ObAH8NGqOnex6pKk+Rjnlx93xI+eLErQJ9kD+EvgJcAm4Noka6rqpoWuy2+nStL0Fmvo5mhgQ1V9u6p+ClwCrFykuiRJ01isoZvDgNsHnm8C/s3gDElWAav6p/cn+eYitWW+lgDfG3cjdhL2xfbsj+3ZH9sbqT/ynnnV8ZRRZlqsoM8kZbXdk6rzgfMXqf4Fk2RdVa0Ydzt2BvbF9uyP7dkf29uZ+mOxhm42AUcMPD8c2LxIdUmSprFYQX8tsDzJU5M8DjgVWLNIdUmSprEoQzdV9WCS1wNfpju98oKqWr8Yde0AO/3w0g5kX2zP/tie/bG9naY/UlUzzyVJ2mX5zVhJapxBL0mNM+h7SU5Jsj7Jw0lWDE17S5INSb6Z5PiB8hP6sg1JVu/4Vu8YSc5J8t0k1/e3EwemTdo3rdtd/vZTSbIxyTf618O6vuyAJJcn+VZ/v/+427lYklyQZGuSGwfKJt3+dD7Qv1ZuSPL8Hd1eg/4RNwL/EbhysDDJkXRnDT0bOAH4UJI9Bi7z8FLgSOC0ft5Wva+qjupva2HqvhlnI3eE3fBvP5UX96+HiR2j1cAVVbUcuKJ/3qoL6V7zg6ba/pcCy/vbKuDDO6iNP2fQ96rq5qqa7Nu5K4FLquqBqroV2EB3iQcv8zB137TOv/3kVgIX9Y8vAk4aY1sWVVVdCdw1VDzV9q8EPl6dq4H9khyyY1raMehnNtnlHA6bprxVr+8/dl4w8JF8d+uDCbvrdg8q4O+TXNdfzgTg4KraAtDfHzS21o3HVNs/9tfLol2meGeU5B+AJ00y6Q+q6gtTLTZJWTH5m+Que67qdH1D91HzHXTb9w7gvcDvMMKlLhq1u273oBdV1eYkBwGXJ7ll3A3aiY399bJbBX1VHTeHxaa7nEMzl3kYtW+SfAT4Yv90d73Uxe663T9XVZv7+61JPkc3nHVnkkOqaks/NLF1rI3c8aba/rG/Xhy6mdka4NQkeyV5Kt0BlWvYjS7zMDSeeDLdgWuYum9at9v87SeTZJ8k+048Bn6d7jWxBji9n+10YKpPya2aavvXAK/uz755IXDPxBDPjrJb7dFPJ8nJwAeBpcBlSa6vquOran2SS4GbgAeBs6vqoX6ZVi7zMJM/TXIU3cfNjcBrAabrm5Y1domPuTgY+FwS6DLk01X1pSTXApcmOQu4DThljG1cVEkuBo4BliTZBLwNOJfJt38tcCLdyQo/As7c4e31EgiS1DaHbiSpcQa9JDXOoJekxhn0ktQ4g16SGufpldIkkhxId2Eq6L4x/BCwrX9+dH+NG2mX4OmV0gySnAPcX1V/Nu62SHPh0I0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnKdXSlLj3KOXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalx/x/7pOuuKc6MRwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(testStats)\n",
    "plt.title('Est. distribution of test statistic (N = %d)' % N)\n",
    "plt.xlabel('T')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute p-value\n",
    "Now that you have generated 1000 estimates of the test statistic with permuted data labels, you can compute the p-value for $T_{obs}$. As a reminder, the p-value is a measure of how likely it is to get a test statistic _at least_ as extreme as what you observed. Thus, $p$ can be computed as:\n",
    "\n",
    "$$ p = \\frac{\\text{# of permutations where } T_i \\geq T_{obs}}{\\text{total # of permutations}} $$\n",
    "\n",
    "Visually, that would look like the probability that you would observe a test statistic that falls to the right of the red line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEWCAYAAABollyxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGXZJREFUeJzt3XvwXGWd5/H3R5DLcJFgfonkovESnQEHoxUZHGdqMgUKBmcSdgqFUYnIEK2CmtVlVyOuijBAsLzPrswGzRIEQUZFMpOsDoOrFOUwkFApJFyWDEQIicmPS0giiCT57h/P0+Sk07df/7p/nTz5vKq6uvs5l+c5p8/59OnnnO5WRGBmZuV62aAbYGZm/eWgNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIO+A5I+LOmOyvNtkl7Xo3lfJOlb+fE0SSHpwB7N+9W5rQf0Yn4jqHeipNslbZX05bGseyyNZv1K+gdJn+2y3iskfbybaa17klZLmtXH+R8v6Rf9mPc+FfSS1kp6Pu9ctdv/6GC6kPSGXrUjIg6PiEfa1DlL0roO5nV5RPxNL9qV18/JlXk/ltu6oxfzH4H5wJPAkRFxYf1ASddI+rvRVjKaN8Zupu12/dYfKORpPxYRl3bR7iHgbOB/5ecHSfp+blvUB5GSKyU9lW9flKTK8BmSVkp6Lt/P6HTafUn9a9fB+HtsoxFxXET8rOeN2zX/e4HNkv6i1/Pep4I++4u8c9VuFwy6Qd3q1ZH7Xug1wP3hb+P1w4eB5RHxfKXsDuCDwK8bjD8fmAu8BTgeeC/wUUhvEsAtwHXAOGAJcEsubzltPxW8X3TievqxjiNin7kBa4GTmwx7A/Bz4FnS0eT3cvntQAC/AbYB7++gnlcCS4EtwF3ApcAdleEBvCE/ng3cD2wFngD+K3AY8DywM9e5DZgEXAx8n7RjbQH+Jpddl+c1Lc97PrAe2ABcWKn3GuDvKs9nAevy4+/k+p7P9X2yMr8D8ziT8nI9DawBzqvM62LgJuDavCyrgZkt1tEfA3fn9X038MeVNr4I/C634+S66ebXDf+nStt+AAwDjwJ/W5nmBGBFXmcbga/k8sfy8tXW8TsatLPjaYHXAz8FniJtQ9cDR41g/X4YeCSvv0eBDwB/APwW2JGn29zktZwDrMrt/A/g1Cbr/afAB5sMWwfMqiv7BTC/8vxc4M78+N2kbVaV4Y/V6m41bYO6Z+X6L8rrbi3wgcrwg4Ev5flvBP4BOLRu2k+R3qy+Uyn7JLCJtC/MJe1v/4+0DV/U7b6Ry/8x1/csKSeOa7ONriVvz3l5vkbaT9fnxwfXLc+FlbafU2nbHplRGTY5t/PgnmZnL2fW7xutg/4G4DOkTymHAH9SGfZSMHdYz42k0DsMeHN+MZoF/QbgT/PjccDb6je0ynQX5w1obm7noTQO+hty3X9ICr7axtV0Y260ftgziH4OfDOvnxl53idV2vbbvBEeAFxB8536aOAZ4EPAgcBZ+fkrG7WzwfT1y/EyYCXwOeAg4HWkwDwlD/834EP58eHAiY2Wr0ldHU9LOlh4F2knHiLt/F/rZP3m12sL8KY87Bh2BceHqWw/9euA9Gb0bK77ZaSd/febLM8w8PYmwxoF/bPAH1WezwS25sefAP5P3fj/TD64aDVtg7pnAduBr+T192ekg6va+vga6SDjaOAI4J+AK+qmvTJPe2il7HPAy4Hz8rJ/N09/HGl7fV03+0Yu+0ieVy20VzXbRuvnAVwC3AlMyNvKL4BL65bnktz22cBzwLhWmVGpZwtwfKd51cltX+y6+ZGkzZXbebn8RVKXwaSI+G1E3NFiHk3lE2t/BXwuIn4TEfeRPtI28yJwrKQjI+KZiLinTRX/FhE/ioidsfvH76ov5Lp/CfxvUpCOiqSpwJ8An8rrZxXwLVJY19wREcsj9Tl/h/SRvZHTgIcj4jsRsT0ibgAeBLrtW3w7MBQRl0TE7yKd/7gaODMPfxF4g6TxEbEtIu4cwbw7njYi1kTErRHxQkQMk0Lrz0ZQ107gzZIOjYgNEbG6w+nOBRbnundGxBMR8WCTcY8iHQl26nBSYNc8Cxye+9rrh9WGH9HBtM18Nq+/nwPLgPfl8c8DPhERT0fEVuBydr2+kNbd5/O0tf3iReCyiHiRdPA1Hvh6RGzN63Y1qUupKxGxOM/rBdKBzlskvaLDyT8AXBIRm/K28gV235dezMNfjIjlpE8Gb6oMa5UZW0mvc8/si0E/NyKOqtyuzuWfBATclc+Of6TL+Q+RjtAer5T9qsX4f0V6x/6VpJ9Lekeb+T/eZnj9OL8idWuM1iSgtpNV5z258rzax/sccEiT/tJJ7LlO6uc1Eq8BJlXfwEldABPz8HOBNwIPSrpb0ntHMO+Op5U0QdKNkp6QtIXUxTa+k0oi4jfA+4GPARskLZP0+x22cSqpu6YTz7AriDuxDTiy8vxIYFukQ8f6YbXhWzuYtmHb8nqoqW27Q8DvASsrr++Pc3nNcET8tm5+T8WuE9218N9YGf486c1oxCQdIGmhpP/Ir/XaPKij15s994H6/fSpiNheef5cpa3tMuMIYHOH7ejIvhj0DUXEryPivIiYRDqZ8c0ur7QZJn3smlope3WLeu+OiDmkj3A/InX5QPpI33CSDtpQX/f6/Pg3pB2m5lUjmPd64GhJ1ZB4NalbaqTWk8K5aiTzqm/n48CjdW/gR0TEbICIeDgiziKt4yuB70s6rMF89qxoZNNekcuPj4gjSSc4q0evLeuLiJ9ExLtI3TYPkj6VtJ2OtPyvb7cs2b2kN65OrWb3T2ZvyWW1YcfXHaEfXze82bSNjMvrtqa27T5JCuXjKq/vKyKiGtKd7BetjHTf+GvSeZGTgVeQuuFg1+vdrj31+0B1P22pRWYgaRKp+/KhTubVqWKCXtIZkqbkp8+QXqja0cBGUr9vW/kI4ofAxZJ+T9KxwLwmdR4k6QOSXpE/Xm6pq/OVI/goWPXZXPdxwDnA93L5KmC2pKMlvQqov5a66XJGxOOkfsQrJB0i6XjS0e71XbRvOfBGSX8t6UBJ7weOJfXvdqK+nXcBWyR9StKh+WjrzZLeDiDpg5KGImInu450dpDelHfS4rUd4bRHkE+YSpoM/Lc27a7WM1HSX+ageyHPp7otTKlczVLv28A5kk6S9DJJk1t8GlhOXXeSpIMlHZKfHpRf31pgXQv8lzzPSaQThNfkYT/LbfzbPI/aFWw/7WDaZr6Q94s/JV2l84953V8NfFXShNzmyZJOaTOvkRjpvnEE6XV6ivQGcXmb8evdAPx3SUOSxpPOJVzXrpFtMgNS//5Pc3dS7/Syw7/fN9LHq9qZ89rt5jzsi6Qjym2kj8HVqwU+RjoBshl4H+nddxvw6ib1DJFCq+VVN6R33h+T3li2kK4+qZ4EXkzakDaz66qb6+rqeqmMPa+6+TX5CoE8/BBS6G8hHdl9gt1POM0hXdWwmXT1T21+tZOxU/JyPZ3X0ccataOuLQ1PdJL6+1eS+m1X1i33NbQ+GTudtGNuBn6UyyaRdp5f5/V5J7tOfF1HunphG+mIcm5lXpeQQnsz+URrXV0dT0s6wbcyj7uKFGwdrV/SUXztqq/NpBA9Nk93EKm/+mngyUbrCDg9v6ZbSVdEndJk3Y0nnXQ9tG6/iLrbtDxMpH3j6Xz7IrtfZfPWvMzPA/cAb60MazltXbtm5XZ9hnQE/xj5JHhl272cdJJ9C/AA+coqGl+4sFtZXscvLVcuu4N8BRIj3zcOJ11aupXU7XI2u19k0WgbXcuubfIQ4BukXNmQHx/SYnnWkj49tMuMZcBf9jo7lWduZvsISZcDmyLia4NuS43SF7Wui4gp7ca1xiT9IbAoItqd5xv5vB30ZjZaDvq9WzF99GZm1piP6M3MCucjejOzwu0VPx40fvz4mDZt2qCbYWaD8FC+ZPxNb2o9nu1h5cqVT0bEULvx9oqgnzZtGitWrBh0M8xsEGbNSvc/+9kgW7FPktTqW/svcdeNmVnhHPRmZoVz0JuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWOAe9mVnh2n4zNv+p9LWkv+baSfq95K9Luphd/8wOcFGkP8FF0qdJ/160g/TnAj/pQ9vNxsS0BcsGUu/ahacNpF4rTyc/gbAduDAi7sn/N7pS0q152Fcj4kvVkfNf751J+reeScC/Snpj7PqTXzMzG0Ntu24iYkNE3JMfbyX9BdjkFpPMAW6MiBci4lHS36Kd0IvGmpnZyI2oj17SNNJ/TP57LrpA0r2SFksal8smk/7VvmYdDd4YJM2XtELSiuHh4frBZmbWIx0HvaTDgR8AH4+ILcBVwOuBGaQ/x/1ybdQGk+/x7yYRsSgiZkbEzKGhtr+yaWZmXeoo6CW9nBTy10fEDwEiYmNE7IiIncDV7OqeWQdMrUw+BVjfuyabmdlItA16SQK+DTwQEV+plB9TGe104L78eClwpqSDJb0WmA7c1bsmm5nZSHRy1c07gQ8Bv5S0KpddBJwlaQapW2Yt8FGAiFgt6SbgftIVO+f7ihszs8FpG/QRcQeN+92Xt5jmMuCyUbTLzMx6xN+MNTMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrXCd/JWg2cNMWLBt0E8z2WT6iNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwbYNe0lRJ/1fSA5JWS/rPufxoSbdKejjfj8vlkvQNSWsk3Svpbf1eCDMza66TI/rtwIUR8QfAicD5ko4FFgC3RcR04Lb8HOA9wPR8mw9c1fNWm5lZx9oGfURsiIh78uOtwAPAZGAOsCSPtgSYmx/PAa6N5E7gKEnH9LzlZmbWkRH10UuaBrwV+HdgYkRsgPRmAEzIo00GHq9Mti6X1c9rvqQVklYMDw+PvOVmZtaRjoNe0uHAD4CPR8SWVqM2KIs9CiIWRcTMiJg5NDTUaTPMzGyEOgp6SS8nhfz1EfHDXLyx1iWT7zfl8nXA1MrkU4D1vWmumZmNVCdX3Qj4NvBARHylMmgpMC8/ngfcUik/O199cyLwbK2Lx8zMxt6BHYzzTuBDwC8lrcplFwELgZsknQs8BpyRhy0HZgNrgOeAc3raYjMzG5G2QR8Rd9C43x3gpAbjB3D+KNtlZmY94m/GmpkVzkFvZlY4B72ZWeEc9GZmhXPQm5kVzkFvZlY4B72ZWeEc9GZmhXPQm5kVzkFvZlY4B72ZWeEc9GZmhXPQm5kVzkFvZla4Tn6P3swGYNqCZQOpd+3C0wZSr/WPj+jNzArnoDczK5yD3syscA56M7PCOejNzArnoDczK5yD3syscA56M7PCOejNzArnoDczK5yD3syscP6tGxuRQf3+ipl1z0f0ZmaFc9CbmRXOQW9mVjgHvZlZ4doGvaTFkjZJuq9SdrGkJyStyrfZlWGflrRG0kOSTulXw83MrDOdHNFfA5zaoPyrETEj35YDSDoWOBM4Lk/zTUkH9KqxZmY2cm2DPiJuB57ucH5zgBsj4oWIeBRYA5wwivaZmdkojaaP/gJJ9+aunXG5bDLweGWcdbnMzMwGpNugvwp4PTAD2AB8OZerwbjRaAaS5ktaIWnF8PBwl80wM7N2ugr6iNgYETsiYidwNbu6Z9YBUyujTgHWN5nHooiYGREzh4aGummGmZl1oKugl3RM5enpQO2KnKXAmZIOlvRaYDpw1+iaaGZmo9H2t24k3QDMAsZLWgd8HpglaQapW2Yt8FGAiFgt6SbgfmA7cH5E7OhP083MrBNtgz4izmpQ/O0W418GXDaaRpmZWe/4m7FmZoVz0JuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWOAe9mVnh2ga9pMWSNkm6r1J2tKRbJT2c78flckn6hqQ1ku6V9LZ+Nt7MzNrr5Ij+GuDUurIFwG0RMR24LT8HeA8wPd/mA1f1pplmZtattkEfEbcDT9cVzwGW5MdLgLmV8msjuRM4StIxvWqsmZmNXLd99BMjYgNAvp+QyycDj1fGW5fL9iBpvqQVklYMDw932QwzM2un1ydj1aAsGo0YEYsiYmZEzBwaGupxM8zMrKbboN9Y65LJ95ty+TpgamW8KcD67ptnZmaj1W3QLwXm5cfzgFsq5Wfnq29OBJ6tdfGYmdlgHNhuBEk3ALOA8ZLWAZ8HFgI3SToXeAw4I4++HJgNrAGeA87pQ5vNzGwE2gZ9RJzVZNBJDcYN4PzRNsrMzHrH34w1Myucg97MrHAOejOzwjnozcwK56A3Myucg97MrHAOejOzwjnozcwK56A3Myucg97MrHAOejOzwrX9rRsz279MW7BsTOu78ZGnADhzwTLWLjxtTOveX/iI3syscA56M7PCOejNzArnoDczK5yD3syscA56M7PCOejNzArnoDczK5yD3syscA56M7PC+ScQ9kFj/RV1M9u3+YjezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwo/pmrKS1wFZgB7A9ImZKOhr4HjANWAu8LyKeGV0zzcysW704ov/ziJgRETPz8wXAbRExHbgtPzczswHpR9fNHGBJfrwEmNuHOszMrEOjDfoA/kXSSknzc9nEiNgAkO8nNJpQ0nxJKyStGB4eHmUzzMysmdH+euU7I2K9pAnArZIe7HTCiFgELAKYOXNmjLIdZmbWxKiO6CNifb7fBNwMnABslHQMQL7fNNpGmplZ97oOekmHSTqi9hh4N3AfsBSYl0ebB9wy2kaamVn3RtN1MxG4WVJtPt+NiB9Luhu4SdK5wGPAGaNvppmZdavroI+IR4C3NCh/CjhpNI0yM7Pe8TdjzcwK56A3Myucg97MrHAOejOzwjnozcwK56A3Myucg97MrHAOejOzwjnozcwK56A3Myucg97MrHAOejOzwjnozcwK56A3Myucg97MrHCj/c9YM7OembZg2UDqXbvwtIHUO1Z8RG9mVjgHvZlZ4Rz0ZmaFc9CbmRXOQW9mVjgHvZlZ4Rz0ZmaF83X0ozCoa37NzEbCR/RmZoVz0JuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWOAe9mVnh+vaFKUmnAl8HDgC+FREL+1WXmdloDPLLj2Pxpyd9CXpJBwD/E3gXsA64W9LSiLi/13X526lmZq31q+vmBGBNRDwSEb8DbgTm9KkuMzNroV9dN5OBxyvP1wF/VB1B0nxgfn66TdJDfWrLaI0Hnhx0I/YSXhe78/rYXVfr4x21B1e+t6eN2Qt0tD505ajqeE0nI/Ur6NWgLHZ7ErEIWNSn+ntG0oqImDnoduwNvC525/WxO6+P3e1N66NfXTfrgKmV51OA9X2qy8zMWuhX0N8NTJf0WkkHAWcCS/tUl5mZtdCXrpuI2C7pAuAnpMsrF0fE6n7UNQb2+u6lMeR1sTuvj915fexur1kfioj2Y5mZ2T7L34w1Myucg97MrHAO+kzSGZJWS9opaWbdsE9LWiPpIUmnVMpPzWVrJC0Y+1aPDUkXS3pC0qp8m10Z1nDdlG5/ee2bkbRW0i/z9rAilx0t6VZJD+f7cYNuZ79IWixpk6T7KmUNl1/JN/K2cq+kt411ex30u9wH/Cfg9mqhpGNJVw0dB5wKfFPSAZWfeXgPcCxwVh63VF+NiBn5thyar5tBNnIs7IevfTN/nreH2oHRAuC2iJgO3Jafl+oa0jZf1Wz53wNMz7f5wFVj1MaXOOiziHggIhp9O3cOcGNEvBARjwJrSD/x4J95aL5uSufXvrE5wJL8eAkwd4Bt6auIuB14uq642fLPAa6N5E7gKEnHjE1LEwd9e41+zmFyi/JSXZA/di6ufCTf39ZBzf663FUB/IuklfnnTAAmRsQGgHw/YWCtG4xmyz/w7aVvP1O8N5L0r8CrGgz6TETc0myyBmVB4zfJffZa1VbrhvRR81LS8l0KfBn4CB381EWh9tflrnpnRKyXNAG4VdKDg27QXmzg28t+FfQRcXIXk7X6OYdifuah03Uj6Wrgn/PT/fWnLvbX5X5JRKzP95sk3Uzqztoo6ZiI2JC7JjYNtJFjr9nyD3x7cddNe0uBMyUdLOm1pBMqd7Ef/cxDXX/i6aQT19B83ZRuv3ntG5F0mKQjao+Bd5O2iaXAvDzaPKDZp+RSNVv+pcDZ+eqbE4Fna108Y2W/OqJvRdLpwN8DQ8AySasi4pSIWC3pJuB+YDtwfkTsyNOU8jMP7XxR0gzSx821wEcBWq2bkhX2Ex/dmAjcLAlShnw3In4s6W7gJknnAo8BZwywjX0l6QZgFjBe0jrg88BCGi//cmA26WKF54Bzxry9/gkEM7OyuevGzKxwDnozs8I56M3MCuegNzMrnIPezKxwvrzSrAFJryT9MBWkbwzvAIbz8xPyb9yY7RN8eaVZG5IuBrZFxJcG3RazbrjrxsyscA56M7PCOejNzArnoDczK5yD3syscA56M7PC+fJKM7PC+YjezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCvf/AaaJ4poOWJvPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(testStats)\n",
    "plt.title('Est. distribution of test statistic (%d permutations)' % N)\n",
    "plt.xlabel('T')\n",
    "plt.axvline(tObs, color = 'r')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "p = 0.035\n"
     ]
    }
   ],
   "source": [
    "p = sum(testStats>=tObs)/N\n",
    "print('p = %.3g' % p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "A significance threshold at $\\alpha=0.05$ is popularly used for most statistical analyses. When $p<0.05$, this means that there is a less than 5% chance that you would observe a test statistic at least as extreme as $T_{obs}$ under the null distribution. In this case, how should we interpret the results?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bootstrapping\n",
    "\n",
    "Let's say we want to know the true difference in the mean survival post-diagnosis between treated and untreated patients. Given the same data used above, we can use bootstrapping - or sampling with replacement - to estimate a confidence interval between which this true mean lies.\n",
    "\n",
    "To do this, we will sample from each group with replacement and compute the difference in the means of the resampled groups. Each resampled group should be the same size as the original group. We will repeat this process $N = 1000$ times (as above)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_differences = []\n",
    "\n",
    "for i in range(N):\n",
    "    sample_untreated = np.random.choice(untreated, size = untreated.shape)\n",
    "    sample_treated = np.random.choice(treated, size = treated.shape)\n",
    "    \n",
    "    bootstrap_diff = np.mean(sample_treated) - np.mean(sample_untreated)\n",
    "    mean_differences.append(bootstrap_diff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get confidence interval - 95%\n",
    "\n",
    "To do this, we will set the bounds of the CI at the 2.5th percentile and 97.5th percentile. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "95% CI bounds: [-6.68900, 126.96430]\n"
     ]
    }
   ],
   "source": [
    "ci = np.quantile(mean_differences, [0.025, 0.975])\n",
    "print('95%% CI bounds: [%.5f, %.5f]' % (ci[0], ci[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the bounds of the CI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGJ9JREFUeJzt3X20HHWd5/H3h4eoAXcACQwiGHSCDu4qYtYTh4mL4zOzGlwfwB2VdZgJZxd0nNHdwYdRdFZHd33YxVEURgRZVEBljBwGRQYkukRIEHkQGbOIEuFAXE1EM8sl4bt/VF1pL3XvbW5ud9/kvl/n9OmqX1dXfbtu3/50/aqrKlWFJEkT7TLqAiRJc5MBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE4GhCSp026jLmB77LvvvrV48eJRl7HzuvXW5v5JTxptHfPcbRt/BcATFu0x4krmoZ30f2DdunU/rapF0023QwfE4sWLWbt27ajL2HkddVRzf+WVo6xi3jv2k1cDcP6JzxpxJfPQTvo/kORH/UxnF5MkqZMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE4GhCSp0w59JLU0l23ZsoWxsbHtns/WrVsB2LRpU1/TL1iwgIULF273ciUDQhqQsbExVq9evd3z2bw5AH3Pa/ny5QaEZoVdTJKkTgaEJKmTASFJ6mRASJI6DSwgkhyU5IoktyS5Ocmfte2nJvlJkuvb29E9z3lrkvVJbk3ywkHVJkma3iB/xbQVeHNVXZfk0cC6JJe1j32kqj7YO3GSw4DjgKcAjwW+nuTQqto2wBolSZMY2BZEVd1VVde1w/cCtwAHTvGUFcDnq+q+qvohsB545qDqkyRNbSj7IJIsBp4OfLttOjnJDUnOSrJ323YgcEfP0zbQEShJViZZm2Ttxo0bB1i1JM1vAw+IJHsCXwTeVFW/AE4HnggcDtwFfGh80o6n10Maqs6oqqVVtXTRommvuS1JmqGBBkSS3WnC4byq+hJAVd1dVduq6gHgTB7sRtoAHNTz9McBdw6yPknS5Ab5K6YAnwJuqaoP97Qf0DPZy4Cb2uFVwHFJHpHkEGAJcM2g6pMkTW2Qv2I6EngtcGOS69u2twGvTnI4TffR7cCJAFV1c5ILgO/R/ALqJH/BJEmjM7CAqKpv0r1f4ZIpnvNe4L2DqkmS1D+PpJYkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUqfdRl2ApNm3adOmoS9zwYIFLFy4cOjL1eAYENJO5v7772fNmjVDX+7y5csNiJ2MXUySpE4DC4gkByW5IsktSW5O8mdt+z5JLkvyg/Z+77Y9SU5Lsj7JDUmOGFRtkqTpDXILYivw5qr6XWAZcFKSw4BTgMuraglweTsO8GJgSXtbCZw+wNokSdMYWEBU1V1VdV07fC9wC3AgsAI4p53sHOCYdngF8JlqrAH2SnLAoOqTJE1tKPsgkiwGng58G9i/qu6CJkSA/drJDgTu6HnahrZNkjQCAw+IJHsCXwTeVFW/mGrSjrbqmN/KJGuTrN24ceNslSlJmmCgAZFkd5pwOK+qvtQ23z3eddTe39O2bwAO6nn644A7J86zqs6oqqVVtXTRokWDK16S5rlB/oopwKeAW6rqwz0PrQKOb4ePB77c0/669tdMy4DN411RkqThG+SBckcCrwVuTHJ92/Y24P3ABUlOAH4MvLJ97BLgaGA9sAV4/QBrkyRNY2ABUVXfpHu/AsBzO6Yv4KRB1SNJeng8klqS1MmAkCR1MiAkSZ0MCElSJ0/3rZ3ali1bGBsbG8myt23bNpLlSrPFgNBObWxsjNWrV49k2cuWLRvJcqXZYheTJKmTASFJ6mRASJI6GRCSpE4GhCSpkwEhSepkQEiSOhkQkqROBoQkqZMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6TRsQSfYZRiGSpLmlny2Ibye5MMnRSTLwiiRJc0I/AXEocAbwWmB9kvclOXSwZUmSRm3agKjGZVX1auBPgOOBa5J8I8mzBl6hJGkkpr0mdZLHAK+h2YK4G3gDsAo4HLgQOGSQBUqSRmPagACuBs4FjqmqDT3ta5N8YjBlSZJGrZ+AeFJVVdcDVfWBWa5HkjRH9LOT+mtJ9hofSbJ3kq8OsCZJ0hzQT0AsqqpN4yNV9XNgv8GVJEmaC/oJiG1JDh4fSfJ4oLPLSZK08+gnIN4OfDPJuUnOBa4C3jrdk5KcleSeJDf1tJ2a5CdJrm9vR/c89tYk65PcmuSFM3kxkqTZM+1O6qq6NMkRwDIgwJ9X1U/7mPfZwN8Cn5nQ/pGq+mBvQ5LDgOOApwCPBb6e5NCq2tbHciRJA9DvyfoeAfwM2AwcluTZ0z2hqq5qn9OPFcDnq+q+qvohsB54Zp/PlSQNQD8Hyn0AOBa4GXigbS6arqaZODnJ64C1wJvbnd4HAmt6ptnQtnXVsxJYCXDwwQd3TSJJmgX9bEEcQ3MsxB9W1Uva20tnuLzTgSfSHIV9F/Chtr3rJICTHXtxRlUtraqlixYtmmEZkqTp9BMQtwG7z8bCquruqtpWVQ8AZ/JgN9IG4KCeSR8H3Dkby5QkzUw/R1JvAa5Pcjlw33hjVb3x4S4syQFVdVc7+jJg/BdOq4DPJvkwzU7qJcA1D3f+kqTZ009ArGpvD0uSzwFHAfsm2QC8CzgqyeE03Ue3AycCVNXNSS4AvgdsBU7yF0ySNFr9/Mz1nCSPAg6uqlv7nXF7evCJPjXF9O8F3tvv/CVJg9XPJUdfAlwPXNqOH57kYW9RSJJ2LP3spD6VZmfyJoCquh6vASFJO71+AmJrVW2e0Oa5mCRpJ9fPTuqbkvx7YNckS4A3Av97sGVJkkatny2IN9CcI+k+4HPAL4A3DbIoSdLo9fMrpi00Z3R9++DLkSTNFf2ci+kKOvY5VNUfDKQiSdKc0M8+iLf0DD8SeDnNwWySpJ1YP11M6yY0fSvJNwZUjyRpjuini2mfntFdgGcAvz2wiiRJc0I/XUzraPZBhKZr6YfACYMsSpI0ev10MXnUtCTNQ/10Mf27qR6vqi/NXjmSdmSbNm0ayXIXLFjAwoULR7LsnVk/XUwnAL8H/GM7/hzgSprrUxdgQEji/vvvZ82aNdNPOADLly83IAagn4Ao4LDxC/0kOQD4WFW9fqCVSZJGqp9TbSzuuQocwN3AoQOqR5I0R/SzBXFlkq/SnIepgOOAKwZalSRp5Pr5FdPJSV4GPLttOqOqLhpsWZKkUetnCwLgOuDeqvp6koVJHl1V9w6yMEnSaPVzydE/Bb4AfLJtOhD4+0EWJUkavX52Up8EHElzHQiq6gfAfoMsSpI0ev0ExH1VNTY+kmQ3vOSoJO30+gmIbyR5G/CoJM8HLgS+MtiyJEmj1k9AnAJsBG4ETgQuAd4xyKIkSaM35a+YkuwKnFNVrwHOHE5JkqS5YMotiKraBixKsmBI9UiS5oh+joO4neYqcquAX403VtWHB1WUJGn0Jt2CSHJuO3gscHE77aN7bpKkndhUWxDPSPJ44MfAR4dUjyRpjpgqID4BXAocAqztaQ/NcRBPGGBdkqQRm7SLqapOq6rfBT5dVU/ouR1SVdOGQ5KzktyT5Kaetn2SXJbkB+393m17kpyWZH2SG5IcMSuvTpI0Y9MeB1FV/3GG8z4beNGEtlOAy6tqCXB5Ow7wYmBJe1sJnD7DZUqSZkk/B8rNSFVdBfxsQvMK4Jx2+BzgmJ72z1RjDbBXe+U6SdKIDCwgJrH/+NXp2vvxk/4dCNzRM92Gtk2SNCLDDojJpKOt84SASVYmWZtk7caNGwdcliTNX8MOiLvHu47a+3va9g3AQT3TPQ64s2sGVXVGVS2tqqWLFi0aaLGSNJ8NOyBWAce3w8cDX+5pf137a6ZlwObxrihJ0mj0e8nRhy3J54CjgH2TbADeBbwfuCDJCTQH4L2ynfwS4GhgPbAFeP2g6pIk9WdgAVFVr57koed2TFs0V66TJM0RAwsIqdeWLVsYGxubfsJZtm3btqEvU9pZGBAairGxMVavXj305S5btmzoy5R2FnPlZ66SpDnGgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUabdRLDTJ7cC9wDZga1UtTbIPcD6wGLgdeFVV/XwU9UmSRrsF8ZyqOryqlrbjpwCXV9US4PJ2XJI0InOpi2kFcE47fA5wzAhrkaR5b1QBUcDXkqxLsrJt27+q7gJo7/fremKSlUnWJlm7cePGIZUrSfPPSPZBAEdW1Z1J9gMuS/L9fp9YVWcAZwAsXbq0BlWgJM13I9mCqKo72/t7gIuAZwJ3JzkAoL2/ZxS1SZIaQ9+CSLIHsEtV3dsOvwB4D7AKOB54f3v/5WHXJmnHtWnTplmf555btwLwy0nmvWDBAhYuXDjry50rRtHFtD9wUZLx5X+2qi5Nci1wQZITgB8DrxxBbZJ2QPfffz9r1qyZ9fk+a/NmAK5evbrz8eXLlxsQs6mqbgOe1tH+f4HnDrseSVK3Ue2k1ghs2bKFsbGxvqefbvP64di2bdt2z0PScBkQ88jY2BirJ9lU7jLd5vXDsWzZsu2eh6ThmksHykmS5hADQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHXyehCStB0GcS3sfgzjetgGhCTN0KCuhd2PYVwP2y4mSVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUiePpB6BLVu2MDY2NvTlbtu2bejLlLTjMiBGYGxsjNWrVw99ucuWLRv6MiXtuOZcF1OSFyW5Ncn6JKeMuh5Jmq/m1BZEkl2BjwHPBzYA1yZZVVXfm+1ljaqbB+zqkbRjmFMBATwTWF9VtwEk+TywApj1gBhVNw/Y1SNpxzDXupgOBO7oGd/QtkmShmyubUGko61+Y4JkJbCyHf1lklsHXhXsC/x0CMuZm1760ummmN/rpz/bvY4un6VC5qi5/R6a/n9g0GZ7/Ty+n4nmWkBsAA7qGX8ccGfvBFV1BnDGMItKsraqlg5zmTsS18/0XEdTc/1MbVTrZ651MV0LLElySJIFwHHAqhHXJEnz0pzagqiqrUlOBr4K7AqcVVU3j7gsSZqX5lRAAFTVJcAlo65jgqF2ae2AXD/Tcx1NzfUztZGsn1TV9FNJkuadubYPQpI0RxgQU0hyapKfJLm+vR3d89hb29OB3JrkhaOsc5Q8NcpDJbk9yY3te2Zt27ZPksuS/KC933vUdQ5TkrOS3JPkpp62znWSxmnte+qGJEeMrvLhmGT9jPzzx4CY3keq6vD2dglAksNofmH1FOBFwMfb04TMKz2nRnkxcBjw6nbdCJ7TvmfGf5p4CnB5VS2hOaRhvoXp2TT/K70mWycvBpa0t5XA6UOqcZTO5qHrB0b8+WNAzMwK4PNVdV9V/RBYT3OakPnm16dGqaoxYPzUKHqoFcA57fA5wDEjrGXoquoq4GcTmidbJyuAz1RjDbBXkgOGU+loTLJ+JjO0zx8DYnont5u5Z/V0C3hKkIbroVsBX0uyrj3yH2D/qroLoL3fb2TVzR2TrRPfVw8a6efPvA+IJF9PclPHbQXNpu0TgcOBu4APjT+tY1bz8edgroduR1bVETRdJSclefaoC9rB+L5qjPzzZ84dBzFsVfW8fqZLciZwcTs67SlB5gnXQ4equrO9vyfJRTSb/3cnOaCq7mq7S+4ZaZFzw2TrxPcVUFV3jw+P6vNn3m9BTGVCv+fLgPFfGKwCjkvyiCSH0OxMu2bY9c0BnhplgiR7JHn0+DDwApr3zSrg+Hay44Evj6bCOWWydbIKeF37a6ZlwObxrqj5ZC58/sz7LYhp/Lckh9Nsvt0OnAhQVTcnuYDmOhVbgZOqat5dBchTo3TaH7goCTT/X5+tqkuTXAtckOQE4MfAK0dY49Al+RxwFLBvkg3Au4D3071OLgGOptn5ugV4/dALHrJJ1s9Ro/788UhqSVInu5gkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAgNTJLF42enTLI0yWnt8CPaI9ivT3JskuVJbm7HHzXaqrsleU+Svg6qnKt1JLkyyUOua5zkTUkW9oxfkmSv7alzkuUfkOTiSR7rrG07lvXBJH8wW/ObrzwOQkNRVWuBte3o04Hdq+pwgCSfAD5YVZ/uZ15pDjJIVT0wkGI7VNU7h7WsJLtV1dYh1vEm4H/RHHNAVR099eQz9hfAmQOa90QfbZf1j0Na3k7JLYidTPut/ftJ/q49p9R5SZ6X5Fvtefef2U63R3sCsGuTfKc999T481cnua69/V7bflT7Le8L7fzPaz+oJy7/GUm+m+Rq4KSe9qOSXJxkP5oPo8PbLYYTgVcB70xyXjvtf27ruiHJu3vquiXJx4HrgIOSvCDJ1W2dFybZs5329iTvbttvTPLktn3PJJ9u225I8vK2vXM+E17X2UleMdX8J0z/lCTXtK/xhiRLereo2mnekuTUdvjKJO9L8g3g7e0ydmkfW5jkjiS7j9eR5MXtwVK96/cr7fDpSda2W2Xvnub98kbgscAVSa7oeX37bu97qcPLgUvb5zwqyefbdXM+8Ostx676kzw3zWlLxqd5fpIvJdm1XSc3tX+LPweoqh8Bj0ny21O9fk2jqrztRDdgMc3Rlf+K5gvAOuAsmhN8rQD+vp3ufcBr2uG9gH8C9gAWAo9s25cAa9vho4DNNOd92QW4Gvj9juXfAPybdvi/Azf1PP/iicPt+NnAK9rhF9Bcfzftci4Gnt2+rgeAZe10+wJXAXu0438JvLMdvh14Qzv8n4C/a4c/APyPnuXuPdV8Jryu3ho75z9h+o8Cf9QOL6D5AFw8vj7a9rcAp7bDVwIf73nsyzTXlAA4tuc1nA28gmbr/8c9dZ/e8/fcp73ftZ3vU3uWsbSj1tuBfSeOs53vpQnLOARY1zP+FzRH3gM8tV3O0snqb5f5fWBR+9hngZcAzwAu65nvXj3DZwIvH/X/5I58cwti5/TDqrqxmi6Ym2kuylLAjTT/9NB8EJ+S5Hqaf8JHAgcDuwNnJrkRuJDmQkDjrqmqDe18r++ZFwBJfovmH/QbbdO5M6j9Be3tOzRbCk+mCSqAH1VzfQCAZW1t32pfw/HA43vm86X2fl1Pnc+jucARAFX18z7mM5mu+fe6Gnhbkr8EHl9V/9zHPM+fMHxsO3zchMeopgvqUuAlSXYD/pAHz2X0qiTX0azDp/Cbf8OHa3veS70OADb2jD+bZkuSqrqB5ovFuIfU3y7zXOA1afaPPAv4B+A24AlJPprkRcAveuZzD83WkWbIfRA7p/t6hh/oGX+AB//mofl2dWvvE9suj7uBp9F8a/x/k8x3Gw99/4TtP+1wgL+pqk9OqGsx8KsJ011WVa+eZD7jtfbW2VXfdPOZTNf8f62qPpvk2zQf3F9N8ic036x7v5Q9csLTel/fKuBvkuxD8y25qy/9fJpuvJ8B11bVvWlO3vYW4F9X1c+TnN2xnIdjxu+lCf65o46HvFemqf/TwFdo3pMXtiH58yRPA15Isy5eBfxxO/0j2+VqhtyCmL++CrwhafYjJHl62/5bwF3tN8bX0mzm96WqNgGbk/x+2/RHM6zrj3v2JxyYZr/FRGuAI5P8TjvdwiSHTjPvrwEnj4+kuQDLTOYzrSRPAG6rqtNoPuyfShO8+yV5TJJHAP92sudX1S9pztD5P2m647pOxnYlcATwpzy4hfEvaIJmc5L9aa5JMZ17gUf387omMdl7qdc/8ZtbWlfRvj+S/Eua9QNT1F/NadTvBN5B09VGkn2BXarqi8Bf0ayPcYfy4BlQNQMGxPz11zTdSTe0O07/um3/OHB8kjU0/2C/muT5k3k98LE0O6kf9re3qvoaTf/y1W031xfo+PCqqo3AfwA+l+QGmg/6h+wsnuC/Anu3OzS/S9PHP5P59ONY4Ka22+XJNJfQvB94D/Btmn0r359mHucDr2FC99K4NjQupvkQvbht+y5N18zNNPsLvtVHrWcA/zC+k3oGJnsv9db6K+D/jAcxzT6TPdt1/l9oT1fdR/3nAXdU1ffa8QOBK9v1fDbwVoAkuwO/w4O/nNMMeDZXSUOR5GXAM6rqHdsxj78FvlNVn+pjWUdU1V/NdFlyH4SkIamqi5I8ZqbPT7KOZov2zX1MvhsPXqJTM+QWhCSpk/sgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVKn/w+RO1mLIvjpwgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(mean_differences, color = 'grey', alpha = 0.6, edgecolor = 'white')\n",
    "plt.axvline(ci[0], color = 'red')\n",
    "plt.axvline(ci[1], color = 'red')\n",
    "plt.axvline(tObs)\n",
    "plt.xlabel('mean difference in survival time (days)')\n",
    "plt.ylabel('frequency')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}