chap03.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Chapter 3: Probability mass functions](#chapter3)\n",
    "    - [3.1 PMFs](#subchapter3.1)\n",
    "    - [3.2 Plotting PMFs](#subchapter3.2)\n",
    "    - [3.3 Other visualizations](#subchapter3.3)\n",
    "    - [3.4 The class size paradox](#subchapter3.4)\n",
    "    - [3.5 DataFrame indexing](#subchapter3.5)\n",
    "    - [3.6 Exercises](#subchapter3.6)\n",
    "    - [3.7 Glossary](#subchapter3.7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='chapter3'></a>\n",
    "# Chapter 3: Probability mass functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "//anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.\n",
      "  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import math\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import nsfg\n",
    "import first\n",
    "import thinkstats2\n",
    "import thinkplot\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='subchapter3.1'></a>\n",
    "## 3.1 PMFs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to represent a distribution is a Probability mass function, which maps from each value to its probability. A probability is a frequency expressed as a fraction of the sample size, n. To get from frequencies to probabilities, we devide through by n, which is called normalization.\n",
    "\n",
    "Given a hist, we can make a dictionary that maps from each value to its probability: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nn = hist.Total()\\nd = {}\\nfor x, freq in hist.Items():\\n    d[x] = freq / n\\n'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pmf is actually a dictionary that maps a value to its frequency/total values \n",
    "# it is calculated and created this way:\n",
    "'''\n",
    "n = hist.Total()\n",
    "d = {}\n",
    "for x, freq in hist.Items():\n",
    "    d[x] = freq / n\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can use the PMF class provided by thinkstats2. Like Hist, the PMF constructor can take a list, pandas Series, dictionary, Hist, or another PMF object. \n",
    "\n",
    "Here's an example with a simple list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pmf({1: 0.2, 2: 0.4, 3: 0.2, 5: 0.2})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pmf = thinkstats2.Pmf([1,2,2,3,5])\n",
    "pmf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Pmf is normalized so total probability is 1. \n",
    "\n",
    "Pmf and Hist objects are similar in many ways. In fact, they inherit many of their methods from a common parent class. For example, the methods Values and Items work the same for both. The biggest difference is that Hist maps from values to integer counters. A Pmf maps from values to floating point probabilities. \n",
    "\n",
    "To look up the probability associated with the value, use Prob:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What is the probability of 2?\n",
    "pmf.Prob(2)\n",
    "# or:\n",
    "pmf[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can modify an existing Pmf by incrementing the probability associated with a value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pmf({1: 0.2, 2: 0.6000000000000001, 3: 0.2, 5: 0.2})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# increment the probability of 2 by 0.2\n",
    "pmf.Incr(2, 0.2)\n",
    "pmf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6000000000000001"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pmf.Prob(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or you can multiply a probatility by a factor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pmf({1: 0.2, 2: 0.30000000000000004, 3: 0.2, 5: 0.2})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pmf.Mult(2, 0.5)\n",
    "pmf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.30000000000000004"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pmf[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you modify a Pmf, the result may not be normalized. To check, use Total:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8999999999999999"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pmf.Total()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To normalize, call Normalize:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pmf.Normalize()\n",
    "pmf.Total()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='subchapter3.2'></a>\n",
    "## 3.2 Plotting PMFs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two methods used in the book to plot Pmf: \n",
    "- <b>thinkplot.Hist</b> is used to create a bar graph (useful when the number of values in the Pmf is smalle).\n",
    "- <b>thinkplot.Pmf</b> is used to create a step function (useful when there are large numbers of values and the Pmf is smooth). This function also works Hist objects.\n",
    "\n",
    "In addition, <b>pyplot.hist</b> provides a function called hist the takes a sequence of values, computes a histogram, and plots it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "preg = nsfg.ReadFemPreg()\n",
    "live = preg[preg.outcome == 1]\n",
    "first_babies = live[live.birthord == 1]\n",
    "other_babies = live[live.birthord != 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "thinkstats2.Pmf"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_pmf = thinkstats2.Pmf(first_babies.prglngth)\n",
    "other_pmf = thinkstats2.Pmf(other_babies.prglngth)\n",
    "type(first_pmf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "PrePlot takes optional parameters rows and cols to make a grid of figures, in this case one row of two figures. The first figure, on the left, displays the Pmfs using thinkplot.Hist, as we have seen before. \n",
    "\n",
    "The second call to PrePlot resets the color generator: The SubPlot switches to the second figure (on the right) and displays the Pmfs using thinkplot.Pmfs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "//anaconda/lib/python2.7/site-packages/matplotlib/axes/_axes.py:519: UserWarning: No labelled objects found. Use label='...' kwarg on individual plots.\n",
      "  warnings.warn(\"No labelled objects found. \"\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs4AAAF/CAYAAABZkk9hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHypJREFUeJzt3XuUZWdZJ+Df290JpBUzJGHIkBgYggkTwkUuMUjQwlYT\ndCAscQXQhXgZJ44TZWTp4Bg0zSyCd8JoVBKNGHQml6WjwBohEaSgw0XCPUKahLgIJAEkgHIJhE7n\nmz/qdFM0Vae/qnN21amu51mrVp+9z947b+0+eftXX31772qtBQAAGG/LehcAAAAbgeAMAAAdBGcA\nAOggOAMAQAfBGQAAOgjOAADQYfDgXFVnVdXuqrqpql64zDZzVfXeqvrHqnrT0DUBsDQ9G2B5NeR9\nnKtqS5KbkuxIckeS65M8u7W2e9E2RyZ5W5Lvb63dXlXHtNbuHKwoAJakZwOMN/SI82lJbm6t3dpa\n25PkyiRnH7DNjyT5q9ba7UmiAQOsGz0bYIyhg/NxST6+aPm20brFTkpyVFW9qaqur6rnDlwTAEvT\nswHG2LbeBWShhscm+Z4k35Tk7VX19tbaR9a3LACWoGcDm9bQwfn2JCcsWj5+tG6x25Lc2Vr7SpKv\nVNVbkjw6ydc14aoabjI2wBpordV613AQejbAyFI9e+ipGtcneVhVPbiqDk/y7CSvOWCbVyc5o6q2\nVtX2JN+R5MalDtZa27BfF1xwwbrXsJG/nD/nb72/Jj2HG4SePaW/b1/OofO3sc/fcgYdcW6t7a2q\n85Jcm4WQfllr7caqOnfh7XZpa213VV2T5ANJ9ia5tLX2oSHrAuAb6dkA4w0+x7m19vokJx+w7pID\nln8nye8MXQsA4+nZAMvz5MA1Mjc3t94lbGjO32Scv8k5h5uLv+/JOYeTcf4mM9T5G/QBKNNUVW2j\n1ApwoKpKm/2LA6dGzwY2suV6thFnAADoIDgDAEAHwRkAADoIzgAA0EFwBgCADoIzAAB0EJwBAKCD\n4AwAAB0EZwAA6CA4AwBAB8EZAAA6CM4AANBBcAYAgA6CMwAAdBCcAQCgg+AMAAAdBGcAAOggOAMA\nQAfBGQAAOgjOAADQQXAGAIAOgjMAAHQQnAEAoIPgDAAAHbatdwEAABvJdbd8Jm+86c589Z626mMc\nvq2y46RjcsaJR0+xMoYmOMMGcP7VN4x9/8JzHrlGlQAwaWhOkq/e0/LGm+4UnDcYUzUAAFZg0tA8\n7eOwdow4AwCs0oVPe/iK9zn/tbsHqIS1YMQZAAA6CM4AANBBcAYAgA6CMwAAdBCcAQCgg+AMAAAd\nBGcAAOggOAMAQAfBGQAAOgjOAADQQXAGAIAOgjMAAHQQnAEAoIPgDAAAHQRnAADoIDgDAEAHwRkA\nADoMHpyr6qyq2l1VN1XVC5d4/7ur6l+q6j2jrxcNXRMAS9OzAZa3bciDV9WWJBcn2ZHkjiTXV9Wr\nW2u7D9j0La21pw9ZCwDj6dkA4w094nxakptba7e21vYkuTLJ2UtsVwPXAcDB6dkAYwwdnI9L8vFF\ny7eN1h3oiVX1vqr6f1V1ysA1AbA0PRtgjEGnanR6d5ITWmt3VdVTk/xNkpPWuSYAlqZnA5vW0MH5\n9iQnLFo+frRuv9baFxe9fl1V/WFVHdVa++yBB9u5c+f+13Nzc5mbm5t2vQBTMT8/n/n5+fUuY6X0\nbGBT6u3Z1VobrIiq2prkw1m40OQTSd6Z5DmttRsXbfPA1tqnRq9PS3J1a+0hSxyrDVkrzLLzr75h\n7PsXnvPINaqE1aqqtNZmem6wng19zn/t166XvfBpD1/z/Rnecj170BHn1treqjovybVZmE99WWvt\nxqo6d+HtdmmSH66q/5JkT5IvJ3nWkDUBsDQ9G2C8wec4t9Zen+TkA9Zdsuj1HyT5g6HrAODg9GyA\n5XlyIAAAdBCcAQCgg+AMAAAdBGcAAOggOAMAQAfBGQAAOszCI7cBADaMT37urtzx2btyb2t55ks/\ntuL9v3jEEdlSlQcdtX2A6hiS4Awz4ryXXLHse0eedOoaVgLAOPtC8yTubS13fPauKVXEWjFVAwBg\nBSYNzdM+DmvHiDMAwCr91a98/4r3OfOiXQNUwlow4gwAAB0EZwAA6CA4AwBAB8EZAAA6CM4AANBB\ncAYAgA6CMwAAdBCcAQCgg+AMAAAdBGcAAOggOAMAQAfBGQAAOgjOAADQQXAGAIAOgjMAAHQQnAEA\noIPgDAAAHQRnAADoIDgDAEAHwRkAADoIzgAA0EFwBgCADoIzAAB0EJwBAKCD4AwAAB0EZwAA6CA4\nAwBAB8EZAAA6CM4AANBBcAYAgA6CMwAAdBCcAQCgg+AMAAAdBGcAAOggOAMAQAfBGQAAOgjOAADQ\nYfDgXFVnVdXuqrqpql44ZrsnVNWeqvqhoWsCYGl6NsDyBg3OVbUlycVJzkzyiCTPqaqHL7PdbyS5\nZsh6AFieng0w3tAjzqclubm1dmtrbU+SK5OcvcR2P5fkL5P888D1ALA8PRtgjKGD83FJPr5o+bbR\nuv2q6kFJntFa+6MkNXA9ACxPzwYYYxYuDnx5ksXz6DRigNmlZwOb1raBj397khMWLR8/WrfY45Nc\nWVWV5JgkT62qPa211xx4sJ07d+5/PTc3l7m5uWnXCzAV8/PzmZ+fX+8yVkrPBjal3p5drbXBiqiq\nrUk+nGRHkk8keWeS57TWblxm+1cmeW1r7f8u8V4bslZYb+e95Ipl3zvypFPH7nvhOY+cdjlMWVWl\ntTbTo7N6NvQ586Jd+19f8wtPXvP9Gd5yPXvQEefW2t6qOi/JtVmYFnJZa+3Gqjp34e126YG7DFkP\nAMvTswHGG3qqRlprr09y8gHrLllm258cuh4AlqdnAyxvFi4OBACAmSc4AwBAB8EZAAA6CM4AANBB\ncAYAgA6CMwAAdBCcAQCgg+AMAAAdBGcAAOggOAMAQAfBGQAAOgjOAADQQXAGAIAOgjMAAHQQnAEA\noIPgDAAAHQRnAADoIDgDAEAHwRkAADoIzgAA0EFwBgCADoIzAAB0EJwBAKCD4AwAAB0EZwAA6CA4\nAwBAh67gXFVbhy4EAABmWe+I881V9dtVdcqg1QAAwIzqDc6PTnJTkj+pqndU1X+uqm8ZsC4AAJgp\nXcG5tfaF1toft9a+M8kLk1yQ5BNVdXlVPWzQCgEAYAZ0z3GuqqdX1V8neXmS303y0CSvTfK3A9YH\nAAAzYVvndjcneVOS326tvW3R+r+squ+aflkAADBbeoPzj7XWrlu8oqqe1Fp7a2vt5weoCwAAZkrv\nxYG/t8S6359mIQAAMMvGjjhX1ROTfGeSB1TVCxa99S1J3NsZAIBN42BTNQ5P8s2j7e63aP3nk/zw\nUEUBAMCsGRucW2tvTvLmqvqz1tqta1QTAADMnINN1Xh5a+2/Jbm4qtqB77fWnj5YZQAAMEMONlXj\nz0d//s7QhQAAwCw72FSNd4/+fPPalAMAALPpYFM1bkjyDVM09mmtPWrqFQEAwAw62FSN/7gmVQAA\nwIw72FQNd9IAAIAc5MmBVXXd6M8vVNXnD/xzbUoEAID1d7AR5zNGf95v3HYAAHCoO9gc5/2q6rFJ\nzsjCxYLXtdbeO1hVAAAwY8ZO1dinqn4tyeVJjk5yTJI/q6oXDVkYAADMkq7gnORHkzyhtXZBa+2C\nJKcneW7PjlV1VlXtrqqbquqFS7z/9Kp6f1W9t6reWVVP6i8fgGnSswGW1ztV444k903yldHyfZLc\nfrCdqmpLkouT7Bgd4/qqenVrbfeizd7QWnvNaPtHJrk6yX/orAuAKdGzAcY72ANQfj8Lc5r/NckH\nq+rvRsvfl+SdHcc/LcnN+25rV1VXJjk7yf4m3Fq7a9H235zk3pV8AwBMjZ4NMMbBRpzfNfrz3Un+\netH6+c7jH5fk44uWb8tCY/46VfWMJL+e5AFJfrDz2ABMl54NMMbBbkd3+VoU0Vr7myR/U1VnJHlJ\nFka0v8HOnTv3v56bm8vc3NxalAewYvPz85mfn1/vMgahZwOHmt6e3TXHuaq+LQujC6dkYa5zkqS1\n9tCD7Hp7khMWLR+fMXOjW2vXVdVDq+qo1tpnD3x/cRMGmGUHBsUXv/jF61dMPz0b2JR6e3bvXTVe\nmeSPktyT5ClJXpXkLzr2uz7Jw6rqwVV1eJJnJ3nN4g2q6sRFrx+b5PClGjAAg9OzAcbovavGEa21\nN1ZVjS4a2VlV707ya+N2aq3trarzklybhZB+WWvtxqo6d+HtdmmSZ1bVjyX5apIvJzln1d8NAKum\nZwOM1xuc7x7dpujmUVO9PQtXUx9Ua+31SU4+YN0li17/VpLf6qwDgAHp2QDL652q8fwk25P8fJLH\nZeHhJ88bqigAAJg1XSPOrbXrk/03x//51toXBq0KAABmTNeIc1U9vqpuSPKBJDeMHrf6uGFLAwCA\n2dE7x/lPk/xsa21Xkozu3fnKJI8aqjAAAJglvXOc9+4LzcnCvTuzcGs6AADYFMaOOI/u0Zkkb66q\nS5JckaQleVb6H7sNAAAb3sGmavzuAcsXLHrdplwLAADMrLHBubX2lLUqBAAAZlnvXTWOrKqXVdW7\nRl+/W1VHDl0cAADMit6LA/80yRey8GjVc5J8Pgt31QAAgE2h93Z0J7bWnrlo+cVV9b4hCgIAgFnU\nO+L85dG9m5MkVfWkJF8epiQAAJg9vSPOP5PkVYvmNX8uyfOGKQkAAGbPQYNzVW1JcnJr7dFV9S1J\n0lr7/OCVAQDADDnoVI3W2r1J/vvo9eeFZgAANqPeOc5vqKpfrKpvraqj9n0NWhkAAMyQ3jnOz8rC\nkwJ/9oD1D51uOQAAMJt6g/MpWQjNZ2QhQO9K8oqhigIAgFnTG5wvz8JDT35vtPwjo3XnDFEUAADM\nmt7gfGpr7ZRFy2+qqg8NURAAAMyi3osD31NVp+9bqKrvSPKuYUoCAIDZ0zvi/Lgkb6uqj42WT0jy\n4aq6IUlrrT1qkOoAAGBG9AbnswatAgAAZlxXcG6t3Tp0IQAAMMt65zgDAMCmJjgDAEAHwRkAADoI\nzgAA0EFwBgCADoIzAAB0EJwBAKCD4AwAAB0EZwAA6CA4AwBAB8EZAAA6CM4AANBBcAYAgA6CMwAA\ndBCcAQCgg+AMAAAdBGcAAOggOAMAQAfBGQAAOgjOAADQQXAGAIAOgjMAAHQYPDhX1VlVtbuqbqqq\nFy7x/o9U1ftHX9dV1SOHrgmApenZAMsbNDhX1ZYkFyc5M8kjkjynqh5+wGb/lOS7WmuPTvKSJH88\nZE0ALE3PBhhv6BHn05Lc3Fq7tbW2J8mVSc5evEFr7R2ttX8dLb4jyXED1wTA0vRsgDGGDs7HJfn4\nouXbMr7J/qckrxu0IgCWo2cDjLFtvQvYp6qekuQnkpyx3rUAMJ6eDWxGQwfn25OcsGj5+NG6r1NV\nj0pyaZKzWmufW+5gO3fu3P96bm4uc3Nz06oTYKrm5+czPz+/3mWslJ4NbEq9Pbtaa4MVUVVbk3w4\nyY4kn0jyziTPaa3duGibE5K8MclzW2vvGHOsNmStsN7Oe8kVy7535Emnjt33wnPc2GDWVVVaa7Xe\ndYyjZ0OfMy/atf/1Nb/w5DXfn+Et17MHHXFure2tqvOSXJuF+dSXtdZurKpzF95ulyb51SRHJfnD\nqqoke1prpw1ZFwDfSM8GGG/wOc6ttdcnOfmAdZcsev3TSX566DoAODg9G2B5nhwIAAAdBGcAAOgg\nOAMAQAfBGQAAOgjOAADQQXAGAIAOgjMAAHQQnAEAoIPgDAAAHQRnAADoIDgDAEAHwRkAADoIzgAA\n0EFwBgCADoIzAAB0EJwBAKCD4AwAAB0EZwAA6CA4AwBAB8EZAAA6CM4AANBBcAYAgA6CMwAAdBCc\nAQCgg+AMAAAdBGcAAOggOAMAQAfBGQAAOgjOAADQQXAGAIAOgjMAAHQQnAEAoIPgDAAAHQRnAADo\nIDgDAEAHwRkAADoIzgAA0EFwBgCADtvWuwAAgM3q/NfuXtV+h2+r7DjpmJxx4tFTrohxBGfYBM6/\n+oax7194ziPXqBIAqrW0qiTJuz7y6VUdY0tV7vjMXYLzGhOc4RBx3kuuWPa9I086dQ0rAWCc7e3e\n3JUt+8PzatzbWj5255emWBU9BGcAgDX0vCc9JFftuiV379m7qv2/eMQRSRbCM2tLcAYAWENnn/6Q\nnH36Q1a9/5kX7ZpeMayIu2oAAEAHwRkAADoIzgAA0EFwBgCADoMH56o6q6p2V9VNVfXCJd4/uare\nVlVfqaoXDF0PAMvTswGWN+hdNapqS5KLk+xIckeS66vq1a21xY/J+UySn0vyjCFrAWA8PRtgvKFH\nnE9LcnNr7dbW2p4kVyY5e/EGrbU7W2vvTnLPwLUAMJ6eDTDG0MH5uCQfX7R822gdALNHzwYYw8WB\nAADQYegnB96e5IRFy8eP1q3Kzp0797+em5vL3Nzcag8FMKj5+fnMz8+vdxkrpWcDm1Jvzx46OF+f\n5GFV9eAkn0jy7CTPGbN9jTvY4iYMMMsODIovfvGL16+Yfno2sCn19uxBg3NrbW9VnZfk2ixMC7ms\ntXZjVZ278Ha7tKoemORdSe6X5N6qen6SU1prXxyyNgC+np4NMN7QI85prb0+yckHrLtk0etPJfnW\noesA4OD0bIDluTgQAAA6CM4AANBBcAYAgA6CMwAAdBCcAQCgw+B31QAAYBjPfOm1q9rvPodtzbOe\nfGLOPv0h0y3oECc4A0mS815yxbLvXfyicc/AAGAtbanKva1NdIy79+zNVbtuEZxXyFQNAIAN5EFH\nbc+WGvvgzi5379k7hWo2FyPOAAAbyLH3355j7789SXLh0x6+4v1XO70DI84AANBFcAYAgA6magAA\nm8p1t3wmb7zpznz1nskusGPzEZxhDZx/9Q1j37/wnEeuUSUATCs014R3tmDjMVUDANhUphWaD7vn\nnilUw0ZixBkA2LRWd1eKjw1QCRuBEWcAAOhgxBkA2FQ++bm7csdn78q9rRk9ZkUEZ5iScY+sPvKk\nU9ewEgDG2ReaJ3Wfw7ZOoRo2ElM1AIBNZVqh+VlPPnEK1bCRGHEGADatv/qV71/vEthAjDgDAEAH\nwRkAADqYqgEAbCgemc16EZyhg0dmA8wOj8xmvQjOAMCG8rFPf2niW8odKo/MPv+1u1e8zxePOOKQ\n+f7XmuAMI+7DvDwj7sAsWRyav/nLX171cTbqfZgP31YTj7i3quzZti3PfOm1q9p/3+34zj79IRPV\nsdEIzsBUjPvB4+IXPWcNKwEOdZv9Psw7TjpmoukqW6pyb2tpVauu4e49e3PVrlsEZwCAjWIz3of5\njBOPzhknHr3q/X/qVe+ZytMT796zd6L9NyLBGQBgEzn2/ttz7P23J0kufNrDV7z/aqd3HAoEZwBg\nTb36HR/NVbtuWf2I5RFHTLcg6CQ4c8gwxxZgY5goNC+yZYI5urAagjMAsCITjxhPwZaqPOio7ev2\n32dzEpyZGUaMATaGaYXm+xy2Nf/nl3aseL/V3LsYpkFwBgBW5Autsue+953odmZbqvJvjtouBLOh\nCM5sCJM+gMMDPACmZ8+2bftD8+Mf9oB1q+PwbeY4s7a2rHcBAMDGMslI87Qcvq2y46Rj1rsMNhkj\nzgDAqq3mPsDMjtVMlfniEUekWsth99wzQEWzTXAGBmeqDMDsOHxbrfpx3fu0quzZtvlipKkaAACb\nyI6TjpnK/PBZmLKz1jbfjwoAAJvYGScenTNOPHrV+5950aenWM3GIjgDALAqz3zptava7z6Hbc2z\nnnxizj79IdMtaGCCMzATxj0AJ/EQHIBZsaUq97bJ5kjfvWdvrtp1i+AMALNsWo+L3qgjZjCpBx21\nPXd89q6phOeNRnCeIo+MBph9l7/1o7lr62Fp2w6f6Dhfai2Xv/WjgjObzrH3355j7789yepuR7ja\n6R2zQHBeI+Nux+VWXABr567aMpW7AbSq3LXKm1O97HW784YPfzp7Jxiw21rJ9578gLzgqe6jzPrZ\nbPeBFpwPIeNGvI886dSx+wrvzDo/fDIti0Pzah8X/a6PfPobjrUSk4bmJNnbFo4jOLPWNvN9oDde\nxQAwJat96t2kt+OaNDRP+ziwEjtOOiZvvOnOqYTnjWbw4FxVZyV5eRYetnJZa+03l9jm95I8NcmX\nkvx4a+19Q9cFwDfSs9feNb/w5BXvc+ZFu/a/Xs180a9u25Y927ZtyODC+tvM94EeNDhX1ZYkFyfZ\nkeSOJNdX1atba7sXbfPUJCe21r6tqr4jySuSnD5kXevh1g9enwc/4gnrXcZYk17cOORUkY1w/maZ\n8ze5+fn5zM3NrXcZg1qrnj2Nu1oMfUeLWf/7Xnw7sC8eccS61bF1TO6e9XM46zbL+Vv8Q+BKHGyO\n/1Dnb+gR59OS3NxauzVJqurKJGcnWTyT/Owkr0qS1to/VNWRVfXA1tqnVvofm+W7Wtz6oY0dXMbN\nL02Gn2O60c/fetsM5+9g94Ee98Nbz+d3k/wjtiY9e9LQ/NVt2/Klrdvyinfclnd++iurPs44K/n7\nXu0//JM44Zhvysfu/NLEtwObxL7gspxN8v/MYA7l87e1Jp9mdLA5/hs1OB+X5OOLlm/LQmMet83t\no3UrDs5DWosLk1Y7YuvCKGBK1qRnf6FV9tz3vhNPE2it7b9IbzVqgtA5jX/49x1nNZ77xBMmnmN6\n+LbKjpOOmehX7rAa33vyA6Z2gexyP7h+5O235u1jfqhd7V1pNs3FgT3Bd5KpBsDGdrAR6/d/8FPZ\ns0wf8cPr0pYdiT3ssP0vV3NXi/fccufEI63VWra3e1e9/zT+4T/YiO04k84xhfX0gqc+fKK7wfzA\ny3dNJXRfs/vTuWb3yn74rjbgr3mq6vQkO1trZ42WfzlJW3yxSVW9IsmbWmtXjZZ3J/nuA3/tV1Wu\nHQY2tNbaTF+JpWcDfM1SPXvoEefrkzysqh6c5BNJnp3kwMnGr0nyX5NcNWra/7LUXLlZ/wcH4BCg\nZwOMMWhwbq3trarzklybr93a6MaqOnfh7XZpa+1vq+oHquojWbi10U8MWRMAS9OzAcYbdKoGAAAc\nKrasdwGHoqq6T1X9Q1W9t6puqKoLRuvvX1XXVtWHq+qaqjpyvWudRWPO3wVVdVtVvWf0ddZ61zrL\nqmrL6Dy9ZrTs87cCo/P33kXnz+fvEKVnT0bPng49ezJr1bMF5wG01u5O8pTW2rcneUySp1bVaUl+\nOckbWmsnJ/n7JP9jHcucWWPOX5K8rLX22NHX69evyg3h+Uk+tGjZ529lnp/kgwes8/k7BOnZk9Gz\np0bPnsya9GzBeSCttbtGL++ThbnkLQsPDrh8tP7yJM9Yh9I2hGXOX5K44KhDVR2f5AeS/Mmi1T5/\nnZY5f4nP3yFLz56Mnj0ZPXsya9mzBeeB7PuVQZJPJvm71tr1SfY/Xau19skk/3Y9a5xly5y/JDmv\nqt5XVX/i11ZjXZTkl/K1f7wSn7+VWOr8JT5/hyw9ezJ69sT07MmsWc8WnAfSWrt39Gur45OcVlWP\nyDf+hboycxlLnL9Tkvxhkoe21h6Theb8svWscVZV1Q8m+VRr7X0Z/9O2z98Sxpw/n79DmJ49GT17\n9fTsyax1zxacB9Za+3yS+SRnJflUVT0wSarq2CT/vI6lbQiLz19r7dPta7eB+eMkT1i3wmbbk5I8\nvar+KckVSb6nqv48ySd9/rosdf5e5fO3OejZk9GzV0XPnsya9mzBeQBVdcy+XwlU1RFJvi/JjVl4\ncMCPjzZ7XpJXr0uBM26Z87d71Dj2+aEk/7ge9c261tqvtNZOaK09NAsPsPj71tpzk7w2Pn8Htcz5\n+zGfv0OXnj0ZPXsyevZk1rpnD/3kwM3q3yW5vKq2ZOGHk6tGDw14R5Krq+onk9ya5Jz1LHKGLXf+\nXlVVj0lyb5KPJjl3HWvciH4jPn+T+C2fv0OWnj0ZPXsYevZkBunZHoACAAAdTNUAAIAOgjMAAHQQ\nnAEAoIPgDAAAHQRnAADoIDgDAEAHwRmWUVVvqqrHrncdABycns1aEJwBAKCD4Mwho6p+sarOG72+\nqKreOHr9lKr6i6r6vqp6W1W9q6quqqrto/cfW1XzVXV9Vb2uqh54wHGrql5ZVf+zqraMXn+gqt5f\nVc9f++8UYOPTs9mIBGcOJbuSPHn0+nFJvqmqto7WfSDJi5LsaK09Psm7k7ygqrYl+f0kz2ytPSHJ\nK5O8dNExD0vyv5Pc1Fr7tSSPSXJca+1RrbVHj7YHYOX0bDacbetdAEzRu5M8rqrul+Tu0fITstCE\nX5PklCRvrarKQnN9e5KTk5ya5O9G67ckuWPRMS9JclVr7ddHy/+U5N9X1f9K8rdJrh38uwI4NOnZ\nbDiCM4eM1to9VfXRJD+e5K1ZGLF4SpITs9A8r22t/ejifarq1CT/2Fp70jKHfWuSp1TVy1prd7fW\n/qWqHp3kzCTnJjknyU8N8f0AHMr0bDYiUzU41OxK8otJ3pLkuiQ/k+S9Sf4hyZOq6sQkqartVfVt\nST6c5AFVdfpo/baqOmXR8S5L8rokV1fV1qo6OsnW1tpfJ/nVJN++Rt8XwKFIz2ZDEZw51OxKcmyS\nt7fW/jnJl5O8pbV2ZxZGNa6oqvcneVuSk1tre5L8cJLfrKr3ZaFhP3F0rJYkrbWLRutfleS4JPNV\n9d4kf57kl9fqGwM4BOnZbCjVWlvvGgAAYOYZcQYAgA6CMwAAdBCcAQCgg+AMAAAdBGcAAOggOAMA\nQAfBGQAAOgjOAADQ4f8Dyis6XDtYN6oAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x119ee9c90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x125bcf9d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "width = 0.45\n",
    "\n",
    "thinkplot.PrePlot(2, cols=2)\n",
    "thinkplot.Hist(first_pmf, align='right', width=width)\n",
    "thinkplot.Hist(other_pmf, align='left', width=width)\n",
    "# The axis option ensures that the two figures are on the same axes\n",
    "# this is important if we want to compare the figures\n",
    "thinkplot.Config(xlabel='weeks',\n",
    "                ylabel='probability',\n",
    "                axis=[27,46,0,0.6])\n",
    "thinkplot.PrePlot(2)\n",
    "thinkplot.SubPlot(2)\n",
    "thinkplot.Pmfs([first_pmf, other_pmf])\n",
    "thinkplot.Show(xlabel='weeks',\n",
    "              axis=[27,46,0,0.6])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='subchapter3.3'></a>\n",
    "## 3.3 Other visualizations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the NSFG data, the biggest differences in the distributions are near the mode. Let's zoom in on that part of the graph, and transform the data to emphasize differences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# weeks = range of weeks\n",
    "weeks = range(35, 46)\n",
    "# diffs = difference between pmfs in %\n",
    "diffs = []\n",
    "for week in weeks: \n",
    "    p1 = first_pmf.Prob(week)\n",
    "    p2 = other_pmf.Prob(week)\n",
    "    diff = 100 * (p1 - p2)\n",
    "    diffs.append(diff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEACAYAAACnJV25AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADSBJREFUeJzt3X+IZWUdx/HPZ9sKoTIz2sLBQkoDkyxsiwI9WdJmpEER\nLtUSQpaxuVSYpQszA0ssVki19FdtWNCWlJRB5a7YMYi2lnRj0638I8wkjaIfRCRu++2Pe3a5jXNn\n9p7nuXO933m/YODeu2ee8zzM8N5n7pyz64gQACCPDdOeAACgLsIOAMkQdgBIhrADQDKEHQCSIewA\nkEy1sNveYPte23fUGhMAML6aO/Ydkh6oOB4AoIcqYbc9J+lySV+uMR4AoL9aO/ZbJF0vidtYAWDK\nisNu+22SHouIw5LcfQAApsSl/1aM7U9Leq+kY5JOk/RsSbdHxLYlx7GbB4AeImKsDXPxjj0iboyI\nsyPiHElXSbp7adSHjk37MT8/P/U5sD7WxvryffTBdewAkMzGmoNFxD2S7qk5JgBgPOzYK2maZtpT\nmKjM68u8Non1rUfFvzw95RPZsVbnAoAsbCvW+penAICnFsIOAMkQdgBIhrADQDKEHQCSqXodO4DV\nbd+1b2Jj79m5dWJjY3awYweAZAg7ACRD2AEgGcIOAMkQdgBIhrADQDKEHQCSIewAkAxhB4BkCDsA\nJEPYASAZwg4AyRB2AEiGsANAMoQdAJIh7ACQDGEHgGQIOwAkQ9gBIBnCDgDJFIfd9pztu23fb/uI\n7etqTAwA0M/GCmMck/SxiDhs+1mSfml7f0T8psLYAIAxFe/YI+LRiDjcPf6XpKOSziodFwDQT9X3\n2G2/RNKFkn5ec1wAwKmr8VaMJKl7G+bbknZ0O3cATwHbd+2byLh7dm6dyLgoVyXstjdqEPWvR8T3\nRh23sLBw8nHTNGqapsbpASCNtm3Vtm3RGLV27HslPRARn1/poOGwAwCebOmmd3Fxcewxalzu+AZJ\n75F0qe37bN9re0vpuACAfop37BHxU0lPqzAXAEAF3HkKAMkQdgBIhrADQDKEHQCSIewAkAxhB4Bk\nCDsAJEPYASAZwg4AyRB2AEiGsANAMoQdAJIh7ACQDGEHgGQIOwAkQ9gBIBnCDgDJEHYASIawA0Ay\nxf/nKQa279o3kXH37Nw6kXEB5EXYAVTFJmf6eCsGAJIh7ACQDGEHgGQIOwAkQ9gBIJm0V8Xwm3kA\n6xU7dgBIhrADQDJVwm57i+3f2P6d7RtqjAkA6Kc47LY3SNoj6S2Szpe01fbLS8cFAPRTY8e+WdKD\nEfFQRDwh6ZuSrqwwLgCghxpXxZwl6eGh53/UIPYAMHFrfQXcLFxx54goG8B+p6S3RMQ13fP3Stoc\nEdctOS7m5+dPPm+aRk3TFJ17Pcv8zTypcz1VzgespG1btW178vni4qIiwuOMUWPH/oiks4eez3Wv\nPcnCwkKF0wFAXks3vYuLi2OPUSPshyS91PaLJf1J0lWS2KZgZrCrRjbFYY+I/9reLmm/Br+M/UpE\nHC2eGQCglyr/pEBE/EjSeTXGAgCU4c5TAEiGsANAMoQdAJIh7ACQTNp/jz07LtEDMAo7dgBIhrAD\nQDKEHQCSIewAkAxhB4BkCDsAJEPYASAZrmPHKeG6eWB2sGMHgGQIOwAkQ9gBIBnCDgDJEHYASIaw\nA0AyhB0AkiHsAJAMYQeAZAg7ACRD2AEgGcIOAMkQdgBIhrADQDKEHQCSKQq77ZttH7V92PZ3bD+n\n1sQAAP2U7tj3Szo/Ii6U9KCkT5VPCQBQoijsEXFXRBzvnh6UNFc+JQBAiZrvsV8t6YcVxwMA9LDq\n/3lq+4CkTcMvSQpJN0XE97tjbpL0RER8Y6WxFhYWTj5umkZN04w/YwBIrG1btW1bNIYjomwA+/2S\nPiDp0oh4fIXjovRcWB+279o3sbH5T7kxa2wrIjzO56y6Y1/lhFskXS/p4pWiDgBYO6XvsX9R0rMk\nHbB9r+0vVZgTAKBA0Y49Il5WayIAgDq48xQAkiHsAJAMYQeAZAg7ACRD2AEgGcIOAMkQdgBIhrAD\nQDKEHQCSIewAkAxhB4BkCDsAJEPYASAZwg4AyRB2AEiGsANAMoQdAJIh7ACQDGEHgGQIOwAkQ9gB\nIBnCDgDJEHYASIawA0AyhB0AkiHsAJAMYQeAZAg7ACRTJey2P277uO3n1RgPANBfcdhtz0m6TNJD\n5dMBAJSqsWO/RdL1FcYBAFRQFHbbV0h6OCKOVJoPAKDQxtUOsH1A0qbhlySFpJ2SbtTgbZjhPxtp\nYWHh5OOmadQ0zanPFADWgbZt1bZt0RiOiH6faL9C0l2S/q1B0OckPSJpc0T8eZnjo++5sL5s37Vv\nYmPv2bl1YmMDk2BbEbHipnmpVXfso0TEryW9cOjkv5f06oj4W98xAQDlal7HHlrlrRgAwOT13rEv\nFRHn1BoLANAfd54CQDKEHQCSIewAkAxhB4BkCDsAJEPYASAZwg4AyRB2AEiGsANAMoQdAJIh7ACQ\nDGEHgGQIOwAkQ9gBIBnCDgDJEHYASIawA0AyhB0AkiHsAJAMYQeAZAg7ACRD2AEgGcIOAMkQdgBI\nhrADQDKEHQCSIewAkAxhB4BkisNu+yO2j9o+Ynt3jUkBAPrbWPLJthtJb5d0QUQcs/38KrMCAPRW\numO/VtLuiDgmSRHxl/IpAQBKlIb9XEkX2z5o+8e2L6oxKQBAf6u+FWP7gKRNwy9JCkk7u88/IyJe\nZ/s1km6TdM4kJgoAODWrhj0iLhv1Z7Y/JOn27rhDto/bPjMi/rrc8QsLCycfN02jpmnGnS8ApNa2\nrdq2LRrDEdH/k+1rJJ0VEfO2z5V0ICJePOLYKDkX1o/tu/ZNbOw9O7dObGxgEmwrIjzO5xRdFSPp\nq5L22j4i6XFJ2wrHAwAUKgp7RDwh6X2V5gIAqIA7TwEgGcIOAMkQdgBIhrADQDKEHQCSIewAkEzR\nDUpjnYgblABgbH1uUGLHDgDJEHYASIawA0AyhB0AkiHsAJAMYQeAZAg7ACRD2AEgGcIOAMkQdgBI\nhrADQDKEHQCSIewAkAxhB4BkCDsAJEPYASAZwg4AyRB2AEiGsANAMoQdAJIh7ACQTFHYbb/S9s9s\n32f7F7YvqjUxAEA/pTv2myXNR8SrJM1L+kz5lGZT27bTnsJEZV5f5rVJrG89Kg37cUmnd4+fK+mR\nwvFmVvZvrszry7w2ifWtRxsLP/+jku60/TlJlvT68ikBAEqsGnbbByRtGn5JUki6SdKbJe2IiO/a\nfpekvZIum8REAQCnxhHR/5Ptv0fEc4ee/yMiTh9xbP8TAcA6FhEe5/jSt2IesX1JRNxj+02Sfldr\nYgCAfkrD/gFJX7D9NEn/kXRN+ZQAACWK3ooBADz1VL/z1PYzbf+8u2npiO35JX/+cdvHbT+v9rnX\nwkrrs/0R20e713dPc559jVpfppvRbG+wfa/tO7rnZ9jeb/u3tu+0vezviWZFt777htZ3c/d9edj2\nd2w/Z9pzLLH06zf0+ky35YTl1jduW6qHPSIel/TG7qalCyW91fbmbnJzGlw181Dt866VUeuz3Uh6\nu6QLIuICSZ+d4jR7G7G+1yrXzWg7JD0w9PyTku6KiPMk3S3pU1OZVT07JN0/9Hy/pPMj4kJJDyrH\n+oa/finaMuT/1mf7jRqzLRP5t2Ii4t/dw2dq8D7+ifd7bpF0/STOuZZGrO9aSbsj4lh3zF+mNL1i\ny6zvuJLcjNYF4HJJXx56+UpJt3aPb5X0jrWeVy3LrS8i7oqI493Tg5LmpjG3GkZ8/aQkbRmxvg9p\nzLZMJOwnfhSU9KikAxFxyPYVkh6OiCOTOOdaWm59ks6VdLHtg7Z/nOCtiqXr+6ikz9r+gwa791nd\n9Z0IwPAvlzZFxGOSFBGPSnrBNCZWyXLrG3a1pB+u3XSqe9L6bF+pJG3R8l+/sdsyqR378e5H9jlJ\nm21fIOlGDX6EP2FmL39cZn3na7CzPSMiXifpE5Jum+YcS4xY37Ua3Ix2tgaR3zvNOfZh+22SHouI\nw1r5+28mryhYZn1e8uc3SXoiIr4xjfmVWu7rZ/s0DTYZM9+WFb4/x25L6eWOK4qIf9puNfhR9yWS\nfmXbGgTjl7Y3R8SfJzmHSRpa3xZJD0u6vXv9UPdLnDMj4q/TnGOJJevbFhE7ute/bfsrU51cP2+Q\ndIXtyyWdJunZtr8u6VHbmyLiMdsvlDSr35PLre9rEbHN9vs1+BH/0mlOsNCT1ifpa8rTlmW/furT\nloio+iHp+ZJO7x6fJuknki5fcszvNfgbqPr5J/0xan0aXMO/2L1+rqSHpj3Xyuu7X9Il3etvknRo\n2nMtXOclku7oHt8s6Ybu8Q0avJ859TlWXN+W7ut35rTnNYn1LXl9Ztuywtfvg+O2ZRI79hdJutX2\nBg3e6vlWRPxgyTGhGf1xSSPWZ/vpkvbaPiLpcUnbpjnJAqPW9w9Jn096M9puSbfZvlqDqyrePeX5\n1PZFSc+QdGCwqdXBiPjwdKc0MbPcllH2asy2cIMSACTDf40HAMkQdgBIhrADQDKEHQCSIewAkAxh\nB4BkCDsAJEPYASCZ/wFHRyaYEjm/xgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x125c33110>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "thinkplot.Bar(weeks, diffs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result shows a pattern: first babies are less likely to be born in week 39, and somewhat more likely to be born in weeks 41-42. \n",
    "\n",
    "This is not a clear conclusion for the question. Note that we used the same dataset to identify an apparent difference and then chose a visualization that makes the difference apparent. We can't be sure this effect is real. It might be due to random variation (we'll adress this later). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='subchapter3.4'></a>\n",
    "## 3.4 The class size paradox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another computation you can do with pmf is referred to by the author as \"The class size paradox\", and it is explained [here](http://www.umasocialmedia.com/socialnetworks/glossary/class-size-paradox/). \n",
    "\n",
    "Let's look at the computations behind this paradox.\n",
    "\n",
    "For example, here are the classes sizes and number of classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Num of classes</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Class size</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5-9</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10-14</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15-19</th>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20-24</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25-29</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30-34</th>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35-39</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40-44</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45-49</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Num of classes\n",
       "Class size                \n",
       "5-9                      8\n",
       "10-14                    8\n",
       "15-19                   14\n",
       "20-24                    4\n",
       "25-29                    6\n",
       "30-34                   12\n",
       "35-39                    8\n",
       "40-44                    3\n",
       "45-49                    2"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dict = {'Class size': ['5-9', '10-14', '15-19', '20-24',\n",
    "                    '25-29', '30-34', '35-39', '40-44', '45-49'],\n",
    "            'Num of classes': [8, 8, 14, 4, 6, 12, 8, 3, 2]}\n",
    "df = pd.DataFrame(data_dict)\n",
    "df.set_index('Class size', inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The administration of the school in our example will compute the mean, and report that the average class size is 23.7:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean 23.6923076923\n"
     ]
    }
   ],
   "source": [
    "d = {7:8, 12:8, 17:14, 22:4, \n",
    "    27:6, 32:12, 37:8, 42:3, 47:2}\n",
    "pmf =  thinkstats2.Pmf(d, label='actual')\n",
    "print('mean', pmf.Mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if you survey a group of students and ask them how many students are in their class, the answer will be bigger. \n",
    "\n",
    "Here is the computation that shows why: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def BiasPmf(pmf, label):\n",
    "    new_pmf = pmf.Copy(label=label)\n",
    "    for x, p in pmf.Items():\n",
    "        # for each class size we multiply the number of students \n",
    "        # by x --> the number of students who observed that class size\n",
    "        new_pmf.Mult(x, x)\n",
    "    new_pmf.Normalize()\n",
    "    return new_pmf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot this observed mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEPCAYAAABRHfM8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHdhJREFUeJzt3Xt0VfW57vHvGw0obCMgGIQQpFJpsVUuKYXSboOezUVU\ndFOPgLc6BNGKpdZhtXZ4DFrdm8qp2tFqVahbLSC6OQq4kYZqQ8WOKFG8gwLKRQIKFaogSCDv+WPN\nhEUIZIZkrrmy1vMZI4M17+/6kaxnzd+8mbsjIiLSkJy4CxARkZZBgSEiIqEoMEREJBQFhoiIhKLA\nEBGRUBQYIiISSuSBYWbDzWylmX1gZjfXM32cmb0Z/Cw1s9OTpq0Nxi83s1ejrlVERA7NorwOw8xy\ngA+As4FKYBkwxt1XJs0zEFjh7v80s+FAibsPDKZ9CPR3922RFSkiIqFEvYcxAFjl7uvcvQp4EhiV\nPIO7l7v7P4PBcqBr0mRLQY0iIhJC1B/GXYENScMfc2Ag1DUeeD5p2IHFZrbMzCZEUJ+IiIR0dNwF\n1DCzIcCVwPeTRg92901m1olEcKxw96XxVCgikt2iDoyNQGHScEEw7gDBge6HgeHJxyvcfVPw7xYz\ne4ZEF9dBgWFmuiGWiEgjubs1Zv6oA2MZ0NPMugObgDHA2OQZzKwQmAtc5u5rksa3AXLcfYeZtQWG\nAlMOtSHdRDGhpKSEkpKSuMuIXTa3w/jpFQcMvz7/Yfqdf3VM1SRMH18U6/ZrZPPvRV1mjcoKIOLA\ncPd9ZjYJKCVxvGSGu68ws4mJyf4wcBvQAXjAEu+gyt0HAPnAM8Hew9HATHcvjbJekUwzfXwRJR8/\nR0kMH9h1g0tavsiPYbj7IqBXnXEPJb2eABx0QNvdPwL6RF2fiIiEo1NWM0xxcXHcJaQFtcN+aov9\n1BZNo8DIMPqDSFA77Ke22E9t0TRpc1qtiGSOk08+mXXr1tUOz9BVVLHp3r07a9eubZZ1KTBEpNmt\nW7dOZy6miSM5G+pQ1CUlIiKhKDBERCQUBYaIiISiwBARiUlOTg4ffvhh3GWEpsAQEQmpR48evPji\ni822vuY8IJ0KCgwRkZi0tDPJFBgikpWmTp1Kz549ycvL41vf+hbPPvts7bRHHnmE3r1710574403\nuPzyy1m/fj3nnXceeXl5TJs2jSVLltCtW7cD1pu8F7Js2TK+973v0b59e7p27cr111/P3r17U/o+\nm5OuwxCRlBs9+Q/Nur6591/T6GV69uzJyy+/TH5+Pk8//TSXXXYZq1ev5m9/+xt33HEH8+bNo1+/\nfnz44Yfk5uby+OOP89JLL/HHP/6RIUOGALBkyZLDdisdddRR3HfffXznO99hw4YNjBgxggceeICf\n/OQnR/xe46Q9DBHJSqNHjyY/Px+Aiy66iJ49e/LKK68wY8YMfv7zn9OvXz8Avva1rx2wF9GYbqR+\n/foxYMAAzIzCwkKuvvpqlixZ0rxvJIW0hyEiWenxxx/n3nvvrb1txs6dO9m6dSsbNmzglFNOaZZt\nrFq1ip/97GdUVFSwa9cu9u7dS//+/Ztl3XFQYIhIyh1JF1JzWr9+PVdffTV//etfGTRoEAB9+/YF\noLCwkDVr1tS7XN3up7Zt2/Lll1/WDu/bt48tW7bUDl977bX069ePOXPm0KZNG+6//37mzp3b3G8n\nZdQlJSJZZ+fOneTk5NCxY0eqq6t59NFHeeeddwC46qqrmDZtGq+//joAa9asYcOGDQDk5+cfcN3E\nqaeeyu7du3n++efZu3cvv/rVr9izZ0/t9C+++IK8vDzatGnDypUrefDBB1P4LpufAkNEss43v/lN\nbrzxRgYOHEjnzp159913+f73vw/AD3/4Q375y18ybtw48vLyuPDCC/nss88A+MUvfsGdd95Jhw4d\n+M1vfkNeXh6///3vueqqqygoKOC4446joKCgdjvTpk1j5syZ5OXlMXHiRMaMGXNAHS3tOgxraecB\n18fMPBPeh0hzSH40alzP0jazFneNQaY61P9FML5RiaU9DBERCUWBISIioSgwREQkFAWGiIiEosAQ\nEZFQFBgiIhKKAkNEREJRYIiISCi6l5Q02bwX32TO8xV8tacq1jpat8rl4hFFjDrrjFjrkJbryiuv\npFu3btxxxx1xlxLalClTWL16NU888UTk29IehjRZOoQFwFd7qpjzfEXDM4pkmFTdYkSBIU2WDmFR\nI51qEanPvn374i7hiKlLSppVXLetbu4nuElmW7lyJddeey1vvPEGBQUF3H333Zx33nkAbNmyhaFD\nh1JeXk7//v157LHHKCwsBOCGG25g1qxZ7N69m5NPPpnZs2fTu3dv9uzZw6233srTTz/Nnj17uPDC\nC7n33ntp3bo1S5Ys4dJLL+X666/n3nvvZejQoSxbtoxp06ZxzjnnAIkQOemkkygtLaVPnz6Ul5dz\n44038t5773HyySdz3333ceaZZwKwdu1afvSjH7F8+XIGDhzIqaeemrJ2U2CISMol3yCxOTTmJot7\n9+7lvPPOY/z48SxevJiXXnqJCy64gIqKRE2zZs1i4cKFDBgwgJtuuolLLrmEl156idLSUpYuXcrq\n1as57rjjeP/992nXrh0AN998Mx999BFvvfUWRx99NOPGjeOOO+7grrvuAmDz5s1s376d9evXU11d\nzT333MOsWbNqA2PRokV06tSJPn36sHHjRs4991xmzpzJsGHDeOGFFxg9ejTvv/8+J5xwAuPGjWPw\n4MEsXryY8vJyRo4cyQUXXNCs7XkoCgwRySrl5eXs3LmTm2++GYAhQ4Zw7rnnMmvWLABGjhzJ4MGD\nAbjrrrto164dGzduJDc3ly+++IL33nuPAQMG0KtXr9p1PvLII7z99tscf/zxANxyyy1ccskltYFx\n1FFHMWXKFHJzcwEYO3Ysffv2Zffu3RxzzDHMnj2bsWPHAjBz5kxGjhzJsGHDADj77LMpKipi4cKF\nFBcXU1FRwQsvvEBubi4/+MEPaveMUkHHMEQkq1RWVh7wjG5IPGWvsrIS4IBpbdu2pX379lRWVjJk\nyBAmTZrEddddR35+Ptdccw07duxgy5YtfPnll/Tv358OHTrQoUMHRowYwT/+8Y/a9XTq1Kk2LABO\nOeUUevfuzYIFC9i1axfz58/nkksuAWDdunU89dRTtetq3749L7/8Mps2baKyspL27dtz7LHH1q6r\ne/fukbRTfbSHISIpF9dzOgC6dOlS+wS9GuvXr6dXr1589NFHB0zbsWMHn332GV26dAFg0qRJTJo0\nia1bt3LRRRdxzz33UFJSQps2bXj33Xc56aST6t1mfWcxjRkzhlmzZrFv3z5OO+00evToASQC6/LL\nL+ehhx46aJn169ezbds2du3aVRsa69evJycnNd/9tYchIlnlu9/9Lm3atOHXv/41e/fupaysjOee\ne672aXgLFy7k73//O3v27OG2225j0KBBdO3alYqKCl599VX27t3LscceyzHHHENOTg5mxoQJE/jp\nT39a+zzvjRs3Ulpaetg6xowZQ2lpKQ8++CDjxo2rHX/ppZeyYMECSktLqa6uZvfu3SxZsoTKykoK\nCwspKiri9ttvp6qqiqVLl7JgwYLoGqsOBYaIZJXc3FwWLFjAwoUL6dixI5MmTeKJJ57g1FNPxcwY\nN24cJSUlnHDCCSxfvpw//elPAHz++edMmDCBDh060KNHDzp27MhNN90EwNSpU+nZsycDBw6kXbt2\nDB06lA8++OCwdXTu3JlBgwZRXl7OxRdfXDu+oKCAefPmcffdd9OpUye6d+/OtGnTqK6uBhLHOMrL\nyznhhBO48847ueKKKyJqqYNF/ohWMxsO3EcinGa4+9Q608cBNweDXwA/dve3wiybtA49ojVGyae0\npsNptXHVkC70iFZJ1mIe0WpmOcDvgGHAacBYM/tGndk+BP7V3c8AfgU83IhlRUQkRaLukhoArHL3\nde5eBTwJjEqewd3L3f2fwWA50DXssiIikjpRB0ZXIPl0hI/ZHwj1GQ88f4TLiohIhNLmtFozGwJc\nCXz/SJYvKSmpfV1cXExxcXGz1CUikgnKysooKytr0jqiDoyNQGHScEEw7gBmdjqJYxfD3X1bY5at\nkRwYIiJyoLpfpKdMmdLodUTdJbUM6Glm3c2sFTAGmJ88g5kVAnOBy9x9TWOWFRGR1Il0D8Pd95nZ\nJKCU/afGrjCziYnJ/jBwG9ABeMASl0NWufuAQy0bZb0i0jy6d++esmc0yOE1561DIj+G4e6LgF51\nxj2U9HoCMCHssiKS/tauXZsW14NI89KV3iIiEkranCUlIpmruZ9/0Ritc3M4v28Xhp3eObYaMoX2\nMEQkEq1z0+Pj5auqauYvr4y7jIyQHv+jIpJxzu/bJa1CQ5pOXVIiEolhp3eOvRsozq6wTJQe8S8i\nImlPgSEiIqEoMEREJBQdwxDJMJu3fk7lp9updmf05Hj78Fu3yuXiEUWMOuuMWOuQ5qE9DJEMUxMW\n6eCrPVXMeV4HnjOFAkMkw6RLWNT4ak9V3CVIM1GXlEgGi/P55snPWZfMoD0MEREJRYEhIiKhKDBE\nRCQUBYaIiISiwBARkVAUGCIiEooCQ0REQlFgiIhIKAoMEREJRYEhIiKhKDBERCQUBYaIiISiwBAR\nkVAUGCIiEooCQ0REQlFgiIhIKAoMEREJRYEhIiKhKDBERCQUBYaIiISiwBARkVAUGCIiEooCQ0RE\nQjk66g2Y2XDgPhLhNMPdp9aZ3gt4FOgH3Oruv0mathb4J1ANVLn7gKjrlcbbzrFsy2lLNcb46RWx\n1LAmpxM5OO2rd8ayfZFsEGlgmFkO8DvgbKASWGZm89x9ZdJs/wCuBy6oZxXVQLG7b4uyTmmamrCI\nWzXGtpy2cZchkrGi7pIaAKxy93XuXgU8CYxKnsHdt7r7a8Deepa3FNQoTZQOYVEjnWoRyTRRd0l1\nBTYkDX9MIkTCcmCxme0DHnb3R5qzOGl+08cXxbLdPjesi2W7Itkk8mMYTTTY3TeZWScSwbHC3ZfG\nXZSISDaKOjA2AoVJwwXBuFDcfVPw7xYze4bE3km9gVFSUlL7uri4mOLi4sZXKyKSocrKyigrK2vS\nOqIOjGVATzPrDmwCxgBjDzN/bQe0mbUBctx9h5m1BYYCUw61YHJgiIjIgep+kZ4y5ZAfp4cUaWC4\n+z4zmwSUsv+02hVmNjEx2R82s3ygAjgOqDazyUBvoBPwjJl5UOdMdy+Nsl4RETm0yI9huPsioFed\ncQ8lvf4E6FbPojuAPtFWJyIiYemUVRERCUWBISIioSgwREQkFAWGiIiEosAQEZFQFBgiIhKKAkNE\nREJRYIiISCgKDBERCUWBISIioSgwREQklMMGhpn9V9LrKyKvRkRE0lZDexhnJL2eHGUhIiKS3hoK\nDE9JFSIikvYaur15gZn9lsSDjWpe13L3n0RWmYiIpJWGAuOmpNcVURYiIiLp7bCB4e6PpaoQERFJ\nb4cNDDObf7jp7n5+85YjIiLpqqEuqUHABmA28AqJYxkiIpKFGgqMzsC/AWOBccD/ALPd/d2oC0t3\n8158kznPV/DVnqq4S4lfTqe4K5A0N3ryH2LZ7pqcTuSY0eXEdrFsP9Mc9rRad9/n7ovc/QpgILAa\nKDOzSSmpLo0pLA6WY9oBlf1at8qNuwQAqt2p/HR73GVkhAZvDWJmrc3s34E/AdcBvwWeibqwdKew\nOJC+xUldF48oSqvQkKZr6KD348C3gIXAFHd/JyVVtTBz778m7hJiNX66zriWg4066wxGnXVGwzNG\nqM8Nc2PdfqZp6BjGpcBOErcFmWxmNTFtgLt7XpTFiYhI+mjoOgzdzVZERICGu6SOAa4BegJvAX90\n972pKExERNJLQ3sQjwFFwNvAOcD/jbwiERFJSw0dw+jt7t8GMLMZwKvRlyQiIumooT2M2nNH1RUl\nIpLdGtrDOMPMPg9eG3BsMKyzpEREskxDZ0kdlapCREQkvem0WRERCUWBISIioSgwREQkFAWGiIiE\n0tBZUiLSCH9+azPzl1fyVVV13KWINLvI9zDMbLiZrTSzD8zs5nqm9zKzv5vZbjP7WWOWFUk36RQW\nOeiW3tK8Ig0MM8sBfgcMA04DxprZN+rM9g/geuCeI1hWJK2kU1i0r94ZdxmSYaLukhoArHL3dQBm\n9iQwClhZM4O7bwW2mtm5jV1WJJ1NH18Uy3ZHT9bzSSQaUXdJdQU2JA1/HIyLelkREWlmOktKRERC\nibpLaiNQmDRcEIxr9mVLSkpqXxcXF1NcXBy2RhGRjFdWVkZZWVmT1hF1YCwDeppZd2ATMAYYe5j5\n7UiXTQ4MERE5UN0v0lOmTGn0OiINDHffZ2aTgFIS3V8z3H2FmU1MTPaHzSwfqACOA6rNbDKJ53Ds\nqG/ZKOsVEZFDi/zCPXdfBPSqM+6hpNefAN3CLisiIvHQQW8REQlFtwYRaUabt35O5afbqXbX9RCS\ncbSHIdKMasIiHbRulRt3CZJhFBgizSidwuLiEfFcaS6ZS11SIhGZe/81cZcg0qy0hyEiIqEoMERE\nJBQFhoiIhKLAEBGRUBQYIiISigJDRERC0Wm1IpIVxk+P98r71rk5nN+3C8NO7xxrHU2hPQwRyVg5\npMeFlJB43vv85ZVxl9EkCgwRyVjtq3emXWi0ZOqSEpGM1Y5dtKveBcD08aNjqyPu7rDmoj0MEREJ\nRYEhIiKhKDBERCQUBYaIiISiwBARkVAUGCIiEooCQ0REQlFgiIhIKAoMEREJRYEhIiKhKDBERCQU\nBYaIiISiwBARkVB0t9oW7M9vbWb+8soWf8tkEWkZFBgtWLqFRevc9NhhHT35D/FtPKdTfNuWFiHO\nW5039al/6fEXLkck3cLi/L5dYtt+jlls265PutUj8UqXL1NNfeqf9jAyxPTxRXGXEKsuJ7aj8tPt\nVHv8T1fLMaPLie3iLkPSyPl9u6RNj0BTalBgSEbo3DGPzh3zAD1ZTdLPsNM7H3E3UHNpjt9NBYZk\nHH1oi0QjPTrWRJooXfqIa6RbPSLNQb/VkhHO79slbT6k4z4BQCQqkXdJmdlw4D4S4TTD3afWM89v\ngRHATuBKd18ejF8L/BOoBqrcfUDU9UrLlA59xCKZLtLAMLMc4HfA2UAlsMzM5rn7yqR5RgCnuPvX\nzey7wIPAwGByNVDs7tuirFNERBoW9T78AGCVu69z9yrgSWBUnXlGAY8DuPsrwPFmlh9MsxTUKCIi\nIUT9YdwV2JA0/HEw7nDzbEyax4HFZrbMzCZEVqWIiDQo3U+rHezum8ysE4ngWOHuS+ubsaSkpPZ1\ncXExxcXFqalQRKQF2PT+a2x6/zUASj5+7ojWEXVgbAQKk4YLgnF15+lW3zzuvin4d4uZPUOii6vB\nwBARkQOd1Ks/J/XqD0DJ+CKmTJnS6HVE3SW1DOhpZt3NrBUwBphfZ575wOUAZjYQ2O7un5hZGzP7\nl2B8W2Ao8E7E9YqIyCFEuofh7vvMbBJQyv7TaleY2cTEZH/Y3Rea2TlmtprgtNpg8XzgGTPzoM6Z\n7l4aZb0ikrlivYsx0LpVLhePKGLUWWfEWkdTRH4Mw90XAb3qjHuozvCkepb7COgTbXUikslat8rl\nqz1VcZcBwFd7qpjzfEWLDgydsioiGeviEUW0bpUbdxm10iW8jlS6nyUlInLERp11Rlp8o4+7O6y5\naA9DRERCUWCIiEgoCgwREQlFgSEiIqHooPcR2s6xbMtpSzWmJ7yJSFbQHsYRqgmLdJAuDw4Skcym\nT5ojlE5hoae7iUgqqEuqGUwfXxR3CSIikdMehoiIhKI9DBGRFIrrqu81OZ3IMaPLie2OeB3awxAR\niVi63M+q2p3KT7cf8fIKDBGRiKXTTRCr3Y94WXVJiYhELB1ugtjnhrlNXof2MEREJBQFhoiIhKLA\nEBGRUBQYIiISigJDRERCUWCIiEgoCgwREQlFgSEiIqEoMEREJBQFhoiIhKLAEBGRUBQYIiISigJD\nRERCUWCIiEgoCgwREQlFgSEiIqEoMEREJBQFhoiIhKLAEBGRUBQYIiISSuSBYWbDzWylmX1gZjcf\nYp7fmtkqM3vDzPo0ZlkREUmNSAPDzHKA3wHDgNOAsWb2jTrzjABOcfevAxOBP4RdVg5WVlYWdwlp\nQe2wn9piP7VF00S9hzEAWOXu69y9CngSGFVnnlHA4wDu/gpwvJnlh1xW6tAfRILaYT+1xX5qi6aJ\nOjC6AhuShj8OxoWZJ8yyIiKSIul40NviLkBERA5m7h7dys0GAiXuPjwYvgVwd5+aNM8fgL+6+5xg\neCVwJtCjoWWT1hHdmxARyVDu3qgv6EdHVUhgGdDTzLoDm4AxwNg688wHrgPmBAGz3d0/MbOtIZYF\nGv+mRUSk8SINDHffZ2aTgFIS3V8z3H2FmU1MTPaH3X2hmZ1jZquBncCVh1s2ynpFROTQIu2SEhGR\nzJGOB71Dy+YL+8xshpl9YmZvJY1rb2alZva+mf3ZzI6Ps8ZUMbMCM3vRzN41s7fN7CfB+KxrDzNr\nbWavmNnyoC1uD8ZnXVtA4nouM3vdzOYHw1nZDgBmttbM3gx+N14NxjWqPVpsYOjCPh4l8d6T3QL8\nxd17AS8Cv0h5VfHYC/zM3U8DBgHXBb8LWdce7v4VMMTd+wJ9gBFmNoAsbIvAZOC9pOFsbQeAaqDY\n3fu6+4BgXKPao8UGBll+YZ+7LwW21Rk9CngseP0YcEFKi4qJu2929zeC1zuAFUAB2dseXwYvW5M4\nTulkYVuYWQFwDjA9aXTWtUMS4+DP/Ea1R0sODF3Yd7AT3f0TSHyIAifGXE/KmdnJJL5ZlwP52dge\nQTfMcmAzsNjdl5GdbXEvcBOJwKyRje1Qw4HFZrbMzMYH4xrVHlGfVivxyqozGszsX4D/Bia7+456\nrs/JivZw92qgr5nlAc+Y2Wkc/N4zui3MbCTwibu/YWbFh5k1o9uhjsHuvsnMOgGlZvY+jfy9aMl7\nGBuBwqThgmBcNvskuA8XZtYZ+DTmelLGzI4mERZPuPu8YHTWtgeAu38OlAHDyb62GAycb2YfArOB\ns8zsCWBzlrVDLXffFPy7BXiWRLd+o34vWnJg1F4UaGatSFzYNz/mmlLNOPBWKvOBHwWvrwDm1V0g\ng/0ReM/d708al3XtYWYda850MbNjgX8jcUwnq9rC3W9190J3/xqJz4YX3f0yYAFZ1A41zKxNsAeO\nmbUFhgJv08jfixZ9HYaZDQfuZ/+Fff8Zc0kpY2azgGLgBOAT4HYS3xqeBroB64D/7e7b46oxVcxs\nMPA3En8AHvzcCrwKPEUWtYeZfZvEwcuc4GeOu99lZh3IsraoYWZnAje6+/nZ2g5m1gN4hsTfxtHA\nTHf/z8a2R4sODBERSZ2W3CUlIiIppMAQEZFQFBgiIhKKAkNEREJRYIiISCgKDBERCUWBIVnPzPLN\nbLaZrQrus/OcmdVcFPp2Cuvob2b3pWp7Io2le0mJJC5oetTdx0LtxW/5JG5ombILldz9NeC1VG1P\npLG0hyFZzcyGAHvc/ZGace7+tru/XGe+7mb2NzOrCH4GBuM7m9mS4CE9b5nZ4OBusY8Gw2+a2eR6\ntntR8ICj5WZWFow708wWBK//J1jncjPbbmaXBev9dfCApDfMbEKUbSNSl/YwJNt9i3Df6j8F/pe7\n7zGzniRuaPcdYBywyN3/w8wMaEPi9upd3f10gOCusXXdBgwN7h6aPN0B3H1ksGw/EvfJeha4Ctju\n7t8N7p/2spmVuvu6xr9tkcZTYIiEkws8ZGZ9gH3A14Pxy4AZZpYLzHP3N4M7pPYws/uBhUBpPetb\nCjxmZk8B/6++DZpZR+AJ4Ifu/oWZDQW+bWYXBbPkBXUoMCQl1CUl2e5doCjEfDcAm4O9hiKgFYC7\nvwT8K4lb6/+XmV0a3LztDBK3Fp/IgU98I1jux8AvSdz07TUza588PXgE8WygxN1X1IwGrg8esdnX\n3U9x97809g2LHCkFhmQ1d38RaJX0BDLM7NvBHXCTHQ9sCl5fDhwVzFsIfOruM0gEQ7/gDqBHufsz\nJLqe+tbdrpl9zd2XufvtJLq7utWZZSrwprs/nTTuz8CPg2d/YGZfD25hLpIS6pISgQuB+83sFmAX\nsBb4aZ15HgDmmtnlwCJgRzC+GLjJzKqAL0iESQHwaLCX4MAt9WzzHjOr6db6i7u/FdyGu8aNwDvB\no1Yd+D/u/kjwCNrXg+Mln5Jdz6SWmOn25iIiEoq6pEREJBQFhoiIhKLAEBGRUBQYIiISigJDRERC\nUWCIiEgoCgwREQlFgSEiIqH8f0aws8y0JPQzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x117f3b550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x117f71a10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "biased_pmf = BiasPmf(pmf, label='observed')\n",
    "thinkplot.PrePlot(2)\n",
    "thinkplot.Pmfs([pmf, biased_pmf])\n",
    "thinkplot.Show(xlabel='Class size', ylabel='PMF')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see in this plot, in the biased distribution there are fewer small classes and more large ones. The mean of the biased distribution is 29.1, almost 25% higher than the actual mean.\n",
    "\n",
    "If you choose x random students and ask them about the observed class size you will be facing the exact problem we see above. Here is the function the unbiases a Pmf:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def UnBiasPmf(pmf, label):\n",
    "    new_pmf = pmf.Copy(label=label)\n",
    "    for x, p in pmf.Items():\n",
    "        # devides each probability by x instead of multiplying\n",
    "        new_pmf.Mult(x, 1.0/x)\n",
    "    new_pmf.Normalize()\n",
    "    return new_pmf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='subchapter3.5'></a>\n",
    "## 3.5 DataFrame indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In section [1.4](chap01.ipynb) we read a panda DataFrame and used it to select and modify data columns. Now let's look at row selection. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.75105066, -0.19820176],\n",
       "       [ 1.91521304, -1.72859852],\n",
       "       [-0.79385791, -1.04171821],\n",
       "       [-0.68112962, -0.57233407]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we already imported numpy and pandas in the begining of this chapter\n",
    "array = np.random.randn(4,2)\n",
    "array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.751051</td>\n",
       "      <td>-0.198202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.915213</td>\n",
       "      <td>-1.728599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.793858</td>\n",
       "      <td>-1.041718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.681130</td>\n",
       "      <td>-0.572334</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1\n",
       "0 -0.751051 -0.198202\n",
       "1  1.915213 -1.728599\n",
       "2 -0.793858 -1.041718\n",
       "3 -0.681130 -0.572334"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(array)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>-0.751051</td>\n",
       "      <td>-0.198202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>1.915213</td>\n",
       "      <td>-1.728599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>-0.793858</td>\n",
       "      <td>-1.041718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>-0.681130</td>\n",
       "      <td>-0.572334</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B\n",
       "a -0.751051 -0.198202\n",
       "b  1.915213 -1.728599\n",
       "c -0.793858 -1.041718\n",
       "d -0.681130 -0.572334"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# by default rows and colums are numbered \n",
    "# however, you can provide a column name\n",
    "columns = ['A', 'B']\n",
    "# note that row names are called labels\n",
    "index = ['a', 'b', 'c', 'd']\n",
    "df = pd.DataFrame(array, columns=columns, index=index)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a   -0.751051\n",
       "b    1.915213\n",
       "c   -0.793858\n",
       "d   -0.681130\n",
       "Name: A, dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# simple indexing selects a column and returns a series\n",
    "df['A']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df['A'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A   -0.751051\n",
       "B   -0.198202\n",
       "Name: a, dtype: float64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to select a row by label you can use the loc attribute\n",
    "df.loc['a']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A   -0.751051\n",
       "B   -0.198202\n",
       "Name: a, dtype: float64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# if you know the index position of the row, use iloc\n",
    "df.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>1.915213</td>\n",
       "      <td>-1.728599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>-0.793858</td>\n",
       "      <td>-1.041718</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B\n",
       "b  1.915213 -1.728599\n",
       "c -0.793858 -1.041718"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using a list of labels with loc will result in a DataFrame\n",
    "indices = ['b', 'c']\n",
    "df.loc[indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>-0.751051</td>\n",
       "      <td>-0.198202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>1.915213</td>\n",
       "      <td>-1.728599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>-0.793858</td>\n",
       "      <td>-1.041718</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B\n",
       "a -0.751051 -0.198202\n",
       "b  1.915213 -1.728599\n",
       "c -0.793858 -1.041718"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# you can use a slice to select a range of rows by label\n",
    "# this notation will include the end of the slice\n",
    "df.loc['a':'c']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>1.915213</td>\n",
       "      <td>-1.728599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>-0.793858</td>\n",
       "      <td>-1.041718</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B\n",
       "b  1.915213 -1.728599\n",
       "c -0.793858 -1.041718"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# or by integer position\n",
    "# this notation will not include the end of the slice\n",
    "df.iloc[1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='subchapter3.6'></a>\n",
    "## 3.6 Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open the exercises in the links below. Some cells are already filled in, and you should execute them. Other cells give you instructions for exercises you should try."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<b>Exercise 3.1</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Exercise 1](chap03ex01.ipynb)\n",
    "\n",
    "[Exercise 1 solution](chap03ex01soln.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<b>Exercise 3.2</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Exercise 2](chap03ex02.ipynb)\n",
    "\n",
    "[Exercise 2 solution](chap03ex02soln.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<b>Exercise 3.3</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Exercise 3](chap03ex03.ipynb)\n",
    "\n",
    "[Exercise 3 solution](chap03ex03soln.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<b>Exercise 3.4</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Exercise 4](chap03ex04.ipynb)\n",
    "\n",
    "[Exercise 4 solution](chap03ex04soln.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='subchapter3.7'></a>\n",
    "## 3.7 Glossary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Probability mass function (PMF)</b>: a representation of a distribution as a function that maps from values to probabilities.\n",
    "\n",
    "<b>probability</b>: A frequency expressed as a fraction of the sample size.\n",
    "\n",
    "<b>normalization</b>: The process of dividing a frequency by a sample size to get a probability.\n",
    "\n",
    "<b>index</b>: In a pandas DataFrame, the index is a special column that con- tains the row labels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Next up: [Chapter 4: Cumulative distribution functions](chap04.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}