week1/intro_to_r.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An overly brief intro to R\n",
    "\n",
    "This is meant as a very brief introduction to R, mainly summarizing features of R that are different from most other programming languages. There's a brief tour of vectors and lists before diving into data frames and comparing the old (pre-`tidyverse`) and new (`tidyverse`) way of doing things."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": "true"
   },
   "source": [
    "# Table of Contents\n",
    " <p><div class=\"lev1 toc-item\"><a href=\"#An-overly-brief-intro-to-R\" data-toc-modified-id=\"An-overly-brief-intro-to-R-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>An overly brief intro to R</a></div><div class=\"lev2 toc-item\"><a href=\"#Vectors\" data-toc-modified-id=\"Vectors-11\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Vectors</a></div><div class=\"lev2 toc-item\"><a href=\"#Factors\" data-toc-modified-id=\"Factors-12\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Factors</a></div><div class=\"lev2 toc-item\"><a href=\"#Lists\" data-toc-modified-id=\"Lists-13\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Lists</a></div><div class=\"lev2 toc-item\"><a href=\"#Data-frames\" data-toc-modified-id=\"Data-frames-14\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>Data frames</a></div><div class=\"lev3 toc-item\"><a href=\"#Selecting-rows\" data-toc-modified-id=\"Selecting-rows-141\"><span class=\"toc-item-num\">1.4.1&nbsp;&nbsp;</span>Selecting rows</a></div><div class=\"lev4 toc-item\"><a href=\"#The-old-way:-logical-indexing\" data-toc-modified-id=\"The-old-way:-logical-indexing-1411\"><span class=\"toc-item-num\">1.4.1.1&nbsp;&nbsp;</span>The old way: logical indexing</a></div><div class=\"lev4 toc-item\"><a href=\"#The-new-way:-filter\" data-toc-modified-id=\"The-new-way:-filter-1412\"><span class=\"toc-item-num\">1.4.1.2&nbsp;&nbsp;</span>The new way: filter</a></div><div class=\"lev3 toc-item\"><a href=\"#Reordering-rows\" data-toc-modified-id=\"Reordering-rows-142\"><span class=\"toc-item-num\">1.4.2&nbsp;&nbsp;</span>Reordering rows</a></div><div class=\"lev4 toc-item\"><a href=\"#The-old-way:-vectorized-indexing-with-order\" data-toc-modified-id=\"The-old-way:-vectorized-indexing-with-order-1421\"><span class=\"toc-item-num\">1.4.2.1&nbsp;&nbsp;</span>The old way: vectorized indexing with order</a></div><div class=\"lev4 toc-item\"><a href=\"#The-new-way:-arrange\" data-toc-modified-id=\"The-new-way:-arrange-1422\"><span class=\"toc-item-num\">1.4.2.2&nbsp;&nbsp;</span>The new way: arrange</a></div><div class=\"lev3 toc-item\"><a href=\"#Selecting-columns\" data-toc-modified-id=\"Selecting-columns-143\"><span class=\"toc-item-num\">1.4.3&nbsp;&nbsp;</span>Selecting columns</a></div><div class=\"lev4 toc-item\"><a href=\"#The-old-way:-manual-or-logical-indexing\" data-toc-modified-id=\"The-old-way:-manual-or-logical-indexing-1431\"><span class=\"toc-item-num\">1.4.3.1&nbsp;&nbsp;</span>The old way: manual or logical indexing</a></div><div class=\"lev4 toc-item\"><a href=\"#The-new-way:-select\" data-toc-modified-id=\"The-new-way:-select-1432\"><span class=\"toc-item-num\">1.4.3.2&nbsp;&nbsp;</span>The new way: select</a></div><div class=\"lev3 toc-item\"><a href=\"#Modifying-columns\" data-toc-modified-id=\"Modifying-columns-144\"><span class=\"toc-item-num\">1.4.4&nbsp;&nbsp;</span>Modifying columns</a></div><div class=\"lev4 toc-item\"><a href=\"#The-old-way:-the-dollar-operator\" data-toc-modified-id=\"The-old-way:-the-dollar-operator-1441\"><span class=\"toc-item-num\">1.4.4.1&nbsp;&nbsp;</span>The old way: the dollar operator</a></div><div class=\"lev4 toc-item\"><a href=\"#The-new-way:-mutate\" data-toc-modified-id=\"The-new-way:-mutate-1442\"><span class=\"toc-item-num\">1.4.4.2&nbsp;&nbsp;</span>The new way: mutate</a></div><div class=\"lev3 toc-item\"><a href=\"#Summarizing-columns\" data-toc-modified-id=\"Summarizing-columns-145\"><span class=\"toc-item-num\">1.4.5&nbsp;&nbsp;</span>Summarizing columns</a></div><div class=\"lev4 toc-item\"><a href=\"#The-old-way:-the-dollar-operator\" data-toc-modified-id=\"The-old-way:-the-dollar-operator-1451\"><span class=\"toc-item-num\">1.4.5.1&nbsp;&nbsp;</span>The old way: the dollar operator</a></div><div class=\"lev4 toc-item\"><a href=\"#The-new-way:-summarize\" data-toc-modified-id=\"The-new-way:-summarize-1452\"><span class=\"toc-item-num\">1.4.5.2&nbsp;&nbsp;</span>The new way: summarize</a></div><div class=\"lev3 toc-item\"><a href=\"#Split-/-apply-/-combine-with-group_by\" data-toc-modified-id=\"Split-/-apply-/-combine-with-group_by-146\"><span class=\"toc-item-num\">1.4.6&nbsp;&nbsp;</span>Split / apply / combine with <code>group_by</code></a></div><div class=\"lev2 toc-item\"><a href=\"#Warnings\" data-toc-modified-id=\"Warnings-15\"><span class=\"toc-item-num\">1.5&nbsp;&nbsp;</span>Warnings</a></div><div class=\"lev3 toc-item\"><a href=\"#group_by-has-side-effects,-so-always-ungroup-intermediate-results\" data-toc-modified-id=\"group_by-has-side-effects,-so-always-ungroup-intermediate-results-151\"><span class=\"toc-item-num\">1.5.1&nbsp;&nbsp;</span><code>group_by</code> has side effects, so <strong>always <code>ungroup</code></strong> intermediate results</a></div><div class=\"lev4 toc-item\"><a href=\"#Ungroup-for-performance\" data-toc-modified-id=\"Ungroup-for-performance-1511\"><span class=\"toc-item-num\">1.5.1.1&nbsp;&nbsp;</span>Ungroup for performance</a></div><div class=\"lev4 toc-item\"><a href=\"#Ungroup-for-correctness\" data-toc-modified-id=\"Ungroup-for-correctness-1512\"><span class=\"toc-item-num\">1.5.1.2&nbsp;&nbsp;</span>Ungroup for correctness</a></div><div class=\"lev3 toc-item\"><a href=\"#Arranging-within-groups\" data-toc-modified-id=\"Arranging-within-groups-152\"><span class=\"toc-item-num\">1.5.2&nbsp;&nbsp;</span>Arranging within groups</a></div><div class=\"lev3 toc-item\"><a href=\"#Unintentionally-overwriting-columns\" data-toc-modified-id=\"Unintentionally-overwriting-columns-153\"><span class=\"toc-item-num\">1.5.3&nbsp;&nbsp;</span>Unintentionally overwriting columns</a></div><div class=\"lev3 toc-item\"><a href=\"#Tricky-variable-scoping\" data-toc-modified-id=\"Tricky-variable-scoping-154\"><span class=\"toc-item-num\">1.5.4&nbsp;&nbsp;</span>Tricky variable scoping</a></div><div class=\"lev3 toc-item\"><a href=\"#Forgetting-to-use-vectorized-functions\" data-toc-modified-id=\"Forgetting-to-use-vectorized-functions-155\"><span class=\"toc-item-num\">1.5.5&nbsp;&nbsp;</span>Forgetting to use vectorized functions</a></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading tidyverse: ggplot2\n",
      "Loading tidyverse: tibble\n",
      "Loading tidyverse: tidyr\n",
      "Loading tidyverse: readr\n",
      "Loading tidyverse: purrr\n",
      "Loading tidyverse: dplyr\n",
      "Conflicts with tidy packages -----------------------------------------------------------------------------------------------------------------------------------\n",
      "filter(): dplyr, stats\n",
      "lag():    dplyr, stats\n"
     ]
    }
   ],
   "source": [
    "# load all packages in the tidyverse, although we'll use only dplyr here\n",
    "library(tidyverse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>1</li>\n",
       "\t<li>2</li>\n",
       "\t<li>3</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 1\n",
       "\\item 2\n",
       "\\item 3\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 1\n",
       "2. 2\n",
       "3. 3\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 1 2 3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# vectors hold values of the same type\n",
    "# c() is the \"concatenation\" operator\n",
    "c(1,2,3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>1</li>\n",
       "\t<li>2</li>\n",
       "\t<li>3</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 1\n",
       "\\item 2\n",
       "\\item 3\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 1\n",
       "2. 2\n",
       "3. 3\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 1 2 3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# the colon operator is handy for generating sequential vectors\n",
    "1:3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>1</li>\n",
       "\t<li>3</li>\n",
       "\t<li>5</li>\n",
       "\t<li>7</li>\n",
       "\t<li>9</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 1\n",
       "\\item 3\n",
       "\\item 5\n",
       "\\item 7\n",
       "\\item 9\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 1\n",
       "2. 3\n",
       "3. 5\n",
       "4. 7\n",
       "5. 9\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 1 3 5 7 9"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# seq is good for other intervals\n",
    "seq(1, 10, by=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'1'</li>\n",
       "\t<li>'a'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item '1'\n",
       "\\item 'a'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. '1'\n",
       "2. 'a'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"1\" \"a\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'1'</li>\n",
       "\t<li>'2'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item '1'\n",
       "\\item '2'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. '1'\n",
       "2. '2'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"1\" \"2\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'alice'</li>\n",
       "\t<li>'bob'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'alice'\n",
       "\\item 'bob'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'alice'\n",
       "2. 'bob'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"alice\" \"bob\"  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# watch out for silent type coercion\n",
    "c(1,'a')\n",
    "c(1,'2')\n",
    "c('alice','bob')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong>alice:</strong> 1"
      ],
      "text/latex": [
       "\\textbf{alice:} 1"
      ],
      "text/markdown": [
       "**alice:** 1"
      ],
      "text/plain": [
       "alice \n",
       "    1 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# vectors can have named columns\n",
    "v <- c(alice=1, bob=2)\n",
    "v[\"alice\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Factors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Factor w/ 2 levels \"alice\",\"bob\": 1 2 2 1 1\n"
     ]
    }
   ],
   "source": [
    "# factors are for categorical variables\n",
    "# they have labels that are mapped to levels\n",
    "v <- c('alice','bob','bob','alice','alice')\n",
    "v <- as.factor(v)\n",
    "str(v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<dl class=dl-horizontal>\n",
       "\t<dt>alice</dt>\n",
       "\t\t<dd>3</dd>\n",
       "\t<dt>bob</dt>\n",
       "\t\t<dd>2</dd>\n",
       "</dl>\n"
      ],
      "text/latex": [
       "\\begin{description*}\n",
       "\\item[alice] 3\n",
       "\\item[bob] 2\n",
       "\\end{description*}\n"
      ],
      "text/markdown": [
       "alice\n",
       ":   3bob\n",
       ":   2\n",
       "\n"
      ],
      "text/plain": [
       "alice   bob \n",
       "    3     2 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# R tries to perform the right operation based on the data type\n",
    "# for example, computing a table of counts to summarize categorical data\n",
    "summary(v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol>\n",
       "\t<li>1</li>\n",
       "\t<li><ol class=list-inline>\n",
       "\t<li>'a'</li>\n",
       "\t<li>'b'</li>\n",
       "\t<li>'c'</li>\n",
       "</ol>\n",
       "</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate}\n",
       "\\item 1\n",
       "\\item \\begin{enumerate*}\n",
       "\\item 'a'\n",
       "\\item 'b'\n",
       "\\item 'c'\n",
       "\\end{enumerate*}\n",
       "\n",
       "\\end{enumerate}\n"
      ],
      "text/markdown": [
       "1. 1\n",
       "2. 1. 'a'\n",
       "2. 'b'\n",
       "3. 'c'\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[[1]]\n",
       "[1] 1\n",
       "\n",
       "[[2]]\n",
       "[1] \"a\" \"b\" \"c\"\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<dl>\n",
       "\t<dt>$alice</dt>\n",
       "\t\t<dd>1</dd>\n",
       "\t<dt>$bob</dt>\n",
       "\t\t<dd><ol class=list-inline>\n",
       "\t<li>'a'</li>\n",
       "\t<li>'b'</li>\n",
       "\t<li>'c'</li>\n",
       "</ol>\n",
       "</dd>\n",
       "</dl>\n"
      ],
      "text/latex": [
       "\\begin{description}\n",
       "\\item[\\$alice] 1\n",
       "\\item[\\$bob] \\begin{enumerate*}\n",
       "\\item 'a'\n",
       "\\item 'b'\n",
       "\\item 'c'\n",
       "\\end{enumerate*}\n",
       "\n",
       "\\end{description}\n"
      ],
      "text/markdown": [
       "$alice\n",
       ":   1\n",
       "$bob\n",
       ":   1. 'a'\n",
       "2. 'b'\n",
       "3. 'c'\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "$alice\n",
       "[1] 1\n",
       "\n",
       "$bob\n",
       "[1] \"a\" \"b\" \"c\"\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# lists hold mixed data types, similar to dictionaries in most other languages\n",
    "# they can have integer or named indices\n",
    "# we won't work with them much directly\n",
    "list(1, c('a','b','c'))\n",
    "list(alice=1, bob=c('a','b','c'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data frames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.1   </td><td>3.5   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>4.9   </td><td>3.0   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>4.7   </td><td>3.2   </td><td>1.3   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>4.6   </td><td>3.1   </td><td>1.5   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>5.0   </td><td>3.6   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>5.4   </td><td>3.9   </td><td>1.7   </td><td>0.4   </td><td>setosa</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       " Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species\\\\\n",
       "\\hline\n",
       "\t 5.1    & 3.5    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 4.9    & 3.0    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 4.7    & 3.2    & 1.3    & 0.2    & setosa\\\\\n",
       "\t 4.6    & 3.1    & 1.5    & 0.2    & setosa\\\\\n",
       "\t 5.0    & 3.6    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 5.4    & 3.9    & 1.7    & 0.4    & setosa\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n",
       "1 5.1          3.5         1.4          0.2         setosa \n",
       "2 4.9          3.0         1.4          0.2         setosa \n",
       "3 4.7          3.2         1.3          0.2         setosa \n",
       "4 4.6          3.1         1.5          0.2         setosa \n",
       "5 5.0          3.6         1.4          0.2         setosa \n",
       "6 5.4          3.9         1.7          0.4         setosa "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# data frames hold tabular data, implemented as a list of vectors of the same length\n",
    "\n",
    "# assign the first 6 rows of the built-in iris dataset as a toy dataframe\n",
    "df <- head(iris)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'data.frame':\t6 obs. of  5 variables:\n",
      " $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4\n",
      " $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9\n",
      " $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7\n",
      " $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4\n",
      " $ Species     : Factor w/ 3 levels \"setosa\",\"versicolor\",..: 1 1 1 1 1 1\n"
     ]
    }
   ],
   "source": [
    "# inspect the structure of the data frame\n",
    "str(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "  Sepal.Length    Sepal.Width     Petal.Length    Petal.Width           Species \n",
       " Min.   :4.600   Min.   :3.000   Min.   :1.300   Min.   :0.2000   setosa    :6  \n",
       " 1st Qu.:4.750   1st Qu.:3.125   1st Qu.:1.400   1st Qu.:0.2000   versicolor:0  \n",
       " Median :4.950   Median :3.350   Median :1.400   Median :0.2000   virginica :0  \n",
       " Mean   :4.950   Mean   :3.383   Mean   :1.450   Mean   :0.2333                 \n",
       " 3rd Qu.:5.075   3rd Qu.:3.575   3rd Qu.:1.475   3rd Qu.:0.2000                 \n",
       " Max.   :5.400   Max.   :3.900   Max.   :1.700   Max.   :0.4000                 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# compute summary statistics for each column\n",
    "# notice that the factor in the last column is summarized differently\n",
    "summary(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>5</th><td>5     </td><td>3.6   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       "  & Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species\\\\\n",
       "\\hline\n",
       "\t5 & 5      & 3.6    & 1.4    & 0.2    & setosa\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n",
       "5 5            3.6         1.4          0.2         setosa "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# look at the fifth row\n",
    "df[5, ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>3.5</li>\n",
       "\t<li>3</li>\n",
       "\t<li>3.2</li>\n",
       "\t<li>3.1</li>\n",
       "\t<li>3.6</li>\n",
       "\t<li>3.9</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 3.5\n",
       "\\item 3\n",
       "\\item 3.2\n",
       "\\item 3.1\n",
       "\\item 3.6\n",
       "\\item 3.9\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 3.5\n",
       "2. 3\n",
       "3. 3.2\n",
       "4. 3.1\n",
       "5. 3.6\n",
       "6. 3.9\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 3.5 3.0 3.2 3.1 3.6 3.9"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>3.5</li>\n",
       "\t<li>3</li>\n",
       "\t<li>3.2</li>\n",
       "\t<li>3.1</li>\n",
       "\t<li>3.6</li>\n",
       "\t<li>3.9</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 3.5\n",
       "\\item 3\n",
       "\\item 3.2\n",
       "\\item 3.1\n",
       "\\item 3.6\n",
       "\\item 3.9\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 3.5\n",
       "2. 3\n",
       "3. 3.2\n",
       "4. 3.1\n",
       "5. 3.6\n",
       "6. 3.9\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 3.5 3.0 3.2 3.1 3.6 3.9"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>3.5</li>\n",
       "\t<li>3</li>\n",
       "\t<li>3.2</li>\n",
       "\t<li>3.1</li>\n",
       "\t<li>3.6</li>\n",
       "\t<li>3.9</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 3.5\n",
       "\\item 3\n",
       "\\item 3.2\n",
       "\\item 3.1\n",
       "\\item 3.6\n",
       "\\item 3.9\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 3.5\n",
       "2. 3\n",
       "3. 3.2\n",
       "4. 3.1\n",
       "5. 3.6\n",
       "6. 3.9\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 3.5 3.0 3.2 3.1 3.6 3.9"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# three different ways to look at the second column\n",
    "\n",
    "# by index\n",
    "df[ , 2]\n",
    "\n",
    "# by name\n",
    "df[ , \"Sepal.Width\"]\n",
    "\n",
    "# extracting the column vector\n",
    "df$Sepal.Width"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The old way: logical indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>5.1   </td><td>3.5   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>5.0   </td><td>3.6   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><th scope=row>6</th><td>5.4   </td><td>3.9   </td><td>1.7   </td><td>0.4   </td><td>setosa</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       "  & Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species\\\\\n",
       "\\hline\n",
       "\t1 & 5.1    & 3.5    & 1.4    & 0.2    & setosa\\\\\n",
       "\t5 & 5.0    & 3.6    & 1.4    & 0.2    & setosa\\\\\n",
       "\t6 & 5.4    & 3.9    & 1.7    & 0.4    & setosa\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n",
       "1 5.1          3.5         1.4          0.2         setosa \n",
       "5 5.0          3.6         1.4          0.2         setosa \n",
       "6 5.4          3.9         1.7          0.4         setosa "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# select all rows with Sepal.Length at least 5\n",
    "df[df$Sepal.Length >= 5, ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>5.1   </td><td>3.5   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>5.0   </td><td>3.6   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><th scope=row>6</th><td>5.4   </td><td>3.9   </td><td>1.7   </td><td>0.4   </td><td>setosa</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       "  & Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species\\\\\n",
       "\\hline\n",
       "\t1 & 5.1    & 3.5    & 1.4    & 0.2    & setosa\\\\\n",
       "\t5 & 5.0    & 3.6    & 1.4    & 0.2    & setosa\\\\\n",
       "\t6 & 5.4    & 3.9    & 1.7    & 0.4    & setosa\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n",
       "1 5.1          3.5         1.4          0.2         setosa \n",
       "5 5.0          3.6         1.4          0.2         setosa \n",
       "6 5.4          3.9         1.7          0.4         setosa "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# equivalent to selecting rows 1, 5, and 6 using a logical vector\n",
    "df[c(T,F,F,F,T,T), ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The new way: filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.1   </td><td>3.5   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>5.0   </td><td>3.6   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>5.4   </td><td>3.9   </td><td>1.7   </td><td>0.4   </td><td>setosa</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       " Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species\\\\\n",
       "\\hline\n",
       "\t 5.1    & 3.5    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 5.0    & 3.6    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 5.4    & 3.9    & 1.7    & 0.4    & setosa\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n",
       "1 5.1          3.5         1.4          0.2         setosa \n",
       "2 5.0          3.6         1.4          0.2         setosa \n",
       "3 5.4          3.9         1.7          0.4         setosa "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# nicer syntax for the same thing\n",
    "# note: the second entry of filter is a logical vector\n",
    "filter(df, Sepal.Length >= 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.1   </td><td>3.5   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>5.0   </td><td>3.6   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       " Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species\\\\\n",
       "\\hline\n",
       "\t 5.1    & 3.5    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 5.0    & 3.6    & 1.4    & 0.2    & setosa\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n",
       "1 5.1          3.5         1.4          0.2         setosa \n",
       "2 5.0          3.6         1.4          0.2         setosa "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# filter ANDs conditions when given multiple arguments\n",
    "filter(df, Sepal.Length >= 5, Petal.Length <= 1.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.1   </td><td>3.5   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>4.9   </td><td>3.0   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>4.7   </td><td>3.2   </td><td>1.3   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>5.0   </td><td>3.6   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>5.4   </td><td>3.9   </td><td>1.7   </td><td>0.4   </td><td>setosa</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       " Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species\\\\\n",
       "\\hline\n",
       "\t 5.1    & 3.5    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 4.9    & 3.0    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 4.7    & 3.2    & 1.3    & 0.2    & setosa\\\\\n",
       "\t 5.0    & 3.6    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 5.4    & 3.9    & 1.7    & 0.4    & setosa\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n",
       "1 5.1          3.5         1.4          0.2         setosa \n",
       "2 4.9          3.0         1.4          0.2         setosa \n",
       "3 4.7          3.2         1.3          0.2         setosa \n",
       "4 5.0          3.6         1.4          0.2         setosa \n",
       "5 5.4          3.9         1.7          0.4         setosa "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# other logical combinations must be manually specified as one argument\n",
    "# make sure to use the vector-friendly | and & (not || and &&)\n",
    "filter(df, Sepal.Length >= 5 | Petal.Length <= 1.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "100"
      ],
      "text/latex": [
       "100"
      ],
      "text/markdown": [
       "100"
      ],
      "text/plain": [
       "[1] 100"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# count the rows in the full iris dataframe where the species starts with a v\n",
    "nrow(filter(iris, grepl('^v', Species)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reordering rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The old way: vectorized indexing with order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>4</th><td>4.6   </td><td>3.1   </td><td>1.5   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><th scope=row>3</th><td>4.7   </td><td>3.2   </td><td>1.3   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><th scope=row>2</th><td>4.9   </td><td>3.0   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>5.0   </td><td>3.6   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><th scope=row>1</th><td>5.1   </td><td>3.5   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><th scope=row>6</th><td>5.4   </td><td>3.9   </td><td>1.7   </td><td>0.4   </td><td>setosa</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       "  & Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species\\\\\n",
       "\\hline\n",
       "\t4 & 4.6    & 3.1    & 1.5    & 0.2    & setosa\\\\\n",
       "\t3 & 4.7    & 3.2    & 1.3    & 0.2    & setosa\\\\\n",
       "\t2 & 4.9    & 3.0    & 1.4    & 0.2    & setosa\\\\\n",
       "\t5 & 5.0    & 3.6    & 1.4    & 0.2    & setosa\\\\\n",
       "\t1 & 5.1    & 3.5    & 1.4    & 0.2    & setosa\\\\\n",
       "\t6 & 5.4    & 3.9    & 1.7    & 0.4    & setosa\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n",
       "4 4.6          3.1         1.5          0.2         setosa \n",
       "3 4.7          3.2         1.3          0.2         setosa \n",
       "2 4.9          3.0         1.4          0.2         setosa \n",
       "5 5.0          3.6         1.4          0.2         setosa \n",
       "1 5.1          3.5         1.4          0.2         setosa \n",
       "6 5.4          3.9         1.7          0.4         setosa "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df[order(df$Sepal.Length), ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The new way: arrange"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>4.6   </td><td>3.1   </td><td>1.5   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>4.7   </td><td>3.2   </td><td>1.3   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>4.9   </td><td>3.0   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>5.0   </td><td>3.6   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>5.1   </td><td>3.5   </td><td>1.4   </td><td>0.2   </td><td>setosa</td></tr>\n",
       "\t<tr><td>5.4   </td><td>3.9   </td><td>1.7   </td><td>0.4   </td><td>setosa</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       " Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species\\\\\n",
       "\\hline\n",
       "\t 4.6    & 3.1    & 1.5    & 0.2    & setosa\\\\\n",
       "\t 4.7    & 3.2    & 1.3    & 0.2    & setosa\\\\\n",
       "\t 4.9    & 3.0    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 5.0    & 3.6    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 5.1    & 3.5    & 1.4    & 0.2    & setosa\\\\\n",
       "\t 5.4    & 3.9    & 1.7    & 0.4    & setosa\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species\n",
       "1 4.6          3.1         1.5          0.2         setosa \n",
       "2 4.7          3.2         1.3          0.2         setosa \n",
       "3 4.9          3.0         1.4          0.2         setosa \n",
       "4 5.0          3.6         1.4          0.2         setosa \n",
       "5 5.1          3.5         1.4          0.2         setosa \n",
       "6 5.4          3.9         1.7          0.4         setosa "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# nicer syntax for the same thing\n",
    "arrange(df, Sepal.Length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The old way: manual or logical indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.1</td><td>3.5</td></tr>\n",
       "\t<tr><td>4.9</td><td>3.0</td></tr>\n",
       "\t<tr><td>4.7</td><td>3.2</td></tr>\n",
       "\t<tr><td>4.6</td><td>3.1</td></tr>\n",
       "\t<tr><td>5.0</td><td>3.6</td></tr>\n",
       "\t<tr><td>5.4</td><td>3.9</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Sepal.Length & Sepal.Width\\\\\n",
       "\\hline\n",
       "\t 5.1 & 3.5\\\\\n",
       "\t 4.9 & 3.0\\\\\n",
       "\t 4.7 & 3.2\\\\\n",
       "\t 4.6 & 3.1\\\\\n",
       "\t 5.0 & 3.6\\\\\n",
       "\t 5.4 & 3.9\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width\n",
       "1 5.1          3.5        \n",
       "2 4.9          3.0        \n",
       "3 4.7          3.2        \n",
       "4 4.6          3.1        \n",
       "5 5.0          3.6        \n",
       "6 5.4          3.9        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# extract the Sepal columns\n",
    "df[ , c(\"Sepal.Length\", \"Sepal.Width\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.1</td><td>3.5</td></tr>\n",
       "\t<tr><td>4.9</td><td>3.0</td></tr>\n",
       "\t<tr><td>4.7</td><td>3.2</td></tr>\n",
       "\t<tr><td>4.6</td><td>3.1</td></tr>\n",
       "\t<tr><td>5.0</td><td>3.6</td></tr>\n",
       "\t<tr><td>5.4</td><td>3.9</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Sepal.Length & Sepal.Width\\\\\n",
       "\\hline\n",
       "\t 5.1 & 3.5\\\\\n",
       "\t 4.9 & 3.0\\\\\n",
       "\t 4.7 & 3.2\\\\\n",
       "\t 4.6 & 3.1\\\\\n",
       "\t 5.0 & 3.6\\\\\n",
       "\t 5.4 & 3.9\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width\n",
       "1 5.1          3.5        \n",
       "2 4.9          3.0        \n",
       "3 4.7          3.2        \n",
       "4 4.6          3.1        \n",
       "5 5.0          3.6        \n",
       "6 5.4          3.9        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# same as above, but matches any column starting with \"Sepal\"\n",
    "df[, grepl(\"Sepal\", names(df))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The new way: select"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.1</td><td>3.5</td></tr>\n",
       "\t<tr><td>4.9</td><td>3.0</td></tr>\n",
       "\t<tr><td>4.7</td><td>3.2</td></tr>\n",
       "\t<tr><td>4.6</td><td>3.1</td></tr>\n",
       "\t<tr><td>5.0</td><td>3.6</td></tr>\n",
       "\t<tr><td>5.4</td><td>3.9</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Sepal.Length & Sepal.Width\\\\\n",
       "\\hline\n",
       "\t 5.1 & 3.5\\\\\n",
       "\t 4.9 & 3.0\\\\\n",
       "\t 4.7 & 3.2\\\\\n",
       "\t 4.6 & 3.1\\\\\n",
       "\t 5.0 & 3.6\\\\\n",
       "\t 5.4 & 3.9\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width\n",
       "1 5.1          3.5        \n",
       "2 4.9          3.0        \n",
       "3 4.7          3.2        \n",
       "4 4.6          3.1        \n",
       "5 5.0          3.6        \n",
       "6 5.4          3.9        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# nicer syntax for the same thing\n",
    "select(df, Sepal.Length, Sepal.Width)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.1</td><td>3.5</td></tr>\n",
       "\t<tr><td>4.9</td><td>3.0</td></tr>\n",
       "\t<tr><td>4.7</td><td>3.2</td></tr>\n",
       "\t<tr><td>4.6</td><td>3.1</td></tr>\n",
       "\t<tr><td>5.0</td><td>3.6</td></tr>\n",
       "\t<tr><td>5.4</td><td>3.9</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Sepal.Length & Sepal.Width\\\\\n",
       "\\hline\n",
       "\t 5.1 & 3.5\\\\\n",
       "\t 4.9 & 3.0\\\\\n",
       "\t 4.7 & 3.2\\\\\n",
       "\t 4.6 & 3.1\\\\\n",
       "\t 5.0 & 3.6\\\\\n",
       "\t 5.4 & 3.9\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width\n",
       "1 5.1          3.5        \n",
       "2 4.9          3.0        \n",
       "3 4.7          3.2        \n",
       "4 4.6          3.1        \n",
       "5 5.0          3.6        \n",
       "6 5.4          3.9        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "select(df, matches(\"Sepal\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modifying columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The old way: the dollar operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a new column by adding the sepal and petal widths\n",
    "df$total_width <- df$Sepal.Width + df$Petal.Width"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The new way: mutate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th><th scope=col>total_width</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.1   </td><td>3.5   </td><td>1.4   </td><td>0.2   </td><td>setosa</td><td>3.7   </td></tr>\n",
       "\t<tr><td>4.9   </td><td>3.0   </td><td>1.4   </td><td>0.2   </td><td>setosa</td><td>3.2   </td></tr>\n",
       "\t<tr><td>4.7   </td><td>3.2   </td><td>1.3   </td><td>0.2   </td><td>setosa</td><td>3.4   </td></tr>\n",
       "\t<tr><td>4.6   </td><td>3.1   </td><td>1.5   </td><td>0.2   </td><td>setosa</td><td>3.3   </td></tr>\n",
       "\t<tr><td>5.0   </td><td>3.6   </td><td>1.4   </td><td>0.2   </td><td>setosa</td><td>3.8   </td></tr>\n",
       "\t<tr><td>5.4   </td><td>3.9   </td><td>1.7   </td><td>0.4   </td><td>setosa</td><td>4.3   </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species & total\\_width\\\\\n",
       "\\hline\n",
       "\t 5.1    & 3.5    & 1.4    & 0.2    & setosa & 3.7   \\\\\n",
       "\t 4.9    & 3.0    & 1.4    & 0.2    & setosa & 3.2   \\\\\n",
       "\t 4.7    & 3.2    & 1.3    & 0.2    & setosa & 3.4   \\\\\n",
       "\t 4.6    & 3.1    & 1.5    & 0.2    & setosa & 3.3   \\\\\n",
       "\t 5.0    & 3.6    & 1.4    & 0.2    & setosa & 3.8   \\\\\n",
       "\t 5.4    & 3.9    & 1.7    & 0.4    & setosa & 4.3   \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species total_width\n",
       "1 5.1          3.5         1.4          0.2         setosa  3.7        \n",
       "2 4.9          3.0         1.4          0.2         setosa  3.2        \n",
       "3 4.7          3.2         1.3          0.2         setosa  3.4        \n",
       "4 4.6          3.1         1.5          0.2         setosa  3.3        \n",
       "5 5.0          3.6         1.4          0.2         setosa  3.8        \n",
       "6 5.4          3.9         1.7          0.4         setosa  4.3        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# nicer syntax for the same thing\n",
    "# note: mutate always gives back same number of rows\n",
    "mutate(df, total_width = Sepal.Width + Petal.Width)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summarizing columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The old way: the dollar operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "4.95"
      ],
      "text/latex": [
       "4.95"
      ],
      "text/markdown": [
       "4.95"
      ],
      "text/plain": [
       "[1] 4.95"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# compute the average sepal length\n",
    "mean(df$Sepal.Length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The new way: summarize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>mean_sepal_length</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>4.95</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|l}\n",
       " mean\\_sepal\\_length\\\\\n",
       "\\hline\n",
       "\t 4.95\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  mean_sepal_length\n",
       "1 4.95             "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# nicer syntax for the same thing\n",
    "# note: summarize gives back one row\n",
    "summarize(df, mean_sepal_length=mean(Sepal.Length))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split / apply / combine with `group_by`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We won't even both with the old way.\n",
    "It involved the [aggregate](http://stats.stackexchange.com/questions/8225/how-to-summarize-data-by-group-in-r) function, and it was ugly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classes ‘grouped_df’, ‘tbl_df’, ‘tbl’ and 'data.frame':\t150 obs. of  5 variables:\n",
      " $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...\n",
      " $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...\n",
      " $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...\n",
      " $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...\n",
      " $ Species     : Factor w/ 3 levels \"setosa\",\"versicolor\",..: 1 1 1 1 1 1 1 1 1 1 ...\n",
      " - attr(*, \"vars\")=List of 1\n",
      "  ..$ : symbol Species\n",
      " - attr(*, \"drop\")= logi TRUE\n",
      " - attr(*, \"indices\")=List of 3\n",
      "  ..$ : int  0 1 2 3 4 5 6 7 8 9 ...\n",
      "  ..$ : int  50 51 52 53 54 55 56 57 58 59 ...\n",
      "  ..$ : int  100 101 102 103 104 105 106 107 108 109 ...\n",
      " - attr(*, \"group_sizes\")= int  50 50 50\n",
      " - attr(*, \"biggest_group_size\")= int 50\n",
      " - attr(*, \"labels\")='data.frame':\t3 obs. of  1 variable:\n",
      "  ..$ Species: Factor w/ 3 levels \"setosa\",\"versicolor\",..: 1 2 3\n",
      "  ..- attr(*, \"vars\")=List of 1\n",
      "  .. ..$ : symbol Species\n",
      "  ..- attr(*, \"drop\")= logi TRUE\n"
     ]
    }
   ],
   "source": [
    "# first create \"grouped data frame\" by Species\n",
    "# notice this modifies the internals of the data frame,\n",
    "# adding indices that give the row numbers for each group\n",
    "iris_by_species <- group_by(iris, Species)\n",
    "str(iris_by_species)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Species</th><th scope=col>count</th><th scope=col>mean_sepal_length</th><th scope=col>sd_sepal_length</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>setosa    </td><td>50        </td><td>5.006     </td><td>0.3524897 </td></tr>\n",
       "\t<tr><td>versicolor</td><td>50        </td><td>5.936     </td><td>0.5161711 </td></tr>\n",
       "\t<tr><td>virginica </td><td>50        </td><td>6.588     </td><td>0.6358796 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " Species & count & mean\\_sepal\\_length & sd\\_sepal\\_length\\\\\n",
       "\\hline\n",
       "\t setosa     & 50         & 5.006      & 0.3524897 \\\\\n",
       "\t versicolor & 50         & 5.936      & 0.5161711 \\\\\n",
       "\t virginica  & 50         & 6.588      & 0.6358796 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Species    count mean_sepal_length sd_sepal_length\n",
       "1 setosa     50    5.006             0.3524897      \n",
       "2 versicolor 50    5.936             0.5161711      \n",
       "3 virginica  50    6.588             0.6358796      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# the summarize function respects these groups, and now returns one row per group\n",
    "# compute the average and standard deviation of sepal length for each species\n",
    "summarize(iris_by_species,\n",
    "          count = n(),\n",
    "          mean_sepal_length = mean(Sepal.Length),\n",
    "          sd_sepal_length = sd(Sepal.Length))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Species</th><th scope=col>count</th><th scope=col>mean_sepal_length</th><th scope=col>sd_sepal_length</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>setosa    </td><td>50        </td><td>5.006     </td><td>0.3524897 </td></tr>\n",
       "\t<tr><td>versicolor</td><td>50        </td><td>5.936     </td><td>0.5161711 </td></tr>\n",
       "\t<tr><td>virginica </td><td>50        </td><td>6.588     </td><td>0.6358796 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " Species & count & mean\\_sepal\\_length & sd\\_sepal\\_length\\\\\n",
       "\\hline\n",
       "\t setosa     & 50         & 5.006      & 0.3524897 \\\\\n",
       "\t versicolor & 50         & 5.936      & 0.5161711 \\\\\n",
       "\t virginica  & 50         & 6.588      & 0.6358796 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Species    count mean_sepal_length sd_sepal_length\n",
       "1 setosa     50    5.006             0.3524897      \n",
       "2 versicolor 50    5.936             0.5161711      \n",
       "3 virginica  50    6.588             0.6358796      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# we can do the same thing without the intermediate data frame,\n",
    "# reading it inside-out\n",
    "summarize(group_by(iris, Species),\n",
    "          count = n(),\n",
    "          mean_sepal_length = mean(Sepal.Length),\n",
    "          sd_sepal_length = sd(Sepal.Length))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Species</th><th scope=col>count</th><th scope=col>mean_sepal_length</th><th scope=col>sd_sepal_length</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>setosa    </td><td>50        </td><td>5.006     </td><td>0.3524897 </td></tr>\n",
       "\t<tr><td>versicolor</td><td>50        </td><td>5.936     </td><td>0.5161711 </td></tr>\n",
       "\t<tr><td>virginica </td><td>50        </td><td>6.588     </td><td>0.6358796 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " Species & count & mean\\_sepal\\_length & sd\\_sepal\\_length\\\\\n",
       "\\hline\n",
       "\t setosa     & 50         & 5.006      & 0.3524897 \\\\\n",
       "\t versicolor & 50         & 5.936      & 0.5161711 \\\\\n",
       "\t virginica  & 50         & 6.588      & 0.6358796 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Species    count mean_sepal_length sd_sepal_length\n",
       "1 setosa     50    5.006             0.3524897      \n",
       "2 versicolor 50    5.936             0.5161711      \n",
       "3 virginica  50    6.588             0.6358796      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# reading inside-out is awkward\n",
    "# the pipe operator (`%>%`) to the rescue\n",
    "# df %>% f() is equivalent to f(df)\n",
    "# df %>% f(x, .) is equivalent to f(x, df)\n",
    "iris %>%\n",
    "  group_by(Species) %>%\n",
    "  summarize(count = n(),\n",
    "            mean_sepal_length = mean(Sepal.Length),\n",
    "            sd_sepal_length = sd(Sepal.Length))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.8       </td><td>4.0       </td><td>1.2       </td><td>0.2       </td><td>setosa    </td></tr>\n",
       "\t<tr><td>7.0       </td><td>3.2       </td><td>4.7       </td><td>1.4       </td><td>versicolor</td></tr>\n",
       "\t<tr><td>7.9       </td><td>3.8       </td><td>6.4       </td><td>2.0       </td><td>virginica </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       " Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species\\\\\n",
       "\\hline\n",
       "\t 5.8        & 4.0        & 1.2        & 0.2        & setosa    \\\\\n",
       "\t 7.0        & 3.2        & 4.7        & 1.4        & versicolor\\\\\n",
       "\t 7.9        & 3.8        & 6.4        & 2.0        & virginica \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species   \n",
       "1 5.8          4.0         1.2          0.2         setosa    \n",
       "2 7.0          3.2         4.7          1.4         versicolor\n",
       "3 7.9          3.8         6.4          2.0         virginica "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# the other dplyr verbs respect group_by as well\n",
    "# for instance, select the flower for each species with the largest sepal length\n",
    "iris %>%\n",
    "    group_by(Species) %>%\n",
    "    filter(Sepal.Length == max(Sepal.Length))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th><th scope=col>z_sepal_length</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.1        </td><td>3.5        </td><td>1.4        </td><td>0.2        </td><td>setosa     </td><td> 0.26667447</td></tr>\n",
       "\t<tr><td>4.9        </td><td>3.0        </td><td>1.4        </td><td>0.2        </td><td>setosa     </td><td>-0.30071802</td></tr>\n",
       "\t<tr><td>4.7        </td><td>3.2        </td><td>1.3        </td><td>0.2        </td><td>setosa     </td><td>-0.86811050</td></tr>\n",
       "\t<tr><td>4.6        </td><td>3.1        </td><td>1.5        </td><td>0.2        </td><td>setosa     </td><td>-1.15180675</td></tr>\n",
       "\t<tr><td>5.0        </td><td>3.6        </td><td>1.4        </td><td>0.2        </td><td>setosa     </td><td>-0.01702177</td></tr>\n",
       "\t<tr><td>5.4        </td><td>3.9        </td><td>1.7        </td><td>0.4        </td><td>setosa     </td><td> 1.11776320</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species & z\\_sepal\\_length\\\\\n",
       "\\hline\n",
       "\t 5.1         & 3.5         & 1.4         & 0.2         & setosa      &  0.26667447\\\\\n",
       "\t 4.9         & 3.0         & 1.4         & 0.2         & setosa      & -0.30071802\\\\\n",
       "\t 4.7         & 3.2         & 1.3         & 0.2         & setosa      & -0.86811050\\\\\n",
       "\t 4.6         & 3.1         & 1.5         & 0.2         & setosa      & -1.15180675\\\\\n",
       "\t 5.0         & 3.6         & 1.4         & 0.2         & setosa      & -0.01702177\\\\\n",
       "\t 5.4         & 3.9         & 1.7         & 0.4         & setosa      &  1.11776320\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species z_sepal_length\n",
       "1 5.1          3.5         1.4          0.2         setosa   0.26667447   \n",
       "2 4.9          3.0         1.4          0.2         setosa  -0.30071802   \n",
       "3 4.7          3.2         1.3          0.2         setosa  -0.86811050   \n",
       "4 4.6          3.1         1.5          0.2         setosa  -1.15180675   \n",
       "5 5.0          3.6         1.4          0.2         setosa  -0.01702177   \n",
       "6 5.4          3.9         1.7          0.4         setosa   1.11776320   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# you can use any vectorized function inside of summarize, mutate, etc\n",
    "# so long as it returns a vector of the appropriate length (1 or N rows)\n",
    "# for instance, compute z-scores for the sepal lengths within species\n",
    "iris %>%\n",
    "    group_by(Species) %>%\n",
    "    mutate(z_sepal_length = (Sepal.Length - mean(Sepal.Length)) / sd(Sepal.Length)) %>%\n",
    "    head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Sepal.Length</th><th scope=col>Sepal.Width</th><th scope=col>Petal.Length</th><th scope=col>Petal.Width</th><th scope=col>Species</th><th scope=col>sepal_length_rank</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>7.9       </td><td>3.8       </td><td>6.4       </td><td>2.0       </td><td>virginica </td><td>1         </td></tr>\n",
       "\t<tr><td>7.7       </td><td>3.8       </td><td>6.7       </td><td>2.2       </td><td>virginica </td><td>2         </td></tr>\n",
       "\t<tr><td>7.7       </td><td>2.6       </td><td>6.9       </td><td>2.3       </td><td>virginica </td><td>3         </td></tr>\n",
       "\t<tr><td>7.0       </td><td>3.2       </td><td>4.7       </td><td>1.4       </td><td>versicolor</td><td>1         </td></tr>\n",
       "\t<tr><td>6.9       </td><td>3.1       </td><td>4.9       </td><td>1.5       </td><td>versicolor</td><td>2         </td></tr>\n",
       "\t<tr><td>6.8       </td><td>2.8       </td><td>4.8       </td><td>1.4       </td><td>versicolor</td><td>3         </td></tr>\n",
       "\t<tr><td>5.8       </td><td>4.0       </td><td>1.2       </td><td>0.2       </td><td>setosa    </td><td>1         </td></tr>\n",
       "\t<tr><td>5.7       </td><td>4.4       </td><td>1.5       </td><td>0.4       </td><td>setosa    </td><td>2         </td></tr>\n",
       "\t<tr><td>5.7       </td><td>3.8       </td><td>1.7       </td><td>0.3       </td><td>setosa    </td><td>3         </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       " Sepal.Length & Sepal.Width & Petal.Length & Petal.Width & Species & sepal\\_length\\_rank\\\\\n",
       "\\hline\n",
       "\t 7.9        & 3.8        & 6.4        & 2.0        & virginica  & 1         \\\\\n",
       "\t 7.7        & 3.8        & 6.7        & 2.2        & virginica  & 2         \\\\\n",
       "\t 7.7        & 2.6        & 6.9        & 2.3        & virginica  & 3         \\\\\n",
       "\t 7.0        & 3.2        & 4.7        & 1.4        & versicolor & 1         \\\\\n",
       "\t 6.9        & 3.1        & 4.9        & 1.5        & versicolor & 2         \\\\\n",
       "\t 6.8        & 2.8        & 4.8        & 1.4        & versicolor & 3         \\\\\n",
       "\t 5.8        & 4.0        & 1.2        & 0.2        & setosa     & 1         \\\\\n",
       "\t 5.7        & 4.4        & 1.5        & 0.4        & setosa     & 2         \\\\\n",
       "\t 5.7        & 3.8        & 1.7        & 0.3        & setosa     & 3         \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Sepal.Length Sepal.Width Petal.Length Petal.Width Species    sepal_length_rank\n",
       "1 7.9          3.8         6.4          2.0         virginica  1                \n",
       "2 7.7          3.8         6.7          2.2         virginica  2                \n",
       "3 7.7          2.6         6.9          2.3         virginica  3                \n",
       "4 7.0          3.2         4.7          1.4         versicolor 1                \n",
       "5 6.9          3.1         4.9          1.5         versicolor 2                \n",
       "6 6.8          2.8         4.8          1.4         versicolor 3                \n",
       "7 5.8          4.0         1.2          0.2         setosa     1                \n",
       "8 5.7          4.4         1.5          0.4         setosa     2                \n",
       "9 5.7          3.8         1.7          0.3         setosa     3                "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# there are plenty of helper functions for easy vectorized operations\n",
    "# for instance, select 3 flowers with the largest sepal length for each species\n",
    "# n, row_number, desc, ifelse, cumsum, lead, lag, rank, and n_distinct are especially useful\n",
    "# more here: https://cran.r-project.org/web/packages/dplyr/vignettes/window-functions.html\n",
    "iris %>%\n",
    "    group_by(Species) %>%\n",
    "    arrange(desc(Sepal.Length)) %>%\n",
    "    mutate(sepal_length_rank = row_number()) %>%\n",
    "    filter(sepal_length_rank <= 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Warnings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `group_by` has side effects, so **always `ungroup`** intermediate results\n",
    "It's common to store grouped data in an intermediate variable and do something with it downstream. Forgetting about the implicit grouping downstream can lead to unexpected issues, both in performance and correctness of code.\n",
    "\n",
    "The solution is to always ungroup when creating intermediate tables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ungroup for performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>91481</td><td>47   </td></tr>\n",
       "\t<tr><td>93708</td><td>42   </td></tr>\n",
       "\t<tr><td>28614</td><td>86   </td></tr>\n",
       "\t<tr><td>83045</td><td>16   </td></tr>\n",
       "\t<tr><td>64175</td><td>53   </td></tr>\n",
       "\t<tr><td>51910</td><td>41   </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " group\\_id & val\\\\\n",
       "\\hline\n",
       "\t 91481 & 47   \\\\\n",
       "\t 93708 & 42   \\\\\n",
       "\t 28614 & 86   \\\\\n",
       "\t 83045 & 16   \\\\\n",
       "\t 64175 & 53   \\\\\n",
       "\t 51910 & 41   \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  group_id val\n",
       "1 91481    47 \n",
       "2 93708    42 \n",
       "3 28614    86 \n",
       "4 83045    16 \n",
       "5 64175    53 \n",
       "6 51910    41 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create a dummy dataframe with 100,000 groups and 1,000,000 rows\n",
    "# and partition by group_id\n",
    "set.seed(42)\n",
    "df <- data.frame(group_id=sample(1:1e5, 1e6, replace=T),\n",
    "                 val=sample(1:100, 1e6, replace=T)) %>%\n",
    "  group_by(group_id)\n",
    "\n",
    "head(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "   user  system elapsed \n",
       "  2.063   0.042   2.364 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# filter rows with a value of 1 naively\n",
    "# this takes a long time because it first loops over groups and then values within group\n",
    "system.time(df %>% filter(val == 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "   user  system elapsed \n",
       "  0.007   0.004   0.013 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ungroup before filtering for a huge speedup\n",
    "# this is substantially faster because it ignores groups\n",
    "df <- df %>%\n",
    "    ungroup()\n",
    "system.time(df %>% filter(val == 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ungroup for correctness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>c </td><td> 5</td></tr>\n",
       "\t<tr><td>c </td><td> 7</td></tr>\n",
       "\t<tr><td>a </td><td> 8</td></tr>\n",
       "\t<tr><td>c </td><td> 2</td></tr>\n",
       "\t<tr><td>b </td><td> 3</td></tr>\n",
       "\t<tr><td>b </td><td>10</td></tr>\n",
       "\t<tr><td>c </td><td> 4</td></tr>\n",
       "\t<tr><td>a </td><td> 1</td></tr>\n",
       "\t<tr><td>b </td><td> 6</td></tr>\n",
       "\t<tr><td>c </td><td> 9</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " group\\_id & val\\\\\n",
       "\\hline\n",
       "\t c  &  5\\\\\n",
       "\t c  &  7\\\\\n",
       "\t a  &  8\\\\\n",
       "\t c  &  2\\\\\n",
       "\t b  &  3\\\\\n",
       "\t b  & 10\\\\\n",
       "\t c  &  4\\\\\n",
       "\t a  &  1\\\\\n",
       "\t b  &  6\\\\\n",
       "\t c  &  9\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   group_id val\n",
       "1  c         5 \n",
       "2  c         7 \n",
       "3  a         8 \n",
       "4  c         2 \n",
       "5  b         3 \n",
       "6  b        10 \n",
       "7  c         4 \n",
       "8  a         1 \n",
       "9  b         6 \n",
       "10 c         9 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create a smaller toy data frame with three groups and random values\n",
    "set.seed(42)\n",
    "df <- data.frame(group_id = sample(c('a','b','c'), 10, replace = T),\n",
    "                 val = sample(1:10, 10))\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th><th scope=col>num_in_group</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>c </td><td> 5</td><td>5 </td></tr>\n",
       "\t<tr><td>c </td><td> 7</td><td>5 </td></tr>\n",
       "\t<tr><td>a </td><td> 8</td><td>2 </td></tr>\n",
       "\t<tr><td>c </td><td> 2</td><td>5 </td></tr>\n",
       "\t<tr><td>b </td><td> 3</td><td>3 </td></tr>\n",
       "\t<tr><td>b </td><td>10</td><td>3 </td></tr>\n",
       "\t<tr><td>c </td><td> 4</td><td>5 </td></tr>\n",
       "\t<tr><td>a </td><td> 1</td><td>2 </td></tr>\n",
       "\t<tr><td>b </td><td> 6</td><td>3 </td></tr>\n",
       "\t<tr><td>c </td><td> 9</td><td>5 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " group\\_id & val & num\\_in\\_group\\\\\n",
       "\\hline\n",
       "\t c  &  5 & 5 \\\\\n",
       "\t c  &  7 & 5 \\\\\n",
       "\t a  &  8 & 2 \\\\\n",
       "\t c  &  2 & 5 \\\\\n",
       "\t b  &  3 & 3 \\\\\n",
       "\t b  & 10 & 3 \\\\\n",
       "\t c  &  4 & 5 \\\\\n",
       "\t a  &  1 & 2 \\\\\n",
       "\t b  &  6 & 3 \\\\\n",
       "\t c  &  9 & 5 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   group_id val num_in_group\n",
       "1  c         5  5           \n",
       "2  c         7  5           \n",
       "3  a         8  2           \n",
       "4  c         2  5           \n",
       "5  b         3  3           \n",
       "6  b        10  3           \n",
       "7  c         4  5           \n",
       "8  a         1  2           \n",
       "9  b         6  3           \n",
       "10 c         9  5           "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's say you use mutate to add a column with a count of the number of observations per group\n",
    "df1 <- df %>%\n",
    "    group_by(group_id) %>%\n",
    "    mutate(num_in_group = n())\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th><th scope=col>num_in_group</th><th scope=col>frac</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>c         </td><td> 5        </td><td>5         </td><td>0.18518519</td></tr>\n",
       "\t<tr><td>c         </td><td> 7        </td><td>5         </td><td>0.25925926</td></tr>\n",
       "\t<tr><td>a         </td><td> 8        </td><td>2         </td><td>0.88888889</td></tr>\n",
       "\t<tr><td>c         </td><td> 2        </td><td>5         </td><td>0.07407407</td></tr>\n",
       "\t<tr><td>b         </td><td> 3        </td><td>3         </td><td>0.15789474</td></tr>\n",
       "\t<tr><td>b         </td><td>10        </td><td>3         </td><td>0.52631579</td></tr>\n",
       "\t<tr><td>c         </td><td> 4        </td><td>5         </td><td>0.14814815</td></tr>\n",
       "\t<tr><td>a         </td><td> 1        </td><td>2         </td><td>0.11111111</td></tr>\n",
       "\t<tr><td>b         </td><td> 6        </td><td>3         </td><td>0.31578947</td></tr>\n",
       "\t<tr><td>c         </td><td> 9        </td><td>5         </td><td>0.33333333</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " group\\_id & val & num\\_in\\_group & frac\\\\\n",
       "\\hline\n",
       "\t c          &  5         & 5          & 0.18518519\\\\\n",
       "\t c          &  7         & 5          & 0.25925926\\\\\n",
       "\t a          &  8         & 2          & 0.88888889\\\\\n",
       "\t c          &  2         & 5          & 0.07407407\\\\\n",
       "\t b          &  3         & 3          & 0.15789474\\\\\n",
       "\t b          & 10         & 3          & 0.52631579\\\\\n",
       "\t c          &  4         & 5          & 0.14814815\\\\\n",
       "\t a          &  1         & 2          & 0.11111111\\\\\n",
       "\t b          &  6         & 3          & 0.31578947\\\\\n",
       "\t c          &  9         & 5          & 0.33333333\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   group_id val num_in_group frac      \n",
       "1  c         5  5            0.18518519\n",
       "2  c         7  5            0.25925926\n",
       "3  a         8  2            0.88888889\n",
       "4  c         2  5            0.07407407\n",
       "5  b         3  3            0.15789474\n",
       "6  b        10  3            0.52631579\n",
       "7  c         4  5            0.14814815\n",
       "8  a         1  2            0.11111111\n",
       "9  b         6  3            0.31578947\n",
       "10 c         9  5            0.33333333"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# later you'd like to normalize the val column\n",
    "df1 <- df1 %>%\n",
    "    mutate(frac = val / sum(val))\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "3"
      ],
      "text/latex": [
       "3"
      ],
      "text/markdown": [
       "3"
      ],
      "text/plain": [
       "[1] 3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# glancing at the last column shows that frac sums to more than 1\n",
    "# in fact, it sums to 3 because the sum() inside the mutate\n",
    "# was applied within each of the 3 groups\n",
    "sum(df1$frac)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th><th scope=col>num_in_group</th><th scope=col>frac</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>c         </td><td> 5        </td><td>5         </td><td>0.09090909</td></tr>\n",
       "\t<tr><td>c         </td><td> 7        </td><td>5         </td><td>0.12727273</td></tr>\n",
       "\t<tr><td>a         </td><td> 8        </td><td>2         </td><td>0.14545455</td></tr>\n",
       "\t<tr><td>c         </td><td> 2        </td><td>5         </td><td>0.03636364</td></tr>\n",
       "\t<tr><td>b         </td><td> 3        </td><td>3         </td><td>0.05454545</td></tr>\n",
       "\t<tr><td>b         </td><td>10        </td><td>3         </td><td>0.18181818</td></tr>\n",
       "\t<tr><td>c         </td><td> 4        </td><td>5         </td><td>0.07272727</td></tr>\n",
       "\t<tr><td>a         </td><td> 1        </td><td>2         </td><td>0.01818182</td></tr>\n",
       "\t<tr><td>b         </td><td> 6        </td><td>3         </td><td>0.10909091</td></tr>\n",
       "\t<tr><td>c         </td><td> 9        </td><td>5         </td><td>0.16363636</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " group\\_id & val & num\\_in\\_group & frac\\\\\n",
       "\\hline\n",
       "\t c          &  5         & 5          & 0.09090909\\\\\n",
       "\t c          &  7         & 5          & 0.12727273\\\\\n",
       "\t a          &  8         & 2          & 0.14545455\\\\\n",
       "\t c          &  2         & 5          & 0.03636364\\\\\n",
       "\t b          &  3         & 3          & 0.05454545\\\\\n",
       "\t b          & 10         & 3          & 0.18181818\\\\\n",
       "\t c          &  4         & 5          & 0.07272727\\\\\n",
       "\t a          &  1         & 2          & 0.01818182\\\\\n",
       "\t b          &  6         & 3          & 0.10909091\\\\\n",
       "\t c          &  9         & 5          & 0.16363636\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   group_id val num_in_group frac      \n",
       "1  c         5  5            0.09090909\n",
       "2  c         7  5            0.12727273\n",
       "3  a         8  2            0.14545455\n",
       "4  c         2  5            0.03636364\n",
       "5  b         3  3            0.05454545\n",
       "6  b        10  3            0.18181818\n",
       "7  c         4  5            0.07272727\n",
       "8  a         1  2            0.01818182\n",
       "9  b         6  3            0.10909091\n",
       "10 c         9  5            0.16363636"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ungrouping the intermediate data avoids this problem\n",
    "# compute the number in each group and ungroup\n",
    "df2 <- df %>%\n",
    "    group_by(group_id) %>%\n",
    "    mutate(num_in_group = n()) %>%\n",
    "    ungroup()\n",
    "\n",
    "# then normalize the val column\n",
    "df2 <- df2 %>%\n",
    "    mutate(frac = val / sum(val))\n",
    "\n",
    "# check that things look good\n",
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "1"
      ],
      "text/latex": [
       "1"
      ],
      "text/markdown": [
       "1"
      ],
      "text/plain": [
       "[1] 1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# frac sums to 1\n",
    "sum(df2$frac)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Arranging within groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th><th scope=col>rank</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>a </td><td> 1</td><td>1 </td></tr>\n",
       "\t<tr><td>c </td><td> 2</td><td>1 </td></tr>\n",
       "\t<tr><td>b </td><td> 3</td><td>1 </td></tr>\n",
       "\t<tr><td>c </td><td> 4</td><td>2 </td></tr>\n",
       "\t<tr><td>c </td><td> 5</td><td>3 </td></tr>\n",
       "\t<tr><td>b </td><td> 6</td><td>2 </td></tr>\n",
       "\t<tr><td>c </td><td> 7</td><td>4 </td></tr>\n",
       "\t<tr><td>a </td><td> 8</td><td>2 </td></tr>\n",
       "\t<tr><td>c </td><td> 9</td><td>5 </td></tr>\n",
       "\t<tr><td>b </td><td>10</td><td>3 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " group\\_id & val & rank\\\\\n",
       "\\hline\n",
       "\t a  &  1 & 1 \\\\\n",
       "\t c  &  2 & 1 \\\\\n",
       "\t b  &  3 & 1 \\\\\n",
       "\t c  &  4 & 2 \\\\\n",
       "\t c  &  5 & 3 \\\\\n",
       "\t b  &  6 & 2 \\\\\n",
       "\t c  &  7 & 4 \\\\\n",
       "\t a  &  8 & 2 \\\\\n",
       "\t c  &  9 & 5 \\\\\n",
       "\t b  & 10 & 3 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   group_id val rank\n",
       "1  a         1  1   \n",
       "2  c         2  1   \n",
       "3  b         3  1   \n",
       "4  c         4  2   \n",
       "5  c         5  3   \n",
       "6  b         6  2   \n",
       "7  c         7  4   \n",
       "8  a         8  2   \n",
       "9  c         9  5   \n",
       "10 b        10  3   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# arrange works within groups, reordering rows on a per-group basis\n",
    "# but it doesn't display the output sorted by group\n",
    "df %>%\n",
    "    group_by(group_id) %>%\n",
    "    arrange(val) %>%\n",
    "    mutate(rank = row_number())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th><th scope=col>rank</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>a </td><td> 1</td><td>1 </td></tr>\n",
       "\t<tr><td>a </td><td> 8</td><td>2 </td></tr>\n",
       "\t<tr><td>b </td><td> 3</td><td>1 </td></tr>\n",
       "\t<tr><td>b </td><td> 6</td><td>2 </td></tr>\n",
       "\t<tr><td>b </td><td>10</td><td>3 </td></tr>\n",
       "\t<tr><td>c </td><td> 2</td><td>1 </td></tr>\n",
       "\t<tr><td>c </td><td> 4</td><td>2 </td></tr>\n",
       "\t<tr><td>c </td><td> 5</td><td>3 </td></tr>\n",
       "\t<tr><td>c </td><td> 7</td><td>4 </td></tr>\n",
       "\t<tr><td>c </td><td> 9</td><td>5 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " group\\_id & val & rank\\\\\n",
       "\\hline\n",
       "\t a  &  1 & 1 \\\\\n",
       "\t a  &  8 & 2 \\\\\n",
       "\t b  &  3 & 1 \\\\\n",
       "\t b  &  6 & 2 \\\\\n",
       "\t b  & 10 & 3 \\\\\n",
       "\t c  &  2 & 1 \\\\\n",
       "\t c  &  4 & 2 \\\\\n",
       "\t c  &  5 & 3 \\\\\n",
       "\t c  &  7 & 4 \\\\\n",
       "\t c  &  9 & 5 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   group_id val rank\n",
       "1  a         1  1   \n",
       "2  a         8  2   \n",
       "3  b         3  1   \n",
       "4  b         6  2   \n",
       "5  b        10  3   \n",
       "6  c         2  1   \n",
       "7  c         4  2   \n",
       "8  c         5  3   \n",
       "9  c         7  4   \n",
       "10 c         9  5   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# it's easy to forget this\n",
    "# so it's good practice to use the group variable(s) as the first argument(s) to arrange\n",
    "# this also displays the output in grouped order\n",
    "df %>%\n",
    "    group_by(group_id) %>%\n",
    "    arrange(group_id, val) %>%\n",
    "    mutate(rank = row_number())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unintentionally overwriting columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>mean_val</th><th scope=col>sd_val</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.5    </td><td>3.02765</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " mean\\_val & sd\\_val\\\\\n",
       "\\hline\n",
       "\t 5.5     & 3.02765\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  mean_val sd_val \n",
       "1 5.5      3.02765"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# it's easy to generate multiple transformations at once\n",
    "df %>%\n",
    "  summarize(mean_val = mean(val),\n",
    "            sd_val = sd(val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>val</th><th scope=col>sd</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>5.5     </td><td>1.739253</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " val & sd\\\\\n",
       "\\hline\n",
       "\t 5.5      & 1.739253\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  val sd      \n",
       "1 5.5 1.739253"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# but sometimes we get lazy about variable names and overwrite things\n",
    "# for instance, this implicitly clobbers the val column\n",
    "# before the sd() function is called, returning the wrong result\n",
    "df %>%\n",
    "  summarize(val = mean(val),\n",
    "            sd = sd(val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "3.02765035409749"
      ],
      "text/latex": [
       "3.02765035409749"
      ],
      "text/markdown": [
       "3.02765035409749"
      ],
      "text/plain": [
       "[1] 3.02765"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sd(df$val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tricky variable scoping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>c </td><td> 5</td></tr>\n",
       "\t<tr><td>c </td><td> 7</td></tr>\n",
       "\t<tr><td>a </td><td> 8</td></tr>\n",
       "\t<tr><td>c </td><td> 2</td></tr>\n",
       "\t<tr><td>b </td><td> 3</td></tr>\n",
       "\t<tr><td>b </td><td>10</td></tr>\n",
       "\t<tr><td>c </td><td> 4</td></tr>\n",
       "\t<tr><td>a </td><td> 1</td></tr>\n",
       "\t<tr><td>b </td><td> 6</td></tr>\n",
       "\t<tr><td>c </td><td> 9</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " group\\_id & val\\\\\n",
       "\\hline\n",
       "\t c  &  5\\\\\n",
       "\t c  &  7\\\\\n",
       "\t a  &  8\\\\\n",
       "\t c  &  2\\\\\n",
       "\t b  &  3\\\\\n",
       "\t b  & 10\\\\\n",
       "\t c  &  4\\\\\n",
       "\t a  &  1\\\\\n",
       "\t b  &  6\\\\\n",
       "\t c  &  9\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   group_id val\n",
       "1  c         5 \n",
       "2  c         7 \n",
       "3  a         8 \n",
       "4  c         2 \n",
       "5  b         3 \n",
       "6  b        10 \n",
       "7  c         4 \n",
       "8  a         1 \n",
       "9  b         6 \n",
       "10 c         9 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's say you want to find all rows with a certain value (e.g., 10)\n",
    "# so you put that value into a variable\n",
    "val <- 10\n",
    "\n",
    "# then you filter, but you get every row back\n",
    "filter(df, val == val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>b </td><td>10</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " group\\_id & val\\\\\n",
       "\\hline\n",
       "\t b  & 10\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  group_id val\n",
       "1 b        10 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# filter has the column df$val in scope\n",
    "# and so val == val evaluates to true for every row\n",
    "# use more specific names to avoid such problems\n",
    "my_val <- 10\n",
    "filter(df, val == my_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Forgetting to use vectorized functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "'A or B'"
      ],
      "text/latex": [
       "'A or B'"
      ],
      "text/markdown": [
       "'A or B'"
      ],
      "text/plain": [
       "[1] \"A or B\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "'A or B'"
      ],
      "text/latex": [
       "'A or B'"
      ],
      "text/markdown": [
       "'A or B'"
      ],
      "text/plain": [
       "[1] \"A or B\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "'C'"
      ],
      "text/latex": [
       "'C'"
      ],
      "text/markdown": [
       "'C'"
      ],
      "text/plain": [
       "[1] \"C\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# sometimes it's easy to write a function that operates on one value\n",
    "# but all functions used inside mutate and summarize must work on vectors\n",
    "\n",
    "# for instance, let's say you have a simple function that \n",
    "# collapses groups 'a' and 'b' into one group and 'c' into another\n",
    "\n",
    "# create a simple function to do this\n",
    "f <- function(x) {\n",
    "  if (x == 'a' || x == 'b')\n",
    "      'A or B'\n",
    "  else\n",
    "      'C'\n",
    "}\n",
    "\n",
    "# verify that it does the right thing on individual values\n",
    "f('a')\n",
    "f('b')\n",
    "f('c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th><th scope=col>new_col</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>c </td><td> 5</td><td>C </td></tr>\n",
       "\t<tr><td>c </td><td> 7</td><td>C </td></tr>\n",
       "\t<tr><td>a </td><td> 8</td><td>C </td></tr>\n",
       "\t<tr><td>c </td><td> 2</td><td>C </td></tr>\n",
       "\t<tr><td>b </td><td> 3</td><td>C </td></tr>\n",
       "\t<tr><td>b </td><td>10</td><td>C </td></tr>\n",
       "\t<tr><td>c </td><td> 4</td><td>C </td></tr>\n",
       "\t<tr><td>a </td><td> 1</td><td>C </td></tr>\n",
       "\t<tr><td>b </td><td> 6</td><td>C </td></tr>\n",
       "\t<tr><td>c </td><td> 9</td><td>C </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " group\\_id & val & new\\_col\\\\\n",
       "\\hline\n",
       "\t c  &  5 & C \\\\\n",
       "\t c  &  7 & C \\\\\n",
       "\t a  &  8 & C \\\\\n",
       "\t c  &  2 & C \\\\\n",
       "\t b  &  3 & C \\\\\n",
       "\t b  & 10 & C \\\\\n",
       "\t c  &  4 & C \\\\\n",
       "\t a  &  1 & C \\\\\n",
       "\t b  &  6 & C \\\\\n",
       "\t c  &  9 & C \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   group_id val new_col\n",
       "1  c         5  C      \n",
       "2  c         7  C      \n",
       "3  a         8  C      \n",
       "4  c         2  C      \n",
       "5  b         3  C      \n",
       "6  b        10  C      \n",
       "7  c         4  C      \n",
       "8  a         1  C      \n",
       "9  b         6  C      \n",
       "10 c         9  C      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# now use it inside of mutate\n",
    "# notice that the new column is all 'C's\n",
    "df %>%\n",
    "      mutate(new_col = f(group_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "'A or B'"
      ],
      "text/latex": [
       "'A or B'"
      ],
      "text/markdown": [
       "'A or B'"
      ],
      "text/plain": [
       "[1] \"A or B\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# this is because the function wasn't made to handle vectors\n",
    "f(c('a','b','c'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<dl class=dl-horizontal>\n",
       "\t<dt>a</dt>\n",
       "\t\t<dd>'A or B'</dd>\n",
       "\t<dt>b</dt>\n",
       "\t\t<dd>'A or B'</dd>\n",
       "\t<dt>c</dt>\n",
       "\t\t<dd>'C'</dd>\n",
       "</dl>\n"
      ],
      "text/latex": [
       "\\begin{description*}\n",
       "\\item[a] 'A or B'\n",
       "\\item[b] 'A or B'\n",
       "\\item[c] 'C'\n",
       "\\end{description*}\n"
      ],
      "text/markdown": [
       "a\n",
       ":   'A or B'b\n",
       ":   'A or B'c\n",
       ":   'C'\n",
       "\n"
      ],
      "text/plain": [
       "       a        b        c \n",
       "\"A or B\" \"A or B\"      \"C\" "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# we can use the Vectorize function to fix this\n",
    "g <- Vectorize(f)\n",
    "\n",
    "g(c('a','b','c'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th><th scope=col>new_col</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>c     </td><td> 5    </td><td>C     </td></tr>\n",
       "\t<tr><td>c     </td><td> 7    </td><td>C     </td></tr>\n",
       "\t<tr><td>a     </td><td> 8    </td><td>A or B</td></tr>\n",
       "\t<tr><td>c     </td><td> 2    </td><td>C     </td></tr>\n",
       "\t<tr><td>b     </td><td> 3    </td><td>A or B</td></tr>\n",
       "\t<tr><td>b     </td><td>10    </td><td>A or B</td></tr>\n",
       "\t<tr><td>c     </td><td> 4    </td><td>C     </td></tr>\n",
       "\t<tr><td>a     </td><td> 1    </td><td>A or B</td></tr>\n",
       "\t<tr><td>b     </td><td> 6    </td><td>A or B</td></tr>\n",
       "\t<tr><td>c     </td><td> 9    </td><td>C     </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " group\\_id & val & new\\_col\\\\\n",
       "\\hline\n",
       "\t c      &  5     & C     \\\\\n",
       "\t c      &  7     & C     \\\\\n",
       "\t a      &  8     & A or B\\\\\n",
       "\t c      &  2     & C     \\\\\n",
       "\t b      &  3     & A or B\\\\\n",
       "\t b      & 10     & A or B\\\\\n",
       "\t c      &  4     & C     \\\\\n",
       "\t a      &  1     & A or B\\\\\n",
       "\t b      &  6     & A or B\\\\\n",
       "\t c      &  9     & C     \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   group_id val new_col\n",
       "1  c         5  C      \n",
       "2  c         7  C      \n",
       "3  a         8  A or B \n",
       "4  c         2  C      \n",
       "5  b         3  A or B \n",
       "6  b        10  A or B \n",
       "7  c         4  C      \n",
       "8  a         1  A or B \n",
       "9  b         6  A or B \n",
       "10 c         9  C      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# and now things work correctly inside of mutate\n",
    "df %>%\n",
    "      mutate(new_col = g(group_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'AB'</li>\n",
       "\t<li>'AB'</li>\n",
       "\t<li>'C'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'AB'\n",
       "\\item 'AB'\n",
       "\\item 'C'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'AB'\n",
       "2. 'AB'\n",
       "3. 'C'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"AB\" \"AB\" \"C\" "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ifelse is a handy shortcut for this sort of thing\n",
    "# where you have ifelse(<condition>, <result if true>, <result if false>)\n",
    "v <- c('a','b','c')\n",
    "ifelse(v == 'a' | v == 'b', 'AB', 'C')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>group_id</th><th scope=col>val</th><th scope=col>new_col</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>c </td><td> 5</td><td>C </td></tr>\n",
       "\t<tr><td>c </td><td> 7</td><td>C </td></tr>\n",
       "\t<tr><td>a </td><td> 8</td><td>AB</td></tr>\n",
       "\t<tr><td>c </td><td> 2</td><td>C </td></tr>\n",
       "\t<tr><td>b </td><td> 3</td><td>AB</td></tr>\n",
       "\t<tr><td>b </td><td>10</td><td>AB</td></tr>\n",
       "\t<tr><td>c </td><td> 4</td><td>C </td></tr>\n",
       "\t<tr><td>a </td><td> 1</td><td>AB</td></tr>\n",
       "\t<tr><td>b </td><td> 6</td><td>AB</td></tr>\n",
       "\t<tr><td>c </td><td> 9</td><td>C </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " group\\_id & val & new\\_col\\\\\n",
       "\\hline\n",
       "\t c  &  5 & C \\\\\n",
       "\t c  &  7 & C \\\\\n",
       "\t a  &  8 & AB\\\\\n",
       "\t c  &  2 & C \\\\\n",
       "\t b  &  3 & AB\\\\\n",
       "\t b  & 10 & AB\\\\\n",
       "\t c  &  4 & C \\\\\n",
       "\t a  &  1 & AB\\\\\n",
       "\t b  &  6 & AB\\\\\n",
       "\t c  &  9 & C \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   group_id val new_col\n",
       "1  c         5  C      \n",
       "2  c         7  C      \n",
       "3  a         8  AB     \n",
       "4  c         2  C      \n",
       "5  b         3  AB     \n",
       "6  b        10  AB     \n",
       "7  c         4  C      \n",
       "8  a         1  AB     \n",
       "9  b         6  AB     \n",
       "10 c         9  C      "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# which is handy inside of mutate\n",
    "# and avoids having to define lots of functions\n",
    "# just be careful to use | and & instead of || and && here as well\n",
    "df %>%\n",
    "  mutate(new_col = ifelse(group_id == 'a' | group_id == 'b', 'AB', 'C'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.3.1"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "512px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": true,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}