Chapter 5 Data Transformation.Rmd

---
title: "R4ds Chapter 5 Data Transformation Yuqing Xue Personal Note"
author: "Yuqing Xue"
date: "October 22, 2017"
output:
  pdf_document: default
  html_document: default
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

<br>

The following personal code and notes are based on the materials from Hadley Wickham's 'R for Data Science', Chapter 5. <http://r4ds.had.co.nz/transform.html#introduction-2>

<br>

* [5.1 Introduction](#intro)

* [5.2 Filter](#filter)

* [5.2.4 Exercises](#ex_5_2_4)

* [5.3 Arrange](#arrange)

* [5.3.1 Exercises](#ex_5_3_1)

* [5.4 Select](#select)

* [5.4.1 Exercises](#ex_5_4_1)

* [5.5 Mutate](#mutate)

* [5.5.1 Useful Creation Functions](#useful_creation)

* [5.5.2 Exercises](#ex_5_5_2)

* [5.6 Summarise](#summarise)

* [5.6.1 Combining multiple operations with the pipe](#summPipe)

* [5.6.3 Counts](#count)

* [5.6.4 Useful Summary Functions](#summary)

* [5.6.5 Grouping by multiple variables](#rollingup)

* [5.6.6 Ungrouping](#ungrouping)





<br>

### 5.1 Introduction {#intro}


#### `nycflights13`
```{r message=FALSE, warning=FALSE}
##########################################
####### Dataset: nycflights13::flights
########################################## 
library(nycflights13)
library(tidyverse)
nycflights13::flights
```

<br>

### 5.2 Filter {#filter}

#### Explore, try `%in%`
```{r explore}

jan1 <- filter(flights, month == 1, day == 1)

nov_and_dec = filter(flights, month == 11 | month == 12)
## A useful short-hand for this problem is x %in% y. This will select every row where x is one of the values in y.
nov_dec <- filter(flights, month %in% c(11, 12))
```

```{r NYC to San Fransisco Bay Area}

NYCtoSF <-  
  flights %>% 
    filter(dest %in% c("SFO", "OAK", "SJC")) %>%
      select(year, month, day, carrier, origin, dest, air_time)
NYCtoSF

NYCtoSF2 <-  
  flights %>% 
    filter(dest == "SFO" | dest == "OAK" |  dest == "SJC") %>%
      select(year, month, day, carrier, origin, dest, air_time)
NYCtoSF2

# H.W. As well as & and |, R also has && and ||. Don’t use them here! You’ll learn when you should use them in conditional execution.

``` 

<br>

###  5.2.4 Exercises  {#ex_5_2_4}
##### 1. Find all flights that
##### 1.1 Had an arrival delay of two or more hours
##### 1.2. Flew to Houston (IAH or HOU)
##### 1.3. Were operated by United, American, or Delta
##### 1.4. Departed in summer (July, August, and September)
##### 1.5. Arrived more than two hours late, but didn’t leave late
##### 1.6. Were delayed by at least an hour, but made up over 30 minutes in flight
##### 1.7. Departed between midnight and 6am (inclusive)

```{r exercise 1.1}
 
Ex1.1 <- 
  flights %>%
    filter(arr_delay >= 120) %>%
      arrange(desc(arr_delay)) %>%
        select(month, day, carrier, origin, dest, dep_delay, arr_delay)
Ex1.1
```

```{r exercise 1.2 - 1.4}
Ex1.2 = filter(flights, dest %in% c("IAH","HOU"))
Ex1.3 = filter(flights, carrier %in% c("UA","AA","DL"))
Ex1.4 = filter(flights, month %in% c(7,8,9))
```

```{r exercise 1.5 - 1.7}
Ex1.5 = filter(flights, arr_delay >= 120 & dep_delay <= 0 )
Ex1.6 = filter(flights, dep_delay >= 60 & dep_delay - arr_delay >= 30)
Ex1.7 = filter(flights, dep_time <= 600)
```

<br>

#### 2. Another useful dplyr filtering helper is `between()`. What does it do? Can you use it to simplify the code needed to answer the previous challenges?

```{r exercise 2}
?between
Ex1.4b = filter(flights, between(month, 7, 9))
```

<br>

#### 3. How many flights have a missing dep_time? What other variables are missing? What might these rows represent?

```{r exercise 3}
sum(is.na(flights$dep_time)) # 8255

flights_missing_dep_time <- flights %>% filter(is.na(dep_time))
flights_missing_dep_time 

flights_cancelled <- flights %>% 
  filter(is.na(dep_time) & is.na(dep_delay) & is.na(arr_time) & is.na(arr_delay) & is.na(arr_time))
# same 8255 flights
```

Beside `dep_time`, `dep_delay`, `arr_time`, `arr_delay` and `air_time` are also missing, for all 8255 flights. One possible explanation for it is that those flights were cancelled.

<br>

#### 4. Why is `NA ^ 0` not missing? Why is `NA | TRUE` not missing? Why is `FALSE & NA` not missing? Can you figure out the general rule? (`NA * 0` is a tricky counterexample!)

`NA ^ 0` returns 1. Any number to the power of 0 is 1, whether the number is missing or not does not matter.

`NA | TRUE` returns TRUE, since the `|` operator returns TRUE if either of the terms are TRUE. In this case, the right half is TRUE, so the whole expression will always return TRUE. 

`FALSE & NA` returns FALSE, because operator `&` returns TRUE when both terms are true. The left half is FALSE, so the whole expression returns FALSE despite the NA on the right half.

```{r}
NA * 0
```

`NA * 0` returns 0, whicn may rendered the general rule we discovered from the previous not definite: if NA represent a value that is `Inf`, and we know `Inf` * 0 should not be a number, i.e., `NaN`.

\vspace*{3\baselineskip}

### 5.3 Arrange {#arrange}

`arrange()` works similarly to `filter()` except that instead of selecting rows, it changes their order. It takes a data frame and a set of column names (or more complicated expressions) to order by. 
__If you provide more than one column name, each additional column will be used to break ties in the values of preceding columns__:

```{r}
arrange(flights, year, month, day)
arrange(flights, desc(arr_delay))
```

```{r}
df <- tibble(x = c(5, 2, NA))
arrange(df, x)
```
In this way, `x` then becomes the column name of the tibble `df`.

<br>

#### 5.3.1 Exercise {#ex_5_3_1}
#### 1. How could you use `arrange()` to sort all missing values to the start? (Hint: use `is.na()`).

```{r}
arrange(df, !is.na(x))
```
It seems that R recognized `FALSE` as 0 and `TRUE` as 1, which makes `FALSE` is less than `TRUE`, hence can be sorted to the start use `arrange`.


#### 2. Sort `flights` to find the most delayed flights. Find the flights that left earliest.

```{r}
colnames(flights)
flights %>% 
  arrange(desc(arr_delay))

flights %>%
  arrange(dep_time)
```

which may lead to an idea to create a function:

idea: do all the common topRanking (smallest/largest, variable) and put it on Shiny

#### 3. Sort `flights` to find the fastest flights.

__*time form needs to be clearer, answer not reliable*__

Idea: create a variable `travelTime` to measure the flight time (`arr_time - dep_time`. Noted that `dep_time` and `arr_time` are coded in a form that is intuitive to read and comprehend, but a little harder to make further calculation with, because they’re not continuous. A more convenient representation of number of minutes since midnight are yet to be computed. So `travelTime` are neither real minutes nor hours and minutes like `dep_time` and `arr_time`. More details and discussion can be found in exersice 5.5.2 problem 2. <http://r4ds.had.co.nz/transform.html#exercises-10>). 

Tricky: The way how `dep_time` and `arr_time` is coded needs a slightly modification.
```{r}
flights %>% 
  mutate(travelTime = ifelse(arr_time - dep_time < 0, arr_time + 2400 - dep_time, arr_time - dep_time)) %>%
    select(origin, dest, dep_time, sched_dep_time, arr_time, sched_arr_time, travelTime) %>%
      arrange(travelTime)

# and the slowest flights 
flights %>%
  mutate(travelTime = ifelse(arr_time - dep_time < 0, arr_time + 2400 - dep_time, arr_time - dep_time)) %>%
    select(origin, dest, dep_time, sched_dep_time, arr_time, sched_arr_time, travelTime) %>%
      arrange(desc(travelTime))

flights_Testimate <- flights %>%
  mutate(travelTime = ifelse(arr_time - dep_time < 0, arr_time + 2400 - dep_time, arr_time - dep_time)) 
```
The most unexpected flights, like the one from EWR to ORD that took 19 hours 25 minutes, can be spotted and further investigated.


#### 4. Which flights travelled the longest? Which travelled the shortest?
Idea: sort variable `distance`
```{r}
# The longest flights
flights_Testimate %>%
  select(carrier:distance, travelTime) %>%
    arrange(desc(distance))

# The longest flight where destination is not HNL
flights_Testimate %>%
  select(carrier:distance, travelTime) %>%
    filter(dest != 'HNL') %>%
      arrange(desc(distance))

# Besides HNL and ANC
flights_Testimate %>%
  select(carrier:distance, travelTime) %>%
    filter(dest != 'HNL' & dest != "ANC") %>%
      arrange(desc(distance))

# Besides HNL, ANC and JFK to SFO
flights_Testimate %>%
  select(carrier:distance, travelTime) %>%
    filter(dest != 'HNL' & dest != "ANC" & dest != "SFO") %>%
      arrange(desc(distance))
```
The second longest flight is from EWR to ANC. The third one is JFK to SFO, and the forth longest flight within United States is from JFK to OAK.

```{r}
# The shortest flights
flights_Testimate %>%
  select(carrier:distance, travelTime) %>%
    arrange(distance)

```

```{r}
flights_Testimate %>%
  filter(distance == 17)
```

### 5.4 Select {#select}

`select()` can be used to rename variables, but it’s rarely useful because it drops all of the variables not explicitly mentioned. Instead, use `rename()`, which is a variant of `select()` that keeps all the variables that aren’t explicitly mentioned


```{r}
# change tailnum to tail_num

flights %>%
  rename(tail_num = tailnum)
```

Nothing changes the original dataset `flights_Testimate`, which still has that variable named `tailnum`.

Another option is to use `select()` in conjunction with the `everything()` helper. This is useful if you have a handful of variables you’d like to move to the start of the data frame.

Move some main information to the start of the dataframe and keep the rest:
```{r}
flights %>%
  select(carrier:dest, everything())
```


### 5.4.1 Exercise {#ex_5_4_1}

#### 1.Brainstorm as many ways as possible to select `dep_time`, `dep_delay`, `arr_time`, and `arr_delay` from `flights`.

1.0. Just list all the variables to select 
```{r}
flights %>%
  select(dep_time, dep_delay, arr_time, arr_delay)
```

1.1. `starts_with` and `ends_with`

```{r}
flights %>%
  select(starts_with("dep"), starts_with("arr"))

flights %>%
  select(ends_with("time"), ends_with("delay")) %>% 
    select(-c(starts_with("sched"), starts_with("air")))
```

1.2. Fancier: `contains` 
```{r}
flights %>% select(dep_time:arr_delay, -c(contains("sched")))
```


1.3. Start learning `matches` syntax
```{r}
flights %>% select(matches("^dep|^arr"))
```

#### 2. What happens if you include the name of a variable multiple times in a `select()` call?

```{r}
flights %>% select(dep_time, dep_time, dep_time)
```

Just select the variable once despite multiple select call.


#### 3. What does the one_of() function do? Why might it be helpful in conjunction with this vector? `vars <- c("year", "month", "day", "dep_delay", "arr_delay")`
```{r}
vars <- c("year", "month", "day", "dep_delay", "arr_delay")
flights %>% select(one_of(vars))
```

#### 4. Does the result of running the following code surprise you? How do the select helpers deal with case by default? How can you change that default?

```{r}
select(flights, contains("TIME"))
select(flights, contains("TIME", ignore.case = FALSE))
```

`contains` does not treat argument with case sensitive by default. Use `ignore.case = FALSE` to cange that setting.


### 5.5 Add new variables with `mutate()`  {#mutate}

```{r}
flights_sml <- select(flights, 
  year:day, 
  ends_with("delay"), 
  distance, 
  air_time
)
mutate(flights_sml,
  gain = arr_delay - dep_delay,
  speed = distance / air_time * 60
)
```

If you only want to keep the new variables, use `transmute()`:


```{r}
transmute(flights,
  gain = arr_delay - dep_delay,
  hours = air_time / 60,
  gain_per_hour = gain / hours  # minutes gain per airtime hour
)
```


#### 5.5.1 Useful creation functions {#useful_creation}

The key property is that the function `mutate` must be vectorised: it must take a vector of values as input, return a vector with the same number of values as output. 

If one parameter is shorter than the other, it will be automatically extended to be the same length. This is most useful when one of the arguments is a single number: air_time / 60, hours * 60 + minute, etc.

Ranking: The default gives smallest values the small ranks; use `desc(x)` to give the largest values the smallest ranks.
```{r}
y <- c(70, 10, 10, NA, 30, 40)
min_rank(y)
min_rank(desc(y))  # order by value, more common

```


#### 5.5.2 Exercises {#ex_5_5_2}

1. Currently `dep_time` and `sched_dep_time` are convenient to look at, but hard to compute with because they’re not really continuous numbers. Convert them to a more convenient representation of number of minutes since midnight.

Modular arithmetic: `%/%` (integer division) and `%%` (remainder), where `x == y * (x %/% y) + (x %% y)`

```{r}
flights_5.5.2.1 = flights_Testimate %>% 
    mutate(dep_time_totalMinutes = dep_time %/% 100 * 60 + dep_time %% 100, sched_dep_time_totalMinutes = sched_dep_time %/% 100 * 60 + sched_dep_time %% 100) %>%
    select(year:dep_delay,dep_time_totalMinutes,sched_dep_time_totalMinutes,arr_time:time_hour )
```
Use `select` to rearrange column orders.


2. Compare `air_time` with `arr_time - dep_time`. What do you expect to see? What do you see? What do you need to do to fix it?

```{r}
flights_Testimate %>% 
  mutate(arr_dep = arr_time - dep_time) %>%
    select(carrier:air_time, arr_dep) 
```

We expect to see `air_time` equals `arr_time - dep_time`, however they are not remotely equal, and they shouldn't be, because as mentioned in problem 1, `dep_time` and `arr_time` (and their corresponding scheduked version) are not recorded in a manner that easy to add or substract. So a more convenient version of representation is preferred.

```{r}
flights_Testimate = flights_Testimate %>%
  mutate(dep_time_totalMinutes = dep_time %/% 100 * 60 + dep_time %% 100, 
         sched_dep_time_totalMinutes = sched_dep_time %/% 100 * 60 + sched_dep_time %% 100, 
         arr_time_totalMinutes = arr_time %/% 100 * 60 + arr_time %% 100, 
         sched_arr_time_totalMinutes = sched_arr_time %/% 100 * 60 + sched_arr_time %% 100) %>%
   select(year:arr_delay,dep_time_totalMinutes, sched_dep_time_totalMinutes, arr_time_totalMinutes, 
          sched_arr_time_totalMinutes, air_time, carrier:travelTime)

```

```{r}
flights_Testimate %>%
  mutate(air_time_minutes = arr_time_totalMinutes - dep_time_totalMinutes) %>%
    select(carrier:distance, sched_dep_time_totalMinutes, sched_arr_time_totalMinutes, dep_time_totalMinutes, arr_time_totalMinutes, air_time_minutes, air_time)
```


3. Compare `dep_time`, `sched_dep_time`, and `dep_delay`. How would you expect those three numbers to be related?

```{r}
flights_Testimate %>%
  mutate(dep_delay_copy = dep_time - sched_dep_time) %>%
    select(dep_time, sched_dep_time, dep_delay_copy, dep_delay)
```

```{r}
flights_Testimate %>%
  mutate(dep_delay_copy = dep_time_totalMinutes - sched_dep_time_totalMinutes) %>%
    select(dep_time, sched_dep_time, dep_delay_copy, dep_delay)

```


4. Find the 10 most delayed flights using a ranking function. How do you want to handle ties? Carefully read the documentation for `min_rank()`.

My first answer: arr_delay
```{r}
flights_Testimate %>%
  mutate(delay_rank = min_rank(desc(arr_delay))) %>%
    select(carrier:dest, arr_delay, delay_rank) %>%
      arrange(delay_rank)
```

Improved version (without having to create a new variable)
```{r}
flights_Testimate %>%
  filter(min_rank(desc(arr_delay)) <= 10) %>%
    select(carrier:dest, arr_delay) %>%
      arrange(desc(arr_delay))
```

5. What does 1:3 + 1:10 return? Why?

```{r}
1:3 + 1:10
```

It returns a 10 dimensional vector along with a warning message. Looks like the shorter vector 1:3 is repeated out to the length of the longer 1:10 one.


\vspace*{3\baselineskip}

#### 5.6 Grouped summaries with `summarise` {#summarise}

It collapses a data frame to a single row.

`summarise()` is not terribly useful unless we pair it with `group_by()`

```{r}
by_day <- group_by(flights, year, month, day)
summarise(by_day, delay = mean(dep_delay, na.rm = TRUE))
```

#### 5.6.1 Combining multiple operations with the pipe {#summPipe}

Imagine that we want to explore the relationship between the distance and average delay for each location. Using what you know about dplyr, you might write code like this:

```{r}
delays <- flights %>% 
  group_by(dest) %>% 
  summarise(
    count = n(),
    dist = mean(distance, na.rm = TRUE),
    delay = mean(arr_delay, na.rm = TRUE)
  ) %>% 
  filter(count > 20, dest != "HNL")

# It looks like delays increase with distance up to ~750 miles and then decrease. Maybe as flights get longer there's more ability to make up delays in the air?
ggplot(data = delays, mapping = aes(x = dist, y = delay)) +
  geom_point(aes(size = count), alpha = 1/3) +
  geom_smooth(se = FALSE)
#> `geom_smooth()` using method = 'loess'
```

There are three steps to prepare this data:

Group flights by destination.

Summarise to compute distance, average delay, and number of flights.

Filter to remove noisy points and Honolulu airport, which is almost twice as far away as the next closest airport.

This code is a little frustrating to write because we have to give each intermediate data frame a name, even though we don’t care about it. Naming things is hard, so this slows down our analysis.

<br>

##### 5.6.2 Missing Values

Naive failure try:
```{r}
flights %>% 
  group_by(year, month, day) %>% 
  summarise(mean = mean(dep_delay))
```

All aggregation functions have an `na.rm` argument which removes the missing values prior to computation:

```{r}
flights %>% 
  group_by(year, month, day) %>% 
  summarise(mean = mean(dep_delay, na.rm = TRUE))
```

Departure delay by day:

```{r}
not_cancelled <- flights %>% 
  filter(!is.na(dep_delay), !is.na(arr_delay))

not_cancelled %>% 
  group_by(year, month, day) %>% 
  summarise(mean = mean(dep_delay))
```

<br>

#### Useful tool alert!
#### 5.6.3 Counts {#count}

##### Whenever you do any aggregation, it’s always a good idea to include either a count `n()`, or a count of non-missing values `(sum(!is.na(x)))`. That way you can check that you’re not drawing conclusions based on very small amounts of data. For example, let’s look at the planes (identified by their tail number) that have the highest average delays:

__*planes identified by their tail number*__

```{r}
# look at the planes (identified by their tail number) that have the highest average delays
delays <- not_cancelled %>% 
  group_by(tailnum) %>%   
  summarise(
    delay = mean(arr_delay)
  )

ggplot(data = delays, mapping = aes(x = delay)) + 
  geom_freqpoly(binwidth = 10)
```


There are some planes that have an average delay of 5 hours (300 minutes), but the story is actually a little more nuanced. We can get more insight if we draw a scatterplot of number of flights vs. average delay:

__Use of `n = n()`__

__Testimate: Displays number of cases within each group you defined by `group_by()`. In this case, if we are interested in delay performance for each plane, which is identified by their tail number (`group_by(tailnum)`) ,then under column of `n` shows the number of flights each plane took.__ 

```{r}
# draw a scatterplot of number of flights vs. average delay
delays <- not_cancelled %>% 
  group_by(tailnum) %>% 
  summarise(
    delay = mean(arr_delay, na.rm = TRUE),
    n = n()  
  )

ggplot(data = delays, mapping = aes(x = n, y = delay)) + 
  geom_point(alpha = 1/10)
```


Not surprisingly, there is much greater variation in the average delay when there are few flights. 
The shape of this plot is very characteristic: whenever you plot a mean (or other summary) vs. group size, you’ll see that the variation decreases as the sample size increases.


When looking at this sort of plot, it’s often useful to filter out the groups with the smallest numbers of observations, so you can see more of the pattern and less of the extreme variation in the smallest groups. 


```{r}
# filter out the groups with the smallest numbers of observations to see more of the pattern and less of the extreme variation in the smallest groups
delays %>% 
  filter(n > 25) %>%  
  ggplot(mapping = aes(x = n, y = delay)) + 
    geom_point(alpha = 1/10)
```


RStudio tip: a useful keyboard shortcut is Cmd/Ctrl + Shift + P. This resends the previously sent chunk from the editor to the console. This is very convenient when you’re (e.g.) exploring the value of n in the example above. You send the whole block once with Cmd/Ctrl + Enter, then you modify the value of n and press Cmd/Ctrl + Shift + P to resend the complete block.


### Baseball performance

Here I use data from the Lahman package to compute the batting average (number of hits / number of attempts) of every major league baseball player.

```{r}
# Convert to a tibble so it prints nicely
library(Lahman)

batting <- as_tibble(Lahman::Batting)

batters <- batting %>% 
  group_by(playerID) %>% 
  summarise(
    ba = sum(H, na.rm = TRUE) / sum(AB, na.rm = TRUE),
    ab = sum(AB, na.rm = TRUE)
  )

batters %>% 
  filter(ab > 100) %>% 
  ggplot(mapping = aes(x = ab, y = ba)) +
    geom_point() + 
    geom_smooth(se = FALSE)
#> `geom_smooth()` using method = 'gam'
```

you see two patterns:

1. As above, the variation in our aggregate decreases as we get more data points.

2. There’s a positive correlation between skill (`ba`) and opportunities to hit the ball (`ab`). This is because teams control who gets to play, and obviously they’ll pick their best players.


#### 5.6.4  Useful Summary Functions{#summary}

It’s sometimes useful to combine aggregation with logical subsetting. 

__*Question: Calculate the average positive delay (by day).*__
```{r}
# Calculate the average positive delay (by day)
not_cancelled %>% 
  group_by(year, month, day) %>% 
  summarise(
    avg_delay1 = mean(arr_delay),
    avg_delay2 = mean(arr_delay[arr_delay > 0]) # the average positive delay
  )
```


##### Measures of Position
`first(x)`, `nth(x, 2)`, `last(x)`. These work similarly to `x[1]`, `x[2]`, and `x[length(x)]` but let you set a default value if that position does not exist (i.e. you’re trying to get the 3rd element from a group that only has two elements). For example, we can find the first and last departure for each day:

```{r}
not_cancelled %>% 
  group_by(year, month, day) %>% 
  summarise(
    first_dep = first(dep_time), # smallest number
    last_dep = last(dep_time)
  )
```


These functions are complementary to filtering on ranks. Filtering gives you all variables, with each observation in a separate row

__*Question: Find the first and last flight of each day.*__

__Testimate trick: display the first and last (earliest and latest) flight of the day (2 rows for each day): `mutate(r = min_rank(dep_time)) %>% filter(r %in% range(r))` __

```{r}
# Find the first and last flight of each day
not_cancelled %>% 
  group_by(year, month, day) %>% 
  mutate(r = min_rank((dep_time))) %>% 
  filter(r %in% range(r))
```

__ Shiny Idea: The upper bound of `r` is the number of flights that day (600 ~ 1000) __

__ `n_distinct()`: To count the number of non-missing values, use `sum(!is.na(x))`. To count the number of distinct (unique) values, use `n_distinct(x)`.__  

__*Question: Which destinations `(group_by(dest))` have the most carriers?*__

```{r}
# Which destinations have the most carriers?
not_cancelled %>% 
  group_by(dest) %>% 
  summarise(carriers = n_distinct(carrier)) %>% 
  arrange(desc(carriers))
```

`count`
```{r}
not_cancelled %>% 
  count(dest)
```
Testimate: similar to `table()`, with even straightforward command.

You can optionally provide a weight variable. For example, you could use this to “count” (sum) the total number of miles a plane flew:

```{r}
not_cancelled %>% 
  count(tailnum, wt = distance) # wt
```

To verify, we may simply calculate / observe like this: 
```{r}
not_cancelled %>%
  filter(tailnum == "D942DN") %>%
    select(tailnum, distance)
```


Counts and proportions of logical values: `sum(x)` gives the number of TRUEs in x, and `mean(x)` gives the proportion.

__*Question: How many flights left before 5am? (these usually indicate delayed flights from the previous day)*__
```{r}
# How many flights left before 5am? (these usually indicate delayed flights from the previous day)

not_cancelled %>%
  group_by(year, month, day) %>%
  summarize(n_early = sum(dep_time < 500))

```

__*Question: What proportion of flights are delayed by more than an hour?*__
```{r}
# What proportion of flights are delayed by more than an hour?
not_cancelled %>% 
  group_by(year, month, day) %>% 
  summarise(hour_perc = round(mean(arr_delay > 60), digit = 3))
```



#### 5.6.5  Grouping by multiple variables {#rollingup}

When you group by multiple variables, each summary peels off one level of the grouping. That makes it easy to progressively roll up a dataset:

```{r}
daily <- group_by(flights, year, month, day)
(per_day  <- summarise(daily, flights = n()))
(per_month <- summarise(per_day, flights = sum(flights)))
(per_year  <- summarise(per_month, flights = sum(flights)))
```


Be careful when progressively rolling up summaries: it’s OK for sums and counts, but you need to think about weighting means and variances, and it’s not possible to do it exactly for rank-based statistics like the median. In other words, the sum of groupwise sums is the overall sum, but the median of groupwise medians is not the overall median.

#### 5.6.6 Ungrouping {#ungrouping}

```{r}
daily %>% 
  ungroup() %>%             # no longer grouped by date
  summarise(flights = n())
```