index.Rmd

--- 
title: "ModernDive"
subtitle: "An Introduction to Statistical and Data Sciences via R"
author: "Chester Ismay and Albert Y. Kim"
date: "`r format(Sys.time(), '%B %d, %Y')`"
site: bookdown::bookdown_site
documentclass: book
bibliography: [bib/books.bib, bib/packages.bib, bib/articles.bib]
biblio-style: apalike
link-citations: no
github-repo: ismayc/moderndiver-book
description: "Combining statistical and computational thinking to make sense of data. An evolution of the traditional introductory statistics curriculum, more focused on reproducible research, data visualization, and modern data analysis techniques and tools including resampling and bootstrapping using R, RStudio, and R Markdown"
---

```{r set-options, include=FALSE}
options(width = 72, digits = 4)
knitr::opts_chunk$set(tidy = FALSE, out.width='\\textwidth', fig.align = "center")

# Version number
version <- "0.1.3"

# Packages needed for following code in book
needed_pkgs <- c(
  # Data packages:
  "nycflights13", "ggplot2movies", "fivethirtyeight", "okcupiddata", "gapminder",
  # Other packages
  "ggplot2", "tibble", "tidyr", "dplyr", "readr",
  "dygraphs", "rmarkdown", "knitr", "mosaic")
new.pkgs <- needed_pkgs[!(needed_pkgs %in% installed.packages())]
if(length(new.pkgs)) {
  install.packages(new.pkgs, repos = "http://cran.rstudio.com")
}

# Additional packages needed to create the book, not for following code in the
# book
needed_pkgs2 <- c("devtools", "webshot", "tufte", "mvtnorm", "stringr")
new.pkgs2 <- needed_pkgs2[!(needed_pkgs2 %in% installed.packages())]
if(length(new.pkgs2)) {
  install.packages(new.pkgs2, repos = "http://cran.rstudio.com")
}

# Check that phantomjs is installed
if(is.null(webshot:::find_phantom()))
  webshot::install_phantomjs()

# Automatically create a bib database for R packages
knitr::write_bib(c(
  .packages(), 'bookdown', 'knitr', 'rmarkdown', 
  'nycflights13', 'devtools', 'ggplot2', 'webshot', 
  'dygraphs', 'tufte', 'okcupiddata', 'mosaic',
   'dplyr', 'ggplot2movies', 'fivethirtyeight', 'tibble', 'readr'
), 'bib/packages.bib')

# Add all simulation results here
dir.create("rds")

# Add all chapter R scripts here
dir.create("docs/scripts")
# Note order matters here:
chapter_titles <- c("getting-started", "viz", "tidy", "wrangling", "regression", "sim", "hypo", "ci")
chapter_numbers <- stringr::str_pad(2:(length(chapter_titles)+1), 2, "left", pad="0")

# for(i in 1:length(chapter_numbers)){
#   Rmd_file <- stringr::str_c(chapter_numbers[i], "-", chapter_titles[i], ".Rmd")
#   R_file <- stringr::str_c("docs/scripts/", chapter_numbers[i], "-", chapter_titles[i], ".R")
#   knitr::purl(Rmd_file, R_file)
# }
```


# Introduction {#intro}

<img src="https://cran.r-project.org/Rlogo.svg" alt="Drawing" style="height: 150px;"/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<img src="https://www.rstudio.com/wp-content/uploads/2014/07/RStudio-Logo-Blue-Gradient.png" alt="Drawing" style="height: 150px;"/>

**Help! I'm new to R and RStudio and I need to learn about them! However I'm completely new to coding! What do I do?** If you're asking yourself this question, then you've come to the right place. Start with our [Introduction for Students](#intro-for-students).

* *Are you an instructor hoping to use this book in your courses? Then click [here](#intro-instructors) for more information on how to teach with this book.*
* *Are you looking to connect with and contribute to ModernDive? Then click [here](#connect-contribute) for information on our mailing list, our GitHub repository, and other links.*
* *Are you curious about the publishing of this book? Then click [here](#about-book) for more information on the open-source technology used, in particular R Markdown and the bookdown package.*





## Introduction for Students {#intro-for-students}

This book assumes no prerequisites: no algebra, no calculus, and no prior programming/coding experience. This is intended to be a gentle introduction to the practice of analyzing data and answering questions using data the way data scientists, statisticians, data journalists, and other researchers would.

### What you will learn from this book {#learning-goals}

We hope that by the end of this book, you'll have learned

1. How to use R to explore data.  
1. How to answer statistical questions using tools like confidence intervals and hypothesis tests. 
1. How to effectively create "data stories" using these tools. 

What do we mean by data stories? We mean any analysis that answers questions with data, such as ["How strong is the relationship between per capita income and crime in Chicago neighborhoods?"](http://rpubs.com/ry_lisa_elana/chicago) and ["How many f**ks does Quentin Tarantino give (as measured by the amount swearing in his films)?"](https://ismayc.github.io/soc301_s2017/group_projects/group4.html) For other examples of such analyses, look at the final projects for two courses that have used ModernDive previously:

* Middlebury College [MATH 116 Introduction to Statistical and Data Sciences](https://rudeboybert.github.io/MATH116/PS/final_project/final_project_outline.html#past_examples) using student collected data.
* Pacific University [SOC 301 Social Statistics](https://ismayc.github.io/soc301_s2017/group-projects/index.html) using data from [FiveThirtyEight.com](https://fivethirtyeight.com/).

This book will help you develop your "data science toolbox", including tools such as data visualization, data formatting, data wrangling, and data modeling via regression. With these tools, you'll be able to perform the entirety of the "data/science pipeline" (see Chapter \@ref(pipeline) for more details). 

In particular, this book will lean heavily on data visualization.  In today's world, we are bombarded with graphics that attempt to convey ideas.  We will explore what makes a good graphic and what the standard ways are to convey relationships with data.  You'll also see the use of visualization to introduce concepts like mean, median, standard deviation, distributions, etc.  In general, we'll use visualization as a way of building almost all of the ideas in this book.

For the statistical topics in this book, we have intentionally avoided using mathematical formulas as possible and instead have focused on developing a conceptual understanding via data visualization, statistical computing, and simulations.  We hope this is a more intuitive experience than the way statistics has traditionally been taught in the past (and how it is commonly perceived).

Finally, you'll learn the importance of literate programming.  By this we mean you'll learn how to write code that is useful not just for a computer to execute but also for readers to understand exactly what your analysis is doing and how you did it.  This part of a greater effort to encourage reproducible research (see Chapter \@ref(reproducible) for more details). Hal Abelson coined the phrase that we will follow throughout this book:

> "Programs must be written for people to read, and only incidentally for machines to execute."

We understand that there may be challenging moments as you learn to program but we are here to help you and you should know that there is a huge community of R users that are always happy to help everyone along as well. 

At this point, if you are interested in more in-depth discussions on the data/science pipeline and reproducible research, then continue with Chapters \@ref(pipeline) and \@ref(reproducible) below. Otherwise, let's get started with R and RStudio in Chapter \@ref(getting-started).


### Data/science pipeline {#pipeline}

You may think of statistics as just being a bunch of numbers.  We commonly hear the phrase "statistician" when listening to broadcasts of sporting events.  Statistics (in particular, data analysis), in addition to describing numbers like with baseball batting averages, plays a vital role in all of the sciences.  You'll commonly hear the phrase "statistically significant" thrown around in the media.  You'll see things that say "Science now shows that chocolate is good for you."  Underpinning these claims is data analysis.  By the end of this book, you'll be able to better understand whether these claims should be trusted or whether we should be wary.  Inside data analysis are many sub-fields that we will discuss throughout this book (not necessarily in this order):

- data collection
- data manipulation
- data visualization
- data modeling
- inference
- correlation and regression
- interpretation of results
- data storytelling

These sub-fields are summarized in what Grolemund and Wickham term the "data/science pipeline" in Figure \@ref(fig:pipeline-figure).

```{r pipeline-figure, echo=FALSE, fig.align='center', fig.cap="Data/Science Pipeline"}
knitr::include_graphics("images/tidy1.png")
```

We will begin with a discussion on what is meant by tidy data and then dig into the gray **Understand** portion of the cycle and conclude by talking about interpreting and discussing the results of our models via **communication**.  These steps are vital to any statistical analysis.  But why should you care about statistics?  "Why did they make me take this class?"

There's a reason so many fields require a statistics course. Scientific knowledge grows through an understanding of statistical significance and data analysis. You needn't be intimidated by statistics.  It's not the beast that it used to be and, paired with computation, you'll see how reproducible research in the sciences particularly increases scientific knowledge.


### Reproducible Research {#reproducible}

> "The most important tool is the _mindset_, when starting, that the end product will be reproducible." – Keith Baggerly

Another goal of this book is to help readers understand the importance of reproducible analyses. The hope is to get readers into the habit of making their analyses reproducible from the very beginning.  This means we'll be trying to help you build new habits.  This will take practice and be difficult at times. You'll see just why it is so important for you to keep track of your code and well-document it to help yourself later and any potential collaborators as well.  

Copying and pasting results from one program into a word processor is not the way that efficient and effective scientific research is conducted.  It's much more important for time to be spent on data collection and data analysis and not on copying and pasting plots back and forth across a variety of programs.

In a traditional analyses if an error was made with the original data, we'd need to step through the entire process again:  recreate the plots and copy and paste all of the new plots and our statistical analysis into your document.  This is error prone and a frustrating use of time.  We'll see how to use R Markdown to get away from this tedious activity so that we can spend more time doing science.

> "We are talking about _computational_ reproducibility." - Yihui Xie

Reproducibility means a lot of things in terms of different scientific fields.  Are experiments conducted in a way that another researcher could follow the steps and get similar results?  In this book, we will focus on what is known as **computational reproducibility**.  This refers to being able to pass all of one's data analysis, datasets, and conclusions to someone else and have them get exactly the same results on their machine.  This allows for time to be spent doing actual science and interpreting of results and assumptions instead of the more error prone way of starting from scratch or following a list of steps that may be different from machine to machine.

<!--
Additionally, this book will focus on computational thinking, data thinking, and inferential thinking. We'll see throughout the book how these three modes of thinking can build effective ways to work with, to describe, and to convey statistical knowledge.  
-->

 



## Introduction for Instructors {#intro-instructors}

This book is inspired by three books:

- "Mathematical Statistics with Resampling and R" [@hester2011],
- "OpenIntro: Intro Stat with Randomization and Simulation" [@isrs2014], and 
- "R for Data Science" [@rds2016].

The first book, while designed for upper-level undergraduates and graduate students, provides an excellent resource on how to use resampling to impart statistical concepts like sampling distributions using computers instead of large-sample approximations and other mathematical formulas.  The last two books are free options to learning introductory statistics and data science, providing an alternative to the many traditionally expensive introductory statistics textbooks. 

When looking over the large number of introductory statistics textbooks that currently exist, we found that there wasn't one that incorporated many newly developed R packages directly into the text, in particular the many packages included in the [`tidyverse`](http://tidyverse.org/) collection of packages, such as `ggplot2`, `dplyr`, and `broom`. Additionally, there wasn't an open-source and easily reproducible textbook available that exposed new learners all of three of the learning goals listed at the outset of Chapter \@ref(learning-goals).



### Who is this book for?

This book is intended for instructors of traditional introductory statistics classes using RStudio, either the desktop or server version, who would like to inject more data science topics into their syllabus. We assume that students taking the class will have no prior algebra, calculus, nor programming/coding experience.

Here are some principles and beliefs we kept in mind while writing this text. If you agree with them, this might be the book for you.

1. **Blur the lines between lecture and lab**
    + With increased availability and accessibility of laptops and open-source non-proprietary statistical software, the strict dichotomy between lab and lecture can be loosened.
    + It's much harder for students to understand the importance of using software if they only use it once a week or less.  They forget the syntax in much the same way someone learning a foreign language forgets the rules. Frequent reinforcement is key.
1. **Focus on the entire data/science research pipeline**
    + We believe that the entirety of Grolemund and Wickham's [data/science pipeline](http://r4ds.had.co.nz/introduction.html) should be taught.
    + We believe in ["minimizing prerequisites to research"](https://arxiv.org/abs/1507.05346): students should be answering questions with data as soon as possible.
1. **It's all about the data**
    + We leverage R packages for rich, real, and realistic datasets that at the same time are easy-to-load, such as the `nycflights13` and `fivethirtyeight` packages.
    + We believe that [data visualization is a gateway drug for statistics](http://escholarship.org/uc/item/84v3774z) and that the Grammar of Graphics as implemented in the `ggplot2` package is the best way to impart such lessons. However, we often hear: "You can't teach `ggplot2` for data visualization in intro stats!" We, like [David Robinson](http://varianceexplained.org/r/teach_ggplot2_to_beginners/), are much more optimistic.
    + `dplyr` has made data wrangling much more [accessible](http://chance.amstat.org/2015/04/setting-the-stage/) to novices, and hence much more interesting datasets can be explored. 
1. **Use simulation/resampling to introduce statistical inference, not probability/mathematical formulas**
    + Instead of using formulas, large-sample approximations, and probability tables, we teach statistical concepts using resampling-based inference, embedded in tools like the [`mosaic`](https://github.com/ProjectMOSAIC/mosaic) package's `shuffle()`, `resample()`, and `do()` functions.
    + This allows for a de-emphasis of traditional probability topics, freeing up room in the syllabus for other topics.
1. **Don't fence off students from the computation pool, throw them in!**
    + Computing skills are essential to working with data in the 21st century. Given this fact, we feel that to shield students from computing is to ultimately do them a disservice.
    + We are not teaching a course on coding/programming per se, but rather just enough of the computational and algorithmic thinking necessary for data analysis.
1. **Complete reproducibility and customizability**
    + We are frustrated when textbooks give examples, but not the source code and the data itself. We give you the source code for all examples as well as the whole book!
    + Ultimately the best textbook is one you've written yourself. You know best your audience, their background, and their priorities. You know best your own style and the types of examples and problems you like best. Customizability is the ultimate end. For more about how to make this book your own, see [About this Book](#about-book).







## Connect and Contribute {#connect-contribute}

If you would like to connect with ModernDive, check out the following links:

* If you would like to receive period updates about ModernDive (roughly every 3 months), please sign up for our [mailing list](http://eepurl.com/cBkItf).
* We're on Twitter at [ModernDive](https://twitter.com/ModernDive)!
* Contact Albert at [albert@moderndive.com](mailto:albert@moderndive.com) and Chester [chester@moderndive.com](mailto:chester@moderndive.com)

If you're interested in contributing, check out the following links:

* Please let us know of any typos, issues, and bugs on our [issues](https://github.com/ismayc/moderndiver-book/issues) page.
* The source code that created ModernDive is available on [GitHub](https://github.com/ismayc/moderndiver-book). See [About this Book](#about-book) for more details.






## About This Book {#about-book}

### Bookdown

This book was written using RStudio's [bookdown](https://bookdown.org/) package by Yihui Xie [@R-bookdown]. This package simplifies the publishing of books by having all content written in [R Markdown](http://rmarkdown.rstudio.com/html_document_format.html). 

The source code is available on [GitHub](https://github.com/ismayc/moderndiver-book). If you click on the **release** tab near the top of the page, you can download all of the source code for whichever release version you'd like to work with and use. Feel free to modify the book as you wish for your own needs, but please list the authors at the top of `index.Rmd` as "Chester Ismay, Albert Y. Kim, and YOU!"

If you find typos or other errors or have suggestions on how to better word something in the book, please create a pull request too!  We also welcome issue creation.  Let's all work together to make this book as great as possible for as many students and instructors as possible.

Could this be a new paradigm for textbooks? Instead of the traditional model of textbook companies publishing updated *editions* of the textbook every few years, we apply a software design influenced model of publishing more easily updated *versions*.  We can then leverage open-source communities of instructors and developers for ideas, tools, resources, and feedback.


### Contributors

The authors would like to thank Nina Sonneborn and Kristin Bott for their contributions. A special thanks goes to Dr. Yana Weinstein at [The Learning Scientists](http://www.learningscientists.org/yana-weinstein/) for her valuable feedback.



### Colophon 

* ModernDive was last updated `r paste("by", Sys.info()[["user"]], "on", format(Sys.time(), "%A, %B %d, %Y %X %Z"))` and is at Version `r version`. 
* ModernDive is written using the CC0 1.0 Universal License; more information on this licence is available [here](https://creativecommons.org/publicdomain/zero/1.0/).
* ModernDive uses the following versions of R packages (and their dependent packages):

```{r colophon, echo=FALSE}
knitr::kable(devtools::session_info(needed_pkgs)$packages, 
             booktabs = TRUE,
             longtable = TRUE)
```