01-Statistical-Data-Science.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PSTAT 134/234 - Statistical Data Science\n",
    "\n",
    "---\n",
    "\n",
    "## Instructor: Sang-Yun Oh\n",
    "\n",
    "- Lectures: MW 11 am - 12:15 pm\n",
    "\n",
    "- Office: South Hall 5514\n",
    "\n",
    "- Office hours: Tuesday 4-6 pm\n",
    "\n",
    "\n",
    "## Teaching Assistant: Sergio Rodriguez \n",
    "\n",
    "- Sections: F 9 - 9:50 am / 12 - 12:50 pm\n",
    "\n",
    "- Office: South Hall 6432-W\n",
    "\n",
    "- Office hours: Thursday 1-3 pm\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Course Information \n",
    "\n",
    "---\n",
    "\n",
    "## Grading\n",
    "\n",
    "* Attendance in lectures and sections are required (20%)  \n",
    "    Total of five will be dropped. No exceptions\n",
    "\n",
    "* Individual in-class midterm (20%)\n",
    "\n",
    "* Individual assignments (30%)\n",
    "\n",
    "* Group final project & presentations (30%)\n",
    "\n",
    "\n",
    "## Textbooks\n",
    "\n",
    "- [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/) by Jake Vanderplas\n",
    "\n",
    "- [R for Data Science](http://r4ds.had.co.nz/) by Hadley Wickham and Garrett Grolemund\n",
    "\n",
    "- Other resources as necessary\n",
    "\n",
    "\n",
    "## Learn by doing\n",
    "\n",
    "- Critical statistical thinking is crucial\n",
    "\n",
    "- Significant programming is required\n",
    "\n",
    "- Many software tools will be new \n",
    "    e.g., R, Python, command line tools, etc\n",
    "\n",
    "- Proactive attitude is a must!  \n",
    "    e.g., asking questions, discussing, experimenting, RTM (read-the-manual)\n",
    "\n",
    "- Diverse backgrounds mean you will have different strengths!  \n",
    "    Help each other, and assess your own areas of improvement\n",
    "\n",
    "- You don't have to be an expert at everything\n",
    "\n",
    "- But you have to be willing to dig deeper on your own"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data Science\n",
    "\n",
    "---\n",
    "\n",
    "- A Forbes magazine editorial on [History of Data Science](https://www.forbes.com/sites/gilpress/2013/05/28/a-very-short-history-of-data-science/#7131af3355cf)  \n",
    "\n",
    "- Isolated statistical/machine learning algorithms are not enough in real usage scenarios \n",
    "\n",
    "- Real world challenges are much more broad \n",
    "\n",
    "## Data challenges\n",
    "\n",
    "- Data collection (text files, web pages, pdf files, APIs, ...)\n",
    "\n",
    "- Data storage (disk space, redundancy, data structure, databases, security, privacy, ...)\n",
    "\n",
    "- Data access (ease of access, cloud vs. local, security, ...)\n",
    "\n",
    "- Data uniformity and consistency (cleaning, wrangling, entity resolution, ...)\n",
    "\n",
    "## Analysis challenges\n",
    "\n",
    "- Not all data are useful: e.g., \"signal to noise\", curse of dimensionality\n",
    "\n",
    "- Critical thinking for formulating questions  \n",
    "    e.g., prediction of medical diagnosis based on predictors\n",
    "\n",
    "- Derived features may be redundant and lead to unstable results \n",
    "\n",
    "- Effects of outliers on algorithms\n",
    "\n",
    "- Getting a “feel” for the data (visualization and summary statistics)\n",
    "\n",
    "- Setting a direction\n",
    "\n",
    "- Getting a simple but complete analysis done is better than doing one part “perfectly”\n",
    "\n",
    "## Interpretation challenges\n",
    "\n",
    "- What results do you obtain from your analysis?\n",
    "\n",
    "- Do you believe it? What can you conclude? (analysis outcome vs. conclusion)\n",
    "\n",
    "- Refining your analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Course outline\n",
    "\n",
    "---\n",
    "\n",
    "* **Week 1** (4/2-4/6): Data and uncertainty   \n",
    "    - Computing: Jupyter notebook and Python primer\n",
    "    - Reading: [Chapter 1 (skim)-2](https://jakevdp.github.io/PythonDataScienceHandbook/index.html#1.-IPython:-Beyond-Normal-Python) in Vanderplas\n",
    "    \n",
    "* **Week 2** (4/9-4/13): Data scraping, transformation, and wrangling\n",
    "    - Computing: Shell commands and Pandas\n",
    "    - Reading: [Chapter 3](https://jakevdp.github.io/PythonDataScienceHandbook/index.html#3.-Data-Manipulation-with-Pandas) in Vanderplas  \n",
    "        [The Unix Shell](http://swcarpentry.github.io/shell-novice/) by Software Carpentry  \n",
    "        \n",
    "* **Week 3-4** (4/16-4/27): Visualization and exploratory analysis\n",
    "    - Computing: Matplotlib and Scikit-learn\n",
    "    - Reading: [Chapter 4 (skim) - 5](https://jakevdp.github.io/PythonDataScienceHandbook/index.html#4.-Visualization-with-Matplotlib)\n",
    "\n",
    "* **In-class midterm** (4/30)\n",
    "\n",
    "* **Week 5-6** (5/2-5/11): Finance data module\n",
    "\n",
    "* **Week 7-8** (5/14-5/24): Health data module\n",
    "               \n",
    "* **Week 9** (5/28-6/1): Text data module\n",
    "\n",
    "* **Week 10** (6/4-6/8): Final project presentations\n",
    "\n",
    "* **Final Projects** (6/14): Final project presentations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Computational Environment\n",
    "\n",
    "---\n",
    "\n",
    "## Github\n",
    "\n",
    "* [Github Student Account](https://education.github.com/pack)\n",
    "\n",
    "\n",
    "## Jupyterhub\n",
    "\n",
    "* [Course Jupyter Hub](https://jupyterhub.lsit.ucsb.edu)\n",
    "\n",
    "* PSTAT 134/234 coursework only\n",
    "\n",
    "* Your work can be inspected by teaching staff at any time\n",
    "\n",
    "* Sign the [privacy policy](https://goo.gl/forms/pwa0FKNy6F0ZT8U32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Jupyter Notebook\n",
    "\n",
    "---\n",
    "\n",
    "- Interactive python environment for code and text\n",
    "\n",
    "- Accessed using a web browser\n",
    "\n",
    "- \"Kernels\" accessible through Jupyter include Python, shell, R, Julia, others  \n",
    "    [Jupyter kernels](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels)\n",
    "    \n",
    "- Cells contain code (depends on kernel or magic), formatted text (markdown)\n",
    "\n",
    "## Writing formatted text\n",
    "\n",
    "- Text formatting with [markdown syntax](https://guides.github.com/features/mastering-markdown/)\n",
    "\n",
    "- Math equations with latex commands:  \n",
    "    e.g., `$$ \\hat \\mu = \\frac{1}{n}\\sum_{i=1}^n x_i $$` produces:\n",
    "    $$ \\hat \\mu = \\frac{1}{n}\\sum_{i=1}^n x_i,\\qquad \\hat\\sigma^2 = \\frac{1}{n}\\sum_{i=1}^n (x_i-\\hat\\mu)^2 $$\n",
    "\n",
    "- Images with `![Not Lazy](http://thoughtfulcampaigner.org/wp-content/uploads/2017/10/im-not-lazy-im-just-in-energy-saving-mode-sleepy-cat.jpg)`\n",
    "    ![Not Lazy](http://thoughtfulcampaigner.org/wp-content/uploads/2017/10/im-not-lazy-im-just-in-energy-saving-mode-sleepy-cat.jpg)\n",
    "\n",
    "</br>\n",
    "\n",
    "- Headings, tables, lists, bold, italic, etc.\n",
    "\n",
    "- Markdown syntax can have variations: e.g., [Github Flavored Markdown (GFM)](https://guides.github.com/features/mastering-markdown/#GitHub-flavored-markdown)\n",
    "\n",
    "## Coding in Python and more\n",
    "\n",
    "- Python interpreter\n",
    "\n",
    "- IPython is short for interactive Python with additional functionality  \n",
    "    e.g., tab completion, syntax highlighting, magic, etc. (Chapter 1 in Vanderplas)  \n",
    "    [demo]\n",
    "\n",
    "- IPython's \"[line, cell] magic\" commands  \n",
    "    e.g., debugging, code timing, shell commands, execute external script, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "application/json": {
       "cell": {
        "!": "OSMagics",
        "HTML": "Other",
        "SVG": "Other",
        "bash": "Other",
        "capture": "ExecutionMagics",
        "debug": "ExecutionMagics",
        "file": "Other",
        "html": "DisplayMagics",
        "javascript": "DisplayMagics",
        "js": "DisplayMagics",
        "latex": "DisplayMagics",
        "markdown": "DisplayMagics",
        "perl": "Other",
        "prun": "ExecutionMagics",
        "pypy": "Other",
        "python": "Other",
        "python2": "Other",
        "python3": "Other",
        "ruby": "Other",
        "script": "ScriptMagics",
        "sh": "Other",
        "svg": "DisplayMagics",
        "sx": "OSMagics",
        "system": "OSMagics",
        "time": "ExecutionMagics",
        "timeit": "ExecutionMagics",
        "writefile": "OSMagics"
       },
       "line": {
        "alias": "OSMagics",
        "alias_magic": "BasicMagics",
        "autocall": "AutoMagics",
        "automagic": "AutoMagics",
        "autosave": "KernelMagics",
        "bookmark": "OSMagics",
        "cat": "Other",
        "cd": "OSMagics",
        "clear": "KernelMagics",
        "colors": "BasicMagics",
        "config": "ConfigMagics",
        "connect_info": "KernelMagics",
        "cp": "Other",
        "debug": "ExecutionMagics",
        "dhist": "OSMagics",
        "dirs": "OSMagics",
        "doctest_mode": "BasicMagics",
        "ed": "Other",
        "edit": "KernelMagics",
        "env": "OSMagics",
        "gui": "BasicMagics",
        "hist": "Other",
        "history": "HistoryMagics",
        "killbgscripts": "ScriptMagics",
        "ldir": "Other",
        "less": "KernelMagics",
        "lf": "Other",
        "lk": "Other",
        "ll": "Other",
        "load": "CodeMagics",
        "load_ext": "ExtensionMagics",
        "loadpy": "CodeMagics",
        "logoff": "LoggingMagics",
        "logon": "LoggingMagics",
        "logstart": "LoggingMagics",
        "logstate": "LoggingMagics",
        "logstop": "LoggingMagics",
        "ls": "Other",
        "lsmagic": "BasicMagics",
        "lx": "Other",
        "macro": "ExecutionMagics",
        "magic": "BasicMagics",
        "man": "KernelMagics",
        "matplotlib": "PylabMagics",
        "mkdir": "Other",
        "more": "KernelMagics",
        "mv": "Other",
        "notebook": "BasicMagics",
        "page": "BasicMagics",
        "pastebin": "CodeMagics",
        "pdb": "ExecutionMagics",
        "pdef": "NamespaceMagics",
        "pdoc": "NamespaceMagics",
        "pfile": "NamespaceMagics",
        "pinfo": "NamespaceMagics",
        "pinfo2": "NamespaceMagics",
        "pip": "BasicMagics",
        "popd": "OSMagics",
        "pprint": "BasicMagics",
        "precision": "BasicMagics",
        "profile": "BasicMagics",
        "prun": "ExecutionMagics",
        "psearch": "NamespaceMagics",
        "psource": "NamespaceMagics",
        "pushd": "OSMagics",
        "pwd": "OSMagics",
        "pycat": "OSMagics",
        "pylab": "PylabMagics",
        "qtconsole": "KernelMagics",
        "quickref": "BasicMagics",
        "recall": "HistoryMagics",
        "rehashx": "OSMagics",
        "reload_ext": "ExtensionMagics",
        "rep": "Other",
        "rerun": "HistoryMagics",
        "reset": "NamespaceMagics",
        "reset_selective": "NamespaceMagics",
        "rm": "Other",
        "rmdir": "Other",
        "run": "ExecutionMagics",
        "save": "CodeMagics",
        "sc": "OSMagics",
        "set_env": "OSMagics",
        "store": "StoreMagics",
        "sx": "OSMagics",
        "system": "OSMagics",
        "tb": "ExecutionMagics",
        "time": "ExecutionMagics",
        "timeit": "ExecutionMagics",
        "unalias": "OSMagics",
        "unload_ext": "ExtensionMagics",
        "who": "NamespaceMagics",
        "who_ls": "NamespaceMagics",
        "whos": "NamespaceMagics",
        "xdel": "NamespaceMagics",
        "xmode": "BasicMagics"
       }
      },
      "text/plain": [
       "Available line magics:\n",
       "%alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode\n",
       "\n",
       "Available cell magics:\n",
       "%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile\n",
       "\n",
       "Automagic is ON, % prefix IS NOT needed for line magics."
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%lsmagic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Line magic starts with `%`\n",
    "\n",
    "- Debugging example (python debugger): `%pdb`  \n",
    "    [Demo]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Automatic pdb calling has been turned OFF\n",
      "x is lowercase: True\n",
      "y is lowercase: True\n"
     ]
    }
   ],
   "source": [
    "%pdb off\n",
    "\n",
    "def print_string(x, y):\n",
    "    print('x is lowercase:', x.islower())\n",
    "    print('y is lowercase:', y.islower())\n",
    "\n",
    "print_string('a', 'b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Cell magic starts with `%%`\n",
    "\n",
    "- Entire cell is interpreted differently\n",
    "\n",
    "#### Time running time of a cell by `%%timeit`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "109 µs ± 4.06 µs per loop (mean ± std. dev. of 10 runs, 500 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n500 -r10\n",
    "total = []\n",
    "for i in range(1000):\n",
    "    total += [i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "33.5 µs ± 439 ns per loop (mean ± std. dev. of 10 runs, 500 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n500 -r10\n",
    "total = [i for i in range(1000)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Run bash commands: `%%bash`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 440M\n",
      "drwxr-xr-x  4 jovyan users 4.0K Apr 11 05:38 .\n",
      "drwxrwxr-x 10 jovyan  1000 4.0K Apr  9 19:57 ..\n",
      "-rw-r--r--  1 jovyan users 249K Apr 11 05:38 00-Syllabus.html\n",
      "-rw-r--r--  1 jovyan users 4.7K Apr 11 05:38 00-Syllabus.ipynb\n",
      "-rw-r--r--  1 jovyan users 3.1K Apr 11 05:38 00-Syllabus.py\n",
      "-rw-r--r--  1 jovyan users  13K Apr 11 04:37 01-Statistical-Data-Science.ipynb\n",
      "-rw-rw-r--  1 jovyan  1000  20K Apr 11 04:37 02-Data-and-Uncertainty.ipynb\n",
      "-rw-rw-r--  1 jovyan  1000  11K Apr 11 04:36 03-Data-collection-and-manipulation.ipynb\n",
      "-rw-r--r--  1 jovyan users  22M Dec  6  2013 1998zipcode.zip\n",
      "-rw-r--r--  1 jovyan users  17M Dec  6  2013 2001zipcode.zip\n",
      "-rw-r--r--  1 jovyan users  19M Apr 14  2015 2002zipcode.zip\n",
      "-rw-r--r--  1 jovyan users  39M Dec  6  2013 2004zipcode.zip\n",
      "-rw-r--r--  1 jovyan users  44M Sep 21  2017 2005zipcode.zip\n",
      "-rw-r--r--  1 jovyan users  50M Sep 21  2017 2006zipcode.zip\n",
      "-rw-r--r--  1 jovyan users  91M Sep 21  2017 2007zipcode.zip\n",
      "-rw-r--r--  1 jovyan users  64M Dec  6  2013 2008zipcode.zip\n",
      "-rw-r--r--  1 jovyan users  51M Mar 11  2015 2009zipcode.zip\n",
      "-rw-r--r--  1 jovyan users  48M Mar 11  2015 2010zipcode.zip\n",
      "-rw-r--r--  1 jovyan users  11K Aug 11  2011 all_060_in_09.PCT7.csv.zip\n",
      "-rw-r--r--  1 jovyan users  20K Apr  9 14:30 asdf.ipynb\n",
      "-rw-r--r--  1 jovyan users 5.1K Apr  6 00:23 candy-data.csv\n",
      "-rw-r--r--  1 jovyan users 5.1K Apr  9 17:09 candy-data.csv.1\n",
      "drwxr-xr-x  8 jovyan users 4.0K Apr 11 05:24 .git\n",
      "-rw-r--r--  1 jovyan users 1.2K Apr  4 16:13 .gitignore\n",
      "drwxr-xr-x  2 jovyan users 4.0K Apr 11 04:37 .ipynb_checkpoints\n",
      "-rw-r--r--  1 jovyan users  12K Apr  4 16:13 LICENSE\n",
      "-rwxr--r--  1 jovyan users  290 Apr  9 14:25 nbclean.sh\n",
      "-rw-rw-r--  1 jovyan  1000 406K Apr  7 17:47 players.json\n",
      "-rw-r--r--  1 jovyan users   15 Apr 11 05:38 somefile.txt\n",
      "######## hello\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "echo \"######## hello\" > somefile.txt\n",
    "ls -alh\n",
    "cat somefile.txt\n",
    "rm somefile.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- [Who is Jovyan?](https://github.com/jupyter/docker-stacks/issues/358)\n",
    "- [\"In science fiction, a Jovian is an inhabitant of the planet Jupiter.\"](https://en.wikipedia.org/wiki/Jovian_%28fiction%29)\n",
    "\n",
    "### Other magic commands\n",
    "\n",
    "    - `%load`: load outside script\n",
    "    - `%store`: pass variables between notebooks\n",
    "    - [many more](https://www.datacamp.com/community/tutorials/tutorial-jupyter-notebook)\n",
    "\n",
    "## Exporting notebooks\n",
    "\n",
    "- Notebooks can be converted using [`nbconvert`](https://nbconvert.readthedocs.io/en/latest/)\n",
    "    - Slides\n",
    "    - Static web pages  \n",
    "        e.g. auto-updating reporting, blogs, etc\n",
    "\n",
    "- Shell script to automated execution  \n",
    "    `jupyter nbconvert --to html --execute mynotebook.ipynb`\n",
    "\n",
    "\n",
    "## Following Python Data Science Handbook notebooks\n",
    "\n",
    "- [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/) is written in Jupter notebook and [the author made the source code available on Github](https://github.com/jakevdp/PythonDataScienceHandbook)\n",
    "\n",
    "- You can follow the book in our course Jupterhub (file upload)\n",
    "\n",
    "- You can fork the repository to your Github and make changes\n",
    "\n",
    "# Bash Shell\n",
    "\n",
    "---\n",
    "\n",
    "* Bash is a text interface to the operating system (OS)\n",
    "\n",
    "* OS handles file operations, interfacing with network, etc\n",
    "\n",
    "* Bash allows you to execute system operations using text commands\n",
    "\n",
    "    ```bash\n",
    "    echo \"######## hello\" > somefile.txt ## prints string into file 'somefile.txt'\n",
    "    ls -alh                              ## list files in directory\n",
    "    cat somefile.txt                     ## print contents of 'somefile.txt'\n",
    "    rm somefile.txt                      ## remove 'somefile.txt'\n",
    "    ```\n",
    "* Super cool resource: [Explain Shell](https://explainshell.com/)\n",
    "\n",
    "* [Demo] Use git command to clone a repository on Github\n",
    "    \n",
    "* More on command line later    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}