1_numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Numpy\n",
    "* The core library for manipulating and cleaning data in python is called pandas.\n",
    "* The core library underneath pandas, is numpy (numerical python)\n",
    "* Numpy is all about n dimensional arrays and doing things quickly\n",
    "* We won't talk about numpy much beyond today, but it's worth a quick consideration..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3]\n",
      "<class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "a = np.array([1, 2, 3])\n",
    "print(a)\n",
    "print(type(a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "# Things we can do with an ndarray include:\n",
    "print(a.ndim) #number of dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3]\n",
      " [4 5 6]]\n",
      "(2, 3)\n"
     ]
    }
   ],
   "source": [
    "# Lets create something multidimensional\n",
    "b = np.array([[1,2,3],[4,5,6]])\n",
    "print(b)\n",
    "print(b.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What is this ndarray filled with?\n",
    "b.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What is this reallly?\n",
    "type(b[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "float64\n",
      "[2.2 5.  1.1]\n"
     ]
    }
   ],
   "source": [
    "# And some automatic typecasting is available\n",
    "c = np.array([2.2, 5, 1.1])\n",
    "print(c.dtype)\n",
    "print(c)\n",
    "#l=[2.2, 5, 1.1]\n",
    "#print(type(l[0]))\n",
    "#print(type(l[1]))\n",
    "#print(type(l[2]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0.],\n",
       "       [0., 0., 0.]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You'll see this code a lot in examples\n",
    "np.zeros((2,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.27035481, 0.51848331, 0.41234737],\n",
       "       [0.1241825 , 0.54737886, 0.01375916]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.rand(2,3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42,\n",
       "       44, 46, 48])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.arange(10, 50, 2) # just like range! ten to fifty by twos!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.        , 0.14285714, 0.28571429, 0.42857143, 0.57142857,\n",
       "       0.71428571, 0.85714286, 1.        , 1.14285714, 1.28571429,\n",
       "       1.42857143, 1.57142857, 1.71428571, 1.85714286, 2.        ])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# last one\n",
    "np.linspace( 0, 2, 15 ) # 15 numbers from 0 (inclusive) to 2 (inclusive)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Array Operations\n",
    "* We can do many things on arrays, such as mathematical manipulation (addition, subtraction, square, exponents) as well as use boolean arrays, which are binary values. \n",
    "* We can also do matrix manipulation such as product, transpose, inverse, and so forth.\n",
    "* Key difference, arithmetic operations are **element wise**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 9, 18, 27, 36])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.array([10,20,30,40])\n",
    "b = np.array([1, 2, 3,4])\n",
    "c = a-b\n",
    "c # this is the same as `display(c)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Real word example\n",
    "* My wife is an ultra-🇨🇦. We're talking syrup chugging lumber jack who knows the words to *God Save the Queen*\n",
    "* I'm kinda trying to fit in around here (poorly!)\n",
    "* We bought a house full of Amazon Alexa's..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* And they all talk in celcius. And she refuses to change. But...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* I got to the thermostat first 😏\n",
    "\n",
    "![](datasets/mwahaha.gif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-17.77777778, -23.33333333, -20.55555556, -26.11111111,\n",
       "       -17.77777778])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "farenheit = np.array([0,-10,-5,-15,0])\n",
    "\n",
    "# And the formula for conversion is ((°F − 32) × 5/9 = °C)\n",
    "celcius = (farenheit - 32) * (5/9)\n",
    "celcius"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* What's happening underneath is so beautiful. It's called *broadcasting*, or *vectorization*.\n",
    "* Each item in the ndarray can be operated on individually - there is no need to consider other operations\n",
    "* I believe this is a data science **threshold concept**, and we should call it out:\n",
    "\n",
    "*The fundamental idea behind array programming is that operations apply at once to an entire set of values. This makes it a high-level programming model as it allows the programmer to think and operate on whole aggregates of data, without having to resort to explicit loops of individual scalar operations.* (wikipedia)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* Vectorization allows for:\n",
    "1. Massive parallelization and thus efficiency\n",
    "2. Increased readability\n",
    "3. Added flexibility\n",
    "4. Increased code quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.78 µs ± 31.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit farenheit = np.linspace( -10, 20, 1000 )\n",
    "# Solve this the numpy way, vectorized!\n",
    "celcius = (farenheit - 32) * (5/9)\n",
    "# How could we solve this iteratively?\n",
    "#np.linspace( -10, 20, 1000 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "515 µs ± 9.14 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit farenheit = np.linspace( -10, 20, 1000 )\n",
    "c=[]\n",
    "for i in farenheit:\n",
    "    c.append((i-32)*(5/9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# How else could we solve this iteratively?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "468 µs ± 4.24 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit farenheit = np.linspace( -10, 20, 1000 )\n",
    "[(temp-32)*(5/9) for temp in farenheit]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# One more thought on how could we solve this iteratively?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "552 µs ± 2.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit farenheit = np.linspace( -10, 20, 1000 )\n",
    "for i in range(len(farenheit)):\n",
    "    farenheit[i]=((farenheit[i]-32)*(5/9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Any other ideas on solving this?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "170 ns ± 0.415 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit farenheit = np.linspace( -10, 20, 1000 )\n",
    "celcius=map(lambda temp: (temp-32)*(5/9), farenheit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# wth\n",
    "* How do we better understand this? What's going on underneath?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-17.77777777777778\n",
      "-23.333333333333336\n",
      "-20.555555555555557\n",
      "-26.11111111111111\n",
      "-17.77777777777778\n"
     ]
    }
   ],
   "source": [
    "celcius=map(lambda temp: (temp-32)*(5/9), farenheit)\n",
    "for i in celcius:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Final thoughts\n",
    "1. Vectorization is the process of applying array programming techniques to data. When you vectorize an operation, function(s) are broadcast across elements in an array which allows for parallelization of operations.\n",
    "2. Vectorization is powerful, and it's thinking in a vectorized way which is really important here. This aligns well with functional programming methods, and is a key to being an effective data scientist. We live in a parallel world: #loopsaredead\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Boolean masking\n",
    "* This is a **critical concept** (and not difficult!) in this course.\n",
    "* This will impact how you look at data and understand how queries work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* A Boolean mask is analagous to a bitwise mask!\n",
    "* Has everyone used bitwise masking before?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Ok, it's really simple. You take a range of values which are `True (1)` or `False (0)` and you either AND them or OR them with another range of values which are `True` or `False`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](./datasets/bitmask.png)\n",
    "* shamelessly stolen from https://stackoverflow.com/questions/28282869/shift-masked-bits-to-the-lsb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 1 1 0 0 0 1 1 1 0]\n",
      "[1 0 0 0 0 0 1 0 1 1]\n",
      "[1 0 0 0 0 0 1 0 1 0]\n"
     ]
    }
   ],
   "source": [
    "#bitwise masking\n",
    "a=np.random.randint(2, size=10)\n",
    "b=np.random.randint(2, size=10)\n",
    "print(a)\n",
    "print(b)\n",
    "print(np.bitwise_and(a,b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[False, True, True, False, False, True, False, False, False, False]\n",
      "[True, False, False, False, True, False, True, False, False, False]\n",
      "[True, False, True, True, False, False, False, False, False, True]\n"
     ]
    }
   ],
   "source": [
    "#boolean masking\n",
    "import random\n",
    "a=[random.choice([True,False]) for x in range(0,10)]\n",
    "b=[random.choice([True,False]) for x in range(0,10)]\n",
    "c=[random.choice([True,False]) for x in range(0,10)]\n",
    "\n",
    "print(a)\n",
    "print(b)\n",
    "print((a and b) and c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Why are we doing this?\n",
    "* It's really common to take an array of data and mask it to reveal a result.\n",
    "* This works hand in hand with broadcasting! This is a highly parallelizable result!\n",
    "* We can broadcast individuals values with comparison operators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "a=np.random.randint(5,size=10)\n",
    "a > 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "a[a>3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](https://ibin.co/4v17tWp4dxMC.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Indexing operator\n",
    "* The indexing operator in numpy, and pandas, is incredibly overloaded. You can use it to\n",
    "  * get a single item out of the array, e.g. `a[0]`\n",
    "  * slice a range out of the array, e.g. `a[1:4]`\n",
    "  * apply a boolean mask to an array, e.g. `a[True, False, True]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, 1, 1, 0, 0, 1, 1, 0])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a=np.random.randint(2, size=10)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False,  True,  True,  True,  True, False, False,  True,  True,\n",
       "       False])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask=a>0.5\n",
    "mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#a[mask]\n",
    "a[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You and boolean masking with numpy (and thus pandas!) will become good friends. :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Extended Topic #1\n",
    "# Just for fun, here's how to impl the indexing operator yourself\n",
    "class HomemadePsychologist:\n",
    "    def __getitem__(self, key):\n",
    "        print(\"\"\"That's a good question, what do you think about the question '{}' Why do you think you are wondering about that?\"\"\".format(key))\n",
    "\n",
    "psych = HomemadePsychologist()\n",
    "psych[\"Are dogs fun?\"]"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}