06 - Week/6 - NumPy and Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical: NumPy and Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro to NumPy\n",
    "\n",
    "NumPy is a Python library for scientific computing. It provides high-performance multidimensional data structure called\n",
    "arrays. NumPy is usually imported like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Arrays\n",
    "\n",
    "A NumPy array is a tensor of values all of the same type. The number of dimensions is the rank of the array, the shape\n",
    "of an array is a tuple of integers giving the size of the array along each dimension.\n",
    "\n",
    "We can create a NumPy array (aka a vector) from Python lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1, 2, 3, 4])\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous code cell has created a NumPy array of rank 1. To check that this is a NumPy array we can use the\n",
    "function `type`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we are given an array and we need to know its shape, we can use its attribute `shape`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What shape returns is a tuple containing the length of each dimension. In this case, we have only one dimension of\n",
    "size 4.\n",
    "\n",
    "To access the values of a one-dimensional array we can do like we do when using Python lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now create a 2-dimensional array (aka a matrix in math):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2 = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])\n",
    "a2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we use shape to get the length of the dimensions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a matrix (with rank 2) with 2 rows and 4 columns.\n",
    "\n",
    "To access the individual values of this matrix we can do like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2[0, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will access the value positioned at the first row and first column. Try to access other locations until you get\n",
    "and error.\n",
    "\n",
    "NumPy also provides some handy functions to create commonly used tensors when developing scientific applications.\n",
    "To use these functions we just need to give a tuple as a parameter defining the size of each dimensions.\n",
    "\n",
    "For example, use zeros to create a tensor filled with zeros:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.zeros((4, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use ones to create a tensor filled with ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ones((4, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use random.random to create a tensor filled with random values $\\in [0,1]$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.random((4, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a matrix containing a given value, we can either use the method full and provide the given value,\n",
    "or create a tensor of ones and multiplying it by the given value, or create a tensor of zeros and sum the given value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.full((4, 4), 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ones((4, 4)) * 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.zeros((4, 4)) + 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two code cells above also demonstrate that in NumPy we can easily do arithmetic operations between tensors and\n",
    "scalars.\n",
    "\n",
    "To create a matrix with the diagonal full with ones (aka the identity matrix), we can use the method eye like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.eye(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Array Selection\n",
    "\n",
    "We can select a part of a tensor using the slicing operator of Python. However, in NumPy the range operator can be\n",
    "used with multi-dimensional tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, to select only the 2x2 matrix on the top right, we can do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a[:2, 2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to select the last row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a[2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to select the last two columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a[:,2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use a list of integers to extract only the coordinates we want, for example if we want to extract only the\n",
    "values of the 4 corners of the matrix `a`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a[[0, 0, 2, 2],[0, 3, 0, 3]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Arithmetic Operators with Arrays\n",
    "\n",
    "To sum a scalar to a matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.eye(4) \n",
    "a + 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To subtract a scalar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a - 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To multiply a scalar to a matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a * 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To divide by a scalar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a / 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To exponentiate by a value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(a + 1) ** 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also do matrix operations like the dot product like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 = np.array([4, 3])\n",
    "a2 = np.array([[1, 2],[3, 4]])\n",
    "\n",
    "a1 @ a2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can transpose a matrix like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison Operators with Arrays\n",
    "\n",
    "We can use the Python comparison operators with arrays. However, this will not return a Boolean value but an array o\n",
    "f Boolean values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1, 2, 3, 4])\n",
    "a == 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, greater or equal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a >= 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extended Datatypes\n",
    "\n",
    "NumPy is implemented in C. This allows us to use C data-types like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([1, 2, 3, 4], dtype=np.int32)\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can find details about C datatypes [here](https://docs.scipy.org/doc/numpy/user/basics.types.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 27\n",
    "\n",
    "Compute the following sequence when n is equal to 5:\n",
    "\n",
    "$\n",
    "a_0 = \n",
    "\\begin{bmatrix}\n",
    "1&0\\\\\n",
    "0&1\\\\\n",
    "\\end{bmatrix}\n",
    "$\n",
    "\n",
    "$\n",
    "a_n = \n",
    "\\begin{bmatrix}\n",
    "1\\\\\n",
    "2\\\\\n",
    "\\end{bmatrix} \\cdot \n",
    "\\begin{bmatrix}\n",
    "2&1\\\\\n",
    "\\end{bmatrix} -\n",
    "a_{n-1}$\n",
    "\n",
    "\n",
    "The result should be:\n",
    "\n",
    "$a_5 = \\begin{bmatrix}\n",
    "1&1\\\\\n",
    "4&1\\\\\n",
    "\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro to Pandas\n",
    "\n",
    "Pandas is an open source package providing high-performance easy-to-use data structures and data analysis tools for\n",
    "the Python programming language.\n",
    "\n",
    "Pandas is usually imported using the following import statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas defines two data structures called DataFrame and Series. \n",
    "\n",
    "### Read a CSV\n",
    "\n",
    "To read a CSV file we can use the read_csv method. This will create a DataFrame. To read a CSV you can pass to this\n",
    "method either a file path to a CSV file stored locally, or an URL to a CSV file which pandas will download and parse\n",
    "automatically.\n",
    "\n",
    "For this tutorial we will work with a famous dataset called IRIS. This dataset describes the features of 3 different\n",
    "species of flowers. Each row is a flower and each column is a different measurement made on the flowers with the\n",
    "exception of the last column which represents the species of the flower:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')\n",
    "iris"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the first 5 rows of the DataFrame by using head:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or tail:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like NumPy, we can get the size of the dimensions of the DataFrame using shape:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can analyze the content of the DataFrame using describe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the type of each column (Series) we can use dtypes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select Columns (Series)\n",
    "\n",
    "To access a column we can either use the name of the column as a method or use the squared brackets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.sepal_length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris['sepal_length']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a DataFrame back we need to put the name of the column in a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris[['sepal_length']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also select two columns using a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris[['sepal_length', 'sepal_width']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select the first row we need to use the iloc like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select Rows\n",
    "\n",
    "To select rows based on a condition we can use boolean operators. For example, to select all the flowers that have a\n",
    "sepal length greater or equal than 5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris[iris.sepal_length >= 5.0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing Values\n",
    "\n",
    "We can change the value of a column similarly to the way we access the DataFrame.\n",
    "\n",
    "For example to change all the values of a column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.sepal_length = 0\n",
    "iris"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To change the values of a row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.iloc[0] = [0, 0, 0, 0, 'ok']\n",
    "iris"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistical Functions\n",
    "\n",
    "We can use the method mean, std, mode, max, min, etc. To compute any basic statistics we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.sepal_width.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.sepal_width.mode()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.sepal_width.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting with Pandas\n",
    "\n",
    "Pandas uses matplotlib to plot series. For example we can plot a column using the plot method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.sepal_width.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot the histogram of the column like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.sepal_width.hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 28\n",
    "\n",
    "Reload the IRIS dataset and compute the standard error of each numerical column per flower species, and store them into\n",
    "the DataFrame.\n",
    "\n",
    "The standard error for a normal distribution is computed like this:\n",
    "\n",
    "$$\\mu \\pm 1.96 \\sqrt{\\frac{\\sigma^2}{n}}$$\n",
    "\n",
    "where, $\\mu$ is the mean, $\\sigma^2$ is the variance, 1.96 is the approximate value of the 97.5\n",
    "percentile point of the normal distribution, and $n$ is the sample size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 29\n",
    "\n",
    "Imagine you are given a NumPy matrix and you want to transform this matrix into a pandas DataFrame,\n",
    "how can you do this? (Use the Internet to answer this question).\n",
    "\n",
    "What about when you have a pandas DataFrame and you need a NumPy matrix?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}