10. Introduction to Modeling Libraries in Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "#### The library that you will use for developing models will depend on the application.\n",
    "#### Usually many statistical problems can be solved with simpler techniques. eg - ordinary least squares.\n",
    "#### But other problems may require more advanced machine learning methods.\n",
    "#### We will focus on the features that will be helpful while crossing back and forth in data wrangling and model fitting and scoring."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interfacing between pandas and Model Code\n",
    "#### A common workflow for model development is to use pandas for data loading and cleaning, then switching to modeling library to build the model.\n",
    "#### An important step in model development is 'feature engineering' i.e. data transformation or analytics that extracts information from raw dataset that will be useful in modeling context.\n",
    "#### We often use the data aggregation and GroupBy tools in this step.\n",
    "#### The point of contact between pandas and other analysis libraries is usually 'NumPy arrays'.\n",
    "#### To turn DataFrame into NumPy array, use the '.values' property. To convert back to a DataFrame, you can pass a 2-d ndarray with optional column names to 'DataFrame'.\n",
    "#### NOTE - '.values' is intended for homogenous data (eg - all numeric type). For heterogenous data, the result would be an ndarray of Python 'objects'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x0</th>\n",
       "      <th>x1</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>-1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>-0.01</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.25</td>\n",
       "      <td>3.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>-4.10</td>\n",
       "      <td>1.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x0    x1    y\n",
       "0   1  0.01 -1.5\n",
       "1   2 -0.01  0.0\n",
       "2   3  0.25  3.6\n",
       "3   4 -4.10  1.3\n",
       "4   5  0.00 -2.0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "data = pd.DataFrame({\n",
    "    'x0': [1, 2, 3, 4, 5],\n",
    "    'x1': [0.01, -0.01, 0.25, -4.1, 0.],\n",
    "    'y': [-1.5, 0., 3.6, 1.3, -2.]\n",
    "})\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['x0', 'x1', 'y'], dtype='object')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.  ,  0.01, -1.5 ],\n",
       "       [ 2.  , -0.01,  0.  ],\n",
       "       [ 3.  ,  0.25,  3.6 ],\n",
       "       [ 4.  , -4.1 ,  1.3 ],\n",
       "       [ 5.  ,  0.  , -2.  ]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>one</th>\n",
       "      <th>two</th>\n",
       "      <th>three</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>-1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>-0.01</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.25</td>\n",
       "      <td>3.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>-4.10</td>\n",
       "      <td>1.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   one   two  three\n",
       "0  1.0  0.01   -1.5\n",
       "1  2.0 -0.01    0.0\n",
       "2  3.0  0.25    3.6\n",
       "3  4.0 -4.10    1.3\n",
       "4  5.0  0.00   -2.0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = pd.DataFrame(data.values, columns=['one', 'two', 'three'])\n",
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x0</th>\n",
       "      <th>x1</th>\n",
       "      <th>y</th>\n",
       "      <th>strings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>-0.01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.25</td>\n",
       "      <td>3.6</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>-4.10</td>\n",
       "      <td>1.3</td>\n",
       "      <td>d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>e</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x0    x1    y strings\n",
       "0   1  0.01 -1.5       a\n",
       "1   2 -0.01  0.0       b\n",
       "2   3  0.25  3.6       c\n",
       "3   4 -4.10  1.3       d\n",
       "4   5  0.00 -2.0       e"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Heterogenous data\n",
    "df3 = data.copy()\n",
    "df3['strings'] = ['a', 'b', 'c', 'd', 'e']\n",
    "df3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0.01, -1.5, 'a'],\n",
       "       [2, -0.01, 0.0, 'b'],\n",
       "       [3, 0.25, 3.6, 'c'],\n",
       "       [4, -4.1, 1.3, 'd'],\n",
       "       [5, 0.0, -2.0, 'e']], dtype=object)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ndarray of object data\n",
    "df3.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Some libraries have naive support for pandas to do this work automatically i.e. converting to NumPy and attaching model parameter names.\n",
    "#### In some other scenarios, you may have to perform this 'metadata management' manually.\n",
    "#### One example is to replace 'category' column with 'dummy' variables. And then dropping the category column and join the dummay variables to the main dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x0</th>\n",
       "      <th>x1</th>\n",
       "      <th>y</th>\n",
       "      <th>category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>-0.01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.25</td>\n",
       "      <td>3.6</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>-4.10</td>\n",
       "      <td>1.3</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x0    x1    y category\n",
       "0   1  0.01 -1.5        a\n",
       "1   2 -0.01  0.0        b\n",
       "2   3  0.25  3.6        a\n",
       "3   4 -4.10  1.3        a\n",
       "4   5  0.00 -2.0        b"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['category'] = pd.Categorical(['a', 'b', 'a', 'a', 'b'],\n",
    "                                 categories=['a', 'b'])\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x0</th>\n",
       "      <th>x1</th>\n",
       "      <th>y</th>\n",
       "      <th>category_a</th>\n",
       "      <th>category_b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>-0.01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.25</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>-4.10</td>\n",
       "      <td>1.3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x0    x1    y  category_a  category_b\n",
       "0   1  0.01 -1.5           1           0\n",
       "1   2 -0.01  0.0           0           1\n",
       "2   3  0.25  3.6           1           0\n",
       "3   4 -4.10  1.3           1           0\n",
       "4   5  0.00 -2.0           0           1"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummies = pd.get_dummies(data.category, prefix='category')\n",
    "data_with_dummies = data.drop('category', axis=1).join(dummies)\n",
    "data_with_dummies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Model Descriptions with Patsy\n",
    "#### There are some nuances to using statistical models with datasets having dummy variables.\n",
    "#### A simpler and less error-prone way is to use 'Patsy', a Python library fro describing models (esp. linear models), using a small string-based 'formula syntax'.\n",
    "#### It is well supported for linear models in 'statsmodels'. It's 'formulas' are special string syntax. (eg - y ~ x0 + x1).\n",
    "#### The above syntax does not mean 'add x0 to x1', rather these are 'terms' in the 'design matrix' created for the model.\n",
    "#### The 'patsy.dmatrices' takes a fromula string with a dataset (DataFrame or dict of arrays) and produces design matrices for linear model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x0</th>\n",
       "      <th>x1</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>-1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>-0.01</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.25</td>\n",
       "      <td>3.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>-4.10</td>\n",
       "      <td>1.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x0    x1    y\n",
       "0   1  0.01 -1.5\n",
       "1   2 -0.01  0.0\n",
       "2   3  0.25  3.6\n",
       "3   4 -4.10  1.3\n",
       "4   5  0.00 -2.0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame({\n",
    "    'x0': [1, 2, 3, 4, 5],\n",
    "    'x1': [0.01, -0.01, 0.25, -4.1, 0.],\n",
    "    'y': [-1.5, 0., 3.6, 1.3, -2.]\n",
    "})\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import patsy\n",
    "\n",
    "y, X = patsy.dmatrices('y ~ x0 + x1', data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DesignMatrix with shape (5, 1)\n",
       "     y\n",
       "  -1.5\n",
       "   0.0\n",
       "   3.6\n",
       "   1.3\n",
       "  -2.0\n",
       "  Terms:\n",
       "    'y' (column 0)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DesignMatrix with shape (5, 3)\n",
       "  Intercept  x0     x1\n",
       "          1   1   0.01\n",
       "          1   2  -0.01\n",
       "          1   3   0.25\n",
       "          1   4  -4.10\n",
       "          1   5   0.00\n",
       "  Terms:\n",
       "    'Intercept' (column 0)\n",
       "    'x0' (column 1)\n",
       "    'x1' (column 2)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The DesignMatix instances are NumPy ndarrays with additional metadata.\n",
    "#### The 'Intercept' is a convention for linear models (eg - OLS {Ordinary Least Squares}). By default it is 1. You can suppress it by adding the term '+ 0' to the model.\n",
    "#### Patsy objects can be directly be directly passed into algorithms (eg - numpy.linalg.lstsq for OLS) which will perform the regression.\n",
    "#### The model metadata is retained in 'design_info' attribute which we can reattach the model column names to the fitted coefficients to get a Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.5],\n",
       "       [ 0. ],\n",
       "       [ 3.6],\n",
       "       [ 1.3],\n",
       "       [-2. ]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.asarray(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.  ,  1.  ,  0.01],\n",
       "       [ 1.  ,  2.  , -0.01],\n",
       "       [ 1.  ,  3.  ,  0.25],\n",
       "       [ 1.  ,  4.  , -4.1 ],\n",
       "       [ 1.  ,  5.  ,  0.  ]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.asarray(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DesignMatrix with shape (5, 2)\n",
       "  x0     x1\n",
       "   1   0.01\n",
       "   2  -0.01\n",
       "   3   0.25\n",
       "   4  -4.10\n",
       "   5   0.00\n",
       "  Terms:\n",
       "    'x0' (column 0)\n",
       "    'x1' (column 1)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "patsy.dmatrices('y ~ x0 + x1 + 0', data)[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\adity\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:1: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.\n",
      "To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    }
   ],
   "source": [
    "coef, resid, _, _ = np.linalg.lstsq(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.31290976],\n",
       "       [-0.07910564],\n",
       "       [-0.26546384]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coef"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Intercept    0.312910\n",
       "x0          -0.079106\n",
       "x1          -0.265464\n",
       "dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coef = pd.Series(coef.squeeze(), index=X.design_info.column_names)\n",
    "coef"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Transformation in Patsy Formulas\n",
    "#### We can mix Python code into Patsy formulas. While evaluating the formula, the library will find the functions you use in enclosing scope.\n",
    "#### Patsy has built-in functions for commonly used variable transformations (eg - standardizing {nean 0 and variance 1}, centering {subtracting by mean})."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DesignMatrix with shape (5, 3)\n",
       "  Intercept  x0  np.log(np.abs(x1) + 1)\n",
       "          1   1                 0.00995\n",
       "          1   2                 0.00995\n",
       "          1   3                 0.22314\n",
       "          1   4                 1.62924\n",
       "          1   5                 0.00000\n",
       "  Terms:\n",
       "    'Intercept' (column 0)\n",
       "    'x0' (column 1)\n",
       "    'np.log(np.abs(x1) + 1)' (column 2)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame({\n",
    "    'x0': [1, 2, 3, 4, 5],\n",
    "    'x1': [0.01, -0.01, 0.25, -4.1, 0.],\n",
    "    'y': [-1.5, 0., 3.6, 1.3, -2.]\n",
    "})\n",
    "\n",
    "y, X = patsy.dmatrices('y ~ x0 + np.log(np.abs(x1) + 1)', data)\n",
    "\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DesignMatrix with shape (5, 3)\n",
       "  Intercept  standardize(x0)  center(x1)\n",
       "          1         -1.41421        0.78\n",
       "          1         -0.70711        0.76\n",
       "          1          0.00000        1.02\n",
       "          1          0.70711       -3.33\n",
       "          1          1.41421        0.77\n",
       "  Terms:\n",
       "    'Intercept' (column 0)\n",
       "    'standardize(x0)' (column 1)\n",
       "    'center(x1)' (column 2)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y, X = patsy.dmatrices('y ~ standardize(x0) + center(x1)', data)\n",
    "\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A part of the modeling process is to fit model on a dataset and then evaluate it using another datset (a 'hold-out' portion or nes data observed later).\n",
    "#### When applying transformations (eg - standardize and center), you should be careful using the model for predictions.\n",
    "#### These are called 'stateful' transformations, as you may use statistics (eg - mean, standard deviation) of original dataset when transforming new dataset.\n",
    "#### The function 'patsy.build_design_matrices' can apply transformations to 'out-of-sample' data using saved information from original 'in-sample' dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[DesignMatrix with shape (4, 3)\n",
       "   Intercept  standardize(x0)  center(x1)\n",
       "           1          2.12132        3.87\n",
       "           1          2.82843        0.27\n",
       "           1          3.53553        0.77\n",
       "           1          4.24264        3.07\n",
       "   Terms:\n",
       "     'Intercept' (column 0)\n",
       "     'standardize(x0)' (column 1)\n",
       "     'center(x1)' (column 2)]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_data = pd.DataFrame({\n",
    "    'x0': [6, 7, 8, 9],\n",
    "    'x1': [3.1, -0.5, 0, 2.3],\n",
    "    'y': [1, 2, 3, 4]\n",
    "})\n",
    "\n",
    "new_X = patsy.build_design_matrices([X.design_info], new_data)\n",
    "new_X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The plus (+) symbol does not mean addition in context of Patsy formulas. To add columns from dataset, you must wrap them in special 'l' function.\n",
    "#### Patsy has several other built-in transforms in its 'patsy.builtins' module, which can be explored through its documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DesignMatrix with shape (5, 2)\n",
       "  Intercept  I(x0 + x1)\n",
       "          1        1.01\n",
       "          1        1.99\n",
       "          1        3.25\n",
       "          1       -0.10\n",
       "          1        5.00\n",
       "  Terms:\n",
       "    'Intercept' (column 0)\n",
       "    'I(x0 + x1)' (column 1)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y, X = patsy.dmatrices('y ~ I(x0 + x1)', data)\n",
    "\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Categorical Data and Patsy\n",
    "#### Non-numeric data can be transformed for model design matrix in different ways.\n",
    "#### When we use non-numeric terms in Patsy formula, they are converted to dummy variables by default.\n",
    "#### If you have included Intercept, one of the levels will be left out to avoid collinearity.\n",
    "#### If you omit the Intercept, columns for each category will be included in the design matrix. NOTE - Now it just shows a single column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DesignMatrix with shape (8, 2)\n",
       "  Intercept  key1[T.b]\n",
       "          1          0\n",
       "          1          0\n",
       "          1          1\n",
       "          1          1\n",
       "          1          0\n",
       "          1          1\n",
       "          1          0\n",
       "          1          1\n",
       "  Terms:\n",
       "    'Intercept' (column 0)\n",
       "    'key1' (column 1)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame({\n",
    "    'key1': ['a', 'a', 'b', 'b', 'a', 'b', 'a', 'b'],\n",
    "    'key2': [0, 1, 0, 1, 0, 1, 0, 0],\n",
    "    'v1': [1, 2, 3, 4, 5, 6, 7, 8],\n",
    "    'v2': [-1, 0, 2.5, -0.5, 4.0, -1.2, 0.2, -1.7]\n",
    "})\n",
    "\n",
    "y, X = patsy.dmatrices('v2 ~ key1', data)\n",
    "\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DesignMatrix with shape (8, 2)\n",
       "  key1[a]  key1[b]\n",
       "        1        0\n",
       "        1        0\n",
       "        0        1\n",
       "        0        1\n",
       "        1        0\n",
       "        0        1\n",
       "        1        0\n",
       "        0        1\n",
       "  Terms:\n",
       "    'key1' (columns 0:2)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Omitting Intercept just shows a single column now.\n",
    "y, X = patsy.dmatrices('v2 ~ key1 + 0', data)\n",
    "\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To convert a Numeric columns into categorica, use the 'C' function.\n",
    "#### Things can get complicated when using multiple categorical terms. This can happen because of using interaction terms of the form 'key1:key2'. (which can be used in ANOVA {Analysis of Variance} models).\n",
    "#### Patsy provides other ways to transform categorical data (eg - transformations for terms with ordering), which can be seen in it's online documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key1</th>\n",
       "      <th>key2</th>\n",
       "      <th>v1</th>\n",
       "      <th>v2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>zero</td>\n",
       "      <td>1</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a</td>\n",
       "      <td>one</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b</td>\n",
       "      <td>zero</td>\n",
       "      <td>3</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b</td>\n",
       "      <td>one</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a</td>\n",
       "      <td>zero</td>\n",
       "      <td>5</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>b</td>\n",
       "      <td>one</td>\n",
       "      <td>6</td>\n",
       "      <td>-1.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>a</td>\n",
       "      <td>zero</td>\n",
       "      <td>7</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>b</td>\n",
       "      <td>zero</td>\n",
       "      <td>8</td>\n",
       "      <td>-1.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  key1  key2  v1   v2\n",
       "0    a  zero   1 -1.0\n",
       "1    a   one   2  0.0\n",
       "2    b  zero   3  2.5\n",
       "3    b   one   4 -0.5\n",
       "4    a  zero   5  4.0\n",
       "5    b   one   6 -1.2\n",
       "6    a  zero   7  0.2\n",
       "7    b  zero   8 -1.7"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['key2'] = data['key2'].map({0: 'zero', 1: 'one'})\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DesignMatrix with shape (8, 3)\n",
       "  Intercept  key1[T.b]  key2[T.zero]\n",
       "          1          0             1\n",
       "          1          0             0\n",
       "          1          1             1\n",
       "          1          1             0\n",
       "          1          0             1\n",
       "          1          1             0\n",
       "          1          0             1\n",
       "          1          1             1\n",
       "  Terms:\n",
       "    'Intercept' (column 0)\n",
       "    'key1' (column 1)\n",
       "    'key2' (column 2)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y, X = patsy.dmatrices('v2 ~ key1 + key2', data)\n",
    "\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DesignMatrix with shape (8, 4)\n",
       "  Intercept  key1[T.b]  key2[T.zero]  key1[T.b]:key2[T.zero]\n",
       "          1          0             1                       0\n",
       "          1          0             0                       0\n",
       "          1          1             1                       1\n",
       "          1          1             0                       0\n",
       "          1          0             1                       0\n",
       "          1          1             0                       0\n",
       "          1          0             1                       0\n",
       "          1          1             1                       1\n",
       "  Terms:\n",
       "    'Intercept' (column 0)\n",
       "    'key1' (column 1)\n",
       "    'key2' (column 2)\n",
       "    'key1:key2' (column 3)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y, X = patsy.dmatrices('v2 ~ key1 + key2 + key1:key2', data)\n",
    "\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to statsmodel\n",
    "#### statsmodel is a Python library fro fitting many kinds of statistical models, performing statistical tests and data exploration and visualization.\n",
    "#### It has more 'classical' frequentist statistical methods. The Bayesian and machine learning models are in other libraries.\n",
    "#### The kinds of models in statsmodel are:\n",
    "####     1. Linear models, generalized linear models and robust linear models\n",
    "####     2. Linear mixed effects models\n",
    "####     3. Analysis of variance (ANOVA) methods\n",
    "####     4. Time series processes and state space models\n",
    "####     5. Generalized methods of moments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Estimating Linear Models\n",
    "#### There are several kinds of linear regression models in statsmodels, from basic (eg - ordinary least squares) to more complex (eg - iteratively reweighted least squares).\n",
    "#### Linear models have 2 different main interfaces - array-based and formula-based. Both are accessed through API module imports.\n",
    "#### As an example, we will create normally distributed data. The helper function 'dnorm' generates normally distributed data with given mean and variance.\n",
    "#### We will then generate a linear model from the data with known parameters 'beta'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "import statsmodels.formula.api as smf\n",
    "\n",
    "def dnorm(mean, variance, size=1):\n",
    "    if isinstance(size, int):\n",
    "        size = size,\n",
    "        return mean + np.sqrt(variance) * np.random.randn(*size)\n",
    "    \n",
    "np.random.seed(12345)\n",
    "N = 100\n",
    "X = np.c_[dnorm(0, 0.4, size=N),\n",
    "         dnorm(0, 0.6, size=N),\n",
    "         dnorm(0, 0.2, size=N)]\n",
    "eps = dnorm(0, 0.1, size=N)\n",
    "beta = [0.1, 0.3, 0.5]\n",
    "\n",
    "y = np.dot(X, beta) + eps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.12946849, -1.21275292,  0.50422488],\n",
       "       [ 0.30291036, -0.43574176, -0.25417986],\n",
       "       [-0.32852189, -0.02530153,  0.13835097],\n",
       "       [-0.35147471, -0.71960511, -0.25821463],\n",
       "       [ 1.2432688 , -0.37379916, -0.52262905]])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.42786349, -0.67348041, -0.09087764, -0.48949442, -0.12894109])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To add an intercept column to the existing matrix, we can use 'sm.add_constant' function.\n",
    "#### To use ordinary least squares linear regression, we will use 'sm.OLS'. The 'fit' method will return regression results object containing estimated model parameters and other diagnostics.\n",
    "#### The 'summary' method on the fit output 'results' can give a detailing diagnostic output of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.        , -0.12946849, -1.21275292,  0.50422488],\n",
       "       [ 1.        ,  0.30291036, -0.43574176, -0.25417986],\n",
       "       [ 1.        , -0.32852189, -0.02530153,  0.13835097],\n",
       "       [ 1.        , -0.35147471, -0.71960511, -0.25821463],\n",
       "       [ 1.        ,  1.2432688 , -0.37379916, -0.52262905]])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_model = sm.add_constant(X)\n",
    "X_model[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.17826108, 0.22303962, 0.50095093])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = sm.OLS(y, X)\n",
    "\n",
    "results = model.fit()\n",
    "\n",
    "results.params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            OLS Regression Results                            \n",
      "==============================================================================\n",
      "Dep. Variable:                      y   R-squared:                       0.430\n",
      "Model:                            OLS   Adj. R-squared:                  0.413\n",
      "Method:                 Least Squares   F-statistic:                     24.42\n",
      "Date:                Sat, 01 Sep 2018   Prob (F-statistic):           7.44e-12\n",
      "Time:                        17:49:02   Log-Likelihood:                -34.305\n",
      "No. Observations:                 100   AIC:                             74.61\n",
      "Df Residuals:                      97   BIC:                             82.42\n",
      "Df Model:                           3                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "==============================================================================\n",
      "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
      "------------------------------------------------------------------------------\n",
      "x1             0.1783      0.053      3.364      0.001       0.073       0.283\n",
      "x2             0.2230      0.046      4.818      0.000       0.131       0.315\n",
      "x3             0.5010      0.080      6.237      0.000       0.342       0.660\n",
      "==============================================================================\n",
      "Omnibus:                        4.662   Durbin-Watson:                   2.201\n",
      "Prob(Omnibus):                  0.097   Jarque-Bera (JB):                4.098\n",
      "Skew:                           0.481   Prob(JB):                        0.129\n",
      "Kurtosis:                       3.243   Cond. No.                         1.74\n",
      "==============================================================================\n",
      "\n",
      "Warnings:\n",
      "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n"
     ]
    }
   ],
   "source": [
    "print(results.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In above example, the parameters had generic names (x1, x2). Instead the parmeter names might be in a DataFrame.\n",
    "#### In such a scenario, we can use statsmodels formula API and Patsy formula strings.\n",
    "#### The result returned by statsmodels is a Series with the DataFrame column names attached. Also, we do not need to use 'add_constant' using formulas and pandas objects.\n",
    "#### For a new out-of-sample data, you can compute predicted values with the estimated model parameters.\n",
    "#### There are additional tools for analysis, diagnostics and visualizations and other kinds of linear models in statsmodels which can be explored in its documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col0</th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.129468</td>\n",
       "      <td>-1.212753</td>\n",
       "      <td>0.504225</td>\n",
       "      <td>0.427863</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.302910</td>\n",
       "      <td>-0.435742</td>\n",
       "      <td>-0.254180</td>\n",
       "      <td>-0.673480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.328522</td>\n",
       "      <td>-0.025302</td>\n",
       "      <td>0.138351</td>\n",
       "      <td>-0.090878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.351475</td>\n",
       "      <td>-0.719605</td>\n",
       "      <td>-0.258215</td>\n",
       "      <td>-0.489494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.243269</td>\n",
       "      <td>-0.373799</td>\n",
       "      <td>-0.522629</td>\n",
       "      <td>-0.128941</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       col0      col1      col2         y\n",
       "0 -0.129468 -1.212753  0.504225  0.427863\n",
       "1  0.302910 -0.435742 -0.254180 -0.673480\n",
       "2 -0.328522 -0.025302  0.138351 -0.090878\n",
       "3 -0.351475 -0.719605 -0.258215 -0.489494\n",
       "4  1.243269 -0.373799 -0.522629 -0.128941"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame(X, columns=['col0', 'col1', 'col2'])\n",
    "data['y'] = y\n",
    "data[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Intercept    0.033559\n",
       "col0         0.176149\n",
       "col1         0.224826\n",
       "col2         0.514808\n",
       "dtype: float64"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = smf.ols('y ~ col0 + col1 + col2', data=data).fit()\n",
    "\n",
    "results.params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Intercept    0.952188\n",
       "col0         3.319754\n",
       "col1         4.850730\n",
       "col2         6.303971\n",
       "dtype: float64"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.tvalues"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   -0.002327\n",
       "1   -0.141904\n",
       "2    0.041226\n",
       "3   -0.323070\n",
       "4   -0.100535\n",
       "dtype: float64"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.predict(data[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Estimating Time Series Process\n",
    "#### statsmodels also has a class of models for time series analysis. These include autogressive process, Kalman filtering and other space models, and multivariate autogressive models.\n",
    "#### As an example we will simulate a time series with autoregressive structure and noise with AR(2) structure (i.e. 2 lags) of parameters 0.8 and -0.4.\n",
    "#### You may not know the exact number of lagged terms when fitting AR model. So we can fit with some larger number of lags.\n",
    "#### The estimated parameters in results have intercepts first and estimates for the first 2 lags next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "init_x = 4\n",
    "\n",
    "import random\n",
    "values = [init_x, init_x]\n",
    "N = 1000\n",
    "\n",
    "b0 = 0.8\n",
    "b1 = -0.4\n",
    "noise = dnorm(0, 0.1, N)\n",
    "\n",
    "for i in range(N):\n",
    "    new_x = values[-1] * b0 + values[-2] * b1 + noise[i]\n",
    "    values.append(new_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAXLAGS = 5\n",
    "model = sm.tsa.AR(values)\n",
    "results = model.fit(MAXLAGS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.00616093,  0.78446347, -0.40847891, -0.01364148,  0.01496872,\n",
       "        0.01429462])"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to scikit-learn\n",
    "#### scikit-learn is one of the most widely used and trusted Python machine learning toolkits.\n",
    "#### It contains a broad selection of standard supervised and unsupervised ML methods with tools for model selection and evaluation, data transformation, data loading, and model persistence.\n",
    "#### These models can be used for classification, clustering, prediction and other common tasks.\n",
    "#### An example would be the passenger survival rates on the Titanic which is present in Kaggle. The first step is downloading test and training dataset using pandas.\n",
    "#### Libraries like statsmodel and scikit-learn cannot be fed missing data, so we need to look at columns to see if they contain missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  "
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = pd.read_csv('datasets/titanic/train.csv')\n",
    "test = pd.read_csv('datasets/titanic/test.csv')\n",
    "\n",
    "train[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      0\n",
       "Survived         0\n",
       "Pclass           0\n",
       "Name             0\n",
       "Sex              0\n",
       "Age            177\n",
       "SibSp            0\n",
       "Parch            0\n",
       "Ticket           0\n",
       "Fare             0\n",
       "Cabin          687\n",
       "Embarked         2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      0\n",
       "Pclass           0\n",
       "Name             0\n",
       "Sex              0\n",
       "Age             86\n",
       "SibSp            0\n",
       "Parch            0\n",
       "Ticket           0\n",
       "Fare             1\n",
       "Cabin          327\n",
       "Embarked         0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In statistics and ML, a typical task is to make predictions based on data features. eg - predict if a passenger would survive.\n",
    "#### A model is fitted on training dataset and evaluated on an out-of-sample testing dataset.\n",
    "#### Age can be used as a predictor here, but it has missing data. There are a number of ways to imputing missing data. But a simple one is fill the nulls with median.\n",
    "#### Next we will encode the 'Sex' column as categorical column 'IsFemale'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "impute_value = train['Age'].median()\n",
    "train['Age'] = train['Age'].fillna(impute_value)\n",
    "test['Age'] = test['Age'].fillna(impute_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "train['IsFemale'] = (train['Sex'] == 'female').astype(int)\n",
    "test['IsFemale'] = (test['Sex'] == 'female').astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We next decide on the variables to include in our input and output for the ML algorithm and create NumPy arrays.\n",
    "#### Now these steps are not enough for a 'good' ML model without proper feature engineering.\n",
    "#### We use the 'LogisticRegression' model from scikit-learn and create model instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 3.,  0., 22.],\n",
       "       [ 1.,  1., 38.],\n",
       "       [ 3.,  1., 26.],\n",
       "       [ 1.,  1., 35.],\n",
       "       [ 3.,  0., 35.]])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictors = ['Pclass', 'IsFemale', 'Age']\n",
    "X_train = train[predictors].values\n",
    "X_test = test[predictors].values\n",
    "y_train = train['Survived'].values\n",
    "\n",
    "X_train[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, 1, 0], dtype=int64)"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "model = LogisticRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Just like statsmodels, we can fit the model instance to the training data using the 'fit' method.\n",
    "#### We can then form predictions for test dataset using 'model.predict'. Using the actual values in the test dataset, you can compute accuracy percentage or some other 'error metric'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 1, 0, 1, 0, 1, 0], dtype=int64)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_predict = model.predict(X_test)\n",
    "y_predict[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n(y_true == y_predict).mean()\\n'"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# No actual values provided in test dataset, so using 'y_true' as proxy\n",
    "'''\n",
    "(y_true == y_predict).mean()\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Actually, there are many layers of complexity in model training. \n",
    "#### Many models have parameters that need to be tuned and there are techniques called 'cross-validation' used for parameter tuning to avoid overfitting on training data.\n",
    "#### This yields a better predictive performance or robustness on new data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cross-validation works by splitting training data to simulate out-of-sample prediction.\n",
    "#### Then, by using model accuracy score (eg - mean squared error), we can perform a grid search on model parameters i.e. predictors and predicted variables.\n",
    "#### Some models (eg - Logistic regression) have estimator class with built-ib cross-validation.\n",
    "#### eg - the 'LogisticRegressionCV' class can be used with a aprameter indicating how fine-grained the grid-search has to be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegressionCV(Cs=10, class_weight=None, cv=None, dual=False,\n",
       "           fit_intercept=True, intercept_scaling=1.0, max_iter=100,\n",
       "           multi_class='ovr', n_jobs=1, penalty='l2', random_state=None,\n",
       "           refit=True, scoring=None, solver='lbfgs', tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "model_cv = LogisticRegressionCV(10)\n",
    "\n",
    "model_cv.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To do the cross-validation manually, you can use the 'cross_val_score' helper function, which handles the data splitting process.\n",
    "#### The default scoring metric for cross-validation is model-dependant, but you can choose an explicit scoring function.\n",
    "#### Cross-validated models take longer to train, but often yield better model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.77232143, 0.80269058, 0.77027027, 0.78828829])"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "model = LogisticRegression(C=10)\n",
    "\n",
    "scores = cross_val_score(model, X_train, y_train, cv=4)\n",
    "scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}