MultivariateRegression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multivariate Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's grab a small little data set of Blue Book car values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_excel('http://cdn.sundog-soft.com/Udemy/DataScience/cars.xls')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Price</th>\n",
       "      <th>Mileage</th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Trim</th>\n",
       "      <th>Type</th>\n",
       "      <th>Cylinder</th>\n",
       "      <th>Liter</th>\n",
       "      <th>Doors</th>\n",
       "      <th>Cruise</th>\n",
       "      <th>Sound</th>\n",
       "      <th>Leather</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17314.103129</td>\n",
       "      <td>8221</td>\n",
       "      <td>Buick</td>\n",
       "      <td>Century</td>\n",
       "      <td>Sedan 4D</td>\n",
       "      <td>Sedan</td>\n",
       "      <td>6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>17542.036083</td>\n",
       "      <td>9135</td>\n",
       "      <td>Buick</td>\n",
       "      <td>Century</td>\n",
       "      <td>Sedan 4D</td>\n",
       "      <td>Sedan</td>\n",
       "      <td>6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>16218.847862</td>\n",
       "      <td>13196</td>\n",
       "      <td>Buick</td>\n",
       "      <td>Century</td>\n",
       "      <td>Sedan 4D</td>\n",
       "      <td>Sedan</td>\n",
       "      <td>6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16336.913140</td>\n",
       "      <td>16342</td>\n",
       "      <td>Buick</td>\n",
       "      <td>Century</td>\n",
       "      <td>Sedan 4D</td>\n",
       "      <td>Sedan</td>\n",
       "      <td>6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>16339.170324</td>\n",
       "      <td>19832</td>\n",
       "      <td>Buick</td>\n",
       "      <td>Century</td>\n",
       "      <td>Sedan 4D</td>\n",
       "      <td>Sedan</td>\n",
       "      <td>6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Price  Mileage   Make    Model      Trim   Type  Cylinder  Liter  \\\n",
       "0  17314.103129     8221  Buick  Century  Sedan 4D  Sedan         6    3.1   \n",
       "1  17542.036083     9135  Buick  Century  Sedan 4D  Sedan         6    3.1   \n",
       "2  16218.847862    13196  Buick  Century  Sedan 4D  Sedan         6    3.1   \n",
       "3  16336.913140    16342  Buick  Century  Sedan 4D  Sedan         6    3.1   \n",
       "4  16339.170324    19832  Buick  Century  Sedan 4D  Sedan         6    3.1   \n",
       "\n",
       "   Doors  Cruise  Sound  Leather  \n",
       "0      4       1      1        1  \n",
       "1      4       1      1        0  \n",
       "2      4       1      1        0  \n",
       "3      4       1      0        0  \n",
       "4      4       1      0        1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use pandas to split up this matrix into the feature vectors we're interested in, and the value we're trying to predict.\n",
    "\n",
    "Note how we use pandas.Categorical to convert textual category data (model name) into an ordinal number that we can work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>          <td>Price</td>      <th>  R-squared:         </th> <td>   0.042</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.038</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   11.57</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Tue, 26 Jan 2016</td> <th>  Prob (F-statistic):</th> <td>1.98e-07</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>12:18:05</td>     <th>  Log-Likelihood:    </th> <td> -8519.1</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>   804</td>      <th>  AIC:               </th> <td>1.705e+04</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>   800</td>      <th>  BIC:               </th> <td>1.706e+04</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     3</td>      <th>                     </th>     <td> </td>    \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>    \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th> <th>[95.0% Conf. Int.]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>const</th>     <td> 3.125e+04</td> <td> 1809.549</td> <td>   17.272</td> <td> 0.000</td> <td> 2.77e+04  3.48e+04</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Mileage</th>   <td>   -0.1765</td> <td>    0.042</td> <td>   -4.227</td> <td> 0.000</td> <td>   -0.259    -0.095</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model_ord</th> <td>  -39.0387</td> <td>   39.326</td> <td>   -0.993</td> <td> 0.321</td> <td> -116.234    38.157</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Doors</th>     <td>-1652.9303</td> <td>  402.649</td> <td>   -4.105</td> <td> 0.000</td> <td>-2443.303  -862.558</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>206.410</td> <th>  Durbin-Watson:     </th> <td>   0.080</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.000</td>  <th>  Jarque-Bera (JB):  </th> <td> 470.872</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td> 1.379</td>  <th>  Prob(JB):          </th> <td>5.64e-103</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 5.541</td>  <th>  Cond. No.          </th> <td>1.15e+05</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                  Price   R-squared:                       0.042\n",
       "Model:                            OLS   Adj. R-squared:                  0.038\n",
       "Method:                 Least Squares   F-statistic:                     11.57\n",
       "Date:                Tue, 26 Jan 2016   Prob (F-statistic):           1.98e-07\n",
       "Time:                        12:18:05   Log-Likelihood:                -8519.1\n",
       "No. Observations:                 804   AIC:                         1.705e+04\n",
       "Df Residuals:                     800   BIC:                         1.706e+04\n",
       "Df Model:                           3                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "==============================================================================\n",
       "                 coef    std err          t      P>|t|      [95.0% Conf. Int.]\n",
       "------------------------------------------------------------------------------\n",
       "const       3.125e+04   1809.549     17.272      0.000      2.77e+04  3.48e+04\n",
       "Mileage       -0.1765      0.042     -4.227      0.000        -0.259    -0.095\n",
       "Model_ord    -39.0387     39.326     -0.993      0.321      -116.234    38.157\n",
       "Doors      -1652.9303    402.649     -4.105      0.000     -2443.303  -862.558\n",
       "==============================================================================\n",
       "Omnibus:                      206.410   Durbin-Watson:                   0.080\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):              470.872\n",
       "Skew:                           1.379   Prob(JB):                    5.64e-103\n",
       "Kurtosis:                       5.541   Cond. No.                     1.15e+05\n",
       "==============================================================================\n",
       "\n",
       "Warnings:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "[2] The condition number is large, 1.15e+05. This might indicate that there are\n",
       "strong multicollinearity or other numerical problems.\n",
       "\"\"\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "df['Model_ord'] = pd.Categorical(df.Model).codes\n",
    "X = df[['Mileage', 'Model_ord', 'Doors']]\n",
    "y = df[['Price']]\n",
    "\n",
    "X1 = sm.add_constant(X)\n",
    "est = sm.OLS(y, X1).fit()\n",
    "\n",
    "est.summary()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "The table of coefficients above gives us the values to plug into an equation of form:\n",
    "    B0 + B1 * Mileage + B2 * model_ord + B3 * doors\n",
    "    \n",
    "But in this example, it's pretty clear that mileage is more important than anything based on the std err's.\n",
    "\n",
    "Could we have figured that out earlier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Doors</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>23807.135520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20580.670749</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Price\n",
       "Doors              \n",
       "2      23807.135520\n",
       "4      20580.670749"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.groupby(df.Doors).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Surprisingly, more doors does not mean a higher price! So it's not surprising that it's pretty useless as a predictor here. This is a very small data set however, so we can't really read much meaning into it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mess around with the fake input data, and see if you can create a measurable influence of number of doors on price. Have some fun with it - why stop at 4 doors?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}