Stochastic Gradient Descent (SGD Classifier).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stochastic Gradient Descent (SGD) is a simple yet very efficient approach to discriminative learning of linear classifiers \n",
    "\n",
    "SGD has been successfully applied to large-scale and sparse machine learning problems often encountered in text classification and natural language processing.\n",
    "\n",
    "* The advantages of Stochastic Gradient Descent are:\n",
    "\n",
    "     * Efficiency.\n",
    "     * Ease of implementation (lots of opportunities for code tuning).\n",
    "\n",
    "* The disadvantages of Stochastic Gradient Descent include:\n",
    "\n",
    "     * SGD requires a number of hyperparameters such as the regularization parameter and the number of iterations.\n",
    "     * SGD is sensitive to feature scaling.\n",
    "     \n",
    "     \n",
    "     \n",
    "The concrete loss function can be set via the loss parameter. SGDClassifier supports the following loss functions:\n",
    "* loss=\"hinge\": (soft-margin) linear Support Vector Machine,\n",
    "* loss=\"modified_huber\": smoothed hinge loss,\n",
    "* loss=\"log\": logistic regression,\n",
    "* and all regression losses below.\n",
    "\n",
    "The concrete penalty can be set via the penalty parameter. SGD supports the following penalties:\n",
    " * penalty=\"l2\": L2 norm penalty on coef_.\n",
    " * penalty=\"l1\": L1 norm penalty on coef_.\n",
    " * penalty=\"elasticnet\": Convex combination of L2 and L1; (1 - l1_ratio) * L2 + l1_ratio * L1.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The Algorithm:\n",
    "The hypothesis is this:\n",
    "<img src=\"https://github.com/arijitBhadraMP/MachineLearningAndAI/blob/master/1.png?raw=true\">\n",
    "where the parameter  x which minimizes h(x) is to be estimated. Each summand function  h{j} is typically associated with the  j-th observation in the data set (used for training).\n",
    "\n",
    "The cos function is this:\n",
    "<img src=\"https://github.com/arijitBhadraMP/MachineLearningAndAI/blob/master/2.png?raw=true\">\n",
    "\n",
    "<img src=\"https://github.com/arijitBhadraMP/MachineLearningAndAI/blob/master/3.png?raw=true\">\n",
    "\n",
    "This is gradient descent:\n",
    "<img src=\"https://github.com/arijitBhadraMP/MachineLearningAndAI/blob/master/rep.png?raw=true\">\n",
    "The parameters are initialized there\n",
    "<img src=\"https://github.com/arijitBhadraMP/MachineLearningAndAI/blob/master/5.png?raw=true\">\n",
    "\n",
    "Then when we run the program it grows and takes the parameters to the global minimum\n",
    "\n",
    "\n",
    "---\n",
    "When the value M increases the computation time also increases.\n",
    "\n",
    "#### Batch Gradient Descent:\n",
    "<img src=\"https://github.com/arijitBhadraMP/MachineLearningAndAI/blob/master/6.png?raw=true\">\n",
    "\n",
    "#### Stochastic Gradient Descent :\n",
    "\n",
    "<img src=\"https://github.com/arijitBhadraMP/MachineLearningAndAI/blob/master/7.png?raw=true\">\n",
    "\n",
    "\n",
    "#### What Stochastic Gradient Descent is doing?\n",
    "\n",
    "<img src=\"https://github.com/arijitBhadraMP/MachineLearningAndAI/blob/master/8.png?raw=true\">\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "<img src=\"https://github.com/arijitBhadraMP/MachineLearningAndAI/blob/master/9.png?raw=true\">\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import linear_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = np.array([[.0,.0], [.1,.1], [.2,.2], [.3,.3]])\n",
    "Y = np.array([0,1,2,3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0. ,  0. ],\n",
       "       [ 0.1,  0.1],\n",
       "       [ 0.2,  0.2],\n",
       "       [ 0.3,  0.3]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/arijitbhadra/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/stochastic_gradient.py:128: FutureWarning: max_iter and tol parameters have been added in <class 'sklearn.linear_model.stochastic_gradient.SGDClassifier'> in 0.19. If both are left unset, they default to max_iter=5 and tol=None. If tol is not None, max_iter defaults to max_iter=1000. From 0.21, default max_iter will be 1000, and default tol will be 1e-3.\n",
      "  \"and default tol will be 1e-3.\" % type(self), FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SGDClassifier(alpha=0.0001, average=False, class_weight=None, epsilon=0.1,\n",
       "       eta0=0.0, fit_intercept=True, l1_ratio=0.15,\n",
       "       learning_rate='optimal', loss='hinge', max_iter=None, n_iter=None,\n",
       "       n_jobs=1, penalty='l2', power_t=0.5, random_state=None,\n",
       "       shuffle=True, tol=None, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = linear_model.SGDClassifier()\n",
    "clf.fit(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\n"
     ]
    }
   ],
   "source": [
    "print(clf.predict([[1,.3]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}