lab-06_1_softmax_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 6-1: Softmax Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: Seungjae Lee (이승재)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    We use elemental PyTorch to implement linear regression here. However, in most actual applications, abstractions such as <code>nn.Module</code> or <code>nn.Linear</code> are used.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x105eccfb0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For reproducibility\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert numbers to probabilities with softmax."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ P(class=i) = \\frac{e^i}{\\sum e^i} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "z = torch.FloatTensor([1, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch has a `softmax` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0900, 0.2447, 0.6652])\n"
     ]
    }
   ],
   "source": [
    "hypothesis = F.softmax(z, dim=0)\n",
    "print(hypothesis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since they are probabilities, they should add up to 1. Let's do a sanity check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hypothesis.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross Entropy Loss (Low-level)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For multi-class classification, we use the cross entropy loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ L = \\frac{1}{N} \\sum - y \\log(\\hat{y}) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $\\hat{y}$ is the predicted probability and $y$ is the correct probability (0 or 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.2645, 0.1639, 0.1855, 0.2585, 0.1277],\n",
      "        [0.2430, 0.1624, 0.2322, 0.1930, 0.1694],\n",
      "        [0.2226, 0.1986, 0.2326, 0.1594, 0.1868]], grad_fn=<SoftmaxBackward>)\n"
     ]
    }
   ],
   "source": [
    "z = torch.rand(3, 5, requires_grad=True)\n",
    "hypothesis = F.softmax(z, dim=1)\n",
    "print(hypothesis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0, 2, 1])\n"
     ]
    }
   ],
   "source": [
    "y = torch.randint(5, (3,)).long()\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 0., 0., 0., 0.],\n",
       "        [0., 0., 1., 0., 0.],\n",
       "        [0., 1., 0., 0., 0.]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_one_hot = torch.zeros_like(hypothesis)\n",
    "y_one_hot.scatter_(1, y.unsqueeze(1), 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.4689, grad_fn=<MeanBackward1>)\n"
     ]
    }
   ],
   "source": [
    "cost = (y_one_hot * -torch.log(hypothesis)).sum(dim=1).mean()\n",
    "print(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross-entropy Loss with `torch.nn.functional`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch has `F.log_softmax()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.3301, -1.8084, -1.6846, -1.3530, -2.0584],\n",
       "        [-1.4147, -1.8174, -1.4602, -1.6450, -1.7758],\n",
       "        [-1.5025, -1.6165, -1.4586, -1.8360, -1.6776]], grad_fn=<LogBackward>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Low level\n",
    "torch.log(F.softmax(z, dim=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.3301, -1.8084, -1.6846, -1.3530, -2.0584],\n",
       "        [-1.4147, -1.8174, -1.4602, -1.6450, -1.7758],\n",
       "        [-1.5025, -1.6165, -1.4586, -1.8360, -1.6776]],\n",
       "       grad_fn=<LogSoftmaxBackward>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# High level\n",
    "F.log_softmax(z, dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch also has `F.nll_loss()` function that computes the negative loss likelihood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.4689, grad_fn=<MeanBackward1>)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Low level\n",
    "(y_one_hot * -torch.log(F.softmax(z, dim=1))).sum(dim=1).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.4689, grad_fn=<NllLossBackward>)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# High level\n",
    "F.nll_loss(F.log_softmax(z, dim=1), y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch also has `F.cross_entropy` that combines `F.log_softmax()` and `F.nll_loss()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.4689, grad_fn=<NllLossBackward>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "F.cross_entropy(z, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with Low-level Cross Entropy Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = [[1, 2, 1, 1],\n",
    "           [2, 1, 3, 2],\n",
    "           [3, 1, 3, 4],\n",
    "           [4, 1, 5, 5],\n",
    "           [1, 7, 5, 5],\n",
    "           [1, 2, 5, 6],\n",
    "           [1, 6, 6, 6],\n",
    "           [1, 7, 7, 7]]\n",
    "y_train = [2, 2, 2, 1, 1, 1, 0, 0]\n",
    "x_train = torch.FloatTensor(x_train)\n",
    "y_train = torch.LongTensor(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 1.098612\n",
      "Epoch  100/1000 Cost: 0.901535\n",
      "Epoch  200/1000 Cost: 0.839114\n",
      "Epoch  300/1000 Cost: 0.807826\n",
      "Epoch  400/1000 Cost: 0.788472\n",
      "Epoch  500/1000 Cost: 0.774822\n",
      "Epoch  600/1000 Cost: 0.764449\n",
      "Epoch  700/1000 Cost: 0.756191\n",
      "Epoch  800/1000 Cost: 0.749398\n",
      "Epoch  900/1000 Cost: 0.743671\n",
      "Epoch 1000/1000 Cost: 0.738749\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((4, 3), requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W, b], lr=0.1)\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산 (1)\n",
    "    hypothesis = F.softmax(x_train.matmul(W) + b, dim=1) # or .mm or @\n",
    "    y_one_hot = torch.zeros_like(hypothesis)\n",
    "    y_one_hot.scatter_(1, y_train.unsqueeze(1), 1)\n",
    "    cost = (y_one_hot * -torch.log(F.softmax(hypothesis, dim=1))).sum(dim=1).mean()\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with `F.cross_entropy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 1.098612\n",
      "Epoch  100/1000 Cost: 0.761050\n",
      "Epoch  200/1000 Cost: 0.689991\n",
      "Epoch  300/1000 Cost: 0.643229\n",
      "Epoch  400/1000 Cost: 0.604117\n",
      "Epoch  500/1000 Cost: 0.568255\n",
      "Epoch  600/1000 Cost: 0.533922\n",
      "Epoch  700/1000 Cost: 0.500291\n",
      "Epoch  800/1000 Cost: 0.466908\n",
      "Epoch  900/1000 Cost: 0.433507\n",
      "Epoch 1000/1000 Cost: 0.399962\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((4, 3), requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W, b], lr=0.1)\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산 (2)\n",
    "    z = x_train.matmul(W) + b # or .mm or @\n",
    "    cost = F.cross_entropy(z, y_train)\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## High-level Implementation with `nn.Module`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SoftmaxClassifierModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear = nn.Linear(4, 3) # Output이 3!\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.linear(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = SoftmaxClassifierModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try another new dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 1.849513\n",
      "Epoch  100/1000 Cost: 0.689894\n",
      "Epoch  200/1000 Cost: 0.609259\n",
      "Epoch  300/1000 Cost: 0.551218\n",
      "Epoch  400/1000 Cost: 0.500141\n",
      "Epoch  500/1000 Cost: 0.451947\n",
      "Epoch  600/1000 Cost: 0.405051\n",
      "Epoch  700/1000 Cost: 0.358733\n",
      "Epoch  800/1000 Cost: 0.312912\n",
      "Epoch  900/1000 Cost: 0.269521\n",
      "Epoch 1000/1000 Cost: 0.241922\n"
     ]
    }
   ],
   "source": [
    "# optimizer 설정\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.1)\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # H(x) 계산\n",
    "    prediction = model(x_train)\n",
    "\n",
    "    # cost 계산\n",
    "    cost = F.cross_entropy(prediction, y_train)\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 20번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}