softmax_classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Softmax Classifier笔记\n",
    "\n",
    "> 2018年8月10日， 罗周杨\n",
    "\n",
    "支持向量机(SVM)是两个最常用的分类器之一，另一个就是**Softmax classifier**，它和损失函数和SVM不一样。如果说逻辑回归分类器是二分类器，那么softmax就是多分类器。\n",
    "\n",
    "不同于SVM把输出$f(x_{i};W)$作为分数（未经校准并且可能难以解释），softmax的输出是一个**归一化概率(normalized class probabilities)**。\n",
    "\n",
    "Softmax使用**交叉熵(cross-entropy loss)** 代替了**hinge loss(即$\\max(0,-)$函数）**。\n",
    "\n",
    "**cross-entropy loss**公式如下：\n",
    "\n",
    "$$L_{i}=-log(\\frac{e^{f_{y_{i}}}} {\\sum_{j} e^{f_{j}}})$$\n",
    "\n",
    "或者写成：\n",
    "\n",
    "$$L_{i}=-f_{y_{i}}+log\\sum_{j}e^{f_{j}}$$\n",
    "\n",
    "其中$f_{j}$表示第$j$个元素的分数$f$。\n",
    "\n",
    "和Multiclass SVM classifier一样，对于整个数据集的损失定义为：\n",
    "\n",
    "\\begin{align}\n",
    "L&=\\frac{1}{N}\\sum_{j}L_{i}+\\lambda R(W) \\\\\n",
    " &=\\frac{1}{N}\\sum_{j}-log(\\frac{e^{f_{y_{i}}}} {\\sum_{j} e^{f_{j}}})+\\lambda R(W)\n",
    "\\end{align}\n",
    "\n",
    "\n",
    "## Softmax function\n",
    "\n",
    "Softmax function的定义如下：\n",
    "\n",
    "$$f_{j}(z)=\\frac{e^{z_{j}}}{\\sum_{k}e^{z_{k}}}$$\n",
    "\n",
    "它对于真正的分数向量$z$压成一个(0,1)区间的值，这些值的和为1。\n",
    "\n",
    "根据信息论，真实分布$p$和估计分布$q$之间的**交叉熵**定义为：\n",
    "\n",
    "$$ H(p,q)=-\\sum_{x}p(x)logq(x)$$\n",
    "\n",
    "因此，Softmax分类器就是在最小化估计的类别概率（$p=\\frac{e^{f_{y_{i}}}} {\\sum_{j} e^{f_{j}}}$）和真实分布之间的交叉熵。\n",
    "\n",
    "**疑问**：\n",
    "\n",
    "* **交叉熵和softmax函数的更具体的解释**。\n",
    "\n",
    "从概率论的角度，式子：\n",
    "\n",
    "$$\n",
    "P(y_{i}|x_{i};W)=\\frac{e^{f_{y_{i}}}} {\\sum_{j} e^{f_{j}}}\n",
    "$$\n",
    "\n",
    "可以解释成：正确类别y关于输入x和权重矩阵W的**概率**。\n",
    "\n",
    "**疑问**:\n",
    "\n",
    "* **概率论更加具体的解释**。\n",
    "\n",
    "\n",
    "## SVM和Softmax的区别：\n",
    "\n",
    "![](images/svm_vs_softmax.png)\n",
    "\n",
    "\n",
    "* Softmax提供的是每一个类别的\"概率”，打上冒号是因为，这个“概率”直接影响于正则化参数$\\lambda$的值。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}