Autoencoders/DBN-Review.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Belief Network\n",
    "\n",
    "One problem with traditional multilayer perceptrons/artificial neural networks is that backpropagation can often lead to “local minima”. This is when your “error surface” contains multiple grooves and you fall into a groove that is not lowest possible groove as you perform gradient descent.\n",
    "\n",
    "__Deep belief networks__ solve this problem by using an extra step called __pre-training__. Pre-training is done before backpropagation and can lead to an error rate not far from optimal. This puts us in the “neighborhood” of the final solution. Then we use backpropagation to slowly reduce the error rate from there.\n",
    "\n",
    "DBNs can be divided in two major parts. The first one are multiple layers of Restricted Boltzmann Machines (RBMs) to pre-train our network. The second one is a feed-forward backpropagation network, that will further refine the results from the RBM stack.\n",
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/15y15xs7w72eer0on3gbi8zu6835imru.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin by importing the necessary libraries and utilities functions to implement a Deep Belief Network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import math for calculations.\n",
    "import math\n",
    "# Tensorflow library. Used to implement machine learning models.\n",
    "import tensorflow as tf\n",
    "# Numpy contains helpful functions for efficient mathematical calculations.\n",
    "import numpy as np\n",
    "# Image library for image manipulation.\n",
    "from PIL import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constructing the Layers of RBMs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, let's detail Restricted Boltzmann Machines.\n",
    "\n",
    "#### What are Restricted Boltzmann Machines?\n",
    "RBMs are shallow neural nets that learn to reconstruct data by themselves in an unsupervised fashion.\n",
    "\n",
    "#### How it works?\n",
    "Simply, RBM takes the inputs and translates them to a set of numbers that represents them. Then, these numbers can be translated back to reconstruct the inputs. Through several forward and backward passes, the RBM will be trained, and a trained RBM can reveal which features are the most important ones when detecting patterns.   \n",
    "\n",
    "#### Why are RBMs important?\n",
    "It can automatically extract __meaningful__ features from a given input.\n",
    "\n",
    "#### What's the RBM's structure?\n",
    "It only possesses two layers. A visible input layer and a hidden layer where the features are learned.\n",
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/7th91vjz32jhslacdym7ll3udq2zixjb.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To implement DBNs in TensorFlow, we will implement a class for the Restricted Boltzmann Machines (RBMs). The class below implements an intuitive way of creating and using RBM's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Class that defines the behavior of the RBM.\n",
    "class RBM(object):\n",
    "    \n",
    "    def __init__(self, input_size, output_size):\n",
    "        # Defining the hyperparameters.\n",
    "        self._input_size = input_size  # Size of input.\n",
    "        self._output_size = output_size  # Size of output.\n",
    "        self.epochs = 5  # Amount of training iterations.\n",
    "        self.learning_rate = 1.0  # The step used in gradient descent.\n",
    "        self.batchsize = 100  # The size of how much data will be used for training per sub iteration.\n",
    "        \n",
    "        # Initializing weights and biases as matrices full of zeroes.\n",
    "        self.w = np.zeros([input_size, output_size], np.float32)  # Creates and initializes the weights with 0.\n",
    "        self.hb = np.zeros([output_size], np.float32)  # Creates and initializes the hidden biases with 0.\n",
    "        self.vb = np.zeros([input_size], np.float32)  # Creates and initializes the visible biases with 0.\n",
    "\n",
    "\n",
    "    # Fits the result from the weighted visible layer plus the bias into a sigmoid curve.\n",
    "    def prob_h_given_v(self, visible, w, hb):\n",
    "        # Sigmoid.\n",
    "        return tf.nn.sigmoid(tf.matmul(visible, w) + hb)\n",
    "\n",
    "    # Fits the result from the weighted hidden layer plus the bias into a sigmoid curve.\n",
    "    def prob_v_given_h(self, hidden, w, vb):\n",
    "        return tf.nn.sigmoid(tf.matmul(hidden, tf.transpose(w)) + vb)\n",
    "    \n",
    "    # Generate the sample probability.\n",
    "    def sample_prob(self, probs):\n",
    "        return tf.nn.relu(tf.sign(probs - tf.random_uniform(tf.shape(probs))))\n",
    "\n",
    "    # Training method for the model.\n",
    "    def train(self, X):\n",
    "        # Create the placeholders for the parameters.\n",
    "        _w = tf.placeholder(\"float\", [self._input_size, self._output_size])\n",
    "        _hb = tf.placeholder(\"float\", [self._output_size])\n",
    "        _vb = tf.placeholder(\"float\", [self._input_size])\n",
    "        \n",
    "        prv_w = np.zeros([self._input_size, self._output_size], np.float32)  # Creates and initializes the weights with 0.\n",
    "        prv_hb = np.zeros([self._output_size], np.float32)  # Creates and initializes the hidden biases with 0.\n",
    "        prv_vb = np.zeros([self._input_size], np.float32)  # Creates and initializes the visible biases with 0.\n",
    "\n",
    "        cur_w = np.zeros([self._input_size, self._output_size], np.float32)\n",
    "        cur_hb = np.zeros([self._output_size], np.float32)\n",
    "        cur_vb = np.zeros([self._input_size], np.float32)\n",
    "        v0 = tf.placeholder(\"float\", [None, self._input_size])\n",
    "        \n",
    "        # Initialize with sample probabilities.\n",
    "        h0 = self.sample_prob(self.prob_h_given_v(v0, _w, _hb))\n",
    "        v1 = self.sample_prob(self.prob_v_given_h(h0, _w, _vb))\n",
    "        h1 = self.prob_h_given_v(v1, _w, _hb)\n",
    "        \n",
    "        # Create the Gradients.\n",
    "        positive_grad = tf.matmul(tf.transpose(v0), h0)\n",
    "        negative_grad = tf.matmul(tf.transpose(v1), h1)\n",
    "        \n",
    "        # Update learning rates for the layers.\n",
    "        update_w = _w + self.learning_rate * (positive_grad - negative_grad) / tf.to_float(tf.shape(v0)[0])\n",
    "        update_vb = _vb +  self.learning_rate * tf.reduce_mean(v0 - v1, 0)\n",
    "        update_hb = _hb +  self.learning_rate * tf.reduce_mean(h0 - h1, 0)\n",
    "        \n",
    "        # Find the error rate.\n",
    "        err = tf.reduce_mean(tf.square(v0 - v1))\n",
    "        \n",
    "        # Training loop.\n",
    "        with tf.Session() as sess:\n",
    "            sess.run(tf.global_variables_initializer())\n",
    "            # For each epoch.\n",
    "            for epoch in range(self.epochs):\n",
    "                # For each step/batch.\n",
    "                for start, end in zip(range(0, len(X), self.batchsize),range(self.batchsize,len(X), self.batchsize)):\n",
    "                    batch = X[start:end]\n",
    "                    # Update the rates.\n",
    "                    cur_w = sess.run(update_w, feed_dict={v0: batch, _w: prv_w, _hb: prv_hb, _vb: prv_vb})\n",
    "                    cur_hb = sess.run(update_hb, feed_dict={v0: batch, _w: prv_w, _hb: prv_hb, _vb: prv_vb})\n",
    "                    cur_vb = sess.run(update_vb, feed_dict={v0: batch, _w: prv_w, _hb: prv_hb, _vb: prv_vb})\n",
    "                    prv_w = cur_w\n",
    "                    prv_hb = cur_hb\n",
    "                    prv_vb = cur_vb\n",
    "                error = sess.run(err, feed_dict={v0: X, _w: cur_w, _vb: cur_vb, _hb: cur_hb})\n",
    "                print 'Epoch: %d' % epoch,'reconstruction error: %f' % error\n",
    "            self.w = prv_w\n",
    "            self.hb = prv_hb\n",
    "            self.vb = prv_vb\n",
    "\n",
    "    # Create the expected output for the DBN.\n",
    "    def rbm_output(self, X):\n",
    "        input_X = tf.constant(X)\n",
    "        _w = tf.constant(self.w)\n",
    "        _hb = tf.constant(self.hb)\n",
    "        out = tf.nn.sigmoid(tf.matmul(input_X, _w) + _hb)\n",
    "        with tf.Session() as sess:\n",
    "            sess.run(tf.global_variables_initializer())\n",
    "            return sess.run(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The MNIST Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the MNIST dataset, which is a commonly used dataset comprised of handwritten digits for benchmarking a model. We will import the images using \"One Hot Encoding\" to encode the handwritten images into values varying from 0 to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://ibm.box.com/shared/static/1yvz587r3jot1n8itvqqbcivv3ay40qh.png\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "# Getting the MNIST data provided by Tensorflow.\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# Loading the MNIST data.\n",
    "mnist = input_data.read_data_sets(\"/tmp/data\", one_hot=True)\n",
    "trX, trY, teX, teY = mnist.train.images, mnist.train.labels, mnist.test.images, mnist.test.labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the Deep Belief Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the RBM class created and the MNIST dataset loaded, we can start creating the DBN. We are going to use 3 RBMs, one with 500 hidden units, the second one with 200 and the last one with 50. We are generating a **deep hierarchical representation of the training data**. The cell below accomplishes this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RBM: 0   784 -> 500\n",
      "RBM: 1   500 -> 200\n",
      "RBM: 2   200 -> 50\n"
     ]
    }
   ],
   "source": [
    "RBM_hidden_sizes = [500, 200, 50]\n",
    "\n",
    "# Since we are training, set input as training data.\n",
    "inpX = trX\n",
    "\n",
    "# Create a list to hold the RBMs.\n",
    "rbm_list = []\n",
    "\n",
    "# input_size is the number of inputs in the training set.\n",
    "input_size = inpX.shape[1]\n",
    "\n",
    "# For each RBM we want to generate.\n",
    "for i, size in enumerate(RBM_hidden_sizes):\n",
    "    print 'RBM:', i, ' ', input_size, '->', size\n",
    "    rbm_list.append(RBM(input_size, size))\n",
    "    input_size = size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RBM Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now begin the pre-training step and train each of the RBMs in the stack by individiually calling the `train` function, getting the current RBMs output and using it as the next RBM's input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New RBM:\n",
      "Epoch: 0 reconstruction error: 0.060590\n",
      "Epoch: 1 reconstruction error: 0.052178\n",
      "Epoch: 2 reconstruction error: 0.049212\n",
      "Epoch: 3 reconstruction error: 0.046610\n",
      "Epoch: 4 reconstruction error: 0.045432\n",
      "New RBM:\n",
      "Epoch: 0 reconstruction error: 0.036951\n",
      "Epoch: 1 reconstruction error: 0.032743\n",
      "Epoch: 2 reconstruction error: 0.030631\n",
      "Epoch: 3 reconstruction error: 0.029075\n",
      "Epoch: 4 reconstruction error: 0.028527\n",
      "New RBM:\n",
      "Epoch: 0 reconstruction error: 0.051912\n",
      "Epoch: 1 reconstruction error: 0.048075\n",
      "Epoch: 2 reconstruction error: 0.047272\n",
      "Epoch: 3 reconstruction error: 0.047029\n",
      "Epoch: 4 reconstruction error: 0.045988\n"
     ]
    }
   ],
   "source": [
    "# For each RBM in the list.\n",
    "for rbm in rbm_list:\n",
    "    print 'New RBM:'\n",
    "    # Train a new one.\n",
    "    rbm.train(inpX) \n",
    "    # Return the output layer.\n",
    "    inpX = rbm.rbm_output(inpX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can convert the learned representation of input data into a supervised prediction, e.g. a linear classifier. Specifically, we use the output of the last hidden layer of the DBN to classify digits using a shallow Neural Network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The class below implements the Neural Network that makes use of the pre-trained RBMs from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import math\n",
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "class NN(object):\n",
    "    \n",
    "    def __init__(self, sizes, X, Y):\n",
    "        # Initialize hyperparameters.\n",
    "        self._sizes = sizes\n",
    "        self._X = X\n",
    "        self._Y = Y\n",
    "        self.w_list = []\n",
    "        self.b_list = []\n",
    "        self._learning_rate = 1.0\n",
    "        self._momentum = 0.0\n",
    "        self._epoches = 10\n",
    "        self._batchsize = 100\n",
    "        input_size = X.shape[1]\n",
    "        \n",
    "        # Initialization loop.\n",
    "        for size in self._sizes + [Y.shape[1]]:\n",
    "            # Define upper limit for the uniform distribution range.\n",
    "            max_range = 4 * math.sqrt(6. / (input_size + size))\n",
    "            \n",
    "            # Initialize weights through a random uniform distribution.\n",
    "            self.w_list.append(\n",
    "                np.random.uniform(-max_range, max_range, [input_size, size]).astype(np.float32))\n",
    "            \n",
    "            # Initialize bias as zeroes.\n",
    "            self.b_list.append(np.zeros([size], np.float32))\n",
    "            input_size = size\n",
    "      \n",
    "    # Load data from rbm.\n",
    "    def load_from_rbms(self, dbn_sizes, rbm_list):\n",
    "        # Check if expected sizes are correct.\n",
    "        assert len(dbn_sizes) == len(self._sizes)\n",
    "        \n",
    "        for i in range(len(self._sizes)):\n",
    "            # Check if for each RBM the expected sizes are correct.\n",
    "            assert dbn_sizes[i] == self._sizes[i]\n",
    "        \n",
    "        # If everything is correct, bring over the weights and biases.\n",
    "        for i in range(len(self._sizes)):\n",
    "            self.w_list[i] = rbm_list[i].w\n",
    "            self.b_list[i] = rbm_list[i].hb\n",
    "\n",
    "    # Training method.\n",
    "    def train(self):\n",
    "        # Create placeholders for weights, biases.\n",
    "        _a = [None] * (len(self._sizes) + 2)\n",
    "        _w = [None] * (len(self._sizes) + 1)\n",
    "        _b = [None] * (len(self._sizes) + 1)\n",
    "        _a[0] = tf.placeholder(\"float\", [None, self._X.shape[1]])\n",
    "        y = tf.placeholder(\"float\", [None, self._Y.shape[1]])\n",
    "        \n",
    "        # Define variables and activation functoin.\n",
    "        for i in range(len(self._sizes) + 1):\n",
    "            _w[i] = tf.Variable(self.w_list[i])\n",
    "            _b[i] = tf.Variable(self.b_list[i])\n",
    "        for i in range(1, len(self._sizes) + 2):\n",
    "            _a[i] = tf.nn.sigmoid(tf.matmul(_a[i - 1], _w[i - 1]) + _b[i - 1])\n",
    "        \n",
    "        # Define the cost function.\n",
    "        cost = tf.reduce_mean(tf.square(_a[-1] - y))\n",
    "        \n",
    "        # Define the training operation (Momentum Optimizer minimizing the Cost function).\n",
    "        train_op = tf.train.MomentumOptimizer(\n",
    "            self._learning_rate, self._momentum).minimize(cost)\n",
    "        \n",
    "        # Prediction operation.\n",
    "        predict_op = tf.argmax(_a[-1], 1)\n",
    "        \n",
    "        # Training Loop.\n",
    "        with tf.Session() as sess:\n",
    "            # Initialize Variables.\n",
    "            sess.run(tf.global_variables_initializer())\n",
    "            \n",
    "            # For each epoch.\n",
    "            for i in range(self._epoches):\n",
    "                \n",
    "                # For each step.\n",
    "                for start, end in zip(\n",
    "                    range(0, len(self._X), self._batchsize), range(self._batchsize, len(self._X), self._batchsize)):\n",
    "                    \n",
    "                    # Run the training operation on the input data.\n",
    "                    sess.run(train_op, feed_dict={\n",
    "                        _a[0]: self._X[start:end], y: self._Y[start:end]})\n",
    "                \n",
    "                for j in range(len(self._sizes) + 1):\n",
    "                    # Retrieve weights and biases.\n",
    "                    self.w_list[j] = sess.run(_w[j])\n",
    "                    self.b_list[j] = sess.run(_b[j])\n",
    "                \n",
    "                print \"Accuracy rating for epoch \" + str(i) + \": \" + str(np.mean(np.argmax(self._Y, axis=1) ==\n",
    "                              sess.run(predict_op, feed_dict={_a[0]: self._X, y: self._Y})))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's execute the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy rating for epoch 0: 0.5136909090909091\n",
      "Accuracy rating for epoch 1: 0.6073636363636363\n",
      "Accuracy rating for epoch 2: 0.6675636363636364\n",
      "Accuracy rating for epoch 3: 0.7445636363636363\n",
      "Accuracy rating for epoch 4: 0.7884727272727273\n",
      "Accuracy rating for epoch 5: 0.8050909090909091\n",
      "Accuracy rating for epoch 6: 0.8152181818181818\n",
      "Accuracy rating for epoch 7: 0.838690909090909\n",
      "Accuracy rating for epoch 8: 0.9047090909090909\n",
      "Accuracy rating for epoch 9: 0.9127454545454545\n"
     ]
    }
   ],
   "source": [
    "nNet = NN(RBM_hidden_sizes, trX, trY)\n",
    "nNet.load_from_rbms(RBM_hidden_sizes, rbm_list)\n",
    "nNet.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "• http://deeplearning.net/tutorial/DBN.html  \n",
    "• https://github.com/myme5261314/dbn_tf"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15rc1"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}