StrassenNetsCIFAR10.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# StrassenNets: Deep Learning with a Multiplication Budget\n",
    "\n",
    "This notebook provides a step-by-step implementation to reproduce part of the experiments in *StrassenNets: Deep Learning with a Multiplication Budget* by Michael Tschannen, Aran Khanna, and Anima Anandkumar (http://bit.ly/2BJZj5a).\n",
    "\n",
    "#### Paper abstract\n",
    "\n",
    "A large fraction of the arithmetic operations required to evaluate deep neural networks (DNNs) are due to matrix multiplications, both in convolutional and fully connected layers. Matrix multiplications can be cast as $2$-layer sum-product (SP) networks (arithmetic circuits), disentangling multiplications and additions. We leverage this observation for end-to-end learning of low-cost (in terms of multiplications) approximations of linear operations in DNN layers. Specifically, we propose to replace matrix multiplication operations by sum-product networks, with widths corresponding to the budget of multiplications we want to allocate to each layer, and learning the edges of the SP networks from data. Experiments on CIFAR-10 and ImageNet show that this method applied to ResNet yields significantly higher accuracy than existing methods for a given multiplication budget, or leads to the same or higher accuracy compared to existing methods while using significantly fewer multiplications. Furthermore, our approach allows fine-grained control of the tradeoff between arithmetic complexity and accuracy of DNN models. Finally, we demonstrate that the proposed framework can rediscover Strassen's algorithm, i.e., it can learn to multiply $2 \\times 2$ matrices using only $7$ multiplications instead of $8$. \n",
    "\n",
    "\n",
    "#### Learning fast matrix multiplications via sum-product networks\n",
    "$\\renewcommand{\\vec}{\\mathrm{vec}}$Given square matrices $A, B \\in \\mathbb{R}^{n \\times n}$, the product $C = A B$ can be represented as a $2$-layer sum-product (SP) network (or ``Strassen network'')\n",
    "\n",
    "$$\n",
    "    \\label{eq:strnet}\n",
    "    \\vec(C) = W_C [ (W_B \\vec(B)) \\odot (W_A \\vec(A))] \\qquad \\qquad (1)\n",
    "$$\n",
    "\n",
    "where $W_A, W_B \\in \\mathbb{K}^{r \\times n^2}$ and $W_C \\in \\mathbb{K}^{n^2 \\times r}$, $\\mathbb{K}:= \\{-1,0,1\\}$ are fixed, and $\\odot$ denotes the element-wise product.\n",
    "\n",
    "<img src=\"files/spnetwork.png\" alt=\"spnetwork\" style=\"width: 400px;\"/>\n",
    "\n",
    "1. $A, B\\in \\mathbb{R}^{2 \\times 2}$: Strassen's matrix multiplication algorithm (https://en.wikipedia.org/wiki/Strassen_algorithm) tells us ternary weights that satisfy (1) for $r=7$ (instead of $r = 8$).\n",
    "2. General case $A \\in \\mathbb{R}^{k \\times m}$, $B \\in \\mathbb{R}^{m \\times n}$: $C = A B$ can be written in the form (1) if $r \\geq nmk$.\n",
    "3. If $A \\in \\mathbb{R}^{k \\times m}$ is fixed and $B \\in \\mathbb{R}^{m \\times n}$ concentrates on low-dimensional subspace of $\\mathbb{R}^{k \\times m}$: Can find $W_A, W_B$ s.t. (1) holds approximately even when $r \\ll nmk$\n",
    "\n",
    "*Idea:* Leverage item 3 to learn low-cost (in terms of multiplications) approximate linear operations in neural network layers by associating $A$ with (pretrained) weights/filters and $B$ with corresponding activations/feature maps, and by learning $W_A$, $W_B$, $W_C$ end-to-end. Please see the paper for a more in-depth description of the method.\n",
    "\n",
    "In this notebook, we demonstrate this idea by compressing the 2D convolution layers of ResNet-20 trained on CIFAR-10.\n",
    "\n",
    "#### Outline of the notebook:\n",
    "1. Implement the ternary quantization operator to learn ternary weights $W_B$ and $W_C$\n",
    "2. Implement the SP 2D convolution layer (i.e. a Strassen network for 2D convolutions)\n",
    "3. Implement residual blocks with SP layers and use these to construct a SP-ResNet-20\n",
    "4. Define iterators and the training loop\n",
    "5. Train the network\n",
    "\n",
    "#### Requirements\n",
    "\n",
    "The code is based on Apache MXNet 0.12 (http://mxnet.incubator.apache.org/install/index.html) and the gluon package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Implement the ternary quantization operator\n",
    "\n",
    "Following the tutorial from [1] to define custom gluon operators, we define an operator implementing the ternarization method proposed in [2]. Using the notation from [2], this method quantizes the entries of a given input tensor to values $\\{-\\alpha, 0, \\alpha\\}$ ($\\alpha$ corresponds to `scale` below) based on a threshold $\\Delta$ (corresponding to `thresh` below) on the full precision entries. $\\alpha$ and $\\Delta$ are determinined by approximately solving an optimization problem, see [2]. The operator will be used to quanzie $W_B$ and $W_C$ (the factor $\\alpha$ can be absorbed into $\\tilde a = W_A \\vec(A)$ after training to ensure that the entries of $W_B$ and $W_C$ are in $\\{-1,0,1\\}$). \n",
    "\n",
    "[1] https://github.com/apache/incubator-mxnet/blob/master/docs/tutorials/gluon/customop.md\n",
    "\n",
    "[2] F. Li, B. Zhang, B. Liu, Ternary Weight Networks, 2016, https://arxiv.org/abs/1605.04711v2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mxnet as mx\n",
    "import numpy as np\n",
    "\n",
    "# define the quantization operator\n",
    "class TerQuant(mx.operator.CustomOp):\n",
    "    # define forward pass as described in [1]\n",
    "    def forward(self, is_train, req, in_data, out_data, aux):\n",
    "        full_data = in_data[0]\n",
    "\n",
    "        # compute $\\Delta$ as in [1], Eq. (6)\n",
    "        thresh = mx.nd.mean(mx.nd.abs(full_data), axis=()).asnumpy()[0] * 0.7\n",
    "\n",
    "        # ternarize weights\n",
    "        quant_data = mx.nd.greater(full_data, thresh*mx.nd.ones(full_data.shape, full_data.context))\\\n",
    "            - mx.nd.lesser(full_data, (-1.0)*thresh*mx.nd.ones(full_data.shape, full_data.context))\n",
    "\n",
    "        # compute $\\alpha$ as in [1], Eq. (5)\n",
    "        scale = mx.nd.sum(mx.nd.multiply(full_data, quant_data), axis=()).asnumpy()[0]\\\n",
    "            / mx.nd.sum(mx.nd.abs(quant_data), axis=()).asnumpy()[0]\n",
    "\n",
    "        # rescale ternary weights\n",
    "        quant_data = scale * quant_data\n",
    "        self.assign(out_data[0], req[0], quant_data)\n",
    "\n",
    "    # define backward pass: simply pass on gradient from previous operation (derivative of the identity function)\n",
    "    def backward(self, req, out_grad, in_data, out_data, in_grad, aux):\n",
    "        self.assign(in_grad[0], req[0], out_grad[0])\n",
    "\n",
    "# wrap the operator and provide functions to get information about input, output etc.\n",
    "@mx.operator.register('ter_quant')\n",
    "class TerQuantProp(mx.operator.CustomOpProp):\n",
    "    def __init__(self):\n",
    "        super(TerQuantProp, self).__init__(True)\n",
    "\n",
    "    def list_arguments(self):\n",
    "        return ['data']\n",
    "\n",
    "    def list_outputs(self):\n",
    "        return ['output']\n",
    "\n",
    "    def infer_shape(self, in_shapes):\n",
    "        data_shape = in_shapes[0]\n",
    "        output_shape = data_shape\n",
    "        return (data_shape,), (output_shape,), ()\n",
    "\n",
    "    def create_operator(self, ctx, in_shapes, in_dtypes):\n",
    "        return TerQuant()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Implement the SP 2DConvLayer\n",
    "$\\newcommand{\\cin}{c_\\mathrm{in}}\\newcommand{\\cout}{c_\\mathrm{out}}$We are now ready to implement the SP 2D convolution layer (as a gluon `HybridBlock` object). Specifically, we compress the computation of $\\cout \\times p \\times p$ output elements of the convolution from $\\cin \\times (p - 1 + k) \\times (p - 1 + k)$ input elements by applying (see figure below)\n",
    "\n",
    "1. a $p$-strided 2D convolution with a *ternary* filter of size $r \\times \\cin \\times (p - 1 + k) \\times (p - 1 + k)$ (corresponding to multiplication with $W_B$),\n",
    "2. a channel-wise multiplication with an $r \\times 1 \\times 1$ *full-precision* filter $\\tilde a = W_A \\vec(A)$ (corresponding to the filters of the original convolution),\n",
    "3. a $1/p$-strided transposed 2D convolution with a *ternary* filter of size $\\cout \\times r \\times p \\times p$ (corresponding to multiplication with $W_C$).\n",
    "\n",
    "<img src=\"files/sp2dconv.png\" alt=\"spnetwork\" style=\"width: 600px;\"/>\n",
    "\n",
    "As we will pretrain the convolutions described in items 1 and 3 above with full-precision weights, we include a variable `mode` that determines whether quantization is active or not. Furthermore, we include the original convolution to selectively replace the SP convolutions with ordinary ones (not used in this notebook)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mxnet import gluon, autograd\n",
    "from mxnet.gluon.block import HybridBlock\n",
    "from mxnet.base import numeric_types\n",
    "from math import ceil\n",
    "\n",
    "class SumProd2DConv(mx.gluon.HybridBlock):\n",
    "    r\"\"\"\n",
    "    nbr_mul : int\n",
    "        Width of the sum-product network, corresponding to the number of multiplications r used to compute an\n",
    "        $p_x$ x $p_y$ x channels activation element\n",
    "    target_layer_shape:\n",
    "        Shape of the filter to be replaced by the sum-product network\n",
    "    target_layer_key:\n",
    "        Name of the target layer in the original ResNet (to enable loading from a pretrained network)\n",
    "    out_patch : int\n",
    "        size ($p_x$, $p_y$) of the output patch for spatial compression\n",
    "    kernel, stride, pad : specifics of the 2D convolution\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, nbr_mul, target_layer_shape, target_layer_key, out_patch=(1,1), kernel=(3,3), stride=(1,1), pad=(1,1), prefix=None, **kwargs):\n",
    "        # Use same prefix as parent layer for loading the shared paramters to be compressed\n",
    "        super(SumProd2DConv, self).__init__(prefix=prefix, **kwargs)\n",
    "\n",
    "        if isinstance(stride, numeric_types):\n",
    "            stride = (stride,)*len(kernel)\n",
    "        if isinstance(pad, numeric_types):\n",
    "            pad = (pad,)*len(kernel)\n",
    "\n",
    "        self._nbr_mul = nbr_mul\n",
    "        # corresponds to $p$ in the description above\n",
    "        self._out_patch = out_patch\n",
    "        self._kernel = kernel\n",
    "        self._stride = stride\n",
    "        self._pad = pad\n",
    "\n",
    "        # compute kernel shapes and stride for $W_B$\n",
    "        self._sp_data_weight_kernel = ((self._out_patch[0]-1)*self._stride[0]+self._kernel[0], (self._out_patch[1]-1)*self._stride[1]+self._kernel[1])\n",
    "        self._sp_data_weight_stride = (self._out_patch[0]*self._stride[0], self._out_patch[1]*self._stride[1])\n",
    "\n",
    "        # filters for the original convolution\n",
    "        self.filter_weights = self.params.get(target_layer_key, shape=target_layer_shape)\n",
    "\n",
    "        # Assuming the default NCHW layout, determine shapes of $W_B$ and $W_C$\n",
    "        filter_weights_shape = list(target_layer_shape)\n",
    "        sp_data_weights_shape = filter_weights_shape.copy()\n",
    "        sp_data_weights_shape[0] = nbr_mul\n",
    "        sp_data_weights_shape[2] = self._sp_data_weight_kernel[0]\n",
    "        sp_data_weights_shape[3] = self._sp_data_weight_kernel[1]\n",
    "        sp_out_weights_shape = filter_weights_shape.copy()\n",
    "        sp_out_weights_shape[0] = nbr_mul\n",
    "        sp_out_weights_shape[1] = filter_weights_shape[0]\n",
    "        sp_out_weights_shape[2] = self._out_patch[0]\n",
    "        sp_out_weights_shape[3] = self._out_patch[1]\n",
    "\n",
    "        ## SP network weights\n",
    "        # $\\tilde a = W_A \\vec(A)$\n",
    "        self.sp_in_weights = self.params.get('sp_in_weights', shape=(1,self._nbr_mul,1,1))\n",
    "        # $W_B$\n",
    "        self.sp_data_weights = self.params.get('sp_data_weights', shape=sp_data_weights_shape)\n",
    "        # $W_C$\n",
    "        self.sp_out_weights =  self.params.get('sp_out_weights', shape=sp_out_weights_shape)\n",
    "\n",
    "        # SP network batchnorm parameters\n",
    "        self.sp_batchnorm_gamma = self.params.get('sp_batchnorm_gamma', shape=(1,self._nbr_mul,1,1))\n",
    "        self.sp_batchnorm_beta = self.params.get('sp_batchnorm_beta', shape=(1,self._nbr_mul,1,1))\n",
    "        self.sp_batchnorm_running_mean = self.params.get('sp_batchnorm_running_mean', shape=(1,self._nbr_mul,1,1))\n",
    "        self.sp_batchnorm_running_var = self.params.get('sp_batchnorm_running_var', shape=(1,self._nbr_mul,1,1))\n",
    "\n",
    "        # mode: 1: original convolution; 2: SP network without ternary quantization; 3: SP network with ternary quantization\n",
    "        self.sp_mode = self.params.get('sp_mode', shape=(1,), dtype=np.uint8)\n",
    "\n",
    "\n",
    "    def forward(self, data):\n",
    "        with data.context:\n",
    "            mode = self.sp_mode.data().asnumpy()[0]\n",
    "            filter_weights = self.filter_weights.data()\n",
    "\n",
    "            if mode == 0:\n",
    "                # perform original convolution\n",
    "                conv_out = mx.nd.Convolution(data=data,\n",
    "                                  weight=filter_weights,\n",
    "                                  bias=None,\n",
    "                                  no_bias=True,\n",
    "                                  kernel=self._kernel,\n",
    "                                  stride=self._stride,\n",
    "                                  pad=self._pad,\n",
    "                                  num_filter=filter_weights.shape[0])\n",
    "\n",
    "                return conv_out\n",
    "            else:\n",
    "                sp_data_weights = self.sp_data_weights.data()\n",
    "                sp_out_weights = self.sp_out_weights.data()\n",
    "\n",
    "                # quantzie $W_B$ and $W_C$ if desired\n",
    "                if mode == 2:\n",
    "                    sp_data_weights = mx.nd.Custom(sp_data_weights, op_type='ter_quant')\n",
    "                    sp_out_weights = mx.nd.Custom(sp_out_weights, op_type='ter_quant')\n",
    "\n",
    "                pad_x = self._pad[0] + ceil((data.shape[2]%self._sp_data_weight_stride[0])/2)\n",
    "                pad_y = self._pad[1] + ceil((data.shape[3]%self._sp_data_weight_stride[1])/2)\n",
    "\n",
    "                # compute $W_B \\vec(B)$\n",
    "                sp_data_mul = mx.nd.Convolution(data=data,\n",
    "                                  weight=sp_data_weights,\n",
    "                                  bias=None,\n",
    "                                  no_bias=True,\n",
    "                                  kernel=self._sp_data_weight_kernel,\n",
    "                                  stride=self._sp_data_weight_stride,\n",
    "                                  pad=(pad_x, pad_y),\n",
    "                                  num_filter = self._nbr_mul)\n",
    "\n",
    "                # apply batchnorm to $W_B \\vec(B)$\n",
    "                sp_data_mul_norm = mx.nd.BatchNorm(data=sp_data_mul,\n",
    "                                              gamma=self.sp_batchnorm_gamma.data(),\n",
    "                                              beta=self.sp_batchnorm_beta.data(),\n",
    "                                              moving_mean=self.sp_batchnorm_running_mean.data(),\n",
    "                                              moving_var=self.sp_batchnorm_running_var.data(),\n",
    "                                              fix_gamma=True)\n",
    "\n",
    "                # apply non-negativity constraint to $\\tilde a = W_A \\vec(A)$\n",
    "                with mx.autograd.pause():\n",
    "                    self.sp_in_weights.set_data(mx.nd.clip(data=self.sp_in_weights.data(), a_min=0.0, a_max=100.0))\n",
    "                \n",
    "                # compute $(W_A \\vec(A)) \\odot (W_B \\vec(B))$\n",
    "                sp_mul = mx.nd.multiply(self.sp_in_weights.data(), sp_data_mul_norm)\n",
    "                \n",
    "                # compute $W_C[(W_A \\vec(A)) \\odot (W_B \\vec(B))]$\n",
    "                sp_out = mx.nd.Deconvolution(data = sp_mul,\n",
    "                                  weight = sp_out_weights,\n",
    "                                  bias = None,\n",
    "                                  no_bias = True,\n",
    "                                  kernel = self._out_patch,\n",
    "                                  stride = self._out_patch,\n",
    "                                  target_shape = (data.shape[2]//self._stride[0], data.shape[3]//self._stride[1]),\n",
    "                                  num_filter = filter_weights.shape[0])\n",
    "\n",
    "                return sp_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Implement the sum-product residual unit\n",
    "\n",
    "We continue by defining the residual unit as proposed in [3] using sum-product 2D convolutions. The code block below is adapted from [4] and looks lengthy, but there are only small modifications compared to [4]. Specifically, we replace ordinary 2D convolutions by sum-product 2D convolutions and add parameters for the sum-product convolutions.\n",
    "\n",
    "[3] K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, 2016, http://arxiv.org/abs/1512.03385\n",
    "\n",
    "[4] https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/gluon/model_zoo/vision/resnet.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mxnet.gluon import nn\n",
    "\n",
    "class BasicBlockV1SumProd(HybridBlock):\n",
    "    r\"\"\"\n",
    "    Parameters\n",
    "    ----------\n",
    "    channels : int\n",
    "        Number of output channels.\n",
    "    stride : int\n",
    "        Stride size.\n",
    "    nbr_mul : int\n",
    "        Width of the sum-product network, corresponding to the number of multiplications r used to compute an\n",
    "        $p_x$ x $p_y$ x channels activation element\n",
    "    out_patch : int\n",
    "        size ($p_x$, $p_y$) of the output patch for spatial compression\n",
    "    conv_idx : int\n",
    "        index of the convolution, used for parameter names\n",
    "    downsample : bool, default False\n",
    "        Whether to downsample the input.\n",
    "    in_channels : int, default 0\n",
    "        Number of input channels. Default is 0, to infer from the graph.\n",
    "    quant_down : bool\n",
    "        Whether to quantize ResNet 1x1 convolution projection layers\n",
    "    \"\"\"\n",
    "    def __init__(self, channels, stride, nbr_mul, out_patch=(1,1), conv_idx=0, downsample=False, in_channels=0, quant_down=True, **kwargs):\n",
    "        super(BasicBlockV1SumProd, self).__init__(**kwargs)\n",
    "        self.body = nn.HybridSequential(prefix='')\n",
    "\n",
    "        prefix1 = 'conv%i_'%conv_idx\n",
    "        # shape of the original convolution\n",
    "        orig_shape1 = (channels, in_channels, 3, 3)\n",
    "        # kernel of the original convolution\n",
    "        kernel = (3,3)\n",
    "\n",
    "        self.body.add(SumProd2DConv(nbr_mul=nbr_mul,\n",
    "                                      target_layer_shape=orig_shape1,\n",
    "                                      target_layer_key='weight',\n",
    "                                      out_patch=out_patch,\n",
    "                                      kernel=kernel,\n",
    "                                      stride=stride,\n",
    "                                      prefix=prefix1))\n",
    "\n",
    "        self.body.add(nn.BatchNorm())\n",
    "        self.body.add(nn.Activation('relu'))\n",
    "        \n",
    "        prefix2 = 'conv%i_'%(conv_idx+1)\n",
    "        orig_shape2 = (channels, channels, 3, 3)\n",
    "\n",
    "        self.body.add(SumProd2DConv(nbr_mul=nbr_mul,\n",
    "                                      target_layer_shape=orig_shape2,\n",
    "                                      target_layer_key='weight',\n",
    "                                      out_patch=out_patch,\n",
    "                                      kernel=kernel,\n",
    "                                      stride=1,\n",
    "                                      prefix=prefix2))\n",
    "\n",
    "        self.body.add(nn.BatchNorm())\n",
    "        \n",
    "        if downsample:\n",
    "            self.downsample = nn.HybridSequential(prefix='')\n",
    "\n",
    "            # Quantize projection layer if required\n",
    "            if quant_down:\n",
    "                prefix_ds = 'conv%i_'%(conv_idx+2)\n",
    "                orig_shape_ds = (channels, in_channels, 1, 1)\n",
    "\n",
    "                self.downsample.add(SumProd2DConv(nbr_mul=nbr_mul,\n",
    "                                              target_layer_shape=orig_shape_ds,\n",
    "                                              target_layer_key='weight',\n",
    "                                              out_patch=(1,1),\n",
    "                                              kernel=(1,1),\n",
    "                                              stride=stride,\n",
    "                                              pad=(0,0),\n",
    "                                              prefix=prefix_ds))\n",
    "            else:\n",
    "                self.downsample.add(nn.Conv2D(channels, kernel_size=1, strides=stride,\n",
    "                                              use_bias=False, in_channels=in_channels))\n",
    "\n",
    "            self.downsample.add(nn.BatchNorm())\n",
    "        else:\n",
    "            self.downsample = None\n",
    "\n",
    "    def hybrid_forward(self, F, x):\n",
    "        residual = x\n",
    "\n",
    "        x = self.body(x)\n",
    "\n",
    "        if self.downsample:\n",
    "            residual = self.downsample(residual)\n",
    "\n",
    "        x = F.Activation(residual+x, act_type='relu')\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Implement the sum-product ResNet\n",
    "\n",
    "Again, we modify the code to construct ResNet (as a gluon `HybridBlock` object) for various dephts from [4], replacing ordinary 2D convolutions by sum-product 2D convolutions, and the fully connected output layer by a sum-product network if desired. As before, we add in some parameters for the sum-product networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mxnet.gluon.model_zoo.vision.resnet import _conv3x3\n",
    "\n",
    "class ResNetV1SumProd(HybridBlock):\n",
    "    r\"\"\"ResNet V1 model from\n",
    "    `\"Deep Residual Learning for Image Recognition\"\n",
    "    <http://arxiv.org/abs/1512.03385>`_ paper.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    block : HybridBlock\n",
    "        Class for the residual block. Options are BasicBlockV1, BottleneckV1.\n",
    "    layers : list of int\n",
    "        Numbers of layers in each block\n",
    "    channels : list of int\n",
    "        Numbers of channels in each block. Length should be one larger than layers list.\n",
    "    nbr_mul : int\n",
    "        Width of the sum-product network, corresponding to the number of multiplications r used to compute an\n",
    "        $p_x$ x $p_y$ x channels activation element\n",
    "    out_patch : int\n",
    "        size ($p_x$, $p_y$) of the output patch for spatial compression\n",
    "    classes : int, default 1000\n",
    "        Number of classification classes.\n",
    "    thumbnail : bool, default False\n",
    "        Enable thumbnail.\n",
    "    quant_down : bool\n",
    "        Whether to quantize ResNet 1x1 convolution projection layers\n",
    "    \"\"\"\n",
    "    def __init__(self, layers, channels, nbr_mul, out_patch, classes=1000, thumbnail=False, quant_down=True, prefix='resnetv10_', **kwargs):\n",
    "        super(ResNetV1SumProd, self).__init__(prefix, **kwargs)\n",
    "        \n",
    "        with self.name_scope():\n",
    "            self.features = nn.HybridSequential(prefix='')\n",
    "            # Thumbnail determines the characteristics of the first layer. If thumbnail==True, 3x3 convolutions are used\n",
    "            # and no pooling is performed (this setting is used for CIFAR). Otherwise, 7x7 filters with max-pooling are used.\n",
    "            if thumbnail:\n",
    "                # replace first layer with a sum-product layer if desired\n",
    "                if nbr_mul[0] > 0:\n",
    "                    orig_shape = (channels[0], 3, 3, 3)\n",
    "\n",
    "                    self.features.add(SumProd2DConv(nbr_mul = nbr_mul[0],\n",
    "                                                  target_layer_shape = orig_shape,\n",
    "                                                  target_layer_key = 'weight',\n",
    "                                                  kernel = (3, 3),\n",
    "                                                  stride = (1, 1),\n",
    "                                                  out_patch = out_patch[0],\n",
    "                                                  prefix = 'conv0_'))\n",
    "                else:\n",
    "                    self.features.add(_conv3x3(channels[0], 1, 3))\n",
    "\n",
    "            else:\n",
    "                # replace first layer with a sum-product layer if desired\n",
    "                if nbr_mul[0] > 0:\n",
    "                    orig_shape = (channels[0], 3, 7, 7)\n",
    "\n",
    "                    self.features.add(SumProd2DConv(nbr_mul = nbr_mul[0],\n",
    "                                                  target_layer_shape = orig_shape,\n",
    "                                                  target_layer_key = 'weight',\n",
    "                                                  kernel = (7, 7),\n",
    "                                                  stride = (2, 2),\n",
    "                                                  pad = (3,3),\n",
    "                                                  out_patch = out_patch[0],\n",
    "                                                  prefix = 'conv0_'))\n",
    "                else:\n",
    "                    self.features.add(nn.Conv2D(channels[0], 7, 2, 3, use_bias=False,\n",
    "                                                in_channels=3))\n",
    "                self.features.add(nn.BatchNorm())\n",
    "                self.features.add(nn.Activation('relu'))\n",
    "                self.features.add(nn.MaxPool2D(3, 2, 1))\n",
    "\n",
    "            for i, num_layer in enumerate(layers):\n",
    "                stride = 1 if i == 0 else 2\n",
    "                self.features.add(self._make_sumprod_layer(num_layer, channels[i+1], stride,\n",
    "                                                   nbr_mul[i+1], out_patch[i+1], i+1, in_channels=channels[i], quant_down=quant_down))\n",
    "\n",
    "            self.classifier = nn.HybridSequential(prefix='')\n",
    "            self.classifier.add(nn.GlobalAvgPool2D())\n",
    "            self.classifier.add(nn.Flatten())\n",
    "            self.classifier.add(nn.Dense(classes, in_units=channels[-1]))\n",
    "\n",
    "    # stack residual blocks as described in [2]\n",
    "    def _make_sumprod_layer(self, layers, channels, stride, nbr_mul, out_patch, stage_index, in_channels=0, quant_down=True):\n",
    "        layer = nn.HybridSequential(prefix='stage%d_'%stage_index)\n",
    "        with layer.name_scope():\n",
    "            layer.add(BasicBlockV1SumProd(channels, stride, nbr_mul, out_patch, 0, channels != in_channels, in_channels=in_channels, quant_down=quant_down, prefix=''))\n",
    "            nbr_conv = 2 if channels == in_channels else 3\n",
    "            for i in range(layers-1):\n",
    "                layer.add(BasicBlockV1SumProd(channels, 1, nbr_mul, out_patch, nbr_conv, False, in_channels=channels, quant_down=quant_down, prefix=''))\n",
    "                nbr_conv += 2\n",
    "        return layer\n",
    "\n",
    "    def hybrid_forward(self, F, x):\n",
    "        x = self.features(x)\n",
    "        x = self.classifier(x)\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Training the network\n",
    "\n",
    "We are now ready to train and evaluate the SP-ResNet-20 on CIFAR-10 as described in the paper.\n",
    "\n",
    "We start by defining the SP-Resnet-20. The parameters below realize the configuration for $p=1$ and $r=c_\\mathrm{out}$ in Table 1 (first row, first column) in the paper, and can be modified for other configurations. Furthermore, the context can be modified below (use `ctx = [mx.cpu(0)]` if you don't have a GPU available, or extend to a list of devices for parallel training)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CIFAR-10 has 10 classes\n",
    "classes = 10\n",
    "# structure of ResNet-20\n",
    "res_units = [3, 3, 3]\n",
    "res_channels = [16, 16, 32, 64]\n",
    "# choose the width of the sum-product convolution networks (i.e., $r$) to be equal to the number of output channels\n",
    "# don't compress the last layer (encoded by nbr_mul=0)\n",
    "nbr_mul = [16, 16, 32, 64, 0]\n",
    "# don't do spatial compression (out. patch=1x1)\n",
    "out_patch = [(1,1)]*4\n",
    "# quantize ResNet 1x1 convolution projection layers\n",
    "quant_down = True\n",
    "# use 3x3 kernels for the first convolution layer, don't do pooling after the first convolution layer\n",
    "thumbnail = True\n",
    "\n",
    "# construct ResNet HybridBlock object\n",
    "sp_resnet = ResNetV1SumProd(res_units, res_channels, nbr_mul, out_patch, classes, thumbnail, quant_down)\n",
    "\n",
    "# define context\n",
    "ctx = [mx.gpu(0)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we initialize the network parameters. As we first train all weights at full precision first, `sp_mode` is set to  1. `sp_in_weights` (corresponding to $\\tilde a = W_A \\mathrm{vec}(A)$) is initialized to 1s and Xavier initialization is used for `sp_data_weights` and `sp_out_weights` (corresponding to $W_B$ and $W_C$, respectively), and for the fully connected output layer. Furthermore, optimization is deactivated for the auxiliary batchnorm variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# helper function to filter and intialize paramters whose name contains a given substring\n",
    "def initialize_by_key(net, key, init, context, force_reinit=False):\n",
    "    for k, p in net.collect_params().items():\n",
    "        if key in k:\n",
    "            p.initialize(init=init, ctx=context, force_reinit=force_reinit)\n",
    "\n",
    "# initialize SP networks\n",
    "initialize_by_key(sp_resnet, 'sp_mode', mx.init.One(), ctx)\n",
    "initialize_by_key(sp_resnet, 'sp_in_weights', mx.init.One(), ctx)\n",
    "initialize_by_key(sp_resnet, 'sp_data_weights', mx.init.Xavier(magnitude=2), ctx)\n",
    "initialize_by_key(sp_resnet, 'sp_out_weights', mx.init.Xavier(magnitude=2), ctx)\n",
    "\n",
    "# intialize batchnorm parameters\n",
    "initialize_by_key(sp_resnet, 'gamma', mx.init.One(), ctx)\n",
    "initialize_by_key(sp_resnet, 'beta', mx.init.Zero(), ctx)\n",
    "initialize_by_key(sp_resnet, 'running_mean', mx.init.Zero(), ctx)\n",
    "initialize_by_key(sp_resnet, 'running_var', mx.init.Zero(), ctx)\n",
    "\n",
    "# initialize fully connected layer\n",
    "initialize_by_key(sp_resnet, 'dense0_weight', mx.init.Xavier(magnitude=2), ctx) #mx.init.Normal(1)\n",
    "initialize_by_key(sp_resnet, 'dense0_bias', mx.init.Zero(), ctx)\n",
    "\n",
    "# deactivate optimization for original convolutions as they are not used here\n",
    "for k, p in sp_resnet.collect_params().items():\n",
    "    if 'conv' in k and 'sp' not in k:\n",
    "        p.initialize(init=mx.init.Xavier(magnitude=2), ctx=ctx)\n",
    "        p.grad_req = 'null'\n",
    "\n",
    "# deactivate optimization for batchnorm auxiliary variables\n",
    "for k, p in sp_resnet.collect_params().items():\n",
    "    if 'running_mean' in k or 'running_var' in k or 'sp_mode' in k:\n",
    "        p.grad_req = 'null'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We continue by defining a testing routine and a training loop. This again looks lengthy but is standard and adapted from [5] with slight modifications.\n",
    "\n",
    "[5] https://github.com/apache/incubator-mxnet/blob/master/example/gluon/image_classification.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "# testing routine\n",
    "def test(net, val_data, context):\n",
    "    metric = mx.metric.Accuracy()\n",
    "    val_data.reset()\n",
    "    for batch in val_data:\n",
    "        data = gluon.utils.split_and_load(batch.data[0], ctx_list=context, batch_axis=0)\n",
    "        label = gluon.utils.split_and_load(batch.label[0], ctx_list=context, batch_axis=0)\n",
    "        outputs = []\n",
    "        for x in data:\n",
    "            outputs.append(net(x))\n",
    "        metric.update(label, outputs)\n",
    "    return metric.get()\n",
    "\n",
    "\n",
    "# training routine\n",
    "def train(net, lr, mom, wd, epochs, scheduler, context, train_data, val_data, log_interval):\n",
    "    # define trainer, metric, loss\n",
    "    trainer = gluon.Trainer(net.collect_params(), 'sgd',\n",
    "                            {'learning_rate': lr, 'wd': wd, 'momentum': mom, 'lr_scheduler': scheduler},\n",
    "                            kvstore = 'device')\n",
    "    metric = mx.metric.Accuracy()\n",
    "    loss = gluon.loss.SoftmaxCrossEntropyLoss()\n",
    "\n",
    "    for epoch in range(epochs):\n",
    "        tic = time.time()\n",
    "        train_data.reset()\n",
    "        metric.reset()\n",
    "        btic = time.time()\n",
    "        # loop through batches\n",
    "        for i, batch in enumerate(train_data):\n",
    "            data = gluon.utils.split_and_load(batch.data[0], ctx_list=ctx, batch_axis=0)\n",
    "            label = gluon.utils.split_and_load(batch.label[0], ctx_list=ctx, batch_axis=0)\n",
    "            outputs = []\n",
    "            Ls = []\n",
    "            with autograd.record():\n",
    "                # forward pass \n",
    "                for x, y in zip(data, label):\n",
    "                    z = net(x)\n",
    "                    L = loss(z, y)\n",
    "                    Ls.append(L)\n",
    "                    outputs.append(z)\n",
    "                # backward pass\n",
    "                for L in Ls:\n",
    "                    L.backward()\n",
    "            # SGD step\n",
    "            trainer.step(batch.data[0].shape[0])\n",
    "            metric.update(label, outputs)\n",
    "            if log_interval and not (i+1)%log_interval:\n",
    "                name, acc = metric.get()\n",
    "                print('Epoch[%d] Batch [%d]\\tSpeed: %f samples/sec\\t%s=%f'%(\n",
    "                               epoch, i, batch_size/(time.time()-btic), name, acc))\n",
    "            btic = time.time()\n",
    "\n",
    "        name, acc = metric.get()\n",
    "        print('[Epoch %d] training: %s=%f'%(epoch, name, acc))\n",
    "        print('[Epoch %d] time cost: %f'%(epoch, time.time()-tic))\n",
    "        name, val_acc = test(net, val_data, context)\n",
    "        print('[Epoch %d] validation: %s=%f'%(epoch, name, val_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we define the iterators, employing the same data augmentation procedures as described in [3], Sec. 4.2. The CIFAR-10 data set should be downloaded automatically if not available locally (otherwise, you can donwload it manually from http://data.mxnet.io/mxnet/data/cifar10.zip, create a subfolder 'data' in the folder containing this notebook, and extract it there)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "mx.random.seed(321)\n",
    "np.random.seed(321)\n",
    "\n",
    "data_shape = (3, 32, 32)\n",
    "batch_size = 128\n",
    "\n",
    "# download CIFAR-10 if necessary\n",
    "import os.path\n",
    "if (not os.path.isfile(\"data/cifar/train.rec\")) or (not os.path.isfile(\"data/cifar/test.rec\")):\n",
    "    zip_file_path = mx.test_utils.download('http://data.mxnet.io/mxnet/data/cifar10.zip', dirname='data')\n",
    "    import zipfile\n",
    "    with zipfile.ZipFile(zip_file_path) as zf:\n",
    "        zf.extractall('data')\n",
    "\n",
    "# training set iterator\n",
    "train_data = mx.io.ImageRecordIter(\n",
    "    path_imgrec   = \"data/cifar/train.rec\",\n",
    "    data_shape    = data_shape,\n",
    "    batch_size    = batch_size,\n",
    "    mean_r        = 125.3,\n",
    "    mean_g        = 123.0,\n",
    "    mean_b        = 113.9,\n",
    "    std_r         = 63.0,\n",
    "    std_g         = 62.1,\n",
    "    std_b         = 66.7,\n",
    "    dtype         = 'float32',\n",
    "    rand_crop     = True,\n",
    "    max_crop_size = 32,\n",
    "    min_crop_size = 32,\n",
    "    pad           = 4,\n",
    "    fill_value    = 0,\n",
    "    shuffle       = True,\n",
    "    rand_mirror   = True,\n",
    "    shuffle_chunk_seed  = 123)\n",
    "\n",
    "# validation set iterator\n",
    "val_data = mx.io.ImageRecordIter(\n",
    "    mean_r      = 125.3,\n",
    "    mean_g      = 123.0,\n",
    "    mean_b      = 113.9,\n",
    "    std_r       = 63.0,\n",
    "    std_g       = 62.1,\n",
    "    std_b       = 66.7,\n",
    "    dtype       = 'float32',\n",
    "    path_imgrec = \"data/cifar/test.rec\",\n",
    "    rand_crop   = False,\n",
    "    rand_mirror = False,\n",
    "    data_shape  = data_shape,\n",
    "    batch_size  = batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify optimizer paramters, regularization, and learning rate schedule (150-50-50 epochs x0.1)... "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 250\n",
    "learning_rate = 0.1\n",
    "momentum = 0.9\n",
    "weight_decay = 0.0001\n",
    "\n",
    "train_set_size = 50000\n",
    "# introduce auxiliary variable to translate steps into epochs\n",
    "schedule_factor = ceil(train_set_size/batch_size)\n",
    "scheduler = mx.lr_scheduler.MultiFactorScheduler([150*schedule_factor, 200*schedule_factor, 250*schedule_factor], factor=0.1)\n",
    "\n",
    "log_interval = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and train the SP-ResNet-20 from scratch with full precision entries for $W_B$, $W_C$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[0] Batch [49]\tSpeed: 3995.050839 samples/sec\taccuracy=0.201719\n",
      "Epoch[0] Batch [99]\tSpeed: 3854.505270 samples/sec\taccuracy=0.241641\n",
      "Epoch[0] Batch [149]\tSpeed: 3657.756799 samples/sec\taccuracy=0.273281\n",
      "Epoch[0] Batch [199]\tSpeed: 3812.353803 samples/sec\taccuracy=0.295234\n",
      "Epoch[0] Batch [249]\tSpeed: 3524.694630 samples/sec\taccuracy=0.314375\n",
      "Epoch[0] Batch [299]\tSpeed: 3733.663291 samples/sec\taccuracy=0.330885\n",
      "Epoch[0] Batch [349]\tSpeed: 2852.707068 samples/sec\taccuracy=0.346763\n",
      "[Epoch 0] training: accuracy=0.358376\n",
      "[Epoch 0] time cost: 16.123975\n",
      "[Epoch 0] validation: accuracy=0.440961\n",
      "Epoch[1] Batch [49]\tSpeed: 3640.888889 samples/sec\taccuracy=0.485156\n",
      "Epoch[1] Batch [99]\tSpeed: 3708.799027 samples/sec\taccuracy=0.492422\n",
      "Epoch[1] Batch [149]\tSpeed: 3799.484165 samples/sec\taccuracy=0.505104\n",
      "Epoch[1] Batch [199]\tSpeed: 3762.656724 samples/sec\taccuracy=0.511836\n",
      "Epoch[1] Batch [249]\tSpeed: 3774.295661 samples/sec\taccuracy=0.518594\n",
      "Epoch[1] Batch [299]\tSpeed: 3988.017560 samples/sec\taccuracy=0.527552\n",
      "Epoch[1] Batch [349]\tSpeed: 1991.316632 samples/sec\taccuracy=0.536629\n",
      "[Epoch 1] training: accuracy=0.540801\n",
      "[Epoch 1] time cost: 14.418656\n",
      "[Epoch 1] validation: accuracy=0.535958\n",
      "Epoch[2] Batch [49]\tSpeed: 4031.470391 samples/sec\taccuracy=0.610938\n",
      "Epoch[2] Batch [99]\tSpeed: 3840.113529 samples/sec\taccuracy=0.613359\n",
      "Epoch[2] Batch [149]\tSpeed: 2664.067685 samples/sec\taccuracy=0.614740\n",
      "Epoch[2] Batch [199]\tSpeed: 3844.403237 samples/sec\taccuracy=0.617656\n",
      "Epoch[2] Batch [249]\tSpeed: 3393.364043 samples/sec\taccuracy=0.621875\n",
      "Epoch[2] Batch [299]\tSpeed: 3842.944761 samples/sec\taccuracy=0.625182\n",
      "Epoch[2] Batch [349]\tSpeed: 3740.244198 samples/sec\taccuracy=0.629375\n",
      "[Epoch 2] training: accuracy=0.632252\n",
      "[Epoch 2] time cost: 14.635556\n",
      "[Epoch 2] validation: accuracy=0.621595\n",
      "Epoch[3] Batch [49]\tSpeed: 3633.816235 samples/sec\taccuracy=0.660000\n",
      "Epoch[3] Batch [99]\tSpeed: 3693.591502 samples/sec\taccuracy=0.666328\n",
      "Epoch[3] Batch [149]\tSpeed: 2788.461781 samples/sec\taccuracy=0.666510\n",
      "Epoch[3] Batch [199]\tSpeed: 3807.163102 samples/sec\taccuracy=0.669336\n",
      "Epoch[3] Batch [249]\tSpeed: 3676.820272 samples/sec\taccuracy=0.670219\n",
      "Epoch[3] Batch [299]\tSpeed: 3938.631433 samples/sec\taccuracy=0.671693\n",
      "Epoch[3] Batch [349]\tSpeed: 3850.690078 samples/sec\taccuracy=0.676004\n",
      "[Epoch 3] training: accuracy=0.678808\n",
      "[Epoch 3] time cost: 14.540682\n",
      "[Epoch 3] validation: accuracy=0.668770\n",
      "Epoch[4] Batch [49]\tSpeed: 3906.333945 samples/sec\taccuracy=0.696406\n",
      "Epoch[4] Batch [99]\tSpeed: 2232.998561 samples/sec\taccuracy=0.702656\n",
      "Epoch[4] Batch [149]\tSpeed: 3726.613949 samples/sec\taccuracy=0.705729\n",
      "Epoch[4] Batch [199]\tSpeed: 3863.575869 samples/sec\taccuracy=0.707461\n",
      "Epoch[4] Batch [249]\tSpeed: 3205.754501 samples/sec\taccuracy=0.709469\n",
      "Epoch[4] Batch [299]\tSpeed: 3715.678202 samples/sec\taccuracy=0.710052\n",
      "Epoch[4] Batch [349]\tSpeed: 2825.353844 samples/sec\taccuracy=0.714107\n",
      "[Epoch 4] training: accuracy=0.715373\n",
      "[Epoch 4] time cost: 14.823233\n",
      "[Epoch 4] validation: accuracy=0.711138\n",
      "Epoch[5] Batch [49]\tSpeed: 3783.098885 samples/sec\taccuracy=0.738125\n",
      "Epoch[5] Batch [99]\tSpeed: 3732.702807 samples/sec\taccuracy=0.738359\n",
      "Epoch[5] Batch [149]\tSpeed: 3864.437989 samples/sec\taccuracy=0.737448\n",
      "Epoch[5] Batch [199]\tSpeed: 3347.680765 samples/sec\taccuracy=0.736367\n",
      "Epoch[5] Batch [249]\tSpeed: 3748.522657 samples/sec\taccuracy=0.737094\n",
      "Epoch[5] Batch [299]\tSpeed: 2742.579523 samples/sec\taccuracy=0.738255\n",
      "Epoch[5] Batch [349]\tSpeed: 2515.336522 samples/sec\taccuracy=0.741607\n",
      "[Epoch 5] training: accuracy=0.741767\n",
      "[Epoch 5] time cost: 14.617910\n",
      "[Epoch 5] validation: accuracy=0.746194\n",
      "Epoch[6] Batch [49]\tSpeed: 3024.232984 samples/sec\taccuracy=0.759531\n",
      "Epoch[6] Batch [99]\tSpeed: 3885.019987 samples/sec\taccuracy=0.759062\n",
      "Epoch[6] Batch [149]\tSpeed: 4083.848656 samples/sec\taccuracy=0.758698\n",
      "Epoch[6] Batch [199]\tSpeed: 3235.799513 samples/sec\taccuracy=0.759258\n",
      "Epoch[6] Batch [249]\tSpeed: 2201.987236 samples/sec\taccuracy=0.760969\n",
      "Epoch[6] Batch [299]\tSpeed: 3896.665713 samples/sec\taccuracy=0.760964\n",
      "Epoch[6] Batch [349]\tSpeed: 3767.541611 samples/sec\taccuracy=0.761540\n",
      "[Epoch 6] training: accuracy=0.761469\n",
      "[Epoch 6] time cost: 15.184485\n",
      "[Epoch 6] validation: accuracy=0.734575\n",
      "Epoch[7] Batch [49]\tSpeed: 1881.743790 samples/sec\taccuracy=0.769844\n",
      "Epoch[7] Batch [99]\tSpeed: 3921.141363 samples/sec\taccuracy=0.773359\n",
      "Epoch[7] Batch [149]\tSpeed: 3949.700295 samples/sec\taccuracy=0.772448\n",
      "Epoch[7] Batch [199]\tSpeed: 3800.963652 samples/sec\taccuracy=0.772266\n",
      "Epoch[7] Batch [249]\tSpeed: 3519.795659 samples/sec\taccuracy=0.772906\n",
      "Epoch[7] Batch [299]\tSpeed: 3608.027688 samples/sec\taccuracy=0.772943\n",
      "Epoch[7] Batch [349]\tSpeed: 3869.786152 samples/sec\taccuracy=0.774665\n",
      "[Epoch 7] training: accuracy=0.774980\n",
      "[Epoch 7] time cost: 14.602977\n",
      "[Epoch 7] validation: accuracy=0.729567\n",
      "Epoch[8] Batch [49]\tSpeed: 3858.466678 samples/sec\taccuracy=0.782188\n",
      "Epoch[8] Batch [99]\tSpeed: 3836.217109 samples/sec\taccuracy=0.782344\n",
      "Epoch[8] Batch [149]\tSpeed: 3879.153115 samples/sec\taccuracy=0.780937\n",
      "Epoch[8] Batch [199]\tSpeed: 1793.365664 samples/sec\taccuracy=0.779336\n",
      "Epoch[8] Batch [249]\tSpeed: 2684.663296 samples/sec\taccuracy=0.780844\n",
      "Epoch[8] Batch [299]\tSpeed: 3926.245709 samples/sec\taccuracy=0.780964\n",
      "Epoch[8] Batch [349]\tSpeed: 4085.775586 samples/sec\taccuracy=0.782879\n",
      "[Epoch 8] training: accuracy=0.782569\n",
      "[Epoch 8] time cost: 14.686705\n",
      "[Epoch 8] validation: accuracy=0.763350\n",
      "Epoch[9] Batch [49]\tSpeed: 2698.440419 samples/sec\taccuracy=0.792813\n",
      "Epoch[16] Batch [349]\tSpeed: 3809.945938 samples/sec\taccuracy=0.829754\n",
      "[Epoch 16] training: accuracy=0.829464\n",
      "[Epoch 16] time cost: 13.826482\n",
      "[Epoch 16] validation: accuracy=0.795787\n",
      "Epoch[17] Batch [49]\tSpeed: 3851.214909 samples/sec\taccuracy=0.825000\n",
      "Epoch[17] Batch [99]\tSpeed: 3755.077302 samples/sec\taccuracy=0.831484\n",
      "Epoch[17] Batch [149]\tSpeed: 3739.957590 samples/sec\taccuracy=0.828438\n",
      "Epoch[17] Batch [199]\tSpeed: 3879.461456 samples/sec\taccuracy=0.827070\n",
      "Epoch[17] Batch [249]\tSpeed: 2950.407562 samples/sec\taccuracy=0.827906\n",
      "Epoch[17] Batch [299]\tSpeed: 3579.950869 samples/sec\taccuracy=0.828359\n",
      "Epoch[17] Batch [349]\tSpeed: 3818.590495 samples/sec\taccuracy=0.829464\n",
      "[Epoch 17] training: accuracy=0.829803\n",
      "[Epoch 17] time cost: 13.992058\n",
      "[Epoch 17] validation: accuracy=0.800080\n",
      "Epoch[18] Batch [49]\tSpeed: 3835.531938 samples/sec\taccuracy=0.827031\n",
      "Epoch[18] Batch [99]\tSpeed: 3944.447880 samples/sec\taccuracy=0.832656\n",
      "Epoch[18] Batch [149]\tSpeed: 3735.351827 samples/sec\taccuracy=0.834688\n",
      "Epoch[18] Batch [199]\tSpeed: 3698.782705 samples/sec\taccuracy=0.833086\n",
      "Epoch[18] Batch [249]\tSpeed: 3524.578931 samples/sec\taccuracy=0.833281\n",
      "Epoch[18] Batch [299]\tSpeed: 3036.531481 samples/sec\taccuracy=0.833802\n",
      "Epoch[18] Batch [349]\tSpeed: 3870.232500 samples/sec\taccuracy=0.834442\n",
      "[Epoch 18] training: accuracy=0.834275\n",
      "[Epoch 18] time cost: 14.280908\n",
      "[Epoch 18] validation: accuracy=0.813101\n",
      "Epoch[19] Batch [49]\tSpeed: 3984.761577 samples/sec\taccuracy=0.837500\n",
      "Epoch[19] Batch [99]\tSpeed: 4100.912134 samples/sec\taccuracy=0.839141\n",
      "Epoch[19] Batch [149]\tSpeed: 3697.891020 samples/sec\taccuracy=0.836875\n",
      "Epoch[19] Batch [199]\tSpeed: 3995.169757 samples/sec\taccuracy=0.834805\n",
      "Epoch[19] Batch [249]\tSpeed: 3724.907459 samples/sec\taccuracy=0.832750\n",
      "Epoch[19] Batch [299]\tSpeed: 3494.434325 samples/sec\taccuracy=0.833620\n",
      "Epoch[19] Batch [349]\tSpeed: 3781.313650 samples/sec\taccuracy=0.834241\n",
      "[Epoch 19] training: accuracy=0.834719\n",
      "[Epoch 19] time cost: 13.666683\n",
      "[Epoch 19] validation: accuracy=0.790064\n",
      "Epoch[20] Batch [49]\tSpeed: 3616.339492 samples/sec\taccuracy=0.844219\n",
      "Epoch[20] Batch [99]\tSpeed: 3804.626972 samples/sec\taccuracy=0.839141\n",
      "Epoch[20] Batch [149]\tSpeed: 3699.012064 samples/sec\taccuracy=0.837865\n",
      "Epoch[20] Batch [199]\tSpeed: 1284.678291 samples/sec\taccuracy=0.837109\n",
      "Epoch[20] Batch [249]\tSpeed: 2443.765815 samples/sec\taccuracy=0.837562\n",
      "Epoch[20] Batch [299]\tSpeed: 3792.934487 samples/sec\taccuracy=0.837266\n",
      "Epoch[20] Batch [349]\tSpeed: 2505.581332 samples/sec\taccuracy=0.838080\n",
      "[Epoch 20] training: accuracy=0.838375\n",
      "[Epoch 20] time cost: 14.469571\n",
      "[Epoch 20] validation: accuracy=0.797175\n",
      "Epoch[21] Batch [49]\tSpeed: 3471.701815 samples/sec\taccuracy=0.843281\n",
      "Epoch[21] Batch [99]\tSpeed: 4069.053449 samples/sec\taccuracy=0.840938\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[21] Batch [149]\tSpeed: 3833.696887 samples/sec\taccuracy=0.839792\n",
      "Epoch[21] Batch [199]\tSpeed: 3585.856919 samples/sec\taccuracy=0.840742\n",
      "Epoch[21] Batch [249]\tSpeed: 2880.641470 samples/sec\taccuracy=0.839781\n",
      "Epoch[21] Batch [299]\tSpeed: 3884.176762 samples/sec\taccuracy=0.839922\n",
      "Epoch[21] Batch [349]\tSpeed: 3509.373077 samples/sec\taccuracy=0.840223\n",
      "[Epoch 21] training: accuracy=0.839423\n",
      "[Epoch 21] time cost: 13.661896\n",
      "[Epoch 21] validation: accuracy=0.792668\n",
      "Epoch[22] Batch [49]\tSpeed: 3322.611644 samples/sec\taccuracy=0.846250\n",
      "Epoch[22] Batch [99]\tSpeed: 3803.279343 samples/sec\taccuracy=0.845000\n",
      "Epoch[22] Batch [149]\tSpeed: 3834.217096 samples/sec\taccuracy=0.846250\n",
      "Epoch[22] Batch [199]\tSpeed: 3686.844429 samples/sec\taccuracy=0.843789\n",
      "Epoch[22] Batch [249]\tSpeed: 3614.489117 samples/sec\taccuracy=0.843375\n",
      "Epoch[22] Batch [299]\tSpeed: 3867.305216 samples/sec\taccuracy=0.844063\n",
      "Epoch[22] Batch [349]\tSpeed: 3826.210585 samples/sec\taccuracy=0.843616\n",
      "[Epoch 22] training: accuracy=0.843470\n",
      "[Epoch 22] time cost: 14.314837\n",
      "[Epoch 22] validation: accuracy=0.824018\n",
      "Epoch[23] Batch [49]\tSpeed: 3877.780192 samples/sec\taccuracy=0.844531\n",
      "Epoch[23] Batch [99]\tSpeed: 3947.231950 samples/sec\taccuracy=0.847578\n",
      "Epoch[23] Batch [149]\tSpeed: 3731.924398 samples/sec\taccuracy=0.844792\n",
      "Epoch[23] Batch [199]\tSpeed: 3120.326591 samples/sec\taccuracy=0.843906\n",
      "Epoch[23] Batch [249]\tSpeed: 2114.805671 samples/sec\taccuracy=0.843125\n",
      "Epoch[23] Batch [299]\tSpeed: 3806.839153 samples/sec\taccuracy=0.843021\n",
      "Epoch[23] Batch [349]\tSpeed: 3796.072291 samples/sec\taccuracy=0.844330\n",
      "[Epoch 23] training: accuracy=0.844331\n",
      "[Epoch 23] time cost: 13.955546\n",
      "[Epoch 23] validation: accuracy=0.811098\n",
      "Epoch[24] Batch [49]\tSpeed: 2721.585854 samples/sec\taccuracy=0.850781\n",
      "Epoch[24] Batch [99]\tSpeed: 3272.405900 samples/sec\taccuracy=0.850156\n",
      "Epoch[24] Batch [149]\tSpeed: 3056.533342 samples/sec\taccuracy=0.850938\n",
      "Epoch[24] Batch [199]\tSpeed: 3082.081807 samples/sec\taccuracy=0.849102\n",
      "Epoch[24] Batch [249]\tSpeed: 3570.475061 samples/sec\taccuracy=0.848500\n",
      "Epoch[24] Batch [299]\tSpeed: 3974.289801 samples/sec\taccuracy=0.847969\n",
      "Epoch[24] Batch [349]\tSpeed: 3858.189391 samples/sec\taccuracy=0.847634\n",
      "[Epoch 24] training: accuracy=0.847367\n",
      "[Epoch 24] time cost: 14.476196\n",
      "[Epoch 24] validation: accuracy=0.806171\n",
      "Epoch[25] Batch [49]\tSpeed: 3814.764714 samples/sec\taccuracy=0.849688\n",
      "Epoch[25] Batch [99]\tSpeed: 2905.287119 samples/sec\taccuracy=0.847656\n",
      "Epoch[25] Batch [149]\tSpeed: 3911.342795 samples/sec\taccuracy=0.845938\n",
      "Epoch[25] Batch [199]\tSpeed: 3885.301143 samples/sec\taccuracy=0.847500\n",
      "Epoch[25] Batch [249]\tSpeed: 3572.280635 samples/sec\taccuracy=0.846406\n",
      "Epoch[25] Batch [299]\tSpeed: 3608.051936 samples/sec\taccuracy=0.846771\n",
      "Epoch[25] Batch [349]\tSpeed: 3720.364448 samples/sec\taccuracy=0.847902\n",
      "[Epoch 25] training: accuracy=0.848046\n",
      "[Epoch 25] time cost: 14.085783\n",
      "[Epoch 25] validation: accuracy=0.782853\n",
      "Epoch[26] Batch [49]\tSpeed: 3646.948339 samples/sec\taccuracy=0.847187\n",
      "Epoch[26] Batch [99]\tSpeed: 3760.679971 samples/sec\taccuracy=0.848203\n",
      "Epoch[26] Batch [149]\tSpeed: 3897.882237 samples/sec\taccuracy=0.848385\n",
      "Epoch[26] Batch [199]\tSpeed: 2931.158069 samples/sec\taccuracy=0.847617\n",
      "Epoch[26] Batch [249]\tSpeed: 3730.213042 samples/sec\taccuracy=0.848500\n",
      "Epoch[26] Batch [299]\tSpeed: 3905.112141 samples/sec\taccuracy=0.849609\n",
      "Epoch[26] Batch [349]\tSpeed: 3824.466170 samples/sec\taccuracy=0.849554\n",
      "[Epoch 26] training: accuracy=0.848638\n",
      "[Epoch 26] time cost: 14.586091\n",
      "[Epoch 26] validation: accuracy=0.792869\n",
      "Epoch[27] Batch [49]\tSpeed: 3479.013407 samples/sec\taccuracy=0.857500\n",
      "Epoch[27] Batch [99]\tSpeed: 3774.906041 samples/sec\taccuracy=0.855000\n",
      "Epoch[27] Batch [149]\tSpeed: 3168.613808 samples/sec\taccuracy=0.855208\n",
      "Epoch[27] Batch [199]\tSpeed: 3997.103168 samples/sec\taccuracy=0.853789\n",
      "Epoch[27] Batch [249]\tSpeed: 3718.251600 samples/sec\taccuracy=0.852344\n",
      "Epoch[27] Batch [299]\tSpeed: 3894.602191 samples/sec\taccuracy=0.852760\n",
      "Epoch[27] Batch [349]\tSpeed: 3755.628936 samples/sec\taccuracy=0.853326\n",
      "[Epoch 27] training: accuracy=0.852562\n",
      "[Epoch 27] time cost: 13.921139\n",
      "[Epoch 27] validation: accuracy=0.793069\n",
      "Epoch[28] Batch [49]\tSpeed: 3819.079445 samples/sec\taccuracy=0.847969\n",
      "Epoch[28] Batch [99]\tSpeed: 3445.654748 samples/sec\taccuracy=0.849219\n",
      "Epoch[28] Batch [149]\tSpeed: 3849.834080 samples/sec\taccuracy=0.850208\n",
      "Epoch[28] Batch [199]\tSpeed: 3132.451788 samples/sec\taccuracy=0.850703\n",
      "Epoch[28] Batch [249]\tSpeed: 2621.849664 samples/sec\taccuracy=0.851812\n",
      "Epoch[28] Batch [299]\tSpeed: 3880.639209 samples/sec\taccuracy=0.852109\n",
      "Epoch[28] Batch [349]\tSpeed: 3818.101670 samples/sec\taccuracy=0.852165\n",
      "[Epoch 28] training: accuracy=0.851922\n",
      "[Epoch 28] time cost: 14.189382\n",
      "[Epoch 28] validation: accuracy=0.835537\n",
      "Epoch[29] Batch [49]\tSpeed: 3757.232221 samples/sec\taccuracy=0.856094\n",
      "Epoch[29] Batch [99]\tSpeed: 4130.793057 samples/sec\taccuracy=0.852812\n",
      "Epoch[29] Batch [149]\tSpeed: 3405.784959 samples/sec\taccuracy=0.851823\n",
      "Epoch[29] Batch [199]\tSpeed: 3810.621998 samples/sec\taccuracy=0.851211\n",
      "Epoch[29] Batch [249]\tSpeed: 3411.455027 samples/sec\taccuracy=0.850594\n",
      "Epoch[29] Batch [299]\tSpeed: 3877.696167 samples/sec\taccuracy=0.851875\n",
      "Epoch[29] Batch [349]\tSpeed: 3890.566275 samples/sec\taccuracy=0.853772\n",
      "[Epoch 29] training: accuracy=0.853305\n",
      "[Epoch 29] time cost: 13.854174\n",
      "[Epoch 29] validation: accuracy=0.818610\n",
      "Epoch[30] Batch [49]\tSpeed: 2406.715823 samples/sec\taccuracy=0.848750\n",
      "Epoch[30] Batch [99]\tSpeed: 3721.963562 samples/sec\taccuracy=0.852734\n",
      "Epoch[30] Batch [149]\tSpeed: 3825.774332 samples/sec\taccuracy=0.852760\n",
      "Epoch[30] Batch [199]\tSpeed: 4003.601214 samples/sec\taccuracy=0.853164\n",
      "Epoch[30] Batch [249]\tSpeed: 3691.331275 samples/sec\taccuracy=0.852719\n",
      "Epoch[30] Batch [299]\tSpeed: 3996.418824 samples/sec\taccuracy=0.852526\n",
      "Epoch[30] Batch [349]\tSpeed: 3613.807877 samples/sec\taccuracy=0.853058\n",
      "[Epoch 30] training: accuracy=0.853880\n",
      "[Epoch 30] time cost: 13.609262\n",
      "[Epoch 30] validation: accuracy=0.824018\n",
      "Epoch[31] Batch [49]\tSpeed: 3842.477183 samples/sec\taccuracy=0.862969\n",
      "Epoch[31] Batch [99]\tSpeed: 2612.841099 samples/sec\taccuracy=0.858203\n",
      "Epoch[31] Batch [149]\tSpeed: 3856.609620 samples/sec\taccuracy=0.855313\n",
      "Epoch[31] Batch [199]\tSpeed: 3798.758293 samples/sec\taccuracy=0.855234\n",
      "Epoch[31] Batch [249]\tSpeed: 3355.023822 samples/sec\taccuracy=0.855906\n",
      "Epoch[31] Batch [299]\tSpeed: 3907.784052 samples/sec\taccuracy=0.856458\n",
      "Epoch[31] Batch [349]\tSpeed: 3760.785346 samples/sec\taccuracy=0.856406\n",
      "[Epoch 31] training: accuracy=0.856310\n",
      "[Epoch 31] time cost: 13.973273\n",
      "[Epoch 31] validation: accuracy=0.829527\n",
      "Epoch[32] Batch [49]\tSpeed: 3822.042985 samples/sec\taccuracy=0.861094\n",
      "Epoch[32] Batch [99]\tSpeed: 3719.230426 samples/sec\taccuracy=0.858516\n",
      "Epoch[32] Batch [149]\tSpeed: 3919.681327 samples/sec\taccuracy=0.857240\n",
      "Epoch[32] Batch [199]\tSpeed: 3893.359479 samples/sec\taccuracy=0.855117\n",
      "Epoch[32] Batch [249]\tSpeed: 2573.451661 samples/sec\taccuracy=0.855656\n",
      "Epoch[32] Batch [299]\tSpeed: 3761.628553 samples/sec\taccuracy=0.855859\n",
      "Epoch[32] Batch [349]\tSpeed: 3792.264689 samples/sec\taccuracy=0.856563\n",
      "[Epoch 32] training: accuracy=0.856937\n",
      "[Epoch 32] time cost: 13.985967\n",
      "[Epoch 32] validation: accuracy=0.817544\n",
      "Epoch[33] Batch [49]\tSpeed: 3834.299247 samples/sec\taccuracy=0.863437\n",
      "Epoch[33] Batch [99]\tSpeed: 3812.462093 samples/sec\taccuracy=0.859297\n",
      "Epoch[33] Batch [149]\tSpeed: 3593.633736 samples/sec\taccuracy=0.860365\n",
      "Epoch[33] Batch [199]\tSpeed: 3551.391210 samples/sec\taccuracy=0.857539\n",
      "Epoch[33] Batch [249]\tSpeed: 3853.979541 samples/sec\taccuracy=0.857875\n",
      "Epoch[33] Batch [299]\tSpeed: 3593.032472 samples/sec\taccuracy=0.857500\n",
      "Epoch[33] Batch [349]\tSpeed: 3891.807204 samples/sec\taccuracy=0.857879\n",
      "[Epoch 33] training: accuracy=0.857677\n",
      "[Epoch 33] time cost: 13.777997\n",
      "[Epoch 33] validation: accuracy=0.829227\n",
      "Epoch[34] Batch [49]\tSpeed: 3865.717972 samples/sec\taccuracy=0.855781\n",
      "Epoch[34] Batch [99]\tSpeed: 1954.987736 samples/sec\taccuracy=0.855234\n",
      "Epoch[34] Batch [149]\tSpeed: 3831.699500 samples/sec\taccuracy=0.856771\n",
      "Epoch[34] Batch [199]\tSpeed: 3946.071443 samples/sec\taccuracy=0.856445\n",
      "Epoch[34] Batch [249]\tSpeed: 3758.573723 samples/sec\taccuracy=0.857062\n",
      "Epoch[34] Batch [299]\tSpeed: 3605.023482 samples/sec\taccuracy=0.858724\n",
      "Epoch[34] Batch [349]\tSpeed: 3952.957420 samples/sec\taccuracy=0.859375\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Epoch 34] training: accuracy=0.859034\n",
      "[Epoch 34] time cost: 14.262979\n",
      "[Epoch 34] validation: accuracy=0.842748\n",
      "Epoch[35] Batch [49]\tSpeed: 3935.542106 samples/sec\taccuracy=0.868594\n",
      "Epoch[35] Batch [99]\tSpeed: 3804.600010 samples/sec\taccuracy=0.865156\n",
      "Epoch[35] Batch [149]\tSpeed: 3641.012350 samples/sec\taccuracy=0.865833\n",
      "Epoch[35] Batch [199]\tSpeed: 3784.085483 samples/sec\taccuracy=0.862773\n",
      "Epoch[35] Batch [249]\tSpeed: 3797.334239 samples/sec\taccuracy=0.862156\n",
      "Epoch[35] Batch [299]\tSpeed: 3832.930519 samples/sec\taccuracy=0.862083\n",
      "Epoch[35] Batch [349]\tSpeed: 3739.775225 samples/sec\taccuracy=0.862634\n",
      "[Epoch 35] training: accuracy=0.861533\n",
      "[Epoch 35] time cost: 14.051878\n",
      "[Epoch 35] validation: accuracy=0.837640\n",
      "Epoch[36] Batch [49]\tSpeed: 3774.799873 samples/sec\taccuracy=0.857656\n",
      "Epoch[36] Batch [99]\tSpeed: 3826.537840 samples/sec\taccuracy=0.859375\n",
      "Epoch[36] Batch [149]\tSpeed: 3756.758978 samples/sec\taccuracy=0.860625\n",
      "Epoch[36] Batch [199]\tSpeed: 3480.908184 samples/sec\taccuracy=0.861680\n",
      "Epoch[36] Batch [249]\tSpeed: 3714.958288 samples/sec\taccuracy=0.862219\n",
      "Epoch[36] Batch [299]\tSpeed: 3474.690225 samples/sec\taccuracy=0.863073\n",
      "Epoch[36] Batch [349]\tSpeed: 3577.612965 samples/sec\taccuracy=0.863125\n",
      "[Epoch 36] training: accuracy=0.862292\n",
      "[Epoch 36] time cost: 13.989904\n",
      "[Epoch 36] validation: accuracy=0.842748\n",
      "Epoch[37] Batch [49]\tSpeed: 3598.234042 samples/sec\taccuracy=0.866875\n",
      "Epoch[37] Batch [99]\tSpeed: 3673.826160 samples/sec\taccuracy=0.864609\n",
      "Epoch[37] Batch [149]\tSpeed: 3327.492265 samples/sec\taccuracy=0.864844\n",
      "Epoch[37] Batch [199]\tSpeed: 3891.017430 samples/sec\taccuracy=0.863164\n",
      "Epoch[37] Batch [249]\tSpeed: 3557.934126 samples/sec\taccuracy=0.864156\n",
      "Epoch[37] Batch [299]\tSpeed: 3808.216319 samples/sec\taccuracy=0.864766\n",
      "Epoch[37] Batch [349]\tSpeed: 4038.171870 samples/sec\taccuracy=0.865469\n",
      "[Epoch 37] training: accuracy=0.864924\n",
      "[Epoch 37] time cost: 13.905051\n",
      "[Epoch 37] validation: accuracy=0.843149\n",
      "Epoch[38] Batch [49]\tSpeed: 3763.448007 samples/sec\taccuracy=0.865938\n",
      "Epoch[38] Batch [99]\tSpeed: 3224.837290 samples/sec\taccuracy=0.860078\n",
      "Epoch[38] Batch [149]\tSpeed: 3731.224108 samples/sec\taccuracy=0.862708\n",
      "Epoch[38] Batch [199]\tSpeed: 3824.220989 samples/sec\taccuracy=0.863828\n",
      "Epoch[38] Batch [249]\tSpeed: 3613.661930 samples/sec\taccuracy=0.862781\n",
      "Epoch[38] Batch [299]\tSpeed: 3557.462608 samples/sec\taccuracy=0.862891\n",
      "Epoch[38] Batch [349]\tSpeed: 3801.824975 samples/sec\taccuracy=0.864732\n",
      "[Epoch 38] training: accuracy=0.864630\n",
      "[Epoch 38] time cost: 14.075155\n",
      "[Epoch 38] validation: accuracy=0.829026\n",
      "Epoch[39] Batch [49]\tSpeed: 2889.540856 samples/sec\taccuracy=0.866094\n",
      "Epoch[39] Batch [99]\tSpeed: 3778.333136 samples/sec\taccuracy=0.862969\n",
      "Epoch[39] Batch [149]\tSpeed: 3845.587342 samples/sec\taccuracy=0.865729\n",
      "Epoch[39] Batch [199]\tSpeed: 4035.197426 samples/sec\taccuracy=0.864766\n",
      "Epoch[39] Batch [249]\tSpeed: 3714.315744 samples/sec\taccuracy=0.865313\n",
      "Epoch[39] Batch [299]\tSpeed: 4024.006776 samples/sec\taccuracy=0.864271\n",
      "Epoch[39] Batch [349]\tSpeed: 3809.594482 samples/sec\taccuracy=0.864888\n",
      "[Epoch 39] training: accuracy=0.864944\n",
      "[Epoch 39] time cost: 14.156489\n",
      "[Epoch 39] validation: accuracy=0.841647\n",
      "Epoch[40] Batch [49]\tSpeed: 3896.241523 samples/sec\taccuracy=0.870313\n",
      "Epoch[40] Batch [99]\tSpeed: 3708.107388 samples/sec\taccuracy=0.866172\n",
      "Epoch[40] Batch [149]\tSpeed: 3666.199429 samples/sec\taccuracy=0.866458\n",
      "Epoch[40] Batch [199]\tSpeed: 4015.849680 samples/sec\taccuracy=0.865352\n",
      "Epoch[40] Batch [249]\tSpeed: 3812.705769 samples/sec\taccuracy=0.866500\n",
      "Epoch[40] Batch [299]\tSpeed: 3968.267749 samples/sec\taccuracy=0.867109\n",
      "Epoch[40] Batch [349]\tSpeed: 3891.750781 samples/sec\taccuracy=0.867545\n",
      "[Epoch 40] training: accuracy=0.867387\n",
      "[Epoch 40] time cost: 13.758765\n",
      "[Epoch 40] validation: accuracy=0.840091\n",
      "Epoch[41] Batch [49]\tSpeed: 3892.597298 samples/sec\taccuracy=0.869844\n",
      "Epoch[41] Batch [99]\tSpeed: 3932.054403 samples/sec\taccuracy=0.868672\n",
      "Epoch[41] Batch [149]\tSpeed: 2741.249187 samples/sec\taccuracy=0.869062\n",
      "Epoch[41] Batch [199]\tSpeed: 3650.395126 samples/sec\taccuracy=0.867148\n",
      "Epoch[41] Batch [249]\tSpeed: 2803.196057 samples/sec\taccuracy=0.866656\n",
      "Epoch[41] Batch [299]\tSpeed: 3779.423672 samples/sec\taccuracy=0.864948\n",
      "Epoch[41] Batch [349]\tSpeed: 3870.678952 samples/sec\taccuracy=0.865603\n",
      "[Epoch 41] training: accuracy=0.865829\n",
      "[Epoch 41] time cost: 13.894866\n",
      "[Epoch 41] validation: accuracy=0.835537\n",
      "Epoch[42] Batch [49]\tSpeed: 3802.498155 samples/sec\taccuracy=0.862812\n",
      "Epoch[42] Batch [99]\tSpeed: 3176.656975 samples/sec\taccuracy=0.865313\n",
      "Epoch[42] Batch [149]\tSpeed: 3757.416292 samples/sec\taccuracy=0.865521\n",
      "Epoch[42] Batch [199]\tSpeed: 3830.933931 samples/sec\taccuracy=0.866406\n",
      "Epoch[42] Batch [249]\tSpeed: 3751.010725 samples/sec\taccuracy=0.864938\n",
      "Epoch[42] Batch [299]\tSpeed: 4012.218251 samples/sec\taccuracy=0.864870\n",
      "Epoch[42] Batch [349]\tSpeed: 3831.972991 samples/sec\taccuracy=0.866094\n",
      "[Epoch 42] training: accuracy=0.866086\n",
      "[Epoch 42] time cost: 13.921444\n",
      "[Epoch 42] validation: accuracy=0.829728\n",
      "Epoch[43] Batch [49]\tSpeed: 3801.609608 samples/sec\taccuracy=0.874375\n",
      "Epoch[43] Batch [99]\tSpeed: 3813.843332 samples/sec\taccuracy=0.867031\n",
      "Epoch[43] Batch [149]\tSpeed: 3601.106161 samples/sec\taccuracy=0.865885\n",
      "Epoch[43] Batch [199]\tSpeed: 2547.998424 samples/sec\taccuracy=0.866680\n",
      "Epoch[43] Batch [249]\tSpeed: 3874.841483 samples/sec\taccuracy=0.864062\n",
      "Epoch[43] Batch [299]\tSpeed: 3033.683178 samples/sec\taccuracy=0.865000\n",
      "Epoch[43] Batch [349]\tSpeed: 3621.511093 samples/sec\taccuracy=0.866049\n",
      "[Epoch 43] training: accuracy=0.866268\n",
      "[Epoch 43] time cost: 14.318908\n",
      "[Epoch 43] validation: accuracy=0.826422\n",
      "Epoch[44] Batch [49]\tSpeed: 3837.341320 samples/sec\taccuracy=0.871875\n",
      "Epoch[44] Batch [99]\tSpeed: 3819.432653 samples/sec\taccuracy=0.867812\n",
      "Epoch[44] Batch [149]\tSpeed: 3506.920236 samples/sec\taccuracy=0.869531\n",
      "Epoch[44] Batch [199]\tSpeed: 3591.253910 samples/sec\taccuracy=0.867539\n",
      "Epoch[44] Batch [249]\tSpeed: 3685.047684 samples/sec\taccuracy=0.865500\n",
      "Epoch[44] Batch [299]\tSpeed: 3678.130156 samples/sec\taccuracy=0.867083\n",
      "Epoch[44] Batch [349]\tSpeed: 3855.750989 samples/sec\taccuracy=0.867768\n",
      "[Epoch 44] training: accuracy=0.867407\n",
      "[Epoch 44] time cost: 14.243127\n",
      "[Epoch 44] validation: accuracy=0.826823\n",
      "Epoch[45] Batch [49]\tSpeed: 3993.089714 samples/sec\taccuracy=0.873437\n",
      "Epoch[45] Batch [99]\tSpeed: 3637.854383 samples/sec\taccuracy=0.871016\n",
      "Epoch[45] Batch [149]\tSpeed: 3797.683436 samples/sec\taccuracy=0.871615\n",
      "Epoch[45] Batch [199]\tSpeed: 3837.478463 samples/sec\taccuracy=0.870195\n",
      "Epoch[45] Batch [249]\tSpeed: 2916.825557 samples/sec\taccuracy=0.870313\n",
      "Epoch[45] Batch [299]\tSpeed: 2152.002854 samples/sec\taccuracy=0.869818\n",
      "Epoch[45] Batch [349]\tSpeed: 3812.624540 samples/sec\taccuracy=0.870893\n",
      "[Epoch 45] training: accuracy=0.870192\n",
      "[Epoch 45] time cost: 14.369019\n",
      "[Epoch 45] validation: accuracy=0.825921\n",
      "Epoch[46] Batch [49]\tSpeed: 3669.105890 samples/sec\taccuracy=0.866563\n",
      "Epoch[46] Batch [99]\tSpeed: 3921.656930 samples/sec\taccuracy=0.867656\n",
      "Epoch[46] Batch [149]\tSpeed: 2935.068075 samples/sec\taccuracy=0.866667\n",
      "Epoch[46] Batch [199]\tSpeed: 3687.097632 samples/sec\taccuracy=0.866992\n",
      "Epoch[46] Batch [249]\tSpeed: 3838.164330 samples/sec\taccuracy=0.868156\n",
      "Epoch[46] Batch [299]\tSpeed: 3943.289009 samples/sec\taccuracy=0.869036\n",
      "Epoch[46] Batch [349]\tSpeed: 3679.920160 samples/sec\taccuracy=0.869509\n",
      "[Epoch 46] training: accuracy=0.869565\n",
      "[Epoch 46] time cost: 13.788782\n",
      "[Epoch 46] validation: accuracy=0.814002\n",
      "Epoch[47] Batch [49]\tSpeed: 3704.065185 samples/sec\taccuracy=0.869219\n",
      "Epoch[47] Batch [99]\tSpeed: 3733.481540 samples/sec\taccuracy=0.869062\n",
      "Epoch[47] Batch [149]\tSpeed: 3952.637286 samples/sec\taccuracy=0.868906\n",
      "Epoch[47] Batch [199]\tSpeed: 3909.149843 samples/sec\taccuracy=0.869102\n",
      "Epoch[47] Batch [249]\tSpeed: 3084.809046 samples/sec\taccuracy=0.868844\n",
      "Epoch[47] Batch [299]\tSpeed: 3759.863520 samples/sec\taccuracy=0.869167\n",
      "Epoch[47] Batch [349]\tSpeed: 2486.791416 samples/sec\taccuracy=0.871071\n",
      "[Epoch 47] training: accuracy=0.870793\n",
      "[Epoch 47] time cost: 14.151264\n",
      "[Epoch 47] validation: accuracy=0.843149\n",
      "Epoch[48] Batch [49]\tSpeed: 3516.982607 samples/sec\taccuracy=0.869687\n",
      "Epoch[48] Batch [99]\tSpeed: 3646.948339 samples/sec\taccuracy=0.872109\n",
      "Epoch[48] Batch [149]\tSpeed: 3908.011618 samples/sec\taccuracy=0.871667\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[48] Batch [199]\tSpeed: 3579.354175 samples/sec\taccuracy=0.872227\n",
      "Epoch[48] Batch [249]\tSpeed: 2934.233921 samples/sec\taccuracy=0.873031\n",
      "Epoch[48] Batch [299]\tSpeed: 3870.093005 samples/sec\taccuracy=0.873516\n",
      "Epoch[48] Batch [349]\tSpeed: 3782.032870 samples/sec\taccuracy=0.873638\n",
      "[Epoch 48] training: accuracy=0.873681\n",
      "[Epoch 48] time cost: 14.680833\n",
      "[Epoch 48] validation: accuracy=0.832971\n",
      "Epoch[49] Batch [49]\tSpeed: 3800.344817 samples/sec\taccuracy=0.875469\n",
      "Epoch[49] Batch [99]\tSpeed: 3724.829928 samples/sec\taccuracy=0.870625\n",
      "Epoch[49] Batch [149]\tSpeed: 3921.427772 samples/sec\taccuracy=0.870052\n",
      "Epoch[49] Batch [199]\tSpeed: 3927.998010 samples/sec\taccuracy=0.868242\n",
      "Epoch[49] Batch [249]\tSpeed: 3784.805758 samples/sec\taccuracy=0.868656\n",
      "Epoch[49] Batch [299]\tSpeed: 3837.341320 samples/sec\taccuracy=0.868620\n",
      "Epoch[49] Batch [349]\tSpeed: 3914.850930 samples/sec\taccuracy=0.871116\n",
      "[Epoch 49] training: accuracy=0.870584\n",
      "[Epoch 49] time cost: 13.562549\n",
      "[Epoch 49] validation: accuracy=0.838341\n",
      "Epoch[50] Batch [49]\tSpeed: 4062.741019 samples/sec\taccuracy=0.878437\n",
      "Epoch[50] Batch [99]\tSpeed: 3790.791965 samples/sec\taccuracy=0.872891\n",
      "Epoch[50] Batch [149]\tSpeed: 3868.364103 samples/sec\taccuracy=0.871979\n",
      "Epoch[50] Batch [199]\tSpeed: 3908.922145 samples/sec\taccuracy=0.870703\n",
      "Epoch[50] Batch [249]\tSpeed: 3680.222868 samples/sec\taccuracy=0.867906\n",
      "Epoch[50] Batch [299]\tSpeed: 2383.518742 samples/sec\taccuracy=0.869036\n",
      "Epoch[50] Batch [349]\tSpeed: 3919.023235 samples/sec\taccuracy=0.870134\n",
      "[Epoch 50] training: accuracy=0.870553\n",
      "[Epoch 50] time cost: 13.810094\n",
      "[Epoch 50] validation: accuracy=0.835737\n",
      "Epoch[51] Batch [49]\tSpeed: 3271.867436 samples/sec\taccuracy=0.871250\n",
      "Epoch[51] Batch [99]\tSpeed: 3686.211572 samples/sec\taccuracy=0.869687\n",
      "Epoch[51] Batch [149]\tSpeed: 3908.666016 samples/sec\taccuracy=0.872292\n",
      "Epoch[51] Batch [199]\tSpeed: 3772.704295 samples/sec\taccuracy=0.870469\n",
      "Epoch[51] Batch [249]\tSpeed: 3791.836142 samples/sec\taccuracy=0.870812\n",
      "Epoch[51] Batch [299]\tSpeed: 3686.844429 samples/sec\taccuracy=0.872109\n",
      "Epoch[51] Batch [349]\tSpeed: 3848.095645 samples/sec\taccuracy=0.873192\n",
      "[Epoch 51] training: accuracy=0.873541\n",
      "[Epoch 51] time cost: 13.921997\n",
      "[Epoch 51] validation: accuracy=0.831130\n",
      "Epoch[52] Batch [49]\tSpeed: 3789.186661 samples/sec\taccuracy=0.870000\n",
      "Epoch[52] Batch [99]\tSpeed: 3956.978058 samples/sec\taccuracy=0.869844\n",
      "Epoch[52] Batch [149]\tSpeed: 3683.404311 samples/sec\taccuracy=0.873698\n",
      "Epoch[52] Batch [199]\tSpeed: 3992.970919 samples/sec\taccuracy=0.874648\n",
      "Epoch[52] Batch [249]\tSpeed: 2867.792935 samples/sec\taccuracy=0.874906\n",
      "Epoch[52] Batch [299]\tSpeed: 3801.582688 samples/sec\taccuracy=0.875182\n",
      "Epoch[52] Batch [349]\tSpeed: 3199.964905 samples/sec\taccuracy=0.875871\n",
      "[Epoch 52] training: accuracy=0.875040\n",
      "[Epoch 52] time cost: 14.241928\n",
      "[Epoch 52] validation: accuracy=0.799379\n",
      "Epoch[53] Batch [49]\tSpeed: 4042.520007 samples/sec\taccuracy=0.874844\n",
      "Epoch[53] Batch [99]\tSpeed: 3939.932132 samples/sec\taccuracy=0.870469\n",
      "Epoch[53] Batch [149]\tSpeed: 3836.820262 samples/sec\taccuracy=0.871979\n",
      "Epoch[53] Batch [199]\tSpeed: 3605.071897 samples/sec\taccuracy=0.871758\n",
      "Epoch[53] Batch [249]\tSpeed: 3542.346244 samples/sec\taccuracy=0.873281\n",
      "Epoch[53] Batch [299]\tSpeed: 3684.744182 samples/sec\taccuracy=0.873516\n",
      "Epoch[53] Batch [349]\tSpeed: 3839.234772 samples/sec\taccuracy=0.873929\n",
      "[Epoch 53] training: accuracy=0.873237\n",
      "[Epoch 53] time cost: 14.001139\n",
      "[Epoch 53] validation: accuracy=0.843750\n",
      "Epoch[54] Batch [49]\tSpeed: 3782.432555 samples/sec\taccuracy=0.877344\n",
      "Epoch[54] Batch [99]\tSpeed: 3783.978799 samples/sec\taccuracy=0.876563\n",
      "Epoch[54] Batch [149]\tSpeed: 3552.025618 samples/sec\taccuracy=0.874062\n",
      "Epoch[54] Batch [199]\tSpeed: 3836.902525 samples/sec\taccuracy=0.872969\n",
      "Epoch[54] Batch [249]\tSpeed: 3730.420395 samples/sec\taccuracy=0.873938\n",
      "Epoch[54] Batch [299]\tSpeed: 3956.803078 samples/sec\taccuracy=0.873307\n",
      "Epoch[54] Batch [349]\tSpeed: 3007.461148 samples/sec\taccuracy=0.874576\n",
      "[Epoch 54] training: accuracy=0.874381\n",
      "[Epoch 54] time cost: 14.379252\n",
      "[Epoch 54] validation: accuracy=0.834135\n",
      "Epoch[55] Batch [49]\tSpeed: 3809.270119 samples/sec\taccuracy=0.877188\n",
      "Epoch[55] Batch [99]\tSpeed: 2993.909871 samples/sec\taccuracy=0.874219\n",
      "Epoch[55] Batch [149]\tSpeed: 3786.701124 samples/sec\taccuracy=0.876146\n",
      "Epoch[55] Batch [199]\tSpeed: 3627.849337 samples/sec\taccuracy=0.875625\n",
      "Epoch[55] Batch [249]\tSpeed: 3604.466800 samples/sec\taccuracy=0.874344\n",
      "Epoch[55] Batch [299]\tSpeed: 3851.353047 samples/sec\taccuracy=0.875078\n",
      "Epoch[55] Batch [349]\tSpeed: 3818.753464 samples/sec\taccuracy=0.876585\n",
      "[Epoch 55] training: accuracy=0.875821\n",
      "[Epoch 55] time cost: 14.035908\n",
      "[Epoch 55] validation: accuracy=0.836238\n",
      "Epoch[56] Batch [49]\tSpeed: 3137.944427 samples/sec\taccuracy=0.870469\n",
      "Epoch[56] Batch [99]\tSpeed: 3842.532186 samples/sec\taccuracy=0.874687\n",
      "Epoch[56] Batch [149]\tSpeed: 3902.869422 samples/sec\taccuracy=0.874219\n",
      "Epoch[56] Batch [199]\tSpeed: 3959.604623 samples/sec\taccuracy=0.875938\n",
      "Epoch[56] Batch [249]\tSpeed: 3736.391685 samples/sec\taccuracy=0.874719\n",
      "Epoch[56] Batch [299]\tSpeed: 3783.498795 samples/sec\taccuracy=0.875234\n",
      "Epoch[56] Batch [349]\tSpeed: 3612.786498 samples/sec\taccuracy=0.876696\n",
      "[Epoch 56] training: accuracy=0.876339\n",
      "[Epoch 56] time cost: 13.543413\n",
      "[Epoch 56] validation: accuracy=0.822290\n",
      "Epoch[57] Batch [49]\tSpeed: 2761.553796 samples/sec\taccuracy=0.872656\n",
      "Epoch[57] Batch [99]\tSpeed: 3585.090664 samples/sec\taccuracy=0.871484\n",
      "Epoch[57] Batch [149]\tSpeed: 3801.663447 samples/sec\taccuracy=0.872240\n",
      "Epoch[57] Batch [199]\tSpeed: 2855.149610 samples/sec\taccuracy=0.873594\n",
      "Epoch[57] Batch [249]\tSpeed: 3649.303352 samples/sec\taccuracy=0.874219\n",
      "Epoch[57] Batch [299]\tSpeed: 3601.855113 samples/sec\taccuracy=0.875104\n",
      "Epoch[57] Batch [349]\tSpeed: 3765.269222 samples/sec\taccuracy=0.875290\n",
      "[Epoch 57] training: accuracy=0.874361\n",
      "[Epoch 57] time cost: 14.469441\n",
      "[Epoch 57] validation: accuracy=0.820713\n",
      "Epoch[58] Batch [49]\tSpeed: 2553.621887 samples/sec\taccuracy=0.878281\n",
      "Epoch[58] Batch [99]\tSpeed: 3831.699500 samples/sec\taccuracy=0.875703\n",
      "Epoch[58] Batch [149]\tSpeed: 3878.032289 samples/sec\taccuracy=0.876250\n",
      "Epoch[58] Batch [199]\tSpeed: 3897.005132 samples/sec\taccuracy=0.875977\n",
      "Epoch[58] Batch [249]\tSpeed: 3783.792116 samples/sec\taccuracy=0.876406\n",
      "Epoch[58] Batch [299]\tSpeed: 4003.034030 samples/sec\taccuracy=0.876953\n",
      "Epoch[58] Batch [349]\tSpeed: 3860.047971 samples/sec\taccuracy=0.877567\n",
      "[Epoch 58] training: accuracy=0.877163\n",
      "[Epoch 58] time cost: 14.120548\n",
      "[Epoch 58] validation: accuracy=0.849960\n",
      "Epoch[59] Batch [49]\tSpeed: 2431.733885 samples/sec\taccuracy=0.878750\n",
      "Epoch[59] Batch [99]\tSpeed: 3827.219801 samples/sec\taccuracy=0.875781\n",
      "Epoch[59] Batch [149]\tSpeed: 2860.138683 samples/sec\taccuracy=0.875677\n",
      "Epoch[59] Batch [199]\tSpeed: 3932.976169 samples/sec\taccuracy=0.876523\n",
      "Epoch[59] Batch [249]\tSpeed: 2903.762883 samples/sec\taccuracy=0.876594\n",
      "Epoch[59] Batch [299]\tSpeed: 3652.456388 samples/sec\taccuracy=0.875729\n",
      "Epoch[59] Batch [349]\tSpeed: 3996.835353 samples/sec\taccuracy=0.877054\n",
      "[Epoch 59] training: accuracy=0.877478\n",
      "[Epoch 59] time cost: 14.736687\n",
      "[Epoch 59] validation: accuracy=0.837340\n",
      "Epoch[60] Batch [49]\tSpeed: 3640.321076 samples/sec\taccuracy=0.871719\n",
      "Epoch[60] Batch [99]\tSpeed: 3687.198923 samples/sec\taccuracy=0.872969\n",
      "Epoch[60] Batch [149]\tSpeed: 4189.231103 samples/sec\taccuracy=0.874167\n",
      "Epoch[60] Batch [199]\tSpeed: 3730.964808 samples/sec\taccuracy=0.874297\n",
      "Epoch[60] Batch [249]\tSpeed: 3744.418025 samples/sec\taccuracy=0.874469\n",
      "Epoch[60] Batch [299]\tSpeed: 3917.450433 samples/sec\taccuracy=0.875260\n",
      "Epoch[60] Batch [349]\tSpeed: 3876.296287 samples/sec\taccuracy=0.876696\n",
      "[Epoch 60] training: accuracy=0.876918\n",
      "[Epoch 60] time cost: 13.505339\n",
      "[Epoch 60] validation: accuracy=0.816907\n",
      "Epoch[61] Batch [49]\tSpeed: 3688.972419 samples/sec\taccuracy=0.880156\n",
      "Epoch[61] Batch [99]\tSpeed: 3997.400782 samples/sec\taccuracy=0.877500\n",
      "Epoch[61] Batch [149]\tSpeed: 4005.034816 samples/sec\taccuracy=0.875833\n",
      "Epoch[61] Batch [199]\tSpeed: 3380.650173 samples/sec\taccuracy=0.875273\n",
      "Epoch[61] Batch [249]\tSpeed: 3802.390430 samples/sec\taccuracy=0.875250\n",
      "Epoch[61] Batch [299]\tSpeed: 4052.804143 samples/sec\taccuracy=0.877109\n",
      "Epoch[61] Batch [349]\tSpeed: 4021.474835 samples/sec\taccuracy=0.878415\n",
      "[Epoch 61] training: accuracy=0.877684\n",
      "[Epoch 61] time cost: 13.523559\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Epoch 61] validation: accuracy=0.825621\n",
      "Epoch[62] Batch [49]\tSpeed: 3851.325418 samples/sec\taccuracy=0.880469\n",
      "Epoch[62] Batch [99]\tSpeed: 3926.733898 samples/sec\taccuracy=0.877812\n",
      "Epoch[62] Batch [149]\tSpeed: 3781.127230 samples/sec\taccuracy=0.878646\n",
      "Epoch[62] Batch [199]\tSpeed: 3224.721070 samples/sec\taccuracy=0.879687\n",
      "Epoch[62] Batch [249]\tSpeed: 3846.055677 samples/sec\taccuracy=0.879906\n",
      "Epoch[62] Batch [299]\tSpeed: 3935.022883 samples/sec\taccuracy=0.880208\n",
      "Epoch[62] Batch [349]\tSpeed: 3929.924472 samples/sec\taccuracy=0.880469\n",
      "[Epoch 62] training: accuracy=0.880874\n",
      "[Epoch 62] time cost: 13.465948\n",
      "[Epoch 62] validation: accuracy=0.814503\n",
      "Epoch[63] Batch [49]\tSpeed: 3823.757956 samples/sec\taccuracy=0.872656\n",
      "Epoch[63] Batch [99]\tSpeed: 1714.590657 samples/sec\taccuracy=0.878516\n",
      "Epoch[63] Batch [149]\tSpeed: 3696.337969 samples/sec\taccuracy=0.878542\n",
      "Epoch[63] Batch [199]\tSpeed: 4061.941818 samples/sec\taccuracy=0.878477\n",
      "Epoch[63] Batch [249]\tSpeed: 3592.743937 samples/sec\taccuracy=0.879031\n",
      "Epoch[63] Batch [299]\tSpeed: 4070.133141 samples/sec\taccuracy=0.879141\n",
      "Epoch[63] Batch [349]\tSpeed: 3961.649918 samples/sec\taccuracy=0.880223\n",
      "[Epoch 63] training: accuracy=0.880068\n",
      "[Epoch 63] time cost: 13.846181\n",
      "[Epoch 63] validation: accuracy=0.823017\n",
      "Epoch[64] Batch [49]\tSpeed: 2877.229650 samples/sec\taccuracy=0.877656\n",
      "Epoch[64] Batch [99]\tSpeed: 4205.837194 samples/sec\taccuracy=0.873516\n",
      "Epoch[64] Batch [149]\tSpeed: 4042.489568 samples/sec\taccuracy=0.875208\n",
      "Epoch[64] Batch [199]\tSpeed: 4080.310330 samples/sec\taccuracy=0.875352\n",
      "Epoch[64] Batch [249]\tSpeed: 2934.394298 samples/sec\taccuracy=0.876625\n",
      "Epoch[64] Batch [299]\tSpeed: 3955.607792 samples/sec\taccuracy=0.876276\n",
      "Epoch[64] Batch [349]\tSpeed: 3915.993144 samples/sec\taccuracy=0.877946\n",
      "[Epoch 64] training: accuracy=0.878916\n",
      "[Epoch 64] time cost: 13.718241\n",
      "[Epoch 64] validation: accuracy=0.848991\n",
      "Epoch[65] Batch [49]\tSpeed: 3770.796426 samples/sec\taccuracy=0.878750\n",
      "Epoch[65] Batch [99]\tSpeed: 4105.396507 samples/sec\taccuracy=0.876719\n",
      "Epoch[65] Batch [149]\tSpeed: 3982.249229 samples/sec\taccuracy=0.878646\n",
      "Epoch[65] Batch [199]\tSpeed: 2626.480072 samples/sec\taccuracy=0.877305\n",
      "Epoch[65] Batch [249]\tSpeed: 3793.953034 samples/sec\taccuracy=0.877062\n",
      "Epoch[65] Batch [299]\tSpeed: 3807.784159 samples/sec\taccuracy=0.876901\n",
      "Epoch[65] Batch [349]\tSpeed: 2017.545639 samples/sec\taccuracy=0.878036\n",
      "[Epoch 65] training: accuracy=0.879076\n",
      "[Epoch 65] time cost: 13.763207\n",
      "[Epoch 65] validation: accuracy=0.847256\n",
      "Epoch[66] Batch [49]\tSpeed: 3978.088665 samples/sec\taccuracy=0.887500\n",
      "Epoch[66] Batch [99]\tSpeed: 4049.747015 samples/sec\taccuracy=0.883984\n",
      "Epoch[66] Batch [149]\tSpeed: 4021.625457 samples/sec\taccuracy=0.883542\n",
      "Epoch[66] Batch [199]\tSpeed: 3892.484408 samples/sec\taccuracy=0.880938\n",
      "Epoch[66] Batch [249]\tSpeed: 3596.739458 samples/sec\taccuracy=0.879750\n",
      "Epoch[66] Batch [299]\tSpeed: 3696.999766 samples/sec\taccuracy=0.879896\n",
      "Epoch[66] Batch [349]\tSpeed: 3865.968503 samples/sec\taccuracy=0.881071\n",
      "[Epoch 66] training: accuracy=0.880789\n",
      "[Epoch 66] time cost: 13.341733\n",
      "[Epoch 66] validation: accuracy=0.847055\n",
      "Epoch[67] Batch [49]\tSpeed: 4155.379778 samples/sec\taccuracy=0.884062\n",
      "Epoch[67] Batch [99]\tSpeed: 4072.386917 samples/sec\taccuracy=0.880156\n",
      "Epoch[67] Batch [149]\tSpeed: 2505.464402 samples/sec\taccuracy=0.879948\n",
      "Epoch[67] Batch [199]\tSpeed: 3847.213231 samples/sec\taccuracy=0.878398\n",
      "Epoch[67] Batch [249]\tSpeed: 3716.912988 samples/sec\taccuracy=0.879219\n",
      "Epoch[67] Batch [299]\tSpeed: 4026.994944 samples/sec\taccuracy=0.880078\n",
      "Epoch[67] Batch [349]\tSpeed: 4064.155762 samples/sec\taccuracy=0.881205\n",
      "[Epoch 67] training: accuracy=0.880854\n",
      "[Epoch 67] time cost: 13.057359\n",
      "[Epoch 67] validation: accuracy=0.835337\n",
      "Epoch[68] Batch [49]\tSpeed: 4089.821833 samples/sec\taccuracy=0.881094\n",
      "Epoch[68] Batch [99]\tSpeed: 3981.038522 samples/sec\taccuracy=0.879531\n",
      "Epoch[68] Batch [149]\tSpeed: 4079.535201 samples/sec\taccuracy=0.880365\n",
      "Epoch[68] Batch [199]\tSpeed: 2620.109476 samples/sec\taccuracy=0.881953\n",
      "Epoch[68] Batch [249]\tSpeed: 2360.536204 samples/sec\taccuracy=0.881875\n",
      "Epoch[68] Batch [299]\tSpeed: 3919.481015 samples/sec\taccuracy=0.881667\n",
      "Epoch[68] Batch [349]\tSpeed: 2994.444232 samples/sec\taccuracy=0.882567\n",
      "[Epoch 68] training: accuracy=0.881953\n",
      "[Epoch 68] time cost: 13.897341\n",
      "[Epoch 68] validation: accuracy=0.848658\n",
      "Epoch[69] Batch [49]\tSpeed: 4115.214717 samples/sec\taccuracy=0.881094\n",
      "Epoch[69] Batch [99]\tSpeed: 1811.935025 samples/sec\taccuracy=0.876719\n",
      "Epoch[69] Batch [149]\tSpeed: 3801.717288 samples/sec\taccuracy=0.878802\n",
      "Epoch[69] Batch [199]\tSpeed: 3708.517217 samples/sec\taccuracy=0.879336\n",
      "Epoch[69] Batch [249]\tSpeed: 3571.258836 samples/sec\taccuracy=0.878313\n",
      "Epoch[69] Batch [299]\tSpeed: 3868.698609 samples/sec\taccuracy=0.878932\n",
      "Epoch[69] Batch [349]\tSpeed: 3889.382490 samples/sec\taccuracy=0.880424\n",
      "[Epoch 69] training: accuracy=0.880108\n",
      "[Epoch 69] time cost: 13.494515\n",
      "[Epoch 69] validation: accuracy=0.840946\n",
      "Epoch[70] Batch [49]\tSpeed: 3734.832115 samples/sec\taccuracy=0.887188\n",
      "Epoch[70] Batch [99]\tSpeed: 3729.902054 samples/sec\taccuracy=0.884141\n",
      "Epoch[70] Batch [149]\tSpeed: 3885.582341 samples/sec\taccuracy=0.884635\n",
      "Epoch[70] Batch [199]\tSpeed: 3476.647835 samples/sec\taccuracy=0.884883\n",
      "Epoch[70] Batch [249]\tSpeed: 3703.196496 samples/sec\taccuracy=0.883750\n",
      "Epoch[70] Batch [299]\tSpeed: 3925.097508 samples/sec\taccuracy=0.883802\n",
      "Epoch[70] Batch [349]\tSpeed: 3902.841050 samples/sec\taccuracy=0.883661\n",
      "[Epoch 70] training: accuracy=0.882932\n",
      "[Epoch 70] time cost: 13.473171\n",
      "[Epoch 70] validation: accuracy=0.843650\n",
      "Epoch[71] Batch [49]\tSpeed: 3914.137385 samples/sec\taccuracy=0.892656\n",
      "Epoch[71] Batch [99]\tSpeed: 3863.186650 samples/sec\taccuracy=0.884219\n",
      "Epoch[71] Batch [149]\tSpeed: 3968.385078 samples/sec\taccuracy=0.884479\n",
      "Epoch[71] Batch [199]\tSpeed: 3950.979239 samples/sec\taccuracy=0.882227\n",
      "Epoch[71] Batch [249]\tSpeed: 3598.282275 samples/sec\taccuracy=0.882500\n",
      "Epoch[71] Batch [299]\tSpeed: 3805.247202 samples/sec\taccuracy=0.882188\n",
      "Epoch[71] Batch [349]\tSpeed: 3668.905296 samples/sec\taccuracy=0.884085\n",
      "[Epoch 71] training: accuracy=0.884716\n",
      "[Epoch 71] time cost: 13.773838\n",
      "[Epoch 71] validation: accuracy=0.832332\n",
      "Epoch[72] Batch [49]\tSpeed: 4040.329565 samples/sec\taccuracy=0.881250\n",
      "Epoch[72] Batch [99]\tSpeed: 3803.521845 samples/sec\taccuracy=0.878359\n",
      "Epoch[72] Batch [149]\tSpeed: 3028.190444 samples/sec\taccuracy=0.879427\n",
      "Epoch[72] Batch [199]\tSpeed: 4036.289570 samples/sec\taccuracy=0.880391\n",
      "Epoch[72] Batch [249]\tSpeed: 3007.343222 samples/sec\taccuracy=0.880500\n",
      "Epoch[72] Batch [299]\tSpeed: 3104.610107 samples/sec\taccuracy=0.881615\n",
      "Epoch[72] Batch [349]\tSpeed: 3780.461595 samples/sec\taccuracy=0.882277\n",
      "[Epoch 72] training: accuracy=0.882013\n",
      "[Epoch 72] time cost: 14.297826\n",
      "[Epoch 72] validation: accuracy=0.836036\n",
      "Epoch[73] Batch [49]\tSpeed: 4016.090006 samples/sec\taccuracy=0.882812\n",
      "Epoch[73] Batch [99]\tSpeed: 3860.547596 samples/sec\taccuracy=0.884687\n",
      "Epoch[73] Batch [149]\tSpeed: 3929.723111 samples/sec\taccuracy=0.885208\n",
      "Epoch[73] Batch [199]\tSpeed: 4024.398909 samples/sec\taccuracy=0.883711\n",
      "Epoch[73] Batch [249]\tSpeed: 3793.202473 samples/sec\taccuracy=0.882969\n",
      "Epoch[73] Batch [299]\tSpeed: 3988.224939 samples/sec\taccuracy=0.882292\n",
      "Epoch[73] Batch [349]\tSpeed: 3810.594951 samples/sec\taccuracy=0.883415\n",
      "[Epoch 73] training: accuracy=0.883652\n",
      "[Epoch 73] time cost: 13.465791\n",
      "[Epoch 73] validation: accuracy=0.828826\n",
      "Epoch[74] Batch [49]\tSpeed: 4097.938417 samples/sec\taccuracy=0.879219\n",
      "Epoch[74] Batch [99]\tSpeed: 4106.526986 samples/sec\taccuracy=0.877344\n",
      "Epoch[74] Batch [149]\tSpeed: 3556.543507 samples/sec\taccuracy=0.877083\n",
      "Epoch[74] Batch [199]\tSpeed: 3991.189854 samples/sec\taccuracy=0.875469\n",
      "Epoch[74] Batch [249]\tSpeed: 3907.926277 samples/sec\taccuracy=0.876500\n",
      "Epoch[74] Batch [299]\tSpeed: 3064.419144 samples/sec\taccuracy=0.877292\n",
      "Epoch[74] Batch [349]\tSpeed: 3999.574706 samples/sec\taccuracy=0.879241\n",
      "[Epoch 74] training: accuracy=0.879908\n",
      "[Epoch 74] time cost: 13.191012\n",
      "[Epoch 74] validation: accuracy=0.849159\n",
      "Epoch[75] Batch [49]\tSpeed: 4143.865388 samples/sec\taccuracy=0.880938\n",
      "Epoch[75] Batch [99]\tSpeed: 3998.323667 samples/sec\taccuracy=0.877969\n",
      "Epoch[75] Batch [149]\tSpeed: 3673.775880 samples/sec\taccuracy=0.879531\n",
      "Epoch[75] Batch [199]\tSpeed: 2243.215095 samples/sec\taccuracy=0.879258\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[75] Batch [249]\tSpeed: 3058.832077 samples/sec\taccuracy=0.879313\n",
      "Epoch[75] Batch [299]\tSpeed: 3934.273135 samples/sec\taccuracy=0.880495\n",
      "Epoch[75] Batch [349]\tSpeed: 2674.578974 samples/sec\taccuracy=0.881942\n",
      "[Epoch 75] training: accuracy=0.881674\n",
      "[Epoch 75] time cost: 13.902251\n",
      "[Epoch 75] validation: accuracy=0.822516\n",
      "Epoch[76] Batch [49]\tSpeed: 3825.229156 samples/sec\taccuracy=0.877812\n",
      "Epoch[76] Batch [99]\tSpeed: 4099.816052 samples/sec\taccuracy=0.883281\n",
      "Epoch[76] Batch [149]\tSpeed: 3941.725613 samples/sec\taccuracy=0.882552\n",
      "Epoch[76] Batch [199]\tSpeed: 3860.825221 samples/sec\taccuracy=0.881484\n",
      "Epoch[76] Batch [249]\tSpeed: 3878.032289 samples/sec\taccuracy=0.880188\n",
      "Epoch[76] Batch [299]\tSpeed: 2346.576592 samples/sec\taccuracy=0.880885\n",
      "Epoch[76] Batch [349]\tSpeed: 3434.654929 samples/sec\taccuracy=0.881830\n",
      "[Epoch 76] training: accuracy=0.882373\n",
      "[Epoch 76] time cost: 13.659685\n",
      "[Epoch 76] validation: accuracy=0.840445\n",
      "Epoch[77] Batch [49]\tSpeed: 3807.163102 samples/sec\taccuracy=0.887969\n",
      "Epoch[77] Batch [99]\tSpeed: 2911.889614 samples/sec\taccuracy=0.882578\n",
      "Epoch[77] Batch [149]\tSpeed: 2816.549301 samples/sec\taccuracy=0.882083\n",
      "Epoch[77] Batch [199]\tSpeed: 3999.098028 samples/sec\taccuracy=0.882266\n",
      "Epoch[77] Batch [249]\tSpeed: 3925.126205 samples/sec\taccuracy=0.882281\n",
      "Epoch[77] Batch [299]\tSpeed: 3892.569075 samples/sec\taccuracy=0.882656\n",
      "Epoch[77] Batch [349]\tSpeed: 2397.687091 samples/sec\taccuracy=0.883371\n",
      "[Epoch 77] training: accuracy=0.882973\n",
      "[Epoch 77] time cost: 13.581352\n",
      "[Epoch 77] validation: accuracy=0.832632\n",
      "Epoch[78] Batch [49]\tSpeed: 3838.850442 samples/sec\taccuracy=0.890781\n",
      "Epoch[78] Batch [99]\tSpeed: 3909.576849 samples/sec\taccuracy=0.884609\n",
      "Epoch[78] Batch [149]\tSpeed: 4149.021322 samples/sec\taccuracy=0.884896\n",
      "Epoch[78] Batch [199]\tSpeed: 3749.569862 samples/sec\taccuracy=0.886289\n",
      "Epoch[78] Batch [249]\tSpeed: 4051.427864 samples/sec\taccuracy=0.885375\n",
      "Epoch[78] Batch [299]\tSpeed: 2525.156094 samples/sec\taccuracy=0.884609\n",
      "Epoch[78] Batch [349]\tSpeed: 3980.300648 samples/sec\taccuracy=0.884375\n",
      "[Epoch 78] training: accuracy=0.885410\n",
      "[Epoch 78] time cost: 13.368570\n",
      "[Epoch 78] validation: accuracy=0.828726\n",
      "Epoch[79] Batch [49]\tSpeed: 3805.301145 samples/sec\taccuracy=0.886563\n",
      "Epoch[79] Batch [99]\tSpeed: 3280.083286 samples/sec\taccuracy=0.885625\n",
      "Epoch[79] Batch [149]\tSpeed: 3892.202211 samples/sec\taccuracy=0.886302\n",
      "Epoch[79] Batch [199]\tSpeed: 3847.378653 samples/sec\taccuracy=0.883047\n",
      "Epoch[79] Batch [249]\tSpeed: 3930.356028 samples/sec\taccuracy=0.883062\n",
      "Epoch[79] Batch [299]\tSpeed: 3670.410282 samples/sec\taccuracy=0.882604\n",
      "Epoch[79] Batch [349]\tSpeed: 3809.702616 samples/sec\taccuracy=0.883594\n",
      "[Epoch 79] training: accuracy=0.883834\n",
      "[Epoch 79] time cost: 13.628694\n",
      "[Epoch 79] validation: accuracy=0.846354\n",
      "Epoch[80] Batch [49]\tSpeed: 4065.048171 samples/sec\taccuracy=0.885625\n",
      "Epoch[80] Batch [99]\tSpeed: 3964.751106 samples/sec\taccuracy=0.883125\n",
      "Epoch[80] Batch [149]\tSpeed: 3942.912523 samples/sec\taccuracy=0.883698\n",
      "Epoch[80] Batch [199]\tSpeed: 3975.466967 samples/sec\taccuracy=0.884414\n",
      "Epoch[80] Batch [249]\tSpeed: 3877.836211 samples/sec\taccuracy=0.885094\n",
      "Epoch[80] Batch [299]\tSpeed: 4017.953509 samples/sec\taccuracy=0.885599\n",
      "Epoch[80] Batch [349]\tSpeed: 3511.645585 samples/sec\taccuracy=0.886786\n",
      "[Epoch 80] training: accuracy=0.886909\n",
      "[Epoch 80] time cost: 13.075113\n",
      "[Epoch 80] validation: accuracy=0.834949\n",
      "Epoch[81] Batch [49]\tSpeed: 3506.668269 samples/sec\taccuracy=0.885156\n",
      "Epoch[81] Batch [99]\tSpeed: 3675.611976 samples/sec\taccuracy=0.884766\n",
      "Epoch[81] Batch [149]\tSpeed: 3925.728935 samples/sec\taccuracy=0.883073\n",
      "Epoch[81] Batch [199]\tSpeed: 2353.561492 samples/sec\taccuracy=0.883828\n",
      "Epoch[81] Batch [249]\tSpeed: 3588.181631 samples/sec\taccuracy=0.883844\n",
      "Epoch[81] Batch [299]\tSpeed: 3903.664015 samples/sec\taccuracy=0.882891\n",
      "Epoch[81] Batch [349]\tSpeed: 4068.066802 samples/sec\taccuracy=0.883460\n",
      "[Epoch 81] training: accuracy=0.883632\n",
      "[Epoch 81] time cost: 13.477776\n",
      "[Epoch 81] validation: accuracy=0.826222\n",
      "Epoch[82] Batch [49]\tSpeed: 3649.477000 samples/sec\taccuracy=0.881719\n",
      "Epoch[82] Batch [99]\tSpeed: 4037.928894 samples/sec\taccuracy=0.878672\n",
      "Epoch[82] Batch [149]\tSpeed: 3853.813551 samples/sec\taccuracy=0.881875\n",
      "Epoch[82] Batch [199]\tSpeed: 4032.802848 samples/sec\taccuracy=0.880586\n",
      "Epoch[82] Batch [249]\tSpeed: 2630.185882 samples/sec\taccuracy=0.881094\n",
      "Epoch[82] Batch [299]\tSpeed: 2441.765189 samples/sec\taccuracy=0.882370\n",
      "Epoch[82] Batch [349]\tSpeed: 3987.217872 samples/sec\taccuracy=0.882589\n",
      "[Epoch 82] training: accuracy=0.883494\n",
      "[Epoch 82] time cost: 13.735600\n",
      "[Epoch 82] validation: accuracy=0.821815\n",
      "Epoch[83] Batch [49]\tSpeed: 4001.870314 samples/sec\taccuracy=0.885781\n",
      "Epoch[83] Batch [99]\tSpeed: 4076.097182 samples/sec\taccuracy=0.882266\n",
      "Epoch[83] Batch [149]\tSpeed: 3260.382668 samples/sec\taccuracy=0.883646\n",
      "Epoch[83] Batch [199]\tSpeed: 4028.324444 samples/sec\taccuracy=0.884414\n",
      "Epoch[83] Batch [249]\tSpeed: 3661.398841 samples/sec\taccuracy=0.882687\n",
      "Epoch[83] Batch [299]\tSpeed: 3750.093684 samples/sec\taccuracy=0.883177\n",
      "Epoch[83] Batch [349]\tSpeed: 2995.981585 samples/sec\taccuracy=0.883951\n",
      "[Epoch 83] training: accuracy=0.883332\n",
      "[Epoch 83] time cost: 13.914133\n",
      "[Epoch 83] validation: accuracy=0.833934\n",
      "Epoch[84] Batch [49]\tSpeed: 4019.216865 samples/sec\taccuracy=0.888594\n",
      "Epoch[84] Batch [99]\tSpeed: 3944.389920 samples/sec\taccuracy=0.882969\n",
      "Epoch[84] Batch [149]\tSpeed: 3960.305629 samples/sec\taccuracy=0.886458\n",
      "Epoch[84] Batch [199]\tSpeed: 3873.331881 samples/sec\taccuracy=0.884727\n",
      "Epoch[84] Batch [249]\tSpeed: 2456.288201 samples/sec\taccuracy=0.885406\n",
      "Epoch[84] Batch [299]\tSpeed: 2497.155312 samples/sec\taccuracy=0.885729\n",
      "Epoch[84] Batch [349]\tSpeed: 3803.333229 samples/sec\taccuracy=0.885804\n",
      "[Epoch 84] training: accuracy=0.886049\n",
      "[Epoch 84] time cost: 14.147071\n",
      "[Epoch 84] validation: accuracy=0.847256\n",
      "Epoch[85] Batch [49]\tSpeed: 3860.408798 samples/sec\taccuracy=0.879687\n",
      "Epoch[85] Batch [99]\tSpeed: 3863.103257 samples/sec\taccuracy=0.882578\n",
      "Epoch[85] Batch [149]\tSpeed: 3932.256002 samples/sec\taccuracy=0.885521\n",
      "Epoch[85] Batch [199]\tSpeed: 3356.492104 samples/sec\taccuracy=0.885352\n",
      "Epoch[85] Batch [249]\tSpeed: 3501.796403 samples/sec\taccuracy=0.884844\n",
      "Epoch[85] Batch [299]\tSpeed: 3846.524127 samples/sec\taccuracy=0.884714\n",
      "Epoch[85] Batch [349]\tSpeed: 4003.720642 samples/sec\taccuracy=0.885804\n",
      "[Epoch 85] training: accuracy=0.885958\n",
      "[Epoch 85] time cost: 13.795506\n",
      "[Epoch 85] validation: accuracy=0.830529\n",
      "Epoch[86] Batch [49]\tSpeed: 2334.485255 samples/sec\taccuracy=0.884375\n",
      "Epoch[86] Batch [99]\tSpeed: 3841.404933 samples/sec\taccuracy=0.883594\n",
      "Epoch[86] Batch [149]\tSpeed: 4039.599946 samples/sec\taccuracy=0.883125\n",
      "Epoch[86] Batch [199]\tSpeed: 3998.502339 samples/sec\taccuracy=0.881797\n",
      "Epoch[86] Batch [249]\tSpeed: 3854.090209 samples/sec\taccuracy=0.882313\n",
      "Epoch[86] Batch [299]\tSpeed: 3935.917187 samples/sec\taccuracy=0.882917\n",
      "Epoch[86] Batch [349]\tSpeed: 3073.119548 samples/sec\taccuracy=0.884085\n",
      "[Epoch 86] training: accuracy=0.884970\n",
      "[Epoch 86] time cost: 13.783408\n",
      "[Epoch 86] validation: accuracy=0.850761\n",
      "Epoch[87] Batch [49]\tSpeed: 3990.270259 samples/sec\taccuracy=0.889844\n",
      "Epoch[87] Batch [99]\tSpeed: 3662.697758 samples/sec\taccuracy=0.881172\n",
      "Epoch[87] Batch [149]\tSpeed: 3870.734766 samples/sec\taccuracy=0.882969\n",
      "Epoch[87] Batch [199]\tSpeed: 4058.533376 samples/sec\taccuracy=0.883086\n",
      "Epoch[87] Batch [249]\tSpeed: 3785.926731 samples/sec\taccuracy=0.882344\n",
      "Epoch[87] Batch [299]\tSpeed: 3980.153106 samples/sec\taccuracy=0.883047\n",
      "Epoch[87] Batch [349]\tSpeed: 3270.372632 samples/sec\taccuracy=0.883304\n",
      "[Epoch 87] training: accuracy=0.883494\n",
      "[Epoch 87] time cost: 13.714016\n",
      "[Epoch 87] validation: accuracy=0.837941\n",
      "Epoch[88] Batch [49]\tSpeed: 4092.846180 samples/sec\taccuracy=0.888750\n",
      "Epoch[88] Batch [99]\tSpeed: 3858.605336 samples/sec\taccuracy=0.887344\n",
      "Epoch[88] Batch [149]\tSpeed: 4000.468786 samples/sec\taccuracy=0.885469\n",
      "Epoch[88] Batch [199]\tSpeed: 2494.556248 samples/sec\taccuracy=0.883281\n",
      "Epoch[88] Batch [249]\tSpeed: 3879.882000 samples/sec\taccuracy=0.883687\n",
      "Epoch[88] Batch [299]\tSpeed: 2728.806823 samples/sec\taccuracy=0.884427\n",
      "Epoch[88] Batch [349]\tSpeed: 3831.754195 samples/sec\taccuracy=0.885290\n",
      "[Epoch 88] training: accuracy=0.884970\n",
      "[Epoch 88] time cost: 13.311323\n",
      "[Epoch 88] validation: accuracy=0.853639\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[89] Batch [49]\tSpeed: 3890.425311 samples/sec\taccuracy=0.884062\n",
      "Epoch[89] Batch [99]\tSpeed: 4026.270132 samples/sec\taccuracy=0.883672\n",
      "Epoch[89] Batch [149]\tSpeed: 2888.856728 samples/sec\taccuracy=0.884062\n",
      "Epoch[89] Batch [199]\tSpeed: 3869.256247 samples/sec\taccuracy=0.884414\n",
      "Epoch[89] Batch [249]\tSpeed: 3650.271028 samples/sec\taccuracy=0.885906\n",
      "Epoch[89] Batch [299]\tSpeed: 3933.293126 samples/sec\taccuracy=0.886875\n",
      "Epoch[89] Batch [349]\tSpeed: 3705.394557 samples/sec\taccuracy=0.888103\n",
      "[Epoch 89] training: accuracy=0.888827\n",
      "[Epoch 89] time cost: 14.023120\n",
      "[Epoch 89] validation: accuracy=0.850361\n",
      "Epoch[90] Batch [49]\tSpeed: 4139.903086 samples/sec\taccuracy=0.878437\n",
      "Epoch[90] Batch [99]\tSpeed: 3574.492573 samples/sec\taccuracy=0.885938\n",
      "Epoch[90] Batch [149]\tSpeed: 3800.748382 samples/sec\taccuracy=0.889323\n",
      "Epoch[90] Batch [199]\tSpeed: 3845.229279 samples/sec\taccuracy=0.888398\n",
      "Epoch[90] Batch [249]\tSpeed: 3785.766552 samples/sec\taccuracy=0.887125\n",
      "Epoch[90] Batch [299]\tSpeed: 3865.439643 samples/sec\taccuracy=0.887552\n",
      "Epoch[90] Batch [349]\tSpeed: 3851.712250 samples/sec\taccuracy=0.888013\n",
      "[Epoch 90] training: accuracy=0.887861\n",
      "[Epoch 90] time cost: 13.793650\n",
      "[Epoch 90] validation: accuracy=0.850160\n",
      "Epoch[91] Batch [49]\tSpeed: 4066.033354 samples/sec\taccuracy=0.888906\n",
      "Epoch[91] Batch [99]\tSpeed: 3943.260046 samples/sec\taccuracy=0.888672\n",
      "Epoch[91] Batch [149]\tSpeed: 3889.100743 samples/sec\taccuracy=0.888021\n",
      "Epoch[91] Batch [199]\tSpeed: 2338.155557 samples/sec\taccuracy=0.885938\n",
      "Epoch[91] Batch [249]\tSpeed: 2974.947424 samples/sec\taccuracy=0.885656\n",
      "Epoch[91] Batch [299]\tSpeed: 3959.371009 samples/sec\taccuracy=0.885755\n",
      "Epoch[91] Batch [349]\tSpeed: 4029.352387 samples/sec\taccuracy=0.886183\n",
      "[Epoch 91] training: accuracy=0.885690\n",
      "[Epoch 91] time cost: 13.587730\n",
      "[Epoch 91] validation: accuracy=0.867688\n",
      "Epoch[92] Batch [49]\tSpeed: 4040.937783 samples/sec\taccuracy=0.898594\n",
      "Epoch[92] Batch [99]\tSpeed: 4040.086330 samples/sec\taccuracy=0.891875\n",
      "Epoch[92] Batch [149]\tSpeed: 3714.624138 samples/sec\taccuracy=0.889062\n",
      "Epoch[92] Batch [199]\tSpeed: 4090.382714 samples/sec\taccuracy=0.887305\n",
      "Epoch[92] Batch [249]\tSpeed: 3658.903510 samples/sec\taccuracy=0.887000\n",
      "Epoch[92] Batch [299]\tSpeed: 3732.209777 samples/sec\taccuracy=0.886823\n",
      "Epoch[92] Batch [349]\tSpeed: 2377.849730 samples/sec\taccuracy=0.887522\n",
      "[Epoch 92] training: accuracy=0.887528\n",
      "[Epoch 92] time cost: 13.725245\n",
      "[Epoch 92] validation: accuracy=0.835737\n",
      "Epoch[93] Batch [49]\tSpeed: 3930.384802 samples/sec\taccuracy=0.882500\n",
      "Epoch[93] Batch [99]\tSpeed: 3920.912266 samples/sec\taccuracy=0.886094\n",
      "Epoch[93] Batch [149]\tSpeed: 3939.527378 samples/sec\taccuracy=0.887188\n",
      "Epoch[93] Batch [199]\tSpeed: 4082.171842 samples/sec\taccuracy=0.885586\n",
      "Epoch[93] Batch [249]\tSpeed: 3881.088065 samples/sec\taccuracy=0.886656\n",
      "Epoch[93] Batch [299]\tSpeed: 3732.910437 samples/sec\taccuracy=0.886536\n",
      "Epoch[93] Batch [349]\tSpeed: 3850.082556 samples/sec\taccuracy=0.887210\n",
      "[Epoch 93] training: accuracy=0.887660\n",
      "[Epoch 93] time cost: 13.691646\n",
      "[Epoch 93] validation: accuracy=0.859675\n",
      "Epoch[94] Batch [49]\tSpeed: 3711.645145 samples/sec\taccuracy=0.891563\n",
      "Epoch[94] Batch [99]\tSpeed: 3709.926696 samples/sec\taccuracy=0.891406\n",
      "Epoch[94] Batch [149]\tSpeed: 3739.749175 samples/sec\taccuracy=0.890990\n",
      "Epoch[94] Batch [199]\tSpeed: 3771.617633 samples/sec\taccuracy=0.890039\n",
      "Epoch[94] Batch [249]\tSpeed: 3703.707440 samples/sec\taccuracy=0.890281\n",
      "Epoch[94] Batch [299]\tSpeed: 3804.141715 samples/sec\taccuracy=0.890781\n",
      "Epoch[94] Batch [349]\tSpeed: 4054.181357 samples/sec\taccuracy=0.890223\n",
      "[Epoch 94] training: accuracy=0.889946\n",
      "[Epoch 94] time cost: 13.519081\n",
      "[Epoch 94] validation: accuracy=0.847155\n",
      "Epoch[95] Batch [49]\tSpeed: 4080.403365 samples/sec\taccuracy=0.884687\n",
      "Epoch[95] Batch [99]\tSpeed: 2739.724390 samples/sec\taccuracy=0.883281\n",
      "Epoch[95] Batch [149]\tSpeed: 3848.233559 samples/sec\taccuracy=0.882448\n",
      "Epoch[95] Batch [199]\tSpeed: 4139.360458 samples/sec\taccuracy=0.884141\n",
      "Epoch[95] Batch [249]\tSpeed: 3872.381994 samples/sec\taccuracy=0.884719\n",
      "Epoch[95] Batch [299]\tSpeed: 3852.596729 samples/sec\taccuracy=0.885234\n",
      "Epoch[95] Batch [349]\tSpeed: 4057.797167 samples/sec\taccuracy=0.886585\n",
      "[Epoch 95] training: accuracy=0.886478\n",
      "[Epoch 95] time cost: 13.231295\n",
      "[Epoch 95] validation: accuracy=0.845853\n",
      "Epoch[96] Batch [49]\tSpeed: 3912.511474 samples/sec\taccuracy=0.883281\n",
      "Epoch[96] Batch [99]\tSpeed: 2907.647337 samples/sec\taccuracy=0.882812\n",
      "Epoch[96] Batch [149]\tSpeed: 4094.875309 samples/sec\taccuracy=0.884427\n",
      "Epoch[96] Batch [199]\tSpeed: 2857.216136 samples/sec\taccuracy=0.885547\n",
      "Epoch[96] Batch [249]\tSpeed: 3977.911813 samples/sec\taccuracy=0.884687\n",
      "Epoch[96] Batch [299]\tSpeed: 3981.569961 samples/sec\taccuracy=0.885677\n",
      "Epoch[96] Batch [349]\tSpeed: 4129.172752 samples/sec\taccuracy=0.886116\n",
      "[Epoch 96] training: accuracy=0.886349\n",
      "[Epoch 96] time cost: 13.732009\n",
      "[Epoch 96] validation: accuracy=0.854826\n",
      "Epoch[97] Batch [49]\tSpeed: 3759.257995 samples/sec\taccuracy=0.888281\n",
      "Epoch[97] Batch [99]\tSpeed: 4057.919850 samples/sec\taccuracy=0.884531\n",
      "Epoch[97] Batch [149]\tSpeed: 4034.833510 samples/sec\taccuracy=0.885104\n",
      "Epoch[97] Batch [199]\tSpeed: 3826.510566 samples/sec\taccuracy=0.887344\n",
      "Epoch[97] Batch [249]\tSpeed: 2894.822640 samples/sec\taccuracy=0.887312\n",
      "Epoch[97] Batch [299]\tSpeed: 3126.031559 samples/sec\taccuracy=0.886927\n",
      "Epoch[97] Batch [349]\tSpeed: 3549.043524 samples/sec\taccuracy=0.887857\n",
      "[Epoch 97] training: accuracy=0.887928\n",
      "[Epoch 97] time cost: 14.354197\n",
      "[Epoch 97] validation: accuracy=0.856470\n",
      "Epoch[98] Batch [49]\tSpeed: 4018.434694 samples/sec\taccuracy=0.890625\n",
      "Epoch[98] Batch [99]\tSpeed: 4062.372118 samples/sec\taccuracy=0.887031\n",
      "Epoch[98] Batch [149]\tSpeed: 3806.029520 samples/sec\taccuracy=0.887969\n",
      "Epoch[98] Batch [199]\tSpeed: 4042.672209 samples/sec\taccuracy=0.887734\n",
      "Epoch[98] Batch [249]\tSpeed: 3587.534243 samples/sec\taccuracy=0.886563\n",
      "Epoch[98] Batch [299]\tSpeed: 3916.507357 samples/sec\taccuracy=0.886120\n",
      "Epoch[98] Batch [349]\tSpeed: 3911.228815 samples/sec\taccuracy=0.887567\n",
      "[Epoch 98] training: accuracy=0.888121\n",
      "[Epoch 98] time cost: 13.672716\n",
      "[Epoch 98] validation: accuracy=0.826522\n",
      "Epoch[99] Batch [49]\tSpeed: 3872.801004 samples/sec\taccuracy=0.887813\n",
      "Epoch[99] Batch [99]\tSpeed: 3101.991137 samples/sec\taccuracy=0.885469\n",
      "Epoch[99] Batch [149]\tSpeed: 3658.205427 samples/sec\taccuracy=0.888542\n",
      "Epoch[99] Batch [199]\tSpeed: 4238.376493 samples/sec\taccuracy=0.888555\n",
      "Epoch[99] Batch [249]\tSpeed: 2770.417582 samples/sec\taccuracy=0.888437\n",
      "Epoch[99] Batch [299]\tSpeed: 4097.187844 samples/sec\taccuracy=0.887839\n",
      "Epoch[99] Batch [349]\tSpeed: 4041.241961 samples/sec\taccuracy=0.887589\n",
      "[Epoch 99] training: accuracy=0.887488\n",
      "[Epoch 99] time cost: 13.556946\n",
      "[Epoch 99] validation: accuracy=0.841246\n",
      "Epoch[100] Batch [49]\tSpeed: 3983.342449 samples/sec\taccuracy=0.892813\n",
      "Epoch[100] Batch [99]\tSpeed: 4056.540095 samples/sec\taccuracy=0.892578\n",
      "Epoch[100] Batch [149]\tSpeed: 3959.721440 samples/sec\taccuracy=0.891302\n",
      "Epoch[100] Batch [199]\tSpeed: 3661.748459 samples/sec\taccuracy=0.889687\n",
      "Epoch[100] Batch [249]\tSpeed: 3050.974967 samples/sec\taccuracy=0.889250\n",
      "Epoch[100] Batch [299]\tSpeed: 3947.928582 samples/sec\taccuracy=0.888828\n",
      "Epoch[100] Batch [349]\tSpeed: 3818.237442 samples/sec\taccuracy=0.890536\n",
      "[Epoch 100] training: accuracy=0.890086\n",
      "[Epoch 100] time cost: 13.495509\n",
      "[Epoch 100] validation: accuracy=0.847155\n",
      "Epoch[101] Batch [49]\tSpeed: 3749.543675 samples/sec\taccuracy=0.889844\n",
      "Epoch[101] Batch [99]\tSpeed: 3854.449923 samples/sec\taccuracy=0.890781\n",
      "Epoch[101] Batch [149]\tSpeed: 2389.257381 samples/sec\taccuracy=0.890000\n",
      "Epoch[101] Batch [199]\tSpeed: 3946.651611 samples/sec\taccuracy=0.889922\n",
      "Epoch[101] Batch [249]\tSpeed: 3567.675283 samples/sec\taccuracy=0.889469\n",
      "Epoch[101] Batch [299]\tSpeed: 3821.362868 samples/sec\taccuracy=0.889401\n",
      "Epoch[101] Batch [349]\tSpeed: 4049.594279 samples/sec\taccuracy=0.889576\n",
      "[Epoch 101] training: accuracy=0.890004\n",
      "[Epoch 101] time cost: 13.506686\n",
      "[Epoch 101] validation: accuracy=0.849058\n",
      "Epoch[102] Batch [49]\tSpeed: 3836.436416 samples/sec\taccuracy=0.884531\n",
      "Epoch[102] Batch [99]\tSpeed: 3833.642136 samples/sec\taccuracy=0.889922\n",
      "Epoch[102] Batch [149]\tSpeed: 3177.785149 samples/sec\taccuracy=0.889896\n",
      "Epoch[102] Batch [199]\tSpeed: 4007.366609 samples/sec\taccuracy=0.889570\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[102] Batch [249]\tSpeed: 3711.132008 samples/sec\taccuracy=0.889906\n",
      "Epoch[102] Batch [299]\tSpeed: 3824.956626 samples/sec\taccuracy=0.890469\n",
      "Epoch[102] Batch [349]\tSpeed: 4085.620121 samples/sec\taccuracy=0.891094\n",
      "[Epoch 102] training: accuracy=0.890205\n",
      "[Epoch 102] time cost: 14.042636\n",
      "[Epoch 102] validation: accuracy=0.854667\n",
      "Epoch[103] Batch [49]\tSpeed: 3968.238418 samples/sec\taccuracy=0.886094\n",
      "Epoch[103] Batch [99]\tSpeed: 3803.521845 samples/sec\taccuracy=0.890000\n",
      "Epoch[103] Batch [149]\tSpeed: 3869.451458 samples/sec\taccuracy=0.891979\n",
      "Epoch[103] Batch [199]\tSpeed: 3888.819036 samples/sec\taccuracy=0.890703\n",
      "Epoch[103] Batch [249]\tSpeed: 3670.435376 samples/sec\taccuracy=0.890656\n",
      "Epoch[103] Batch [299]\tSpeed: 3002.130023 samples/sec\taccuracy=0.890391\n",
      "Epoch[103] Batch [349]\tSpeed: 3747.031400 samples/sec\taccuracy=0.891272\n",
      "[Epoch 103] training: accuracy=0.891286\n",
      "[Epoch 103] time cost: 13.973863\n",
      "[Epoch 103] validation: accuracy=0.841647\n",
      "Epoch[104] Batch [49]\tSpeed: 1884.134413 samples/sec\taccuracy=0.894687\n",
      "Epoch[104] Batch [99]\tSpeed: 3802.067292 samples/sec\taccuracy=0.889219\n",
      "Epoch[104] Batch [149]\tSpeed: 3924.064701 samples/sec\taccuracy=0.888750\n",
      "Epoch[104] Batch [199]\tSpeed: 4081.582180 samples/sec\taccuracy=0.888398\n",
      "Epoch[104] Batch [249]\tSpeed: 3794.489331 samples/sec\taccuracy=0.888781\n",
      "Epoch[104] Batch [299]\tSpeed: 3772.253652 samples/sec\taccuracy=0.888568\n",
      "Epoch[104] Batch [349]\tSpeed: 3015.332536 samples/sec\taccuracy=0.889442\n",
      "[Epoch 104] training: accuracy=0.889706\n",
      "[Epoch 104] time cost: 13.655073\n",
      "[Epoch 104] validation: accuracy=0.847310\n",
      "Epoch[105] Batch [49]\tSpeed: 3944.129122 samples/sec\taccuracy=0.891094\n",
      "Epoch[105] Batch [99]\tSpeed: 3936.350060 samples/sec\taccuracy=0.887188\n",
      "Epoch[105] Batch [149]\tSpeed: 1825.812076 samples/sec\taccuracy=0.888385\n",
      "Epoch[105] Batch [199]\tSpeed: 3841.185058 samples/sec\taccuracy=0.887461\n",
      "Epoch[105] Batch [249]\tSpeed: 3834.929190 samples/sec\taccuracy=0.887344\n",
      "Epoch[105] Batch [299]\tSpeed: 4038.749056 samples/sec\taccuracy=0.887474\n",
      "Epoch[105] Batch [349]\tSpeed: 3708.465984 samples/sec\taccuracy=0.888549\n",
      "[Epoch 105] training: accuracy=0.888687\n",
      "[Epoch 105] time cost: 13.528864\n",
      "[Epoch 105] validation: accuracy=0.857171\n",
      "Epoch[106] Batch [49]\tSpeed: 3777.402689 samples/sec\taccuracy=0.888750\n",
      "Epoch[106] Batch [99]\tSpeed: 3794.569789 samples/sec\taccuracy=0.887813\n",
      "Epoch[106] Batch [149]\tSpeed: 3840.250871 samples/sec\taccuracy=0.887708\n",
      "Epoch[106] Batch [199]\tSpeed: 3715.935381 samples/sec\taccuracy=0.887734\n",
      "Epoch[106] Batch [249]\tSpeed: 3789.935633 samples/sec\taccuracy=0.889188\n",
      "Epoch[106] Batch [299]\tSpeed: 4026.813717 samples/sec\taccuracy=0.889687\n",
      "Epoch[106] Batch [349]\tSpeed: 4045.749148 samples/sec\taccuracy=0.890491\n",
      "[Epoch 106] training: accuracy=0.889864\n",
      "[Epoch 106] time cost: 13.541670\n",
      "[Epoch 106] validation: accuracy=0.808293\n",
      "Epoch[107] Batch [49]\tSpeed: 3690.950610 samples/sec\taccuracy=0.883750\n",
      "Epoch[107] Batch [99]\tSpeed: 3996.151101 samples/sec\taccuracy=0.890234\n",
      "Epoch[107] Batch [149]\tSpeed: 3970.703745 samples/sec\taccuracy=0.890312\n",
      "Epoch[107] Batch [199]\tSpeed: 2791.709759 samples/sec\taccuracy=0.890781\n",
      "Epoch[107] Batch [249]\tSpeed: 3764.793952 samples/sec\taccuracy=0.890406\n",
      "Epoch[107] Batch [299]\tSpeed: 3924.408909 samples/sec\taccuracy=0.890339\n",
      "Epoch[107] Batch [349]\tSpeed: 4012.158283 samples/sec\taccuracy=0.891384\n",
      "[Epoch 107] training: accuracy=0.890645\n",
      "[Epoch 107] time cost: 14.048093\n",
      "[Epoch 107] validation: accuracy=0.837941\n",
      "Epoch[108] Batch [49]\tSpeed: 4015.279021 samples/sec\taccuracy=0.893594\n",
      "Epoch[108] Batch [99]\tSpeed: 3910.516589 samples/sec\taccuracy=0.887422\n",
      "Epoch[108] Batch [149]\tSpeed: 4023.132293 samples/sec\taccuracy=0.886823\n",
      "Epoch[108] Batch [199]\tSpeed: 4064.001938 samples/sec\taccuracy=0.886445\n",
      "Epoch[108] Batch [249]\tSpeed: 3447.004250 samples/sec\taccuracy=0.887625\n",
      "Epoch[108] Batch [299]\tSpeed: 4046.115037 samples/sec\taccuracy=0.887891\n",
      "Epoch[108] Batch [349]\tSpeed: 3945.752424 samples/sec\taccuracy=0.889196\n",
      "[Epoch 108] training: accuracy=0.889007\n",
      "[Epoch 108] time cost: 13.379844\n",
      "[Epoch 108] validation: accuracy=0.850861\n",
      "Epoch[109] Batch [49]\tSpeed: 3688.592240 samples/sec\taccuracy=0.888125\n",
      "Epoch[109] Batch [99]\tSpeed: 4076.932923 samples/sec\taccuracy=0.887891\n",
      "Epoch[109] Batch [149]\tSpeed: 4100.285730 samples/sec\taccuracy=0.887708\n",
      "Epoch[109] Batch [199]\tSpeed: 3986.240910 samples/sec\taccuracy=0.886055\n",
      "Epoch[109] Batch [249]\tSpeed: 3855.169553 samples/sec\taccuracy=0.885563\n",
      "Epoch[109] Batch [299]\tSpeed: 3471.006006 samples/sec\taccuracy=0.887031\n",
      "Epoch[109] Batch [349]\tSpeed: 3253.703945 samples/sec\taccuracy=0.888125\n",
      "[Epoch 109] training: accuracy=0.888642\n",
      "[Epoch 109] time cost: 13.760027\n",
      "[Epoch 109] validation: accuracy=0.853365\n",
      "Epoch[110] Batch [49]\tSpeed: 3225.806117 samples/sec\taccuracy=0.892813\n",
      "Epoch[110] Batch [99]\tSpeed: 3868.113261 samples/sec\taccuracy=0.889062\n",
      "Epoch[110] Batch [149]\tSpeed: 3972.407784 samples/sec\taccuracy=0.889531\n",
      "Epoch[110] Batch [199]\tSpeed: 4012.068334 samples/sec\taccuracy=0.889766\n",
      "Epoch[110] Batch [249]\tSpeed: 3551.743631 samples/sec\taccuracy=0.890406\n",
      "Epoch[110] Batch [299]\tSpeed: 3713.134044 samples/sec\taccuracy=0.889531\n",
      "Epoch[110] Batch [349]\tSpeed: 3851.905695 samples/sec\taccuracy=0.890134\n",
      "[Epoch 110] training: accuracy=0.889506\n",
      "[Epoch 110] time cost: 13.729698\n",
      "[Epoch 110] validation: accuracy=0.828225\n",
      "Epoch[111] Batch [49]\tSpeed: 3865.050049 samples/sec\taccuracy=0.892969\n",
      "Epoch[111] Batch [99]\tSpeed: 4032.681680 samples/sec\taccuracy=0.890781\n",
      "Epoch[111] Batch [149]\tSpeed: 2893.824011 samples/sec\taccuracy=0.890833\n",
      "Epoch[111] Batch [199]\tSpeed: 3752.111766 samples/sec\taccuracy=0.891602\n",
      "Epoch[111] Batch [249]\tSpeed: 3954.180227 samples/sec\taccuracy=0.892406\n",
      "Epoch[111] Batch [299]\tSpeed: 4038.870590 samples/sec\taccuracy=0.892161\n",
      "Epoch[111] Batch [349]\tSpeed: 2966.219575 samples/sec\taccuracy=0.893147\n",
      "[Epoch 111] training: accuracy=0.892909\n",
      "[Epoch 111] time cost: 13.754509\n",
      "[Epoch 111] validation: accuracy=0.845753\n",
      "Epoch[112] Batch [49]\tSpeed: 3489.936633 samples/sec\taccuracy=0.888750\n",
      "Epoch[112] Batch [99]\tSpeed: 4046.572490 samples/sec\taccuracy=0.886016\n",
      "Epoch[112] Batch [149]\tSpeed: 3921.227281 samples/sec\taccuracy=0.888542\n",
      "Epoch[112] Batch [199]\tSpeed: 3991.100842 samples/sec\taccuracy=0.889297\n",
      "Epoch[112] Batch [249]\tSpeed: 3922.631147 samples/sec\taccuracy=0.890375\n",
      "Epoch[112] Batch [299]\tSpeed: 3992.555195 samples/sec\taccuracy=0.890182\n",
      "Epoch[112] Batch [349]\tSpeed: 2966.875256 samples/sec\taccuracy=0.890960\n",
      "[Epoch 112] training: accuracy=0.890865\n",
      "[Epoch 112] time cost: 13.641119\n",
      "[Epoch 112] validation: accuracy=0.837619\n",
      "Epoch[113] Batch [49]\tSpeed: 3807.460104 samples/sec\taccuracy=0.897813\n",
      "Epoch[113] Batch [99]\tSpeed: 4215.745016 samples/sec\taccuracy=0.895703\n",
      "Epoch[113] Batch [149]\tSpeed: 3535.837194 samples/sec\taccuracy=0.894115\n",
      "Epoch[113] Batch [199]\tSpeed: 3878.340451 samples/sec\taccuracy=0.892773\n",
      "Epoch[113] Batch [249]\tSpeed: 3750.303254 samples/sec\taccuracy=0.891875\n",
      "Epoch[113] Batch [299]\tSpeed: 4034.409024 samples/sec\taccuracy=0.892318\n",
      "Epoch[113] Batch [349]\tSpeed: 2534.250880 samples/sec\taccuracy=0.893080\n",
      "[Epoch 113] training: accuracy=0.892323\n",
      "[Epoch 113] time cost: 13.598943\n",
      "[Epoch 113] validation: accuracy=0.826623\n",
      "Epoch[114] Batch [49]\tSpeed: 3828.202252 samples/sec\taccuracy=0.890469\n",
      "Epoch[114] Batch [99]\tSpeed: 3710.106161 samples/sec\taccuracy=0.893906\n",
      "Epoch[114] Batch [149]\tSpeed: 4076.128129 samples/sec\taccuracy=0.893177\n",
      "Epoch[114] Batch [199]\tSpeed: 4032.257646 samples/sec\taccuracy=0.892969\n",
      "Epoch[114] Batch [249]\tSpeed: 3271.907316 samples/sec\taccuracy=0.892125\n",
      "Epoch[114] Batch [299]\tSpeed: 3513.576084 samples/sec\taccuracy=0.892161\n",
      "Epoch[114] Batch [349]\tSpeed: 3867.054512 samples/sec\taccuracy=0.892589\n",
      "[Epoch 114] training: accuracy=0.892167\n",
      "[Epoch 114] time cost: 13.728774\n",
      "[Epoch 114] validation: accuracy=0.866987\n",
      "Epoch[115] Batch [49]\tSpeed: 3846.000573 samples/sec\taccuracy=0.895625\n",
      "Epoch[115] Batch [99]\tSpeed: 3991.694328 samples/sec\taccuracy=0.892891\n",
      "Epoch[115] Batch [149]\tSpeed: 2706.889414 samples/sec\taccuracy=0.894896\n",
      "Epoch[115] Batch [199]\tSpeed: 3941.928206 samples/sec\taccuracy=0.894375\n",
      "Epoch[115] Batch [249]\tSpeed: 3901.281207 samples/sec\taccuracy=0.891844\n",
      "Epoch[115] Batch [299]\tSpeed: 3854.311564 samples/sec\taccuracy=0.892109\n",
      "Epoch[115] Batch [349]\tSpeed: 3687.857451 samples/sec\taccuracy=0.892076\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Epoch 115] training: accuracy=0.892703\n",
      "[Epoch 115] time cost: 13.683660\n",
      "[Epoch 115] validation: accuracy=0.820212\n",
      "Epoch[116] Batch [49]\tSpeed: 3543.515273 samples/sec\taccuracy=0.889531\n",
      "Epoch[116] Batch [99]\tSpeed: 3857.579502 samples/sec\taccuracy=0.889062\n",
      "Epoch[116] Batch [149]\tSpeed: 3847.185662 samples/sec\taccuracy=0.889323\n",
      "Epoch[116] Batch [199]\tSpeed: 4005.841668 samples/sec\taccuracy=0.889961\n",
      "Epoch[116] Batch [249]\tSpeed: 2660.740488 samples/sec\taccuracy=0.889312\n",
      "Epoch[116] Batch [299]\tSpeed: 3964.165605 samples/sec\taccuracy=0.890104\n",
      "Epoch[116] Batch [349]\tSpeed: 3795.052606 samples/sec\taccuracy=0.891473\n",
      "[Epoch 116] training: accuracy=0.891664\n",
      "[Epoch 116] time cost: 14.062855\n",
      "[Epoch 116] validation: accuracy=0.846655\n",
      "Epoch[117] Batch [49]\tSpeed: 3990.477872 samples/sec\taccuracy=0.887969\n",
      "Epoch[117] Batch [99]\tSpeed: 3890.566275 samples/sec\taccuracy=0.886094\n",
      "Epoch[117] Batch [149]\tSpeed: 3918.594164 samples/sec\taccuracy=0.890104\n",
      "Epoch[117] Batch [199]\tSpeed: 3743.008317 samples/sec\taccuracy=0.891523\n",
      "Epoch[117] Batch [249]\tSpeed: 3300.753835 samples/sec\taccuracy=0.892906\n",
      "Epoch[117] Batch [299]\tSpeed: 3760.943418 samples/sec\taccuracy=0.892135\n",
      "Epoch[117] Batch [349]\tSpeed: 3930.931584 samples/sec\taccuracy=0.892612\n",
      "[Epoch 117] training: accuracy=0.892107\n",
      "[Epoch 117] time cost: 14.356192\n",
      "[Epoch 117] validation: accuracy=0.855268\n",
      "Epoch[118] Batch [49]\tSpeed: 3825.583503 samples/sec\taccuracy=0.886250\n",
      "Epoch[118] Batch [99]\tSpeed: 3930.701341 samples/sec\taccuracy=0.885625\n",
      "Epoch[118] Batch [149]\tSpeed: 3763.131454 samples/sec\taccuracy=0.888594\n",
      "Epoch[118] Batch [199]\tSpeed: 3874.030624 samples/sec\taccuracy=0.887695\n",
      "Epoch[118] Batch [249]\tSpeed: 3787.155226 samples/sec\taccuracy=0.888656\n",
      "Epoch[118] Batch [299]\tSpeed: 3949.758411 samples/sec\taccuracy=0.890964\n",
      "Epoch[118] Batch [349]\tSpeed: 3582.626503 samples/sec\taccuracy=0.892589\n",
      "[Epoch 118] training: accuracy=0.891944\n",
      "[Epoch 118] time cost: 13.590377\n",
      "[Epoch 118] validation: accuracy=0.846454\n",
      "Epoch[119] Batch [49]\tSpeed: 3974.525178 samples/sec\taccuracy=0.891875\n",
      "Epoch[119] Batch [99]\tSpeed: 4098.626683 samples/sec\taccuracy=0.888516\n",
      "Epoch[119] Batch [149]\tSpeed: 3879.994161 samples/sec\taccuracy=0.888177\n",
      "Epoch[119] Batch [199]\tSpeed: 3964.604714 samples/sec\taccuracy=0.891914\n",
      "Epoch[119] Batch [249]\tSpeed: 3452.146453 samples/sec\taccuracy=0.892000\n",
      "Epoch[119] Batch [299]\tSpeed: 3991.753686 samples/sec\taccuracy=0.892734\n",
      "Epoch[119] Batch [349]\tSpeed: 4132.191989 samples/sec\taccuracy=0.893683\n",
      "[Epoch 119] training: accuracy=0.893810\n",
      "[Epoch 119] time cost: 13.738540\n",
      "[Epoch 119] validation: accuracy=0.845653\n",
      "Epoch[120] Batch [49]\tSpeed: 4031.197952 samples/sec\taccuracy=0.890781\n",
      "Epoch[120] Batch [99]\tSpeed: 4029.261665 samples/sec\taccuracy=0.889687\n",
      "Epoch[120] Batch [149]\tSpeed: 3768.969862 samples/sec\taccuracy=0.887760\n",
      "Epoch[120] Batch [199]\tSpeed: 3887.946005 samples/sec\taccuracy=0.888789\n",
      "Epoch[120] Batch [249]\tSpeed: 3809.324176 samples/sec\taccuracy=0.888719\n",
      "Epoch[120] Batch [299]\tSpeed: 2186.597504 samples/sec\taccuracy=0.889297\n",
      "Epoch[120] Batch [349]\tSpeed: 3805.705763 samples/sec\taccuracy=0.891250\n",
      "[Epoch 120] training: accuracy=0.891404\n",
      "[Epoch 120] time cost: 13.726249\n",
      "[Epoch 120] validation: accuracy=0.850079\n",
      "Epoch[121] Batch [49]\tSpeed: 3930.183394 samples/sec\taccuracy=0.900312\n",
      "Epoch[121] Batch [99]\tSpeed: 3725.346860 samples/sec\taccuracy=0.896484\n",
      "Epoch[121] Batch [149]\tSpeed: 3633.742450 samples/sec\taccuracy=0.893906\n",
      "Epoch[121] Batch [199]\tSpeed: 3986.004143 samples/sec\taccuracy=0.894336\n",
      "Epoch[121] Batch [249]\tSpeed: 3767.409420 samples/sec\taccuracy=0.895344\n",
      "Epoch[121] Batch [299]\tSpeed: 3433.666413 samples/sec\taccuracy=0.893750\n",
      "Epoch[121] Batch [349]\tSpeed: 3963.843653 samples/sec\taccuracy=0.894353\n",
      "[Epoch 121] training: accuracy=0.894361\n",
      "[Epoch 121] time cost: 14.112600\n",
      "[Epoch 121] validation: accuracy=0.842849\n",
      "Epoch[122] Batch [49]\tSpeed: 4056.938595 samples/sec\taccuracy=0.888750\n",
      "Epoch[122] Batch [99]\tSpeed: 4011.288942 samples/sec\taccuracy=0.882812\n",
      "Epoch[122] Batch [149]\tSpeed: 3622.806305 samples/sec\taccuracy=0.885729\n",
      "Epoch[122] Batch [199]\tSpeed: 2977.306648 samples/sec\taccuracy=0.884727\n",
      "Epoch[122] Batch [249]\tSpeed: 3237.692375 samples/sec\taccuracy=0.887469\n",
      "Epoch[122] Batch [299]\tSpeed: 3887.326672 samples/sec\taccuracy=0.888437\n",
      "Epoch[122] Batch [349]\tSpeed: 3916.250233 samples/sec\taccuracy=0.889464\n",
      "[Epoch 122] training: accuracy=0.889283\n",
      "[Epoch 122] time cost: 13.740631\n",
      "[Epoch 122] validation: accuracy=0.830929\n",
      "Epoch[123] Batch [49]\tSpeed: 3928.314165 samples/sec\taccuracy=0.896875\n",
      "Epoch[123] Batch [99]\tSpeed: 2736.582232 samples/sec\taccuracy=0.895156\n",
      "Epoch[123] Batch [149]\tSpeed: 2626.467222 samples/sec\taccuracy=0.892552\n",
      "Epoch[123] Batch [199]\tSpeed: 3909.263702 samples/sec\taccuracy=0.890977\n",
      "Epoch[123] Batch [249]\tSpeed: 3829.157896 samples/sec\taccuracy=0.890281\n",
      "Epoch[123] Batch [299]\tSpeed: 4035.925457 samples/sec\taccuracy=0.891198\n",
      "Epoch[123] Batch [349]\tSpeed: 4030.229576 samples/sec\taccuracy=0.892299\n",
      "[Epoch 123] training: accuracy=0.892463\n",
      "[Epoch 123] time cost: 13.604674\n",
      "[Epoch 123] validation: accuracy=0.846454\n",
      "Epoch[124] Batch [49]\tSpeed: 4052.620585 samples/sec\taccuracy=0.883906\n",
      "Epoch[124] Batch [99]\tSpeed: 3966.157015 samples/sec\taccuracy=0.891250\n",
      "Epoch[124] Batch [149]\tSpeed: 3753.475855 samples/sec\taccuracy=0.890833\n",
      "Epoch[124] Batch [199]\tSpeed: 3771.829616 samples/sec\taccuracy=0.890000\n",
      "Epoch[124] Batch [249]\tSpeed: 3553.412705 samples/sec\taccuracy=0.890844\n",
      "Epoch[124] Batch [299]\tSpeed: 3832.574810 samples/sec\taccuracy=0.892943\n",
      "Epoch[124] Batch [349]\tSpeed: 3691.356656 samples/sec\taccuracy=0.893080\n",
      "[Epoch 124] training: accuracy=0.892643\n",
      "[Epoch 124] time cost: 13.301790\n",
      "[Epoch 124] validation: accuracy=0.854968\n",
      "Epoch[125] Batch [49]\tSpeed: 3936.234618 samples/sec\taccuracy=0.894219\n",
      "Epoch[125] Batch [99]\tSpeed: 3899.354396 samples/sec\taccuracy=0.893281\n",
      "Epoch[125] Batch [149]\tSpeed: 3780.488216 samples/sec\taccuracy=0.892031\n",
      "Epoch[125] Batch [199]\tSpeed: 3538.727149 samples/sec\taccuracy=0.890820\n",
      "Epoch[125] Batch [249]\tSpeed: 3794.221164 samples/sec\taccuracy=0.891687\n",
      "Epoch[125] Batch [299]\tSpeed: 3838.274091 samples/sec\taccuracy=0.892214\n",
      "Epoch[125] Batch [349]\tSpeed: 3853.343324 samples/sec\taccuracy=0.892969\n",
      "[Epoch 125] training: accuracy=0.893069\n",
      "[Epoch 125] time cost: 13.921700\n",
      "[Epoch 125] validation: accuracy=0.838842\n",
      "Epoch[126] Batch [49]\tSpeed: 3813.247381 samples/sec\taccuracy=0.890000\n",
      "Epoch[126] Batch [99]\tSpeed: 3728.710417 samples/sec\taccuracy=0.892813\n",
      "Epoch[126] Batch [149]\tSpeed: 2856.653623 samples/sec\taccuracy=0.893281\n",
      "Epoch[126] Batch [199]\tSpeed: 3877.220094 samples/sec\taccuracy=0.893516\n",
      "Epoch[126] Batch [249]\tSpeed: 3796.850841 samples/sec\taccuracy=0.894031\n",
      "Epoch[126] Batch [299]\tSpeed: 3804.600010 samples/sec\taccuracy=0.893464\n",
      "Epoch[126] Batch [349]\tSpeed: 4069.670346 samples/sec\taccuracy=0.893393\n",
      "[Epoch 126] training: accuracy=0.893822\n",
      "[Epoch 126] time cost: 13.657172\n",
      "[Epoch 126] validation: accuracy=0.839543\n",
      "Epoch[127] Batch [49]\tSpeed: 2131.903696 samples/sec\taccuracy=0.891094\n",
      "Epoch[127] Batch [99]\tSpeed: 3806.299359 samples/sec\taccuracy=0.892031\n",
      "Epoch[127] Batch [149]\tSpeed: 3916.107402 samples/sec\taccuracy=0.892760\n",
      "Epoch[127] Batch [199]\tSpeed: 3916.364506 samples/sec\taccuracy=0.893711\n",
      "Epoch[127] Batch [249]\tSpeed: 2081.814879 samples/sec\taccuracy=0.895031\n",
      "Epoch[127] Batch [299]\tSpeed: 4117.045076 samples/sec\taccuracy=0.894245\n",
      "Epoch[127] Batch [349]\tSpeed: 3720.261326 samples/sec\taccuracy=0.894911\n",
      "[Epoch 127] training: accuracy=0.894010\n",
      "[Epoch 127] time cost: 13.284156\n",
      "[Epoch 127] validation: accuracy=0.824619\n",
      "Epoch[128] Batch [49]\tSpeed: 3994.159180 samples/sec\taccuracy=0.894062\n",
      "Epoch[128] Batch [99]\tSpeed: 2524.230028 samples/sec\taccuracy=0.894922\n",
      "Epoch[128] Batch [149]\tSpeed: 4058.195914 samples/sec\taccuracy=0.893333\n",
      "Epoch[128] Batch [199]\tSpeed: 3828.802887 samples/sec\taccuracy=0.893633\n",
      "Epoch[128] Batch [249]\tSpeed: 3648.931306 samples/sec\taccuracy=0.893906\n",
      "Epoch[128] Batch [299]\tSpeed: 3822.560036 samples/sec\taccuracy=0.894922\n",
      "Epoch[128] Batch [349]\tSpeed: 3964.165605 samples/sec\taccuracy=0.895045\n",
      "[Epoch 128] training: accuracy=0.895081\n",
      "[Epoch 128] time cost: 13.831303\n",
      "[Epoch 128] validation: accuracy=0.873121\n",
      "Epoch[129] Batch [49]\tSpeed: 4021.203745 samples/sec\taccuracy=0.895156\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[129] Batch [99]\tSpeed: 2373.728454 samples/sec\taccuracy=0.890391\n",
      "Epoch[129] Batch [149]\tSpeed: 3942.130819 samples/sec\taccuracy=0.889479\n",
      "Epoch[129] Batch [199]\tSpeed: 4059.423317 samples/sec\taccuracy=0.890000\n",
      "Epoch[129] Batch [249]\tSpeed: 3798.032698 samples/sec\taccuracy=0.890875\n",
      "Epoch[129] Batch [299]\tSpeed: 3896.552587 samples/sec\taccuracy=0.891615\n",
      "Epoch[129] Batch [349]\tSpeed: 3961.036108 samples/sec\taccuracy=0.893013\n",
      "[Epoch 129] training: accuracy=0.892703\n",
      "[Epoch 129] time cost: 13.522790\n",
      "[Epoch 129] validation: accuracy=0.852063\n",
      "Epoch[130] Batch [49]\tSpeed: 3930.327255 samples/sec\taccuracy=0.894375\n",
      "Epoch[130] Batch [99]\tSpeed: 3778.067247 samples/sec\taccuracy=0.893516\n",
      "Epoch[130] Batch [149]\tSpeed: 3526.685839 samples/sec\taccuracy=0.894583\n",
      "Epoch[130] Batch [199]\tSpeed: 3646.948339 samples/sec\taccuracy=0.894844\n",
      "Epoch[130] Batch [249]\tSpeed: 3624.273702 samples/sec\taccuracy=0.894312\n",
      "Epoch[130] Batch [299]\tSpeed: 4002.556526 samples/sec\taccuracy=0.894401\n",
      "Epoch[130] Batch [349]\tSpeed: 3729.383858 samples/sec\taccuracy=0.895424\n",
      "[Epoch 130] training: accuracy=0.894932\n",
      "[Epoch 130] time cost: 13.740923\n",
      "[Epoch 130] validation: accuracy=0.861579\n",
      "Epoch[131] Batch [49]\tSpeed: 3989.914401 samples/sec\taccuracy=0.895781\n",
      "Epoch[131] Batch [99]\tSpeed: 3777.562162 samples/sec\taccuracy=0.891563\n",
      "Epoch[131] Batch [149]\tSpeed: 3981.363274 samples/sec\taccuracy=0.893437\n",
      "Epoch[131] Batch [199]\tSpeed: 3827.219801 samples/sec\taccuracy=0.893945\n",
      "Epoch[131] Batch [249]\tSpeed: 2869.310302 samples/sec\taccuracy=0.894250\n",
      "Epoch[131] Batch [299]\tSpeed: 3926.360566 samples/sec\taccuracy=0.895312\n",
      "Epoch[131] Batch [349]\tSpeed: 3772.598252 samples/sec\taccuracy=0.895826\n",
      "[Epoch 131] training: accuracy=0.896040\n",
      "[Epoch 131] time cost: 13.890925\n",
      "[Epoch 131] validation: accuracy=0.845853\n",
      "Epoch[132] Batch [49]\tSpeed: 4019.337226 samples/sec\taccuracy=0.897656\n",
      "Epoch[132] Batch [99]\tSpeed: 3932.572843 samples/sec\taccuracy=0.896563\n",
      "Epoch[132] Batch [149]\tSpeed: 3984.968617 samples/sec\taccuracy=0.896406\n",
      "Epoch[132] Batch [199]\tSpeed: 3762.920448 samples/sec\taccuracy=0.894141\n",
      "Epoch[132] Batch [249]\tSpeed: 3567.841249 samples/sec\taccuracy=0.894250\n",
      "Epoch[132] Batch [299]\tSpeed: 4128.696433 samples/sec\taccuracy=0.893802\n",
      "Epoch[132] Batch [349]\tSpeed: 4148.604528 samples/sec\taccuracy=0.894263\n",
      "[Epoch 132] training: accuracy=0.894142\n",
      "[Epoch 132] time cost: 13.748241\n",
      "[Epoch 132] validation: accuracy=0.836538\n",
      "Epoch[133] Batch [49]\tSpeed: 3826.074245 samples/sec\taccuracy=0.893125\n",
      "Epoch[133] Batch [99]\tSpeed: 3822.097405 samples/sec\taccuracy=0.892813\n",
      "Epoch[133] Batch [149]\tSpeed: 3952.957420 samples/sec\taccuracy=0.892917\n",
      "Epoch[133] Batch [199]\tSpeed: 4083.258509 samples/sec\taccuracy=0.892930\n",
      "Epoch[133] Batch [249]\tSpeed: 2794.063462 samples/sec\taccuracy=0.893875\n",
      "Epoch[133] Batch [299]\tSpeed: 2993.225499 samples/sec\taccuracy=0.894792\n",
      "Epoch[133] Batch [349]\tSpeed: 3960.919213 samples/sec\taccuracy=0.894420\n",
      "[Epoch 133] training: accuracy=0.893930\n",
      "[Epoch 133] time cost: 13.534896\n",
      "[Epoch 133] validation: accuracy=0.850160\n",
      "Epoch[134] Batch [49]\tSpeed: 3969.646801 samples/sec\taccuracy=0.891094\n",
      "Epoch[134] Batch [99]\tSpeed: 3715.806787 samples/sec\taccuracy=0.891953\n",
      "Epoch[134] Batch [149]\tSpeed: 3997.728208 samples/sec\taccuracy=0.894375\n",
      "Epoch[134] Batch [199]\tSpeed: 4066.402921 samples/sec\taccuracy=0.894492\n",
      "Epoch[134] Batch [249]\tSpeed: 2581.792839 samples/sec\taccuracy=0.895281\n",
      "Epoch[134] Batch [299]\tSpeed: 4018.555008 samples/sec\taccuracy=0.894661\n",
      "Epoch[134] Batch [349]\tSpeed: 4085.713399 samples/sec\taccuracy=0.895960\n",
      "[Epoch 134] training: accuracy=0.896160\n",
      "[Epoch 134] time cost: 13.208851\n",
      "[Epoch 134] validation: accuracy=0.850661\n",
      "Epoch[135] Batch [49]\tSpeed: 3981.599490 samples/sec\taccuracy=0.899219\n",
      "Epoch[135] Batch [99]\tSpeed: 3838.438745 samples/sec\taccuracy=0.894609\n",
      "Epoch[135] Batch [149]\tSpeed: 3880.246545 samples/sec\taccuracy=0.894271\n",
      "Epoch[135] Batch [199]\tSpeed: 4002.765420 samples/sec\taccuracy=0.893789\n",
      "Epoch[135] Batch [249]\tSpeed: 2795.736711 samples/sec\taccuracy=0.894906\n",
      "Epoch[135] Batch [299]\tSpeed: 3831.808892 samples/sec\taccuracy=0.894557\n",
      "Epoch[135] Batch [349]\tSpeed: 3806.569237 samples/sec\taccuracy=0.895491\n",
      "[Epoch 135] training: accuracy=0.895232\n",
      "[Epoch 135] time cost: 13.931178\n",
      "[Epoch 135] validation: accuracy=0.848858\n",
      "Epoch[136] Batch [49]\tSpeed: 3660.724766 samples/sec\taccuracy=0.897344\n",
      "Epoch[136] Batch [99]\tSpeed: 3917.879254 samples/sec\taccuracy=0.897188\n",
      "Epoch[136] Batch [149]\tSpeed: 3943.897331 samples/sec\taccuracy=0.897760\n",
      "Epoch[136] Batch [199]\tSpeed: 3880.835571 samples/sec\taccuracy=0.898086\n",
      "Epoch[136] Batch [249]\tSpeed: 3751.954435 samples/sec\taccuracy=0.897656\n",
      "Epoch[136] Batch [299]\tSpeed: 3909.349101 samples/sec\taccuracy=0.896589\n",
      "Epoch[136] Batch [349]\tSpeed: 2355.368469 samples/sec\taccuracy=0.896920\n",
      "[Epoch 136] training: accuracy=0.896240\n",
      "[Epoch 136] time cost: 13.776248\n",
      "[Epoch 136] validation: accuracy=0.853145\n",
      "Epoch[137] Batch [49]\tSpeed: 3927.480775 samples/sec\taccuracy=0.893594\n",
      "Epoch[137] Batch [99]\tSpeed: 4134.037485 samples/sec\taccuracy=0.893437\n",
      "Epoch[137] Batch [149]\tSpeed: 3869.144706 samples/sec\taccuracy=0.894427\n",
      "Epoch[137] Batch [199]\tSpeed: 4028.536037 samples/sec\taccuracy=0.894805\n",
      "Epoch[137] Batch [249]\tSpeed: 3751.928214 samples/sec\taccuracy=0.894031\n",
      "Epoch[137] Batch [299]\tSpeed: 3493.820321 samples/sec\taccuracy=0.894036\n",
      "Epoch[137] Batch [349]\tSpeed: 4002.944490 samples/sec\taccuracy=0.894040\n",
      "[Epoch 137] training: accuracy=0.894242\n",
      "[Epoch 137] time cost: 13.383337\n",
      "[Epoch 137] validation: accuracy=0.859375\n",
      "Epoch[138] Batch [49]\tSpeed: 3953.277606 samples/sec\taccuracy=0.897813\n",
      "Epoch[138] Batch [99]\tSpeed: 3917.965029 samples/sec\taccuracy=0.894375\n",
      "Epoch[138] Batch [149]\tSpeed: 3846.303666 samples/sec\taccuracy=0.897031\n",
      "Epoch[138] Batch [199]\tSpeed: 4079.845217 samples/sec\taccuracy=0.895508\n",
      "Epoch[138] Batch [249]\tSpeed: 3761.417716 samples/sec\taccuracy=0.893687\n",
      "Epoch[138] Batch [299]\tSpeed: 3997.162687 samples/sec\taccuracy=0.893828\n",
      "Epoch[138] Batch [349]\tSpeed: 2416.095479 samples/sec\taccuracy=0.893571\n",
      "[Epoch 138] training: accuracy=0.894291\n",
      "[Epoch 138] time cost: 13.607674\n",
      "[Epoch 138] validation: accuracy=0.829728\n",
      "Epoch[139] Batch [49]\tSpeed: 2097.143808 samples/sec\taccuracy=0.900312\n",
      "Epoch[139] Batch [99]\tSpeed: 3837.889954 samples/sec\taccuracy=0.896719\n",
      "Epoch[139] Batch [149]\tSpeed: 3905.396213 samples/sec\taccuracy=0.893125\n",
      "Epoch[139] Batch [199]\tSpeed: 3008.742088 samples/sec\taccuracy=0.893281\n",
      "Epoch[139] Batch [249]\tSpeed: 3748.077772 samples/sec\taccuracy=0.895031\n",
      "Epoch[139] Batch [299]\tSpeed: 4002.049303 samples/sec\taccuracy=0.896146\n",
      "Epoch[139] Batch [349]\tSpeed: 3729.332046 samples/sec\taccuracy=0.896540\n",
      "[Epoch 139] training: accuracy=0.896959\n",
      "[Epoch 139] time cost: 14.546894\n",
      "[Epoch 139] validation: accuracy=0.821615\n",
      "Epoch[140] Batch [49]\tSpeed: 3765.876685 samples/sec\taccuracy=0.900625\n",
      "Epoch[140] Batch [99]\tSpeed: 3917.479018 samples/sec\taccuracy=0.897813\n",
      "Epoch[140] Batch [149]\tSpeed: 3837.670481 samples/sec\taccuracy=0.896771\n",
      "Epoch[140] Batch [199]\tSpeed: 3954.034615 samples/sec\taccuracy=0.895117\n",
      "Epoch[140] Batch [249]\tSpeed: 3977.528687 samples/sec\taccuracy=0.895000\n",
      "Epoch[140] Batch [299]\tSpeed: 4135.661611 samples/sec\taccuracy=0.895495\n",
      "Epoch[140] Batch [349]\tSpeed: 2787.766768 samples/sec\taccuracy=0.896585\n",
      "[Epoch 140] training: accuracy=0.895940\n",
      "[Epoch 140] time cost: 13.727086\n",
      "[Epoch 140] validation: accuracy=0.860176\n",
      "Epoch[141] Batch [49]\tSpeed: 4038.141497 samples/sec\taccuracy=0.890781\n",
      "Epoch[141] Batch [99]\tSpeed: 3952.666387 samples/sec\taccuracy=0.892500\n",
      "Epoch[141] Batch [149]\tSpeed: 3850.248225 samples/sec\taccuracy=0.894687\n",
      "Epoch[141] Batch [199]\tSpeed: 4014.258245 samples/sec\taccuracy=0.893633\n",
      "Epoch[141] Batch [249]\tSpeed: 3981.304224 samples/sec\taccuracy=0.892969\n",
      "Epoch[141] Batch [299]\tSpeed: 3827.274368 samples/sec\taccuracy=0.892135\n",
      "Epoch[141] Batch [349]\tSpeed: 3930.557453 samples/sec\taccuracy=0.893326\n",
      "[Epoch 141] training: accuracy=0.893650\n",
      "[Epoch 141] time cost: 13.572323\n",
      "[Epoch 141] validation: accuracy=0.863882\n",
      "Epoch[142] Batch [49]\tSpeed: 3885.722954 samples/sec\taccuracy=0.890312\n",
      "Epoch[142] Batch [99]\tSpeed: 3965.336780 samples/sec\taccuracy=0.894219\n",
      "Epoch[142] Batch [149]\tSpeed: 2490.887340 samples/sec\taccuracy=0.895365\n",
      "Epoch[142] Batch [199]\tSpeed: 3954.733651 samples/sec\taccuracy=0.893516\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[142] Batch [249]\tSpeed: 3672.971594 samples/sec\taccuracy=0.893750\n",
      "Epoch[142] Batch [299]\tSpeed: 3899.269434 samples/sec\taccuracy=0.893542\n",
      "Epoch[142] Batch [349]\tSpeed: 2706.261749 samples/sec\taccuracy=0.894643\n",
      "[Epoch 142] training: accuracy=0.894501\n",
      "[Epoch 142] time cost: 14.177687\n",
      "[Epoch 142] validation: accuracy=0.853165\n",
      "Epoch[143] Batch [49]\tSpeed: 3920.482781 samples/sec\taccuracy=0.892188\n",
      "Epoch[143] Batch [99]\tSpeed: 3745.306163 samples/sec\taccuracy=0.892969\n",
      "Epoch[143] Batch [149]\tSpeed: 3957.065554 samples/sec\taccuracy=0.897135\n",
      "Epoch[143] Batch [199]\tSpeed: 3847.158094 samples/sec\taccuracy=0.897070\n",
      "Epoch[143] Batch [249]\tSpeed: 3252.600052 samples/sec\taccuracy=0.896344\n",
      "Epoch[143] Batch [299]\tSpeed: 4002.496846 samples/sec\taccuracy=0.896536\n",
      "Epoch[143] Batch [349]\tSpeed: 3964.311964 samples/sec\taccuracy=0.897232\n",
      "[Epoch 143] training: accuracy=0.897736\n",
      "[Epoch 143] time cost: 13.924367\n",
      "[Epoch 143] validation: accuracy=0.853165\n",
      "Epoch[144] Batch [49]\tSpeed: 4016.059964 samples/sec\taccuracy=0.894687\n",
      "Epoch[144] Batch [99]\tSpeed: 3951.415433 samples/sec\taccuracy=0.894062\n",
      "Epoch[144] Batch [149]\tSpeed: 2171.421397 samples/sec\taccuracy=0.894167\n",
      "Epoch[144] Batch [199]\tSpeed: 3703.273129 samples/sec\taccuracy=0.894219\n",
      "Epoch[144] Batch [249]\tSpeed: 2491.083822 samples/sec\taccuracy=0.895500\n",
      "Epoch[144] Batch [299]\tSpeed: 4043.372486 samples/sec\taccuracy=0.895990\n",
      "Epoch[144] Batch [349]\tSpeed: 3886.538715 samples/sec\taccuracy=0.896362\n",
      "[Epoch 144] training: accuracy=0.896240\n",
      "[Epoch 144] time cost: 13.718903\n",
      "[Epoch 144] validation: accuracy=0.863726\n",
      "Epoch[145] Batch [49]\tSpeed: 3876.548191 samples/sec\taccuracy=0.896250\n",
      "Epoch[145] Batch [99]\tSpeed: 3502.436064 samples/sec\taccuracy=0.893516\n",
      "Epoch[145] Batch [149]\tSpeed: 3751.508735 samples/sec\taccuracy=0.894948\n",
      "Epoch[145] Batch [199]\tSpeed: 4047.548737 samples/sec\taccuracy=0.893555\n",
      "Epoch[145] Batch [249]\tSpeed: 2860.687327 samples/sec\taccuracy=0.895188\n",
      "Epoch[145] Batch [299]\tSpeed: 2599.368216 samples/sec\taccuracy=0.895443\n",
      "Epoch[145] Batch [349]\tSpeed: 3253.033635 samples/sec\taccuracy=0.897098\n",
      "[Epoch 145] training: accuracy=0.897099\n",
      "[Epoch 145] time cost: 13.608664\n",
      "[Epoch 145] validation: accuracy=0.826923\n",
      "Epoch[146] Batch [49]\tSpeed: 3985.560280 samples/sec\taccuracy=0.898594\n",
      "Epoch[146] Batch [99]\tSpeed: 4098.970903 samples/sec\taccuracy=0.898516\n",
      "Epoch[146] Batch [149]\tSpeed: 3929.234179 samples/sec\taccuracy=0.896979\n",
      "Epoch[146] Batch [199]\tSpeed: 2051.521670 samples/sec\taccuracy=0.893516\n",
      "Epoch[146] Batch [249]\tSpeed: 2243.421317 samples/sec\taccuracy=0.893656\n",
      "Epoch[146] Batch [299]\tSpeed: 3646.824476 samples/sec\taccuracy=0.895104\n",
      "Epoch[146] Batch [349]\tSpeed: 3737.432122 samples/sec\taccuracy=0.895179\n",
      "[Epoch 146] training: accuracy=0.894912\n",
      "[Epoch 146] time cost: 13.799861\n",
      "[Epoch 146] validation: accuracy=0.829828\n",
      "Epoch[147] Batch [49]\tSpeed: 2141.889034 samples/sec\taccuracy=0.890781\n",
      "Epoch[147] Batch [99]\tSpeed: 4077.397372 samples/sec\taccuracy=0.891797\n",
      "Epoch[147] Batch [149]\tSpeed: 3994.931928 samples/sec\taccuracy=0.891094\n",
      "Epoch[147] Batch [199]\tSpeed: 3910.829936 samples/sec\taccuracy=0.892461\n",
      "Epoch[147] Batch [249]\tSpeed: 4020.149852 samples/sec\taccuracy=0.893062\n",
      "Epoch[147] Batch [299]\tSpeed: 3371.330595 samples/sec\taccuracy=0.894844\n",
      "Epoch[147] Batch [349]\tSpeed: 3783.712115 samples/sec\taccuracy=0.895915\n",
      "[Epoch 147] training: accuracy=0.896280\n",
      "[Epoch 147] time cost: 13.578592\n",
      "[Epoch 147] validation: accuracy=0.831530\n",
      "Epoch[148] Batch [49]\tSpeed: 3584.492255 samples/sec\taccuracy=0.899531\n",
      "Epoch[148] Batch [99]\tSpeed: 3125.321846 samples/sec\taccuracy=0.895625\n",
      "Epoch[148] Batch [149]\tSpeed: 3745.724257 samples/sec\taccuracy=0.894010\n",
      "Epoch[148] Batch [199]\tSpeed: 2888.064166 samples/sec\taccuracy=0.895547\n",
      "Epoch[148] Batch [249]\tSpeed: 3054.707269 samples/sec\taccuracy=0.896312\n",
      "Epoch[148] Batch [299]\tSpeed: 3878.312435 samples/sec\taccuracy=0.897396\n",
      "Epoch[148] Batch [349]\tSpeed: 3987.336324 samples/sec\taccuracy=0.897098\n",
      "[Epoch 148] training: accuracy=0.897339\n",
      "[Epoch 148] time cost: 13.431555\n",
      "[Epoch 148] validation: accuracy=0.855669\n",
      "Epoch[149] Batch [49]\tSpeed: 3925.815055 samples/sec\taccuracy=0.895938\n",
      "Epoch[149] Batch [99]\tSpeed: 2144.824066 samples/sec\taccuracy=0.892813\n",
      "Epoch[149] Batch [149]\tSpeed: 3796.716585 samples/sec\taccuracy=0.892708\n",
      "Epoch[149] Batch [199]\tSpeed: 3853.149735 samples/sec\taccuracy=0.894219\n",
      "Epoch[149] Batch [249]\tSpeed: 3835.696357 samples/sec\taccuracy=0.894531\n",
      "Epoch[149] Batch [299]\tSpeed: 3471.746715 samples/sec\taccuracy=0.895573\n",
      "Epoch[149] Batch [349]\tSpeed: 4010.090469 samples/sec\taccuracy=0.895692\n",
      "[Epoch 149] training: accuracy=0.896214\n",
      "[Epoch 149] time cost: 13.883907\n",
      "[Epoch 149] validation: accuracy=0.840545\n",
      "Epoch[150] Batch [49]\tSpeed: 3985.589868 samples/sec\taccuracy=0.898750\n",
      "Epoch[150] Batch [99]\tSpeed: 3938.920403 samples/sec\taccuracy=0.888359\n",
      "Epoch[150] Batch [149]\tSpeed: 4134.674245 samples/sec\taccuracy=0.895365\n",
      "Epoch[150] Batch [199]\tSpeed: 4087.828833 samples/sec\taccuracy=0.901250\n",
      "Epoch[150] Batch [249]\tSpeed: 3709.772882 samples/sec\taccuracy=0.907031\n",
      "Epoch[150] Batch [299]\tSpeed: 3538.960416 samples/sec\taccuracy=0.910495\n",
      "Epoch[150] Batch [349]\tSpeed: 3774.507945 samples/sec\taccuracy=0.914397\n",
      "[Epoch 150] training: accuracy=0.917599\n",
      "[Epoch 150] time cost: 13.523962\n",
      "[Epoch 150] validation: accuracy=0.902043\n",
      "Epoch[151] Batch [49]\tSpeed: 3978.973163 samples/sec\taccuracy=0.950313\n",
      "Epoch[151] Batch [99]\tSpeed: 3942.796695 samples/sec\taccuracy=0.942813\n",
      "Epoch[151] Batch [149]\tSpeed: 3963.170649 samples/sec\taccuracy=0.940937\n",
      "Epoch[151] Batch [199]\tSpeed: 3881.452837 samples/sec\taccuracy=0.940820\n",
      "Epoch[151] Batch [249]\tSpeed: 3994.634682 samples/sec\taccuracy=0.941187\n",
      "Epoch[151] Batch [299]\tSpeed: 4052.406455 samples/sec\taccuracy=0.941016\n",
      "Epoch[151] Batch [349]\tSpeed: 3973.760303 samples/sec\taccuracy=0.941652\n",
      "[Epoch 151] training: accuracy=0.942348\n",
      "[Epoch 151] time cost: 13.084672\n",
      "[Epoch 151] validation: accuracy=0.905849\n",
      "Epoch[152] Batch [49]\tSpeed: 4048.372811 samples/sec\taccuracy=0.951406\n",
      "Epoch[152] Batch [99]\tSpeed: 3872.074777 samples/sec\taccuracy=0.946094\n",
      "Epoch[152] Batch [149]\tSpeed: 3963.112138 samples/sec\taccuracy=0.944271\n",
      "Epoch[152] Batch [199]\tSpeed: 3987.395553 samples/sec\taccuracy=0.944180\n",
      "Epoch[152] Batch [249]\tSpeed: 2950.861628 samples/sec\taccuracy=0.944750\n",
      "Epoch[152] Batch [299]\tSpeed: 3972.113880 samples/sec\taccuracy=0.945182\n",
      "Epoch[152] Batch [349]\tSpeed: 3939.267223 samples/sec\taccuracy=0.946161\n",
      "[Epoch 152] training: accuracy=0.947071\n",
      "[Epoch 152] time cost: 13.513325\n",
      "[Epoch 152] validation: accuracy=0.906744\n",
      "Epoch[153] Batch [49]\tSpeed: 3953.918133 samples/sec\taccuracy=0.954531\n",
      "Epoch[153] Batch [99]\tSpeed: 4073.622921 samples/sec\taccuracy=0.950547\n",
      "Epoch[153] Batch [149]\tSpeed: 3728.658624 samples/sec\taccuracy=0.950208\n",
      "Epoch[153] Batch [199]\tSpeed: 3966.069116 samples/sec\taccuracy=0.949570\n",
      "Epoch[153] Batch [249]\tSpeed: 4002.646050 samples/sec\taccuracy=0.950094\n",
      "Epoch[153] Batch [299]\tSpeed: 4173.988416 samples/sec\taccuracy=0.950260\n",
      "Epoch[153] Batch [349]\tSpeed: 3807.460104 samples/sec\taccuracy=0.951362\n",
      "[Epoch 153] training: accuracy=0.951606\n",
      "[Epoch 153] time cost: 13.502347\n",
      "[Epoch 153] validation: accuracy=0.908253\n",
      "Epoch[154] Batch [49]\tSpeed: 3038.439055 samples/sec\taccuracy=0.955937\n",
      "Epoch[154] Batch [99]\tSpeed: 3996.359327 samples/sec\taccuracy=0.952656\n",
      "Epoch[154] Batch [149]\tSpeed: 4002.526686 samples/sec\taccuracy=0.952135\n",
      "Epoch[154] Batch [199]\tSpeed: 4022.228056 samples/sec\taccuracy=0.952539\n",
      "Epoch[154] Batch [249]\tSpeed: 2740.927299 samples/sec\taccuracy=0.952688\n",
      "Epoch[154] Batch [299]\tSpeed: 4067.666114 samples/sec\taccuracy=0.953047\n",
      "Epoch[154] Batch [349]\tSpeed: 3858.244845 samples/sec\taccuracy=0.953705\n",
      "[Epoch 154] training: accuracy=0.954026\n",
      "[Epoch 154] time cost: 13.778263\n",
      "[Epoch 154] validation: accuracy=0.911458\n",
      "Epoch[155] Batch [49]\tSpeed: 3333.111354 samples/sec\taccuracy=0.956094\n",
      "Epoch[155] Batch [99]\tSpeed: 3857.607220 samples/sec\taccuracy=0.953047\n",
      "Epoch[155] Batch [149]\tSpeed: 3790.122923 samples/sec\taccuracy=0.952344\n",
      "Epoch[155] Batch [199]\tSpeed: 2895.790719 samples/sec\taccuracy=0.951836\n",
      "Epoch[155] Batch [249]\tSpeed: 3090.722793 samples/sec\taccuracy=0.952406\n",
      "Epoch[155] Batch [299]\tSpeed: 3620.876045 samples/sec\taccuracy=0.953646\n",
      "Epoch[155] Batch [349]\tSpeed: 3905.481443 samples/sec\taccuracy=0.954129\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Epoch 155] training: accuracy=0.954963\n",
      "[Epoch 155] time cost: 13.801093\n",
      "[Epoch 155] validation: accuracy=0.907051\n",
      "Epoch[156] Batch [49]\tSpeed: 3800.990562 samples/sec\taccuracy=0.961406\n",
      "Epoch[156] Batch [99]\tSpeed: 2300.336401 samples/sec\taccuracy=0.957187\n",
      "Epoch[156] Batch [149]\tSpeed: 3811.947770 samples/sec\taccuracy=0.956406\n",
      "Epoch[156] Batch [199]\tSpeed: 3990.922831 samples/sec\taccuracy=0.956094\n",
      "Epoch[156] Batch [249]\tSpeed: 3684.086765 samples/sec\taccuracy=0.955688\n",
      "Epoch[156] Batch [299]\tSpeed: 2220.750654 samples/sec\taccuracy=0.955625\n",
      "Epoch[156] Batch [349]\tSpeed: 3842.972270 samples/sec\taccuracy=0.956272\n",
      "[Epoch 156] training: accuracy=0.956522\n",
      "[Epoch 156] time cost: 13.768610\n",
      "[Epoch 156] validation: accuracy=0.907652\n",
      "Epoch[157] Batch [49]\tSpeed: 3581.407638 samples/sec\taccuracy=0.959063\n",
      "Epoch[157] Batch [99]\tSpeed: 3952.986526 samples/sec\taccuracy=0.956641\n",
      "Epoch[157] Batch [149]\tSpeed: 4229.461398 samples/sec\taccuracy=0.956562\n",
      "Epoch[157] Batch [199]\tSpeed: 3959.838264 samples/sec\taccuracy=0.956406\n",
      "Epoch[157] Batch [249]\tSpeed: 3843.852739 samples/sec\taccuracy=0.956438\n",
      "Epoch[157] Batch [299]\tSpeed: 2980.083107 samples/sec\taccuracy=0.956250\n",
      "Epoch[157] Batch [349]\tSpeed: 4057.735829 samples/sec\taccuracy=0.957277\n",
      "[Epoch 157] training: accuracy=0.957452\n",
      "[Epoch 157] time cost: 13.621229\n",
      "[Epoch 157] validation: accuracy=0.910056\n",
      "Epoch[158] Batch [49]\tSpeed: 4104.831501 samples/sec\taccuracy=0.959063\n",
      "Epoch[158] Batch [99]\tSpeed: 3709.952333 samples/sec\taccuracy=0.958359\n",
      "Epoch[158] Batch [149]\tSpeed: 3989.262158 samples/sec\taccuracy=0.957969\n",
      "Epoch[158] Batch [199]\tSpeed: 3792.773714 samples/sec\taccuracy=0.958125\n",
      "Epoch[158] Batch [249]\tSpeed: 3770.425676 samples/sec\taccuracy=0.958656\n",
      "Epoch[158] Batch [299]\tSpeed: 3920.511410 samples/sec\taccuracy=0.958333\n",
      "Epoch[158] Batch [349]\tSpeed: 3801.717288 samples/sec\taccuracy=0.959241\n",
      "[Epoch 158] training: accuracy=0.959299\n",
      "[Epoch 158] time cost: 13.560778\n",
      "[Epoch 158] validation: accuracy=0.909155\n",
      "Epoch[159] Batch [49]\tSpeed: 3730.368555 samples/sec\taccuracy=0.959844\n",
      "Epoch[159] Batch [99]\tSpeed: 3716.398394 samples/sec\taccuracy=0.961406\n",
      "Epoch[159] Batch [149]\tSpeed: 4119.098276 samples/sec\taccuracy=0.959948\n",
      "Epoch[159] Batch [199]\tSpeed: 3984.702426 samples/sec\taccuracy=0.959414\n",
      "Epoch[159] Batch [249]\tSpeed: 3623.735518 samples/sec\taccuracy=0.959875\n",
      "Epoch[159] Batch [299]\tSpeed: 4158.662959 samples/sec\taccuracy=0.959609\n",
      "Epoch[159] Batch [349]\tSpeed: 2545.944298 samples/sec\taccuracy=0.960647\n",
      "[Epoch 159] training: accuracy=0.960477\n",
      "[Epoch 159] time cost: 13.446020\n",
      "[Epoch 159] validation: accuracy=0.907151\n",
      "Epoch[160] Batch [49]\tSpeed: 4092.877382 samples/sec\taccuracy=0.963750\n",
      "Epoch[160] Batch [99]\tSpeed: 3936.407784 samples/sec\taccuracy=0.960313\n",
      "Epoch[160] Batch [149]\tSpeed: 3868.113261 samples/sec\taccuracy=0.959010\n",
      "Epoch[160] Batch [199]\tSpeed: 3918.994627 samples/sec\taccuracy=0.957891\n",
      "Epoch[160] Batch [249]\tSpeed: 2847.441788 samples/sec\taccuracy=0.958375\n",
      "Epoch[160] Batch [299]\tSpeed: 3609.701553 samples/sec\taccuracy=0.958542\n",
      "Epoch[160] Batch [349]\tSpeed: 3968.678430 samples/sec\taccuracy=0.959442\n",
      "[Epoch 160] training: accuracy=0.959819\n",
      "[Epoch 160] time cost: 13.587126\n",
      "[Epoch 160] validation: accuracy=0.911294\n",
      "Epoch[161] Batch [49]\tSpeed: 4066.495323 samples/sec\taccuracy=0.962344\n",
      "Epoch[161] Batch [99]\tSpeed: 3892.145777 samples/sec\taccuracy=0.960547\n",
      "Epoch[161] Batch [149]\tSpeed: 3881.677346 samples/sec\taccuracy=0.960729\n",
      "Epoch[161] Batch [199]\tSpeed: 3925.011420 samples/sec\taccuracy=0.960859\n",
      "Epoch[161] Batch [249]\tSpeed: 3803.872182 samples/sec\taccuracy=0.960500\n",
      "Epoch[161] Batch [299]\tSpeed: 3800.533137 samples/sec\taccuracy=0.960260\n",
      "Epoch[161] Batch [349]\tSpeed: 4009.491501 samples/sec\taccuracy=0.960982\n",
      "[Epoch 161] training: accuracy=0.961557\n",
      "[Epoch 161] time cost: 13.582100\n",
      "[Epoch 161] validation: accuracy=0.911458\n",
      "Epoch[162] Batch [49]\tSpeed: 4008.144477 samples/sec\taccuracy=0.964219\n",
      "Epoch[162] Batch [99]\tSpeed: 2730.791672 samples/sec\taccuracy=0.962500\n",
      "Epoch[162] Batch [149]\tSpeed: 4001.124698 samples/sec\taccuracy=0.962708\n",
      "Epoch[162] Batch [199]\tSpeed: 3933.206679 samples/sec\taccuracy=0.961953\n",
      "Epoch[162] Batch [249]\tSpeed: 3853.149735 samples/sec\taccuracy=0.962437\n",
      "Epoch[162] Batch [299]\tSpeed: 3955.636937 samples/sec\taccuracy=0.961380\n",
      "Epoch[162] Batch [349]\tSpeed: 4002.675892 samples/sec\taccuracy=0.962500\n",
      "[Epoch 162] training: accuracy=0.962320\n",
      "[Epoch 162] time cost: 13.815033\n",
      "[Epoch 162] validation: accuracy=0.910357\n",
      "Epoch[163] Batch [49]\tSpeed: 3976.792113 samples/sec\taccuracy=0.959063\n",
      "Epoch[163] Batch [99]\tSpeed: 3951.502683 samples/sec\taccuracy=0.962187\n",
      "Epoch[163] Batch [149]\tSpeed: 3874.142447 samples/sec\taccuracy=0.962396\n",
      "Epoch[163] Batch [199]\tSpeed: 2896.212505 samples/sec\taccuracy=0.961758\n",
      "Epoch[163] Batch [249]\tSpeed: 3857.607220 samples/sec\taccuracy=0.961531\n",
      "Epoch[163] Batch [299]\tSpeed: 3858.882682 samples/sec\taccuracy=0.961719\n",
      "Epoch[163] Batch [349]\tSpeed: 4159.113996 samples/sec\taccuracy=0.961987\n",
      "[Epoch 163] training: accuracy=0.962176\n",
      "[Epoch 163] time cost: 13.944541\n",
      "[Epoch 163] validation: accuracy=0.910557\n",
      "Epoch[164] Batch [49]\tSpeed: 3790.952570 samples/sec\taccuracy=0.963906\n",
      "Epoch[164] Batch [99]\tSpeed: 4128.632932 samples/sec\taccuracy=0.964453\n",
      "Epoch[164] Batch [149]\tSpeed: 3992.525504 samples/sec\taccuracy=0.965052\n",
      "Epoch[164] Batch [199]\tSpeed: 3753.817033 samples/sec\taccuracy=0.964414\n",
      "Epoch[164] Batch [249]\tSpeed: 3847.158094 samples/sec\taccuracy=0.964219\n",
      "Epoch[164] Batch [299]\tSpeed: 2811.019080 samples/sec\taccuracy=0.963437\n",
      "Epoch[164] Batch [349]\tSpeed: 4010.180330 samples/sec\taccuracy=0.964509\n",
      "[Epoch 164] training: accuracy=0.964374\n",
      "[Epoch 164] time cost: 13.788559\n",
      "[Epoch 164] validation: accuracy=0.909856\n",
      "Epoch[165] Batch [49]\tSpeed: 2087.854523 samples/sec\taccuracy=0.966719\n",
      "Epoch[165] Batch [99]\tSpeed: 3909.206772 samples/sec\taccuracy=0.965391\n",
      "Epoch[165] Batch [149]\tSpeed: 3716.938722 samples/sec\taccuracy=0.964896\n",
      "Epoch[165] Batch [199]\tSpeed: 3682.595805 samples/sec\taccuracy=0.964375\n",
      "Epoch[165] Batch [249]\tSpeed: 3919.824420 samples/sec\taccuracy=0.963719\n",
      "Epoch[165] Batch [299]\tSpeed: 3908.068513 samples/sec\taccuracy=0.963932\n",
      "Epoch[165] Batch [349]\tSpeed: 3876.100384 samples/sec\taccuracy=0.964621\n",
      "[Epoch 165] training: accuracy=0.964744\n",
      "[Epoch 165] time cost: 13.659609\n",
      "[Epoch 165] validation: accuracy=0.910357\n",
      "Epoch[166] Batch [49]\tSpeed: 3748.470312 samples/sec\taccuracy=0.964219\n",
      "Epoch[166] Batch [99]\tSpeed: 4085.899967 samples/sec\taccuracy=0.965078\n",
      "Epoch[166] Batch [149]\tSpeed: 4045.810125 samples/sec\taccuracy=0.963958\n",
      "Epoch[166] Batch [199]\tSpeed: 3918.365364 samples/sec\taccuracy=0.963359\n",
      "Epoch[166] Batch [249]\tSpeed: 3592.359296 samples/sec\taccuracy=0.963969\n",
      "Epoch[166] Batch [299]\tSpeed: 4058.226590 samples/sec\taccuracy=0.963906\n",
      "Epoch[166] Batch [349]\tSpeed: 3070.939819 samples/sec\taccuracy=0.965022\n",
      "[Epoch 166] training: accuracy=0.965133\n",
      "[Epoch 166] time cost: 13.683833\n",
      "[Epoch 166] validation: accuracy=0.906050\n",
      "Epoch[167] Batch [49]\tSpeed: 2200.534125 samples/sec\taccuracy=0.963750\n",
      "Epoch[167] Batch [99]\tSpeed: 3943.955276 samples/sec\taccuracy=0.962031\n",
      "Epoch[167] Batch [149]\tSpeed: 2630.533838 samples/sec\taccuracy=0.963073\n",
      "Epoch[167] Batch [199]\tSpeed: 3087.487848 samples/sec\taccuracy=0.963047\n",
      "Epoch[167] Batch [249]\tSpeed: 3625.962678 samples/sec\taccuracy=0.963594\n",
      "Epoch[167] Batch [299]\tSpeed: 3936.119182 samples/sec\taccuracy=0.963776\n",
      "Epoch[167] Batch [349]\tSpeed: 3855.557158 samples/sec\taccuracy=0.964442\n",
      "[Epoch 167] training: accuracy=0.964784\n",
      "[Epoch 167] time cost: 13.880819\n",
      "[Epoch 167] validation: accuracy=0.910056\n",
      "Epoch[168] Batch [49]\tSpeed: 4000.289938 samples/sec\taccuracy=0.966719\n",
      "Epoch[168] Batch [99]\tSpeed: 4137.382665 samples/sec\taccuracy=0.965859\n",
      "Epoch[168] Batch [149]\tSpeed: 3755.287429 samples/sec\taccuracy=0.966354\n",
      "Epoch[168] Batch [199]\tSpeed: 3743.660828 samples/sec\taccuracy=0.965469\n",
      "Epoch[168] Batch [249]\tSpeed: 3900.544260 samples/sec\taccuracy=0.965500\n",
      "Epoch[168] Batch [299]\tSpeed: 4143.993331 samples/sec\taccuracy=0.965078\n",
      "Epoch[168] Batch [349]\tSpeed: 4092.970992 samples/sec\taccuracy=0.965469\n",
      "[Epoch 168] training: accuracy=0.966073\n",
      "[Epoch 168] time cost: 13.514310\n",
      "[Epoch 168] validation: accuracy=0.909217\n",
      "Epoch[169] Batch [49]\tSpeed: 4079.101257 samples/sec\taccuracy=0.966875\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[169] Batch [99]\tSpeed: 3813.599511 samples/sec\taccuracy=0.966328\n",
      "Epoch[169] Batch [149]\tSpeed: 3925.298394 samples/sec\taccuracy=0.965990\n",
      "Epoch[169] Batch [199]\tSpeed: 3793.175673 samples/sec\taccuracy=0.965508\n",
      "Epoch[169] Batch [249]\tSpeed: 3715.498197 samples/sec\taccuracy=0.966031\n",
      "Epoch[169] Batch [299]\tSpeed: 4053.660966 samples/sec\taccuracy=0.966276\n",
      "Epoch[169] Batch [349]\tSpeed: 4083.382736 samples/sec\taccuracy=0.967121\n",
      "[Epoch 169] training: accuracy=0.967172\n",
      "[Epoch 169] time cost: 14.012477\n",
      "[Epoch 169] validation: accuracy=0.909555\n",
      "Epoch[170] Batch [49]\tSpeed: 2724.956411 samples/sec\taccuracy=0.969063\n",
      "Epoch[170] Batch [99]\tSpeed: 3710.593368 samples/sec\taccuracy=0.966875\n",
      "Epoch[170] Batch [149]\tSpeed: 3800.183415 samples/sec\taccuracy=0.966823\n",
      "Epoch[170] Batch [199]\tSpeed: 2592.627402 samples/sec\taccuracy=0.966055\n",
      "Epoch[170] Batch [249]\tSpeed: 3768.784657 samples/sec\taccuracy=0.966344\n",
      "Epoch[170] Batch [299]\tSpeed: 3929.090398 samples/sec\taccuracy=0.966458\n",
      "Epoch[170] Batch [349]\tSpeed: 3828.884806 samples/sec\taccuracy=0.966964\n",
      "[Epoch 170] training: accuracy=0.967027\n",
      "[Epoch 170] time cost: 14.151591\n",
      "[Epoch 170] validation: accuracy=0.910657\n",
      "Epoch[171] Batch [49]\tSpeed: 3847.158094 samples/sec\taccuracy=0.969063\n",
      "Epoch[171] Batch [99]\tSpeed: 2416.073733 samples/sec\taccuracy=0.969375\n",
      "Epoch[171] Batch [149]\tSpeed: 3985.501106 samples/sec\taccuracy=0.967969\n",
      "Epoch[171] Batch [199]\tSpeed: 4050.449746 samples/sec\taccuracy=0.967812\n",
      "Epoch[171] Batch [249]\tSpeed: 3797.952093 samples/sec\taccuracy=0.967906\n",
      "Epoch[171] Batch [299]\tSpeed: 4117.866110 samples/sec\taccuracy=0.967865\n",
      "Epoch[171] Batch [349]\tSpeed: 4000.677462 samples/sec\taccuracy=0.968192\n",
      "[Epoch 171] training: accuracy=0.968310\n",
      "[Epoch 171] time cost: 13.333490\n",
      "[Epoch 171] validation: accuracy=0.909455\n",
      "Epoch[172] Batch [49]\tSpeed: 3881.200295 samples/sec\taccuracy=0.969219\n",
      "Epoch[172] Batch [99]\tSpeed: 2667.814113 samples/sec\taccuracy=0.968672\n",
      "Epoch[172] Batch [149]\tSpeed: 4018.915994 samples/sec\taccuracy=0.967812\n",
      "Epoch[172] Batch [199]\tSpeed: 2216.002444 samples/sec\taccuracy=0.967109\n",
      "Epoch[172] Batch [249]\tSpeed: 3537.607897 samples/sec\taccuracy=0.967313\n",
      "Epoch[172] Batch [299]\tSpeed: 3814.900249 samples/sec\taccuracy=0.967109\n",
      "Epoch[172] Batch [349]\tSpeed: 3744.731437 samples/sec\taccuracy=0.967589\n",
      "[Epoch 172] training: accuracy=0.967271\n",
      "[Epoch 172] time cost: 13.793229\n",
      "[Epoch 172] validation: accuracy=0.909054\n",
      "Epoch[173] Batch [49]\tSpeed: 3965.951924 samples/sec\taccuracy=0.969063\n",
      "Epoch[173] Batch [99]\tSpeed: 4031.954819 samples/sec\taccuracy=0.968594\n",
      "Epoch[173] Batch [149]\tSpeed: 3901.337907 samples/sec\taccuracy=0.968542\n",
      "Epoch[173] Batch [199]\tSpeed: 4019.487688 samples/sec\taccuracy=0.968125\n",
      "Epoch[173] Batch [249]\tSpeed: 3784.939173 samples/sec\taccuracy=0.967531\n",
      "Epoch[173] Batch [299]\tSpeed: 3670.962420 samples/sec\taccuracy=0.967500\n",
      "Epoch[173] Batch [349]\tSpeed: 4034.560616 samples/sec\taccuracy=0.968192\n",
      "[Epoch 173] training: accuracy=0.967929\n",
      "[Epoch 173] time cost: 13.515107\n",
      "[Epoch 173] validation: accuracy=0.905849\n",
      "Epoch[174] Batch [49]\tSpeed: 3436.017818 samples/sec\taccuracy=0.969219\n",
      "Epoch[174] Batch [99]\tSpeed: 4034.681897 samples/sec\taccuracy=0.968359\n",
      "Epoch[174] Batch [149]\tSpeed: 3943.173159 samples/sec\taccuracy=0.968229\n",
      "Epoch[174] Batch [199]\tSpeed: 4013.868086 samples/sec\taccuracy=0.967031\n",
      "Epoch[174] Batch [249]\tSpeed: 3769.949104 samples/sec\taccuracy=0.967469\n",
      "Epoch[174] Batch [299]\tSpeed: 3943.404865 samples/sec\taccuracy=0.967240\n",
      "Epoch[174] Batch [349]\tSpeed: 4231.194729 samples/sec\taccuracy=0.967701\n",
      "[Epoch 174] training: accuracy=0.967911\n",
      "[Epoch 174] time cost: 13.902579\n",
      "[Epoch 174] validation: accuracy=0.908554\n",
      "Epoch[175] Batch [49]\tSpeed: 3770.637524 samples/sec\taccuracy=0.972812\n",
      "Epoch[175] Batch [99]\tSpeed: 3738.759520 samples/sec\taccuracy=0.970781\n",
      "Epoch[175] Batch [149]\tSpeed: 4176.066335 samples/sec\taccuracy=0.969479\n",
      "Epoch[175] Batch [199]\tSpeed: 3972.613543 samples/sec\taccuracy=0.968750\n",
      "Epoch[175] Batch [249]\tSpeed: 2904.579800 samples/sec\taccuracy=0.968844\n",
      "Epoch[175] Batch [299]\tSpeed: 3937.909194 samples/sec\taccuracy=0.968333\n",
      "Epoch[175] Batch [349]\tSpeed: 1883.116666 samples/sec\taccuracy=0.968884\n",
      "[Epoch 175] training: accuracy=0.968890\n",
      "[Epoch 175] time cost: 13.542975\n",
      "[Epoch 175] validation: accuracy=0.909355\n",
      "Epoch[176] Batch [49]\tSpeed: 3762.867700 samples/sec\taccuracy=0.969375\n",
      "Epoch[176] Batch [99]\tSpeed: 3084.436866 samples/sec\taccuracy=0.968359\n",
      "Epoch[176] Batch [149]\tSpeed: 4034.105874 samples/sec\taccuracy=0.968385\n",
      "Epoch[176] Batch [199]\tSpeed: 4020.179956 samples/sec\taccuracy=0.968320\n",
      "Epoch[176] Batch [249]\tSpeed: 2498.212730 samples/sec\taccuracy=0.969219\n",
      "Epoch[176] Batch [299]\tSpeed: 3913.167381 samples/sec\taccuracy=0.969531\n",
      "Epoch[176] Batch [349]\tSpeed: 3804.168671 samples/sec\taccuracy=0.969821\n",
      "[Epoch 176] training: accuracy=0.969829\n",
      "[Epoch 176] time cost: 13.836605\n",
      "[Epoch 176] validation: accuracy=0.907832\n",
      "Epoch[177] Batch [49]\tSpeed: 3738.082689 samples/sec\taccuracy=0.972344\n",
      "Epoch[177] Batch [99]\tSpeed: 4084.967297 samples/sec\taccuracy=0.972344\n",
      "Epoch[177] Batch [149]\tSpeed: 3991.100842 samples/sec\taccuracy=0.971094\n",
      "Epoch[177] Batch [199]\tSpeed: 3925.872470 samples/sec\taccuracy=0.970078\n",
      "Epoch[177] Batch [249]\tSpeed: 3652.754594 samples/sec\taccuracy=0.970594\n",
      "Epoch[177] Batch [299]\tSpeed: 3960.743884 samples/sec\taccuracy=0.970417\n",
      "Epoch[177] Batch [349]\tSpeed: 3033.546045 samples/sec\taccuracy=0.970759\n",
      "[Epoch 177] training: accuracy=0.970908\n",
      "[Epoch 177] time cost: 13.443305\n",
      "[Epoch 177] validation: accuracy=0.908153\n",
      "Epoch[178] Batch [49]\tSpeed: 3994.396917 samples/sec\taccuracy=0.972031\n",
      "Epoch[178] Batch [99]\tSpeed: 4122.007847 samples/sec\taccuracy=0.970859\n",
      "Epoch[178] Batch [149]\tSpeed: 3710.875493 samples/sec\taccuracy=0.970417\n",
      "Epoch[178] Batch [199]\tSpeed: 3563.507494 samples/sec\taccuracy=0.970977\n",
      "Epoch[178] Batch [249]\tSpeed: 3566.205973 samples/sec\taccuracy=0.970531\n",
      "Epoch[178] Batch [299]\tSpeed: 3343.594338 samples/sec\taccuracy=0.970156\n",
      "Epoch[178] Batch [349]\tSpeed: 3818.807790 samples/sec\taccuracy=0.970670\n",
      "[Epoch 178] training: accuracy=0.970813\n",
      "[Epoch 178] time cost: 13.406262\n",
      "[Epoch 178] validation: accuracy=0.909756\n",
      "Epoch[179] Batch [49]\tSpeed: 3791.273821 samples/sec\taccuracy=0.968906\n",
      "Epoch[179] Batch [99]\tSpeed: 3852.900863 samples/sec\taccuracy=0.969375\n",
      "Epoch[179] Batch [149]\tSpeed: 3777.057211 samples/sec\taccuracy=0.969844\n",
      "Epoch[179] Batch [199]\tSpeed: 3757.784488 samples/sec\taccuracy=0.969805\n",
      "Epoch[179] Batch [249]\tSpeed: 3501.430988 samples/sec\taccuracy=0.970094\n",
      "Epoch[179] Batch [299]\tSpeed: 3675.561647 samples/sec\taccuracy=0.969792\n",
      "Epoch[179] Batch [349]\tSpeed: 3934.734484 samples/sec\taccuracy=0.970067\n",
      "[Epoch 179] training: accuracy=0.970309\n",
      "[Epoch 179] time cost: 13.760530\n",
      "[Epoch 179] validation: accuracy=0.909555\n",
      "Epoch[180] Batch [49]\tSpeed: 3869.200476 samples/sec\taccuracy=0.971406\n",
      "Epoch[180] Batch [99]\tSpeed: 4006.589043 samples/sec\taccuracy=0.971719\n",
      "Epoch[180] Batch [149]\tSpeed: 4118.529493 samples/sec\taccuracy=0.971562\n",
      "Epoch[180] Batch [199]\tSpeed: 3524.208746 samples/sec\taccuracy=0.970430\n",
      "Epoch[180] Batch [249]\tSpeed: 3682.646326 samples/sec\taccuracy=0.970562\n",
      "Epoch[180] Batch [299]\tSpeed: 4021.444712 samples/sec\taccuracy=0.970677\n",
      "Epoch[180] Batch [349]\tSpeed: 3945.375467 samples/sec\taccuracy=0.971049\n",
      "[Epoch 180] training: accuracy=0.971308\n",
      "[Epoch 180] time cost: 13.409312\n",
      "[Epoch 180] validation: accuracy=0.909756\n",
      "Epoch[181] Batch [49]\tSpeed: 3643.631694 samples/sec\taccuracy=0.970625\n",
      "Epoch[181] Batch [99]\tSpeed: 1960.305808 samples/sec\taccuracy=0.970625\n",
      "Epoch[181] Batch [149]\tSpeed: 2257.372543 samples/sec\taccuracy=0.971042\n",
      "Epoch[181] Batch [199]\tSpeed: 4057.490492 samples/sec\taccuracy=0.971836\n",
      "Epoch[181] Batch [249]\tSpeed: 3969.764212 samples/sec\taccuracy=0.971969\n",
      "Epoch[181] Batch [299]\tSpeed: 2325.072701 samples/sec\taccuracy=0.971667\n",
      "Epoch[181] Batch [349]\tSpeed: 3862.186163 samples/sec\taccuracy=0.972031\n",
      "[Epoch 181] training: accuracy=0.971775\n",
      "[Epoch 181] time cost: 13.865661\n",
      "[Epoch 181] validation: accuracy=0.910657\n",
      "Epoch[182] Batch [49]\tSpeed: 3810.459721 samples/sec\taccuracy=0.971406\n",
      "Epoch[182] Batch [99]\tSpeed: 4024.127424 samples/sec\taccuracy=0.970938\n",
      "Epoch[182] Batch [149]\tSpeed: 3075.954302 samples/sec\taccuracy=0.972344\n",
      "Epoch[182] Batch [199]\tSpeed: 4079.535201 samples/sec\taccuracy=0.972109\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[182] Batch [249]\tSpeed: 3781.846379 samples/sec\taccuracy=0.971531\n",
      "Epoch[182] Batch [299]\tSpeed: 4023.373492 samples/sec\taccuracy=0.971042\n",
      "Epoch[182] Batch [349]\tSpeed: 4052.375867 samples/sec\taccuracy=0.971339\n",
      "[Epoch 182] training: accuracy=0.971407\n",
      "[Epoch 182] time cost: 13.521734\n",
      "[Epoch 182] validation: accuracy=0.911959\n",
      "Epoch[183] Batch [49]\tSpeed: 3094.909823 samples/sec\taccuracy=0.968906\n",
      "Epoch[183] Batch [99]\tSpeed: 3712.851575 samples/sec\taccuracy=0.969297\n",
      "Epoch[183] Batch [149]\tSpeed: 2227.828038 samples/sec\taccuracy=0.969792\n",
      "Epoch[183] Batch [199]\tSpeed: 4015.128874 samples/sec\taccuracy=0.970273\n",
      "Epoch[183] Batch [249]\tSpeed: 3847.930162 samples/sec\taccuracy=0.970625\n",
      "Epoch[183] Batch [299]\tSpeed: 3795.159916 samples/sec\taccuracy=0.970260\n",
      "Epoch[183] Batch [349]\tSpeed: 4254.127670 samples/sec\taccuracy=0.970804\n",
      "[Epoch 183] training: accuracy=0.970913\n",
      "[Epoch 183] time cost: 14.140009\n",
      "[Epoch 183] validation: accuracy=0.909455\n",
      "Epoch[184] Batch [49]\tSpeed: 4021.745962 samples/sec\taccuracy=0.974531\n",
      "Epoch[184] Batch [99]\tSpeed: 3580.571642 samples/sec\taccuracy=0.972656\n",
      "Epoch[184] Batch [149]\tSpeed: 3221.450974 samples/sec\taccuracy=0.972865\n",
      "Epoch[184] Batch [199]\tSpeed: 3999.842888 samples/sec\taccuracy=0.972930\n",
      "Epoch[184] Batch [249]\tSpeed: 3798.220788 samples/sec\taccuracy=0.972938\n",
      "Epoch[184] Batch [299]\tSpeed: 3926.016015 samples/sec\taccuracy=0.972943\n",
      "Epoch[184] Batch [349]\tSpeed: 3926.016015 samples/sec\taccuracy=0.973125\n",
      "[Epoch 184] training: accuracy=0.973246\n",
      "[Epoch 184] time cost: 13.740927\n",
      "[Epoch 184] validation: accuracy=0.908327\n",
      "Epoch[185] Batch [49]\tSpeed: 3801.044385 samples/sec\taccuracy=0.971875\n",
      "Epoch[185] Batch [99]\tSpeed: 4015.218961 samples/sec\taccuracy=0.972109\n",
      "Epoch[185] Batch [149]\tSpeed: 4055.253171 samples/sec\taccuracy=0.972708\n",
      "Epoch[185] Batch [199]\tSpeed: 4099.565601 samples/sec\taccuracy=0.972031\n",
      "Epoch[185] Batch [249]\tSpeed: 3816.717346 samples/sec\taccuracy=0.972094\n",
      "Epoch[185] Batch [299]\tSpeed: 3694.176056 samples/sec\taccuracy=0.971979\n",
      "Epoch[185] Batch [349]\tSpeed: 3958.290905 samples/sec\taccuracy=0.972321\n",
      "[Epoch 185] training: accuracy=0.971967\n",
      "[Epoch 185] time cost: 13.115767\n",
      "[Epoch 185] validation: accuracy=0.905749\n",
      "Epoch[186] Batch [49]\tSpeed: 3897.231444 samples/sec\taccuracy=0.970313\n",
      "Epoch[186] Batch [99]\tSpeed: 4043.981621 samples/sec\taccuracy=0.970703\n",
      "Epoch[186] Batch [149]\tSpeed: 4040.968199 samples/sec\taccuracy=0.971927\n",
      "Epoch[186] Batch [199]\tSpeed: 3982.692354 samples/sec\taccuracy=0.971836\n",
      "Epoch[186] Batch [249]\tSpeed: 3478.675271 samples/sec\taccuracy=0.972812\n",
      "Epoch[186] Batch [299]\tSpeed: 3750.696260 samples/sec\taccuracy=0.972760\n",
      "Epoch[186] Batch [349]\tSpeed: 4146.553841 samples/sec\taccuracy=0.973058\n",
      "[Epoch 186] training: accuracy=0.973117\n",
      "[Epoch 186] time cost: 13.694626\n",
      "[Epoch 186] validation: accuracy=0.909155\n",
      "Epoch[187] Batch [49]\tSpeed: 3713.622046 samples/sec\taccuracy=0.973125\n",
      "Epoch[187] Batch [99]\tSpeed: 4094.875309 samples/sec\taccuracy=0.972578\n",
      "Epoch[187] Batch [149]\tSpeed: 4017.923439 samples/sec\taccuracy=0.971771\n",
      "Epoch[187] Batch [199]\tSpeed: 3892.089344 samples/sec\taccuracy=0.971641\n",
      "Epoch[187] Batch [249]\tSpeed: 2905.507244 samples/sec\taccuracy=0.971938\n",
      "Epoch[187] Batch [299]\tSpeed: 3940.134540 samples/sec\taccuracy=0.971849\n",
      "Epoch[187] Batch [349]\tSpeed: 4084.438974 samples/sec\taccuracy=0.972902\n",
      "[Epoch 187] training: accuracy=0.972606\n",
      "[Epoch 187] time cost: 13.744496\n",
      "[Epoch 187] validation: accuracy=0.907953\n",
      "Epoch[188] Batch [49]\tSpeed: 3956.540636 samples/sec\taccuracy=0.973594\n",
      "Epoch[188] Batch [99]\tSpeed: 3708.517217 samples/sec\taccuracy=0.972422\n",
      "Epoch[188] Batch [149]\tSpeed: 3912.882177 samples/sec\taccuracy=0.973542\n",
      "Epoch[188] Batch [199]\tSpeed: 4138.722253 samples/sec\taccuracy=0.973867\n",
      "Epoch[188] Batch [249]\tSpeed: 3858.494409 samples/sec\taccuracy=0.973938\n",
      "Epoch[188] Batch [299]\tSpeed: 3778.891625 samples/sec\taccuracy=0.974089\n",
      "Epoch[188] Batch [349]\tSpeed: 4072.263358 samples/sec\taccuracy=0.974308\n",
      "[Epoch 188] training: accuracy=0.974884\n",
      "[Epoch 188] time cost: 13.157674\n",
      "[Epoch 188] validation: accuracy=0.909355\n",
      "Epoch[189] Batch [49]\tSpeed: 3953.918133 samples/sec\taccuracy=0.975469\n",
      "Epoch[189] Batch [99]\tSpeed: 2852.282704 samples/sec\taccuracy=0.974375\n",
      "Epoch[189] Batch [149]\tSpeed: 3940.915452 samples/sec\taccuracy=0.973854\n",
      "Epoch[189] Batch [199]\tSpeed: 3728.114884 samples/sec\taccuracy=0.974102\n",
      "Epoch[189] Batch [249]\tSpeed: 3953.685190 samples/sec\taccuracy=0.973531\n",
      "Epoch[189] Batch [299]\tSpeed: 3891.835416 samples/sec\taccuracy=0.973568\n",
      "Epoch[189] Batch [349]\tSpeed: 3746.037888 samples/sec\taccuracy=0.974330\n",
      "[Epoch 189] training: accuracy=0.974439\n",
      "[Epoch 189] time cost: 13.526466\n",
      "[Epoch 189] validation: accuracy=0.906851\n",
      "Epoch[190] Batch [49]\tSpeed: 4000.975608 samples/sec\taccuracy=0.976094\n",
      "Epoch[190] Batch [99]\tSpeed: 3890.284356 samples/sec\taccuracy=0.974609\n",
      "Epoch[190] Batch [149]\tSpeed: 4129.871551 samples/sec\taccuracy=0.975313\n",
      "Epoch[190] Batch [199]\tSpeed: 4075.292717 samples/sec\taccuracy=0.975938\n",
      "Epoch[190] Batch [249]\tSpeed: 3596.546699 samples/sec\taccuracy=0.975906\n",
      "Epoch[190] Batch [299]\tSpeed: 3924.150747 samples/sec\taccuracy=0.975547\n",
      "Epoch[190] Batch [349]\tSpeed: 4051.519587 samples/sec\taccuracy=0.975960\n",
      "[Epoch 190] training: accuracy=0.975723\n",
      "[Epoch 190] time cost: 13.448109\n",
      "[Epoch 190] validation: accuracy=0.907252\n",
      "Epoch[191] Batch [49]\tSpeed: 3962.673359 samples/sec\taccuracy=0.975938\n",
      "Epoch[191] Batch [99]\tSpeed: 3994.723851 samples/sec\taccuracy=0.974297\n",
      "Epoch[191] Batch [149]\tSpeed: 3880.022202 samples/sec\taccuracy=0.973177\n",
      "Epoch[191] Batch [199]\tSpeed: 3972.319608 samples/sec\taccuracy=0.972852\n",
      "Epoch[191] Batch [249]\tSpeed: 3063.999406 samples/sec\taccuracy=0.973031\n",
      "Epoch[191] Batch [299]\tSpeed: 3497.667088 samples/sec\taccuracy=0.972995\n",
      "Epoch[191] Batch [349]\tSpeed: 3881.621216 samples/sec\taccuracy=0.972835\n",
      "[Epoch 191] training: accuracy=0.972957\n",
      "[Epoch 191] time cost: 13.526544\n",
      "[Epoch 191] validation: accuracy=0.909355\n",
      "Epoch[192] Batch [49]\tSpeed: 3989.143592 samples/sec\taccuracy=0.975469\n",
      "Epoch[192] Batch [99]\tSpeed: 3092.948525 samples/sec\taccuracy=0.973281\n",
      "Epoch[192] Batch [149]\tSpeed: 3334.104928 samples/sec\taccuracy=0.974010\n",
      "Epoch[192] Batch [199]\tSpeed: 4010.060516 samples/sec\taccuracy=0.973047\n",
      "Epoch[192] Batch [249]\tSpeed: 3414.536014 samples/sec\taccuracy=0.973031\n",
      "Epoch[192] Batch [299]\tSpeed: 3888.988055 samples/sec\taccuracy=0.972656\n",
      "Epoch[192] Batch [349]\tSpeed: 3856.111014 samples/sec\taccuracy=0.972723\n",
      "[Epoch 192] training: accuracy=0.973006\n",
      "[Epoch 192] time cost: 13.750256\n",
      "[Epoch 192] validation: accuracy=0.907239\n",
      "Epoch[193] Batch [49]\tSpeed: 3800.882923 samples/sec\taccuracy=0.971875\n",
      "Epoch[193] Batch [99]\tSpeed: 3065.311469 samples/sec\taccuracy=0.972891\n",
      "Epoch[193] Batch [149]\tSpeed: 3750.093684 samples/sec\taccuracy=0.971719\n",
      "Epoch[193] Batch [199]\tSpeed: 3804.626972 samples/sec\taccuracy=0.971602\n",
      "Epoch[193] Batch [249]\tSpeed: 3638.815996 samples/sec\taccuracy=0.972094\n",
      "Epoch[193] Batch [299]\tSpeed: 4110.268281 samples/sec\taccuracy=0.972214\n",
      "Epoch[193] Batch [349]\tSpeed: 3950.049016 samples/sec\taccuracy=0.972589\n",
      "[Epoch 193] training: accuracy=0.973066\n",
      "[Epoch 193] time cost: 13.707235\n",
      "[Epoch 193] validation: accuracy=0.905849\n",
      "Epoch[194] Batch [49]\tSpeed: 3704.499683 samples/sec\taccuracy=0.973906\n",
      "Epoch[194] Batch [99]\tSpeed: 3302.540627 samples/sec\taccuracy=0.972812\n",
      "Epoch[194] Batch [149]\tSpeed: 3886.060469 samples/sec\taccuracy=0.972969\n",
      "Epoch[194] Batch [199]\tSpeed: 4143.641508 samples/sec\taccuracy=0.973398\n",
      "Epoch[194] Batch [249]\tSpeed: 3950.572213 samples/sec\taccuracy=0.973969\n",
      "Epoch[194] Batch [299]\tSpeed: 3752.583838 samples/sec\taccuracy=0.974349\n",
      "Epoch[194] Batch [349]\tSpeed: 3926.216996 samples/sec\taccuracy=0.974866\n",
      "[Epoch 194] training: accuracy=0.975000\n",
      "[Epoch 194] time cost: 13.310344\n",
      "[Epoch 194] validation: accuracy=0.905649\n",
      "Epoch[195] Batch [49]\tSpeed: 3967.388003 samples/sec\taccuracy=0.975469\n",
      "Epoch[195] Batch [99]\tSpeed: 3990.448212 samples/sec\taccuracy=0.976406\n",
      "Epoch[195] Batch [149]\tSpeed: 4070.256571 samples/sec\taccuracy=0.975990\n",
      "Epoch[195] Batch [199]\tSpeed: 1848.079917 samples/sec\taccuracy=0.975156\n",
      "Epoch[195] Batch [249]\tSpeed: 3992.317678 samples/sec\taccuracy=0.975750\n",
      "Epoch[195] Batch [299]\tSpeed: 3970.556914 samples/sec\taccuracy=0.975443\n",
      "Epoch[195] Batch [349]\tSpeed: 4100.223100 samples/sec\taccuracy=0.975737\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Epoch 195] training: accuracy=0.975803\n",
      "[Epoch 195] time cost: 13.264545\n",
      "[Epoch 195] validation: accuracy=0.904347\n",
      "Epoch[196] Batch [49]\tSpeed: 3988.017560 samples/sec\taccuracy=0.975938\n",
      "Epoch[196] Batch [99]\tSpeed: 3109.068920 samples/sec\taccuracy=0.975078\n",
      "Epoch[196] Batch [149]\tSpeed: 3784.992541 samples/sec\taccuracy=0.974323\n",
      "Epoch[196] Batch [199]\tSpeed: 3894.630443 samples/sec\taccuracy=0.974258\n",
      "Epoch[196] Batch [249]\tSpeed: 3596.329870 samples/sec\taccuracy=0.974156\n",
      "Epoch[196] Batch [299]\tSpeed: 3972.025717 samples/sec\taccuracy=0.973594\n",
      "Epoch[196] Batch [349]\tSpeed: 3029.198519 samples/sec\taccuracy=0.974464\n",
      "[Epoch 196] training: accuracy=0.974864\n",
      "[Epoch 196] time cost: 13.872098\n",
      "[Epoch 196] validation: accuracy=0.910457\n",
      "Epoch[197] Batch [49]\tSpeed: 3719.333490 samples/sec\taccuracy=0.971250\n",
      "Epoch[197] Batch [99]\tSpeed: 3968.385078 samples/sec\taccuracy=0.972969\n",
      "Epoch[197] Batch [149]\tSpeed: 3812.732846 samples/sec\taccuracy=0.973802\n",
      "Epoch[197] Batch [199]\tSpeed: 3874.841483 samples/sec\taccuracy=0.972852\n",
      "Epoch[197] Batch [249]\tSpeed: 3803.036871 samples/sec\taccuracy=0.973219\n",
      "Epoch[197] Batch [299]\tSpeed: 3904.402141 samples/sec\taccuracy=0.973620\n",
      "Epoch[197] Batch [349]\tSpeed: 3829.977400 samples/sec\taccuracy=0.973862\n",
      "[Epoch 197] training: accuracy=0.974339\n",
      "[Epoch 197] time cost: 13.193766\n",
      "[Epoch 197] validation: accuracy=0.907853\n",
      "Epoch[198] Batch [49]\tSpeed: 4133.178170 samples/sec\taccuracy=0.975938\n",
      "Epoch[198] Batch [99]\tSpeed: 3885.751078 samples/sec\taccuracy=0.976953\n",
      "Epoch[198] Batch [149]\tSpeed: 3948.712587 samples/sec\taccuracy=0.976823\n",
      "Epoch[198] Batch [199]\tSpeed: 3890.650859 samples/sec\taccuracy=0.976172\n",
      "Epoch[198] Batch [249]\tSpeed: 3805.409035 samples/sec\taccuracy=0.976094\n",
      "Epoch[198] Batch [299]\tSpeed: 3976.644831 samples/sec\taccuracy=0.976432\n",
      "Epoch[198] Batch [349]\tSpeed: 3246.993613 samples/sec\taccuracy=0.976496\n",
      "[Epoch 198] training: accuracy=0.976263\n",
      "[Epoch 198] time cost: 13.853307\n",
      "[Epoch 198] validation: accuracy=0.909155\n",
      "Epoch[199] Batch [49]\tSpeed: 3935.022883 samples/sec\taccuracy=0.977031\n",
      "Epoch[199] Batch [99]\tSpeed: 2509.352329 samples/sec\taccuracy=0.976328\n",
      "Epoch[199] Batch [149]\tSpeed: 3893.670083 samples/sec\taccuracy=0.975365\n",
      "Epoch[199] Batch [199]\tSpeed: 3777.269806 samples/sec\taccuracy=0.975625\n",
      "Epoch[199] Batch [249]\tSpeed: 3756.101894 samples/sec\taccuracy=0.976156\n",
      "Epoch[199] Batch [299]\tSpeed: 3867.778857 samples/sec\taccuracy=0.976068\n",
      "Epoch[199] Batch [349]\tSpeed: 3985.175679 samples/sec\taccuracy=0.976429\n",
      "[Epoch 199] training: accuracy=0.976362\n",
      "[Epoch 199] time cost: 13.543883\n",
      "[Epoch 199] validation: accuracy=0.905449\n",
      "Epoch[200] Batch [49]\tSpeed: 3853.868880 samples/sec\taccuracy=0.974063\n",
      "Epoch[200] Batch [99]\tSpeed: 3970.645011 samples/sec\taccuracy=0.974766\n",
      "Epoch[200] Batch [149]\tSpeed: 3562.561627 samples/sec\taccuracy=0.974792\n",
      "Epoch[200] Batch [199]\tSpeed: 3925.843762 samples/sec\taccuracy=0.974727\n",
      "Epoch[200] Batch [249]\tSpeed: 3775.675931 samples/sec\taccuracy=0.975531\n",
      "Epoch[200] Batch [299]\tSpeed: 3995.823933 samples/sec\taccuracy=0.975885\n",
      "Epoch[200] Batch [349]\tSpeed: 3872.074777 samples/sec\taccuracy=0.976920\n",
      "[Epoch 200] training: accuracy=0.977302\n",
      "[Epoch 200] time cost: 13.865263\n",
      "[Epoch 200] validation: accuracy=0.909118\n",
      "Epoch[201] Batch [49]\tSpeed: 3902.585718 samples/sec\taccuracy=0.982969\n",
      "Epoch[201] Batch [99]\tSpeed: 4069.454410 samples/sec\taccuracy=0.983047\n",
      "Epoch[201] Batch [149]\tSpeed: 3928.026749 samples/sec\taccuracy=0.982396\n",
      "Epoch[201] Batch [199]\tSpeed: 3804.573048 samples/sec\taccuracy=0.982031\n",
      "Epoch[201] Batch [249]\tSpeed: 3747.580673 samples/sec\taccuracy=0.982281\n",
      "Epoch[201] Batch [299]\tSpeed: 2576.873180 samples/sec\taccuracy=0.982161\n",
      "Epoch[201] Batch [349]\tSpeed: 3808.621558 samples/sec\taccuracy=0.982254\n",
      "[Epoch 201] training: accuracy=0.982537\n",
      "[Epoch 201] time cost: 13.655720\n",
      "[Epoch 201] validation: accuracy=0.911659\n",
      "Epoch[202] Batch [49]\tSpeed: 3962.995121 samples/sec\taccuracy=0.984375\n",
      "Epoch[202] Batch [99]\tSpeed: 4064.863503 samples/sec\taccuracy=0.983984\n",
      "Epoch[202] Batch [149]\tSpeed: 3978.206576 samples/sec\taccuracy=0.983437\n",
      "Epoch[202] Batch [199]\tSpeed: 3964.516885 samples/sec\taccuracy=0.982500\n",
      "Epoch[202] Batch [249]\tSpeed: 3048.064905 samples/sec\taccuracy=0.982156\n",
      "Epoch[202] Batch [299]\tSpeed: 3602.531854 samples/sec\taccuracy=0.982422\n",
      "Epoch[202] Batch [349]\tSpeed: 4024.097261 samples/sec\taccuracy=0.982902\n",
      "[Epoch 202] training: accuracy=0.983153\n",
      "[Epoch 202] time cost: 13.353276\n",
      "[Epoch 202] validation: accuracy=0.911859\n",
      "Epoch[203] Batch [49]\tSpeed: 4102.196861 samples/sec\taccuracy=0.984062\n",
      "Epoch[203] Batch [99]\tSpeed: 3891.807204 samples/sec\taccuracy=0.982891\n",
      "Epoch[203] Batch [149]\tSpeed: 4007.067510 samples/sec\taccuracy=0.983177\n",
      "Epoch[203] Batch [199]\tSpeed: 2347.879893 samples/sec\taccuracy=0.981836\n",
      "Epoch[203] Batch [249]\tSpeed: 3738.629341 samples/sec\taccuracy=0.982094\n",
      "Epoch[203] Batch [299]\tSpeed: 3747.920779 samples/sec\taccuracy=0.982187\n",
      "Epoch[203] Batch [349]\tSpeed: 3452.390645 samples/sec\taccuracy=0.982812\n",
      "[Epoch 203] training: accuracy=0.982996\n",
      "[Epoch 203] time cost: 14.519499\n",
      "[Epoch 203] validation: accuracy=0.912861\n",
      "Epoch[204] Batch [49]\tSpeed: 3871.572164 samples/sec\taccuracy=0.984531\n",
      "Epoch[204] Batch [99]\tSpeed: 2953.540215 samples/sec\taccuracy=0.983672\n",
      "Epoch[204] Batch [149]\tSpeed: 4038.749056 samples/sec\taccuracy=0.984427\n",
      "Epoch[204] Batch [199]\tSpeed: 3847.792270 samples/sec\taccuracy=0.983516\n",
      "Epoch[204] Batch [249]\tSpeed: 3943.665567 samples/sec\taccuracy=0.983313\n",
      "Epoch[204] Batch [299]\tSpeed: 2931.782330 samples/sec\taccuracy=0.983594\n",
      "Epoch[204] Batch [349]\tSpeed: 4006.290059 samples/sec\taccuracy=0.984286\n",
      "[Epoch 204] training: accuracy=0.984555\n",
      "[Epoch 204] time cost: 13.221662\n",
      "[Epoch 204] validation: accuracy=0.912560\n",
      "Epoch[205] Batch [49]\tSpeed: 3892.456187 samples/sec\taccuracy=0.984688\n",
      "Epoch[205] Batch [99]\tSpeed: 3878.592621 samples/sec\taccuracy=0.985547\n",
      "Epoch[205] Batch [149]\tSpeed: 3984.051887 samples/sec\taccuracy=0.984479\n",
      "Epoch[205] Batch [199]\tSpeed: 3778.732040 samples/sec\taccuracy=0.984023\n",
      "Epoch[205] Batch [249]\tSpeed: 3992.881828 samples/sec\taccuracy=0.984031\n",
      "Epoch[205] Batch [299]\tSpeed: 4073.561103 samples/sec\taccuracy=0.984010\n",
      "Epoch[205] Batch [349]\tSpeed: 3952.928315 samples/sec\taccuracy=0.984643\n",
      "[Epoch 205] training: accuracy=0.984856\n",
      "[Epoch 205] time cost: 13.748440\n",
      "[Epoch 205] validation: accuracy=0.912760\n",
      "Epoch[206] Batch [49]\tSpeed: 2131.082835 samples/sec\taccuracy=0.987656\n",
      "Epoch[206] Batch [99]\tSpeed: 2403.451201 samples/sec\taccuracy=0.986094\n",
      "Epoch[206] Batch [149]\tSpeed: 4005.124450 samples/sec\taccuracy=0.985885\n",
      "Epoch[206] Batch [199]\tSpeed: 3928.342909 samples/sec\taccuracy=0.985234\n",
      "Epoch[206] Batch [249]\tSpeed: 2867.440645 samples/sec\taccuracy=0.984750\n",
      "Epoch[206] Batch [299]\tSpeed: 3984.258854 samples/sec\taccuracy=0.984453\n",
      "Epoch[206] Batch [349]\tSpeed: 3646.675850 samples/sec\taccuracy=0.984487\n",
      "[Epoch 206] training: accuracy=0.984615\n",
      "[Epoch 206] time cost: 13.832187\n",
      "[Epoch 206] validation: accuracy=0.911458\n",
      "Epoch[207] Batch [49]\tSpeed: 4111.590366 samples/sec\taccuracy=0.988125\n",
      "Epoch[207] Batch [99]\tSpeed: 3791.407692 samples/sec\taccuracy=0.987734\n",
      "Epoch[207] Batch [149]\tSpeed: 3378.267621 samples/sec\taccuracy=0.987083\n",
      "Epoch[207] Batch [199]\tSpeed: 3976.114706 samples/sec\taccuracy=0.986523\n",
      "Epoch[207] Batch [249]\tSpeed: 3797.871492 samples/sec\taccuracy=0.986406\n",
      "Epoch[207] Batch [299]\tSpeed: 3859.492984 samples/sec\taccuracy=0.985911\n",
      "Epoch[207] Batch [349]\tSpeed: 4002.138804 samples/sec\taccuracy=0.985737\n",
      "[Epoch 207] training: accuracy=0.985797\n",
      "[Epoch 207] time cost: 13.848589\n",
      "[Epoch 207] validation: accuracy=0.912159\n",
      "Epoch[208] Batch [49]\tSpeed: 3837.012214 samples/sec\taccuracy=0.986563\n",
      "Epoch[208] Batch [99]\tSpeed: 2662.772787 samples/sec\taccuracy=0.986250\n",
      "Epoch[208] Batch [149]\tSpeed: 3788.277592 samples/sec\taccuracy=0.985990\n",
      "Epoch[208] Batch [199]\tSpeed: 3932.544038 samples/sec\taccuracy=0.985117\n",
      "Epoch[208] Batch [249]\tSpeed: 3744.235225 samples/sec\taccuracy=0.984656\n",
      "Epoch[208] Batch [299]\tSpeed: 4032.045662 samples/sec\taccuracy=0.984688\n",
      "Epoch[208] Batch [349]\tSpeed: 3772.174138 samples/sec\taccuracy=0.985000\n",
      "[Epoch 208] training: accuracy=0.985014\n",
      "[Epoch 208] time cost: 13.800565\n",
      "[Epoch 208] validation: accuracy=0.911689\n",
      "Epoch[209] Batch [49]\tSpeed: 4039.265625 samples/sec\taccuracy=0.982344\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[209] Batch [99]\tSpeed: 3399.466288 samples/sec\taccuracy=0.984375\n",
      "Epoch[209] Batch [149]\tSpeed: 2732.556863 samples/sec\taccuracy=0.984896\n",
      "Epoch[209] Batch [199]\tSpeed: 3999.395939 samples/sec\taccuracy=0.984492\n",
      "Epoch[209] Batch [249]\tSpeed: 3767.356546 samples/sec\taccuracy=0.985156\n",
      "Epoch[209] Batch [299]\tSpeed: 4041.181122 samples/sec\taccuracy=0.984896\n",
      "Epoch[209] Batch [349]\tSpeed: 4046.298006 samples/sec\taccuracy=0.985134\n",
      "[Epoch 209] training: accuracy=0.985454\n",
      "[Epoch 209] time cost: 13.563575\n",
      "[Epoch 209] validation: accuracy=0.913762\n",
      "Epoch[210] Batch [49]\tSpeed: 3987.425168 samples/sec\taccuracy=0.988125\n",
      "Epoch[210] Batch [99]\tSpeed: 4028.143097 samples/sec\taccuracy=0.986875\n",
      "Epoch[210] Batch [149]\tSpeed: 4086.895283 samples/sec\taccuracy=0.986250\n",
      "Epoch[210] Batch [199]\tSpeed: 3962.117711 samples/sec\taccuracy=0.985586\n",
      "Epoch[210] Batch [249]\tSpeed: 2522.000761 samples/sec\taccuracy=0.985375\n",
      "Epoch[210] Batch [299]\tSpeed: 3856.415702 samples/sec\taccuracy=0.985313\n",
      "Epoch[210] Batch [349]\tSpeed: 3801.959592 samples/sec\taccuracy=0.985379\n",
      "[Epoch 210] training: accuracy=0.985537\n",
      "[Epoch 210] time cost: 13.477666\n",
      "[Epoch 210] validation: accuracy=0.912760\n",
      "Epoch[211] Batch [49]\tSpeed: 3800.640757 samples/sec\taccuracy=0.986719\n",
      "Epoch[211] Batch [99]\tSpeed: 3894.969507 samples/sec\taccuracy=0.987344\n",
      "Epoch[211] Batch [149]\tSpeed: 3038.937825 samples/sec\taccuracy=0.986354\n",
      "Epoch[211] Batch [199]\tSpeed: 3974.348642 samples/sec\taccuracy=0.986328\n",
      "Epoch[211] Batch [249]\tSpeed: 3751.036932 samples/sec\taccuracy=0.986250\n",
      "Epoch[211] Batch [299]\tSpeed: 3752.111766 samples/sec\taccuracy=0.986328\n",
      "Epoch[211] Batch [349]\tSpeed: 3403.949480 samples/sec\taccuracy=0.986004\n",
      "[Epoch 211] training: accuracy=0.986173\n",
      "[Epoch 211] time cost: 14.006219\n",
      "[Epoch 211] validation: accuracy=0.913061\n",
      "Epoch[212] Batch [49]\tSpeed: 2802.318142 samples/sec\taccuracy=0.985938\n",
      "Epoch[212] Batch [99]\tSpeed: 2984.821629 samples/sec\taccuracy=0.986094\n",
      "Epoch[212] Batch [149]\tSpeed: 2286.064902 samples/sec\taccuracy=0.985990\n",
      "Epoch[212] Batch [199]\tSpeed: 3947.377061 samples/sec\taccuracy=0.986055\n",
      "Epoch[212] Batch [249]\tSpeed: 3870.539425 samples/sec\taccuracy=0.986281\n",
      "Epoch[212] Batch [299]\tSpeed: 3618.703909 samples/sec\taccuracy=0.985755\n",
      "Epoch[212] Batch [349]\tSpeed: 2563.424986 samples/sec\taccuracy=0.985759\n",
      "[Epoch 212] training: accuracy=0.985734\n",
      "[Epoch 212] time cost: 14.245903\n",
      "[Epoch 212] validation: accuracy=0.913261\n",
      "Epoch[213] Batch [49]\tSpeed: 3788.464717 samples/sec\taccuracy=0.987031\n",
      "Epoch[213] Batch [99]\tSpeed: 3758.363228 samples/sec\taccuracy=0.986641\n",
      "Epoch[213] Batch [149]\tSpeed: 3777.482424 samples/sec\taccuracy=0.986406\n",
      "Epoch[213] Batch [199]\tSpeed: 4028.384897 samples/sec\taccuracy=0.986172\n",
      "Epoch[213] Batch [249]\tSpeed: 3049.987002 samples/sec\taccuracy=0.986250\n",
      "Epoch[213] Batch [299]\tSpeed: 3907.727165 samples/sec\taccuracy=0.986302\n",
      "Epoch[213] Batch [349]\tSpeed: 3989.647548 samples/sec\taccuracy=0.986518\n",
      "[Epoch 213] training: accuracy=0.986558\n",
      "[Epoch 213] time cost: 13.434606\n",
      "[Epoch 213] validation: accuracy=0.912961\n",
      "Epoch[214] Batch [49]\tSpeed: 3853.481614 samples/sec\taccuracy=0.985781\n",
      "Epoch[214] Batch [99]\tSpeed: 3735.325838 samples/sec\taccuracy=0.986719\n",
      "Epoch[214] Batch [149]\tSpeed: 3959.283411 samples/sec\taccuracy=0.986458\n",
      "Epoch[214] Batch [199]\tSpeed: 3903.522100 samples/sec\taccuracy=0.986055\n",
      "Epoch[214] Batch [249]\tSpeed: 3865.022224 samples/sec\taccuracy=0.986156\n",
      "Epoch[214] Batch [299]\tSpeed: 2860.855010 samples/sec\taccuracy=0.986094\n",
      "Epoch[214] Batch [349]\tSpeed: 3494.616294 samples/sec\taccuracy=0.986339\n",
      "[Epoch 214] training: accuracy=0.986413\n",
      "[Epoch 214] time cost: 13.769169\n",
      "[Epoch 214] validation: accuracy=0.912861\n",
      "Epoch[215] Batch [49]\tSpeed: 2985.468959 samples/sec\taccuracy=0.988437\n",
      "Epoch[215] Batch [99]\tSpeed: 3856.055621 samples/sec\taccuracy=0.987109\n",
      "Epoch[215] Batch [149]\tSpeed: 3905.623500 samples/sec\taccuracy=0.987396\n",
      "Epoch[215] Batch [199]\tSpeed: 4076.375724 samples/sec\taccuracy=0.987031\n",
      "Epoch[215] Batch [249]\tSpeed: 3061.169180 samples/sec\taccuracy=0.986656\n",
      "Epoch[215] Batch [299]\tSpeed: 2892.358523 samples/sec\taccuracy=0.986771\n",
      "Epoch[215] Batch [349]\tSpeed: 2507.008760 samples/sec\taccuracy=0.986830\n",
      "[Epoch 215] training: accuracy=0.986859\n",
      "[Epoch 215] time cost: 14.202528\n",
      "[Epoch 215] validation: accuracy=0.913862\n",
      "Epoch[216] Batch [49]\tSpeed: 4138.020456 samples/sec\taccuracy=0.986875\n",
      "Epoch[216] Batch [99]\tSpeed: 2427.895915 samples/sec\taccuracy=0.985938\n",
      "Epoch[216] Batch [149]\tSpeed: 4074.457648 samples/sec\taccuracy=0.986406\n",
      "Epoch[216] Batch [199]\tSpeed: 4096.187509 samples/sec\taccuracy=0.985977\n",
      "Epoch[216] Batch [249]\tSpeed: 3646.155756 samples/sec\taccuracy=0.985781\n",
      "Epoch[216] Batch [299]\tSpeed: 3961.562220 samples/sec\taccuracy=0.985677\n",
      "Epoch[216] Batch [349]\tSpeed: 3999.127817 samples/sec\taccuracy=0.985893\n",
      "[Epoch 216] training: accuracy=0.986113\n",
      "[Epoch 216] time cost: 13.577027\n",
      "[Epoch 216] validation: accuracy=0.913766\n",
      "Epoch[217] Batch [49]\tSpeed: 3996.508073 samples/sec\taccuracy=0.989219\n",
      "Epoch[217] Batch [99]\tSpeed: 4066.094944 samples/sec\taccuracy=0.988516\n",
      "Epoch[217] Batch [149]\tSpeed: 3901.593076 samples/sec\taccuracy=0.987396\n",
      "Epoch[217] Batch [199]\tSpeed: 3898.080347 samples/sec\taccuracy=0.987187\n",
      "Epoch[217] Batch [249]\tSpeed: 3621.657674 samples/sec\taccuracy=0.987500\n",
      "Epoch[217] Batch [299]\tSpeed: 3787.662880 samples/sec\taccuracy=0.987240\n",
      "Epoch[217] Batch [349]\tSpeed: 3828.939421 samples/sec\taccuracy=0.987121\n",
      "[Epoch 217] training: accuracy=0.987332\n",
      "[Epoch 217] time cost: 13.566374\n",
      "[Epoch 217] validation: accuracy=0.914163\n",
      "Epoch[218] Batch [49]\tSpeed: 3640.962964 samples/sec\taccuracy=0.985781\n",
      "Epoch[218] Batch [99]\tSpeed: 3893.359479 samples/sec\taccuracy=0.986172\n",
      "Epoch[218] Batch [149]\tSpeed: 3936.927373 samples/sec\taccuracy=0.985625\n",
      "Epoch[218] Batch [199]\tSpeed: 3812.028970 samples/sec\taccuracy=0.985547\n",
      "Epoch[218] Batch [249]\tSpeed: 3994.396917 samples/sec\taccuracy=0.985812\n",
      "Epoch[218] Batch [299]\tSpeed: 4102.040144 samples/sec\taccuracy=0.986146\n",
      "Epoch[218] Batch [349]\tSpeed: 3935.369017 samples/sec\taccuracy=0.986339\n",
      "[Epoch 218] training: accuracy=0.986098\n",
      "[Epoch 218] time cost: 13.533152\n",
      "[Epoch 218] validation: accuracy=0.912560\n",
      "Epoch[219] Batch [49]\tSpeed: 4036.987638 samples/sec\taccuracy=0.985781\n",
      "Epoch[219] Batch [99]\tSpeed: 3853.868880 samples/sec\taccuracy=0.985313\n",
      "Epoch[219] Batch [149]\tSpeed: 3855.917146 samples/sec\taccuracy=0.985833\n",
      "Epoch[219] Batch [199]\tSpeed: 3925.786348 samples/sec\taccuracy=0.986289\n",
      "Epoch[219] Batch [249]\tSpeed: 3223.888404 samples/sec\taccuracy=0.986125\n",
      "Epoch[219] Batch [299]\tSpeed: 3960.889991 samples/sec\taccuracy=0.986042\n",
      "Epoch[219] Batch [349]\tSpeed: 2346.494543 samples/sec\taccuracy=0.986429\n",
      "[Epoch 219] training: accuracy=0.986633\n",
      "[Epoch 219] time cost: 13.971574\n",
      "[Epoch 219] validation: accuracy=0.912961\n",
      "Epoch[220] Batch [49]\tSpeed: 4078.574450 samples/sec\taccuracy=0.985156\n",
      "Epoch[220] Batch [99]\tSpeed: 3929.234179 samples/sec\taccuracy=0.986797\n",
      "Epoch[220] Batch [149]\tSpeed: 3940.510496 samples/sec\taccuracy=0.986719\n",
      "Epoch[220] Batch [199]\tSpeed: 3918.222378 samples/sec\taccuracy=0.986289\n",
      "Epoch[220] Batch [249]\tSpeed: 3941.407296 samples/sec\taccuracy=0.985812\n",
      "Epoch[220] Batch [299]\tSpeed: 3845.752624 samples/sec\taccuracy=0.985781\n",
      "Epoch[220] Batch [349]\tSpeed: 3769.260935 samples/sec\taccuracy=0.985871\n",
      "[Epoch 220] training: accuracy=0.986213\n",
      "[Epoch 220] time cost: 13.593516\n",
      "[Epoch 220] validation: accuracy=0.911759\n",
      "Epoch[221] Batch [49]\tSpeed: 2588.290170 samples/sec\taccuracy=0.988594\n",
      "Epoch[221] Batch [99]\tSpeed: 3926.561582 samples/sec\taccuracy=0.988281\n",
      "Epoch[221] Batch [149]\tSpeed: 3785.179342 samples/sec\taccuracy=0.987969\n",
      "Epoch[221] Batch [199]\tSpeed: 2890.365350 samples/sec\taccuracy=0.987695\n",
      "Epoch[221] Batch [249]\tSpeed: 3711.003746 samples/sec\taccuracy=0.987875\n",
      "Epoch[221] Batch [299]\tSpeed: 3724.804085 samples/sec\taccuracy=0.987604\n",
      "Epoch[221] Batch [349]\tSpeed: 3892.343305 samples/sec\taccuracy=0.987679\n",
      "[Epoch 221] training: accuracy=0.987720\n",
      "[Epoch 221] time cost: 13.759980\n",
      "[Epoch 221] validation: accuracy=0.911659\n",
      "Epoch[222] Batch [49]\tSpeed: 4021.685709 samples/sec\taccuracy=0.987656\n",
      "Epoch[222] Batch [99]\tSpeed: 3991.486588 samples/sec\taccuracy=0.987812\n",
      "Epoch[222] Batch [149]\tSpeed: 4005.841668 samples/sec\taccuracy=0.987969\n",
      "Epoch[222] Batch [199]\tSpeed: 3877.220094 samples/sec\taccuracy=0.988281\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[222] Batch [249]\tSpeed: 3802.148072 samples/sec\taccuracy=0.987469\n",
      "Epoch[222] Batch [299]\tSpeed: 3451.059106 samples/sec\taccuracy=0.987344\n",
      "Epoch[222] Batch [349]\tSpeed: 3828.174955 samples/sec\taccuracy=0.987366\n",
      "[Epoch 222] training: accuracy=0.987432\n",
      "[Epoch 222] time cost: 13.469655\n",
      "[Epoch 222] validation: accuracy=0.912660\n",
      "Epoch[223] Batch [49]\tSpeed: 3755.418770 samples/sec\taccuracy=0.987344\n",
      "Epoch[223] Batch [99]\tSpeed: 3261.472037 samples/sec\taccuracy=0.988125\n",
      "Epoch[223] Batch [149]\tSpeed: 2419.296442 samples/sec\taccuracy=0.987396\n",
      "Epoch[223] Batch [199]\tSpeed: 3783.312160 samples/sec\taccuracy=0.986875\n",
      "Epoch[223] Batch [249]\tSpeed: 3763.975714 samples/sec\taccuracy=0.987062\n",
      "Epoch[223] Batch [299]\tSpeed: 3804.465206 samples/sec\taccuracy=0.987240\n",
      "Epoch[223] Batch [349]\tSpeed: 3734.338522 samples/sec\taccuracy=0.987433\n",
      "[Epoch 223] training: accuracy=0.987440\n",
      "[Epoch 223] time cost: 13.595052\n",
      "[Epoch 223] validation: accuracy=0.912260\n",
      "Epoch[224] Batch [49]\tSpeed: 3964.956072 samples/sec\taccuracy=0.989219\n",
      "Epoch[224] Batch [99]\tSpeed: 3646.378635 samples/sec\taccuracy=0.987187\n",
      "Epoch[224] Batch [149]\tSpeed: 3849.364824 samples/sec\taccuracy=0.987656\n",
      "Epoch[224] Batch [199]\tSpeed: 3998.651244 samples/sec\taccuracy=0.986875\n",
      "Epoch[224] Batch [249]\tSpeed: 3695.930827 samples/sec\taccuracy=0.987094\n",
      "Epoch[224] Batch [299]\tSpeed: 3838.301533 samples/sec\taccuracy=0.987214\n",
      "Epoch[224] Batch [349]\tSpeed: 3946.332498 samples/sec\taccuracy=0.987344\n",
      "[Epoch 224] training: accuracy=0.987132\n",
      "[Epoch 224] time cost: 13.454503\n",
      "[Epoch 224] validation: accuracy=0.912480\n",
      "Epoch[225] Batch [49]\tSpeed: 3833.751639 samples/sec\taccuracy=0.987187\n",
      "Epoch[225] Batch [99]\tSpeed: 3891.525105 samples/sec\taccuracy=0.987656\n",
      "Epoch[225] Batch [149]\tSpeed: 3741.938693 samples/sec\taccuracy=0.987135\n",
      "Epoch[225] Batch [199]\tSpeed: 3749.308006 samples/sec\taccuracy=0.987500\n",
      "Epoch[225] Batch [249]\tSpeed: 3722.841079 samples/sec\taccuracy=0.987219\n",
      "Epoch[225] Batch [299]\tSpeed: 3810.838387 samples/sec\taccuracy=0.987500\n",
      "Epoch[225] Batch [349]\tSpeed: 1966.106400 samples/sec\taccuracy=0.987656\n",
      "[Epoch 225] training: accuracy=0.987552\n",
      "[Epoch 225] time cost: 14.033943\n",
      "[Epoch 225] validation: accuracy=0.913562\n",
      "Epoch[226] Batch [49]\tSpeed: 3483.098770 samples/sec\taccuracy=0.987500\n",
      "Epoch[226] Batch [99]\tSpeed: 3901.649785 samples/sec\taccuracy=0.987422\n",
      "Epoch[226] Batch [149]\tSpeed: 3872.493721 samples/sec\taccuracy=0.987969\n",
      "Epoch[226] Batch [199]\tSpeed: 3903.891101 samples/sec\taccuracy=0.987695\n",
      "Epoch[226] Batch [249]\tSpeed: 3931.708852 samples/sec\taccuracy=0.987531\n",
      "Epoch[226] Batch [299]\tSpeed: 3733.533467 samples/sec\taccuracy=0.987292\n",
      "Epoch[226] Batch [349]\tSpeed: 3915.650410 samples/sec\taccuracy=0.987277\n",
      "[Epoch 226] training: accuracy=0.987099\n",
      "[Epoch 226] time cost: 13.510834\n",
      "[Epoch 226] validation: accuracy=0.911158\n",
      "Epoch[227] Batch [49]\tSpeed: 3883.249637 samples/sec\taccuracy=0.989531\n",
      "Epoch[227] Batch [99]\tSpeed: 4046.419994 samples/sec\taccuracy=0.988828\n",
      "Epoch[227] Batch [149]\tSpeed: 3342.282961 samples/sec\taccuracy=0.988698\n",
      "Epoch[227] Batch [199]\tSpeed: 4101.789421 samples/sec\taccuracy=0.988398\n",
      "Epoch[227] Batch [249]\tSpeed: 3534.487060 samples/sec\taccuracy=0.988437\n",
      "Epoch[227] Batch [299]\tSpeed: 3855.335660 samples/sec\taccuracy=0.988281\n",
      "Epoch[227] Batch [349]\tSpeed: 3783.205519 samples/sec\taccuracy=0.988393\n",
      "[Epoch 227] training: accuracy=0.988131\n",
      "[Epoch 227] time cost: 13.269480\n",
      "[Epoch 227] validation: accuracy=0.913462\n",
      "Epoch[228] Batch [49]\tSpeed: 3779.237438 samples/sec\taccuracy=0.989219\n",
      "Epoch[228] Batch [99]\tSpeed: 3974.525178 samples/sec\taccuracy=0.989297\n",
      "Epoch[228] Batch [149]\tSpeed: 3899.750937 samples/sec\taccuracy=0.989115\n",
      "Epoch[228] Batch [199]\tSpeed: 3698.145743 samples/sec\taccuracy=0.988555\n",
      "Epoch[228] Batch [249]\tSpeed: 3569.383100 samples/sec\taccuracy=0.988500\n",
      "Epoch[228] Batch [299]\tSpeed: 4019.848841 samples/sec\taccuracy=0.988281\n",
      "Epoch[228] Batch [349]\tSpeed: 2585.336184 samples/sec\taccuracy=0.988393\n",
      "[Epoch 228] training: accuracy=0.988491\n",
      "[Epoch 228] time cost: 14.187026\n",
      "[Epoch 228] validation: accuracy=0.912360\n",
      "Epoch[229] Batch [49]\tSpeed: 3316.556574 samples/sec\taccuracy=0.987031\n",
      "Epoch[229] Batch [99]\tSpeed: 3424.795305 samples/sec\taccuracy=0.987734\n",
      "Epoch[229] Batch [149]\tSpeed: 3984.347560 samples/sec\taccuracy=0.987344\n",
      "Epoch[229] Batch [199]\tSpeed: 2950.034684 samples/sec\taccuracy=0.986992\n",
      "Epoch[229] Batch [249]\tSpeed: 3783.845452 samples/sec\taccuracy=0.987313\n",
      "Epoch[229] Batch [299]\tSpeed: 4164.114172 samples/sec\taccuracy=0.987552\n",
      "Epoch[229] Batch [349]\tSpeed: 4178.536553 samples/sec\taccuracy=0.987656\n",
      "[Epoch 229] training: accuracy=0.987740\n",
      "[Epoch 229] time cost: 13.472111\n",
      "[Epoch 229] validation: accuracy=0.911859\n",
      "Epoch[230] Batch [49]\tSpeed: 3769.260935 samples/sec\taccuracy=0.990469\n",
      "Epoch[230] Batch [99]\tSpeed: 3834.655277 samples/sec\taccuracy=0.989609\n",
      "Epoch[230] Batch [149]\tSpeed: 3805.085383 samples/sec\taccuracy=0.988073\n",
      "Epoch[230] Batch [199]\tSpeed: 4115.120088 samples/sec\taccuracy=0.987500\n",
      "Epoch[230] Batch [249]\tSpeed: 3877.276097 samples/sec\taccuracy=0.987531\n",
      "Epoch[230] Batch [299]\tSpeed: 3927.911795 samples/sec\taccuracy=0.987266\n",
      "Epoch[230] Batch [349]\tSpeed: 3564.335540 samples/sec\taccuracy=0.987746\n",
      "[Epoch 230] training: accuracy=0.987712\n",
      "[Epoch 230] time cost: 13.685243\n",
      "[Epoch 230] validation: accuracy=0.912760\n",
      "Epoch[231] Batch [49]\tSpeed: 3712.158424 samples/sec\taccuracy=0.989375\n",
      "Epoch[231] Batch [99]\tSpeed: 4035.106441 samples/sec\taccuracy=0.988594\n",
      "Epoch[231] Batch [149]\tSpeed: 3940.915452 samples/sec\taccuracy=0.988385\n",
      "Epoch[231] Batch [199]\tSpeed: 3958.845479 samples/sec\taccuracy=0.988008\n",
      "Epoch[231] Batch [249]\tSpeed: 1981.877928 samples/sec\taccuracy=0.988031\n",
      "Epoch[231] Batch [299]\tSpeed: 3835.449734 samples/sec\taccuracy=0.988125\n",
      "Epoch[231] Batch [349]\tSpeed: 3776.950923 samples/sec\taccuracy=0.988504\n",
      "[Epoch 231] training: accuracy=0.988321\n",
      "[Epoch 231] time cost: 13.527256\n",
      "[Epoch 231] validation: accuracy=0.911358\n",
      "Epoch[232] Batch [49]\tSpeed: 3772.015120 samples/sec\taccuracy=0.983906\n",
      "Epoch[232] Batch [99]\tSpeed: 3879.545558 samples/sec\taccuracy=0.986016\n",
      "Epoch[232] Batch [149]\tSpeed: 3206.826820 samples/sec\taccuracy=0.986458\n",
      "Epoch[232] Batch [199]\tSpeed: 4025.123047 samples/sec\taccuracy=0.986641\n",
      "Epoch[232] Batch [249]\tSpeed: 3759.889851 samples/sec\taccuracy=0.987156\n",
      "Epoch[232] Batch [299]\tSpeed: 3873.192162 samples/sec\taccuracy=0.987161\n",
      "Epoch[232] Batch [349]\tSpeed: 2694.026114 samples/sec\taccuracy=0.987299\n",
      "[Epoch 232] training: accuracy=0.987232\n",
      "[Epoch 232] time cost: 13.715734\n",
      "[Epoch 232] validation: accuracy=0.912777\n",
      "Epoch[233] Batch [49]\tSpeed: 3583.846198 samples/sec\taccuracy=0.990938\n",
      "Epoch[233] Batch [99]\tSpeed: 4008.114554 samples/sec\taccuracy=0.989141\n",
      "Epoch[233] Batch [149]\tSpeed: 3720.390229 samples/sec\taccuracy=0.988802\n",
      "Epoch[233] Batch [199]\tSpeed: 4076.344773 samples/sec\taccuracy=0.988320\n",
      "Epoch[233] Batch [249]\tSpeed: 3616.047202 samples/sec\taccuracy=0.988250\n",
      "Epoch[233] Batch [299]\tSpeed: 3909.605319 samples/sec\taccuracy=0.988099\n",
      "Epoch[233] Batch [349]\tSpeed: 3725.734653 samples/sec\taccuracy=0.987946\n",
      "[Epoch 233] training: accuracy=0.988031\n",
      "[Epoch 233] time cost: 13.380625\n",
      "[Epoch 233] validation: accuracy=0.912059\n",
      "Epoch[234] Batch [49]\tSpeed: 4121.058622 samples/sec\taccuracy=0.987187\n",
      "Epoch[234] Batch [99]\tSpeed: 4024.368742 samples/sec\taccuracy=0.986563\n",
      "Epoch[234] Batch [149]\tSpeed: 2803.766996 samples/sec\taccuracy=0.987292\n",
      "Epoch[234] Batch [199]\tSpeed: 4105.082595 samples/sec\taccuracy=0.987695\n",
      "Epoch[234] Batch [249]\tSpeed: 3890.425311 samples/sec\taccuracy=0.987812\n",
      "Epoch[234] Batch [299]\tSpeed: 2173.733656 samples/sec\taccuracy=0.987682\n",
      "Epoch[234] Batch [349]\tSpeed: 4156.988533 samples/sec\taccuracy=0.987835\n",
      "[Epoch 234] training: accuracy=0.987700\n",
      "[Epoch 234] time cost: 13.006138\n",
      "[Epoch 234] validation: accuracy=0.912660\n",
      "Epoch[235] Batch [49]\tSpeed: 4024.851465 samples/sec\taccuracy=0.988125\n",
      "Epoch[235] Batch [99]\tSpeed: 3973.966202 samples/sec\taccuracy=0.987578\n",
      "Epoch[235] Batch [149]\tSpeed: 3923.261782 samples/sec\taccuracy=0.988177\n",
      "Epoch[235] Batch [199]\tSpeed: 3974.848868 samples/sec\taccuracy=0.987617\n",
      "Epoch[235] Batch [249]\tSpeed: 3723.848153 samples/sec\taccuracy=0.987406\n",
      "Epoch[235] Batch [299]\tSpeed: 3841.102611 samples/sec\taccuracy=0.987318\n",
      "Epoch[235] Batch [349]\tSpeed: 3874.477953 samples/sec\taccuracy=0.987500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Epoch 235] training: accuracy=0.987732\n",
      "[Epoch 235] time cost: 13.529035\n",
      "[Epoch 235] validation: accuracy=0.911659\n",
      "Epoch[236] Batch [49]\tSpeed: 3724.623195 samples/sec\taccuracy=0.988750\n",
      "Epoch[236] Batch [99]\tSpeed: 3720.467575 samples/sec\taccuracy=0.988359\n",
      "Epoch[236] Batch [149]\tSpeed: 2823.541014 samples/sec\taccuracy=0.987969\n",
      "Epoch[236] Batch [199]\tSpeed: 3250.965303 samples/sec\taccuracy=0.988008\n",
      "Epoch[236] Batch [249]\tSpeed: 2816.061769 samples/sec\taccuracy=0.987844\n",
      "Epoch[236] Batch [299]\tSpeed: 3859.437494 samples/sec\taccuracy=0.987370\n",
      "Epoch[236] Batch [349]\tSpeed: 3940.394809 samples/sec\taccuracy=0.987857\n",
      "[Epoch 236] training: accuracy=0.987972\n",
      "[Epoch 236] time cost: 13.742738\n",
      "[Epoch 236] validation: accuracy=0.913762\n",
      "Epoch[237] Batch [49]\tSpeed: 3817.857304 samples/sec\taccuracy=0.989219\n",
      "Epoch[237] Batch [99]\tSpeed: 3997.311493 samples/sec\taccuracy=0.988516\n",
      "Epoch[237] Batch [149]\tSpeed: 3271.927257 samples/sec\taccuracy=0.989219\n",
      "Epoch[237] Batch [199]\tSpeed: 4053.201909 samples/sec\taccuracy=0.988555\n",
      "Epoch[237] Batch [249]\tSpeed: 3821.172478 samples/sec\taccuracy=0.988062\n",
      "Epoch[237] Batch [299]\tSpeed: 2730.472236 samples/sec\taccuracy=0.987891\n",
      "Epoch[237] Batch [349]\tSpeed: 4051.855940 samples/sec\taccuracy=0.987812\n",
      "[Epoch 237] training: accuracy=0.987700\n",
      "[Epoch 237] time cost: 13.182709\n",
      "[Epoch 237] validation: accuracy=0.913261\n",
      "Epoch[238] Batch [49]\tSpeed: 4074.890604 samples/sec\taccuracy=0.988281\n",
      "Epoch[238] Batch [99]\tSpeed: 3990.388893 samples/sec\taccuracy=0.989297\n",
      "Epoch[238] Batch [149]\tSpeed: 3990.062667 samples/sec\taccuracy=0.989010\n",
      "Epoch[238] Batch [199]\tSpeed: 4075.787735 samples/sec\taccuracy=0.988711\n",
      "Epoch[238] Batch [249]\tSpeed: 3755.839125 samples/sec\taccuracy=0.987969\n",
      "Epoch[238] Batch [299]\tSpeed: 4008.024786 samples/sec\taccuracy=0.987500\n",
      "Epoch[238] Batch [349]\tSpeed: 4043.798861 samples/sec\taccuracy=0.987857\n",
      "[Epoch 238] training: accuracy=0.987852\n",
      "[Epoch 238] time cost: 13.623228\n",
      "[Epoch 238] validation: accuracy=0.913462\n",
      "Epoch[239] Batch [49]\tSpeed: 3988.017560 samples/sec\taccuracy=0.988281\n",
      "Epoch[239] Batch [99]\tSpeed: 3927.825583 samples/sec\taccuracy=0.988047\n",
      "Epoch[239] Batch [149]\tSpeed: 4050.052520 samples/sec\taccuracy=0.987865\n",
      "Epoch[239] Batch [199]\tSpeed: 4145.881401 samples/sec\taccuracy=0.987578\n",
      "Epoch[239] Batch [249]\tSpeed: 3865.050049 samples/sec\taccuracy=0.987313\n",
      "Epoch[239] Batch [299]\tSpeed: 3814.385267 samples/sec\taccuracy=0.987318\n",
      "Epoch[239] Batch [349]\tSpeed: 3854.948100 samples/sec\taccuracy=0.987455\n",
      "[Epoch 239] training: accuracy=0.987780\n",
      "[Epoch 239] time cost: 13.019986\n",
      "[Epoch 239] validation: accuracy=0.913662\n",
      "Epoch[240] Batch [49]\tSpeed: 4016.931374 samples/sec\taccuracy=0.990625\n",
      "Epoch[240] Batch [99]\tSpeed: 2877.831148 samples/sec\taccuracy=0.989844\n",
      "Epoch[240] Batch [149]\tSpeed: 3697.636332 samples/sec\taccuracy=0.988750\n",
      "Epoch[240] Batch [199]\tSpeed: 3740.166028 samples/sec\taccuracy=0.988047\n",
      "Epoch[240] Batch [249]\tSpeed: 2727.074177 samples/sec\taccuracy=0.987844\n",
      "Epoch[240] Batch [299]\tSpeed: 3997.698440 samples/sec\taccuracy=0.988021\n",
      "Epoch[240] Batch [349]\tSpeed: 3876.408240 samples/sec\taccuracy=0.988170\n",
      "[Epoch 240] training: accuracy=0.988271\n",
      "[Epoch 240] time cost: 13.779610\n",
      "[Epoch 240] validation: accuracy=0.912085\n",
      "Epoch[241] Batch [49]\tSpeed: 3819.785927 samples/sec\taccuracy=0.988125\n",
      "Epoch[241] Batch [99]\tSpeed: 3806.353332 samples/sec\taccuracy=0.987656\n",
      "Epoch[241] Batch [149]\tSpeed: 3893.670083 samples/sec\taccuracy=0.987865\n",
      "Epoch[241] Batch [199]\tSpeed: 4043.768403 samples/sec\taccuracy=0.987656\n",
      "Epoch[241] Batch [249]\tSpeed: 3691.889726 samples/sec\taccuracy=0.987812\n",
      "Epoch[241] Batch [299]\tSpeed: 2469.280250 samples/sec\taccuracy=0.988255\n",
      "Epoch[241] Batch [349]\tSpeed: 3779.849417 samples/sec\taccuracy=0.988616\n",
      "[Epoch 241] training: accuracy=0.988571\n",
      "[Epoch 241] time cost: 13.687999\n",
      "[Epoch 241] validation: accuracy=0.913061\n",
      "Epoch[242] Batch [49]\tSpeed: 3875.484819 samples/sec\taccuracy=0.988125\n",
      "Epoch[242] Batch [99]\tSpeed: 3800.694569 samples/sec\taccuracy=0.988203\n",
      "Epoch[242] Batch [149]\tSpeed: 4135.820907 samples/sec\taccuracy=0.987969\n",
      "Epoch[242] Batch [199]\tSpeed: 4128.442442 samples/sec\taccuracy=0.988125\n",
      "Epoch[242] Batch [249]\tSpeed: 2355.368469 samples/sec\taccuracy=0.988125\n",
      "Epoch[242] Batch [299]\tSpeed: 3503.304548 samples/sec\taccuracy=0.988437\n",
      "Epoch[242] Batch [349]\tSpeed: 4021.836346 samples/sec\taccuracy=0.988482\n",
      "[Epoch 242] training: accuracy=0.988281\n",
      "[Epoch 242] time cost: 13.715631\n",
      "[Epoch 242] validation: accuracy=0.912059\n",
      "Epoch[243] Batch [49]\tSpeed: 3184.079995 samples/sec\taccuracy=0.987656\n",
      "Epoch[243] Batch [99]\tSpeed: 3308.157228 samples/sec\taccuracy=0.988125\n",
      "Epoch[243] Batch [149]\tSpeed: 2972.493187 samples/sec\taccuracy=0.988073\n",
      "Epoch[243] Batch [199]\tSpeed: 4075.818677 samples/sec\taccuracy=0.988047\n",
      "Epoch[243] Batch [249]\tSpeed: 3279.983089 samples/sec\taccuracy=0.988594\n",
      "Epoch[243] Batch [299]\tSpeed: 3756.522401 samples/sec\taccuracy=0.988672\n",
      "Epoch[243] Batch [349]\tSpeed: 3915.393398 samples/sec\taccuracy=0.989040\n",
      "[Epoch 243] training: accuracy=0.988971\n",
      "[Epoch 243] time cost: 13.845931\n",
      "[Epoch 243] validation: accuracy=0.913161\n",
      "Epoch[244] Batch [49]\tSpeed: 3087.115021 samples/sec\taccuracy=0.989688\n",
      "Epoch[244] Batch [99]\tSpeed: 3996.924621 samples/sec\taccuracy=0.989453\n",
      "Epoch[244] Batch [149]\tSpeed: 3791.514795 samples/sec\taccuracy=0.988854\n",
      "Epoch[244] Batch [199]\tSpeed: 3778.572468 samples/sec\taccuracy=0.989297\n",
      "Epoch[244] Batch [249]\tSpeed: 3222.379084 samples/sec\taccuracy=0.989125\n",
      "Epoch[244] Batch [299]\tSpeed: 3944.563804 samples/sec\taccuracy=0.988906\n",
      "Epoch[244] Batch [349]\tSpeed: 2915.289193 samples/sec\taccuracy=0.988638\n",
      "[Epoch 244] training: accuracy=0.988751\n",
      "[Epoch 244] time cost: 14.215833\n",
      "[Epoch 244] validation: accuracy=0.912360\n",
      "Epoch[245] Batch [49]\tSpeed: 3747.371407 samples/sec\taccuracy=0.990000\n",
      "Epoch[245] Batch [99]\tSpeed: 4095.968750 samples/sec\taccuracy=0.987969\n",
      "Epoch[245] Batch [149]\tSpeed: 3978.324493 samples/sec\taccuracy=0.987969\n",
      "Epoch[245] Batch [199]\tSpeed: 3791.488019 samples/sec\taccuracy=0.988555\n",
      "Epoch[245] Batch [249]\tSpeed: 3731.665001 samples/sec\taccuracy=0.988594\n",
      "Epoch[245] Batch [299]\tSpeed: 3928.199194 samples/sec\taccuracy=0.988854\n",
      "Epoch[245] Batch [349]\tSpeed: 4031.500665 samples/sec\taccuracy=0.988839\n",
      "[Epoch 245] training: accuracy=0.988962\n",
      "[Epoch 245] time cost: 13.842251\n",
      "[Epoch 245] validation: accuracy=0.913462\n",
      "Epoch[246] Batch [49]\tSpeed: 3737.328052 samples/sec\taccuracy=0.987656\n",
      "Epoch[246] Batch [99]\tSpeed: 3929.579295 samples/sec\taccuracy=0.987500\n",
      "Epoch[246] Batch [149]\tSpeed: 3912.711075 samples/sec\taccuracy=0.988125\n",
      "Epoch[246] Batch [199]\tSpeed: 3028.976343 samples/sec\taccuracy=0.988516\n",
      "Epoch[246] Batch [249]\tSpeed: 3695.523775 samples/sec\taccuracy=0.988594\n",
      "Epoch[246] Batch [299]\tSpeed: 3991.664649 samples/sec\taccuracy=0.988177\n",
      "Epoch[246] Batch [349]\tSpeed: 4009.132206 samples/sec\taccuracy=0.988437\n",
      "[Epoch 246] training: accuracy=0.988671\n",
      "[Epoch 246] time cost: 13.382385\n",
      "[Epoch 246] validation: accuracy=0.913662\n",
      "Epoch[247] Batch [49]\tSpeed: 3904.714509 samples/sec\taccuracy=0.989375\n",
      "Epoch[247] Batch [99]\tSpeed: 2869.801108 samples/sec\taccuracy=0.988437\n",
      "Epoch[247] Batch [149]\tSpeed: 3864.910927 samples/sec\taccuracy=0.988021\n",
      "Epoch[247] Batch [199]\tSpeed: 3935.167098 samples/sec\taccuracy=0.987734\n",
      "Epoch[247] Batch [249]\tSpeed: 3688.465532 samples/sec\taccuracy=0.988031\n",
      "Epoch[247] Batch [299]\tSpeed: 2315.126595 samples/sec\taccuracy=0.988047\n",
      "Epoch[247] Batch [349]\tSpeed: 3890.002478 samples/sec\taccuracy=0.988304\n",
      "[Epoch 247] training: accuracy=0.988401\n",
      "[Epoch 247] time cost: 13.451901\n",
      "[Epoch 247] validation: accuracy=0.913061\n",
      "Epoch[248] Batch [49]\tSpeed: 4114.930842 samples/sec\taccuracy=0.988125\n",
      "Epoch[248] Batch [99]\tSpeed: 3998.383222 samples/sec\taccuracy=0.988594\n",
      "Epoch[248] Batch [149]\tSpeed: 3965.219631 samples/sec\taccuracy=0.988594\n",
      "Epoch[248] Batch [199]\tSpeed: 3386.557194 samples/sec\taccuracy=0.989062\n",
      "Epoch[248] Batch [249]\tSpeed: 3882.659878 samples/sec\taccuracy=0.989000\n",
      "Epoch[248] Batch [299]\tSpeed: 4050.571985 samples/sec\taccuracy=0.989271\n",
      "Epoch[248] Batch [349]\tSpeed: 3057.072886 samples/sec\taccuracy=0.989263\n",
      "[Epoch 248] training: accuracy=0.988891\n",
      "[Epoch 248] time cost: 13.338642\n",
      "[Epoch 248] validation: accuracy=0.913469\n",
      "Epoch[249] Batch [49]\tSpeed: 3196.611563 samples/sec\taccuracy=0.990625\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[249] Batch [99]\tSpeed: 2727.517525 samples/sec\taccuracy=0.990703\n",
      "Epoch[249] Batch [149]\tSpeed: 3851.656984 samples/sec\taccuracy=0.989896\n",
      "Epoch[249] Batch [199]\tSpeed: 4098.814433 samples/sec\taccuracy=0.989570\n",
      "Epoch[249] Batch [249]\tSpeed: 3799.107752 samples/sec\taccuracy=0.989500\n",
      "Epoch[249] Batch [299]\tSpeed: 3805.894614 samples/sec\taccuracy=0.989167\n",
      "Epoch[249] Batch [349]\tSpeed: 3747.292938 samples/sec\taccuracy=0.989420\n",
      "[Epoch 249] training: accuracy=0.989170\n",
      "[Epoch 249] time cost: 13.792157\n",
      "[Epoch 249] validation: accuracy=0.913662\n"
     ]
    }
   ],
   "source": [
    "train(sp_resnet, learning_rate, momentum, weight_decay, epochs, scheduler, ctx, train_data, val_data, log_interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training should have led to a similar validation accuracy as reported in [3], Sec. 4.2. We next activate quantization for $W_B$, $W_C$ (by setting `sp_mode` to 2) and continue training for 40 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[0] Batch [49]\tSpeed: 1015.650668 samples/sec\taccuracy=0.752188\n",
      "Epoch[0] Batch [99]\tSpeed: 929.617626 samples/sec\taccuracy=0.793750\n",
      "Epoch[0] Batch [149]\tSpeed: 1117.950040 samples/sec\taccuracy=0.815729\n",
      "Epoch[0] Batch [199]\tSpeed: 1347.828049 samples/sec\taccuracy=0.829961\n",
      "Epoch[0] Batch [249]\tSpeed: 1122.817951 samples/sec\taccuracy=0.841406\n",
      "Epoch[0] Batch [299]\tSpeed: 1005.670031 samples/sec\taccuracy=0.848385\n",
      "Epoch[0] Batch [349]\tSpeed: 1155.652136 samples/sec\taccuracy=0.854799\n",
      "[Epoch 0] training: accuracy=0.859135\n",
      "[Epoch 0] time cost: 48.579850\n",
      "[Epoch 0] validation: accuracy=0.852264\n",
      "Epoch[1] Batch [49]\tSpeed: 1139.712970 samples/sec\taccuracy=0.900312\n",
      "Epoch[1] Batch [99]\tSpeed: 851.494855 samples/sec\taccuracy=0.899609\n",
      "Epoch[1] Batch [149]\tSpeed: 1065.097493 samples/sec\taccuracy=0.897292\n",
      "Epoch[1] Batch [199]\tSpeed: 1061.093588 samples/sec\taccuracy=0.895664\n",
      "Epoch[1] Batch [249]\tSpeed: 1176.695632 samples/sec\taccuracy=0.896156\n",
      "Epoch[1] Batch [299]\tSpeed: 1138.280025 samples/sec\taccuracy=0.897917\n",
      "Epoch[1] Batch [349]\tSpeed: 845.689745 samples/sec\taccuracy=0.898326\n",
      "[Epoch 1] training: accuracy=0.898557\n",
      "[Epoch 1] time cost: 48.327556\n",
      "[Epoch 1] validation: accuracy=0.862280\n",
      "Epoch[2] Batch [49]\tSpeed: 1064.630714 samples/sec\taccuracy=0.903906\n",
      "Epoch[2] Batch [99]\tSpeed: 1264.171573 samples/sec\taccuracy=0.905937\n",
      "Epoch[2] Batch [149]\tSpeed: 1070.118403 samples/sec\taccuracy=0.905885\n",
      "Epoch[2] Batch [199]\tSpeed: 822.698628 samples/sec\taccuracy=0.905039\n",
      "Epoch[2] Batch [249]\tSpeed: 1046.717667 samples/sec\taccuracy=0.904625\n",
      "Epoch[2] Batch [299]\tSpeed: 1004.349321 samples/sec\taccuracy=0.905729\n",
      "Epoch[2] Batch [349]\tSpeed: 1041.286991 samples/sec\taccuracy=0.906518\n",
      "[Epoch 2] training: accuracy=0.906310\n",
      "[Epoch 2] time cost: 48.396528\n",
      "[Epoch 2] validation: accuracy=0.853065\n",
      "Epoch[3] Batch [49]\tSpeed: 840.248523 samples/sec\taccuracy=0.912031\n",
      "Epoch[3] Batch [99]\tSpeed: 1138.746001 samples/sec\taccuracy=0.911875\n",
      "Epoch[3] Batch [149]\tSpeed: 1093.692971 samples/sec\taccuracy=0.910885\n",
      "Epoch[3] Batch [199]\tSpeed: 1058.678940 samples/sec\taccuracy=0.909531\n",
      "Epoch[3] Batch [249]\tSpeed: 1239.892360 samples/sec\taccuracy=0.907844\n",
      "Epoch[3] Batch [299]\tSpeed: 1152.427252 samples/sec\taccuracy=0.907031\n",
      "Epoch[3] Batch [349]\tSpeed: 816.573448 samples/sec\taccuracy=0.907478\n",
      "[Epoch 3] training: accuracy=0.907452\n",
      "[Epoch 3] time cost: 48.224097\n",
      "[Epoch 3] validation: accuracy=0.858173\n",
      "Epoch[4] Batch [49]\tSpeed: 1244.658302 samples/sec\taccuracy=0.909219\n",
      "Epoch[4] Batch [99]\tSpeed: 937.351549 samples/sec\taccuracy=0.910625\n",
      "Epoch[4] Batch [149]\tSpeed: 1069.414971 samples/sec\taccuracy=0.910625\n",
      "Epoch[4] Batch [199]\tSpeed: 961.796416 samples/sec\taccuracy=0.909375\n",
      "Epoch[4] Batch [249]\tSpeed: 1076.694119 samples/sec\taccuracy=0.910062\n",
      "Epoch[4] Batch [299]\tSpeed: 711.806650 samples/sec\taccuracy=0.909609\n",
      "Epoch[4] Batch [349]\tSpeed: 1077.487169 samples/sec\taccuracy=0.909554\n",
      "[Epoch 4] training: accuracy=0.909966\n",
      "[Epoch 4] time cost: 48.294883\n",
      "[Epoch 4] validation: accuracy=0.856771\n",
      "Epoch[5] Batch [49]\tSpeed: 1126.883391 samples/sec\taccuracy=0.908125\n",
      "Epoch[5] Batch [99]\tSpeed: 984.954120 samples/sec\taccuracy=0.906328\n",
      "Epoch[5] Batch [149]\tSpeed: 1106.783746 samples/sec\taccuracy=0.907552\n",
      "Epoch[5] Batch [199]\tSpeed: 1150.306205 samples/sec\taccuracy=0.908398\n",
      "Epoch[5] Batch [249]\tSpeed: 1037.706430 samples/sec\taccuracy=0.910156\n",
      "Epoch[5] Batch [299]\tSpeed: 1203.786926 samples/sec\taccuracy=0.909948\n",
      "Epoch[5] Batch [349]\tSpeed: 1022.589835 samples/sec\taccuracy=0.910714\n",
      "[Epoch 5] training: accuracy=0.910216\n",
      "[Epoch 5] time cost: 48.623598\n",
      "[Epoch 5] validation: accuracy=0.858674\n",
      "Epoch[6] Batch [49]\tSpeed: 1188.874841 samples/sec\taccuracy=0.911406\n",
      "Epoch[6] Batch [99]\tSpeed: 1059.912091 samples/sec\taccuracy=0.910469\n",
      "Epoch[6] Batch [149]\tSpeed: 1118.483397 samples/sec\taccuracy=0.910937\n",
      "Epoch[6] Batch [199]\tSpeed: 1213.278565 samples/sec\taccuracy=0.911055\n",
      "Epoch[6] Batch [249]\tSpeed: 1060.841985 samples/sec\taccuracy=0.911031\n",
      "Epoch[6] Batch [299]\tSpeed: 1069.287174 samples/sec\taccuracy=0.910521\n",
      "Epoch[6] Batch [349]\tSpeed: 1075.624166 samples/sec\taccuracy=0.910179\n",
      "[Epoch 6] training: accuracy=0.910106\n",
      "[Epoch 6] time cost: 48.839132\n",
      "[Epoch 6] validation: accuracy=0.863034\n",
      "Epoch[7] Batch [49]\tSpeed: 1231.137877 samples/sec\taccuracy=0.904375\n",
      "Epoch[7] Batch [99]\tSpeed: 1114.806849 samples/sec\taccuracy=0.905937\n",
      "Epoch[7] Batch [149]\tSpeed: 870.471144 samples/sec\taccuracy=0.908438\n",
      "Epoch[7] Batch [199]\tSpeed: 909.754870 samples/sec\taccuracy=0.908555\n",
      "Epoch[7] Batch [249]\tSpeed: 1109.489578 samples/sec\taccuracy=0.908125\n",
      "Epoch[7] Batch [299]\tSpeed: 1153.722481 samples/sec\taccuracy=0.907969\n",
      "Epoch[7] Batch [349]\tSpeed: 1162.211703 samples/sec\taccuracy=0.909018\n",
      "[Epoch 7] training: accuracy=0.909347\n",
      "[Epoch 7] time cost: 47.591304\n",
      "[Epoch 7] validation: accuracy=0.865785\n",
      "Epoch[8] Batch [49]\tSpeed: 1132.803957 samples/sec\taccuracy=0.907813\n",
      "Epoch[8] Batch [99]\tSpeed: 1032.904896 samples/sec\taccuracy=0.907422\n",
      "Epoch[8] Batch [149]\tSpeed: 1112.349241 samples/sec\taccuracy=0.908854\n",
      "Epoch[8] Batch [199]\tSpeed: 1109.682211 samples/sec\taccuracy=0.908867\n",
      "Epoch[8] Batch [249]\tSpeed: 1148.163485 samples/sec\taccuracy=0.907875\n",
      "Epoch[8] Batch [299]\tSpeed: 1045.378612 samples/sec\taccuracy=0.908229\n",
      "Epoch[8] Batch [349]\tSpeed: 810.464162 samples/sec\taccuracy=0.907857\n",
      "[Epoch 8] training: accuracy=0.907873\n",
      "[Epoch 8] time cost: 48.243156\n",
      "Epoch[10] Batch [299]\tSpeed: 1185.140269 samples/sec\taccuracy=0.926016\n",
      "Epoch[10] Batch [349]\tSpeed: 979.757597 samples/sec\taccuracy=0.929040\n",
      "[Epoch 10] training: accuracy=0.930467\n",
      "[Epoch 10] time cost: 48.324215\n",
      "[Epoch 10] validation: accuracy=0.896134\n",
      "Epoch[11] Batch [49]\tSpeed: 1084.265707 samples/sec\taccuracy=0.941406\n",
      "Epoch[11] Batch [99]\tSpeed: 1151.517628 samples/sec\taccuracy=0.940000\n",
      "Epoch[11] Batch [149]\tSpeed: 792.572112 samples/sec\taccuracy=0.938906\n",
      "Epoch[11] Batch [199]\tSpeed: 1104.250044 samples/sec\taccuracy=0.939727\n",
      "Epoch[11] Batch [249]\tSpeed: 1091.962307 samples/sec\taccuracy=0.939813\n",
      "Epoch[11] Batch [299]\tSpeed: 1177.237480 samples/sec\taccuracy=0.939948\n",
      "Epoch[11] Batch [349]\tSpeed: 1151.438598 samples/sec\taccuracy=0.941562\n",
      "[Epoch 11] training: accuracy=0.942187\n",
      "[Epoch 11] time cost: 48.293900\n",
      "[Epoch 11] validation: accuracy=0.893229\n",
      "Epoch[12] Batch [49]\tSpeed: 1203.954297 samples/sec\taccuracy=0.940781\n",
      "Epoch[12] Batch [99]\tSpeed: 953.095219 samples/sec\taccuracy=0.939453\n",
      "Epoch[12] Batch [149]\tSpeed: 1135.633870 samples/sec\taccuracy=0.940677\n",
      "Epoch[12] Batch [199]\tSpeed: 1136.862716 samples/sec\taccuracy=0.941641\n",
      "Epoch[12] Batch [249]\tSpeed: 1096.267938 samples/sec\taccuracy=0.942875\n",
      "Epoch[12] Batch [299]\tSpeed: 1036.420328 samples/sec\taccuracy=0.942344\n",
      "Epoch[12] Batch [349]\tSpeed: 1159.435718 samples/sec\taccuracy=0.943839\n",
      "[Epoch 12] training: accuracy=0.944154\n",
      "[Epoch 12] time cost: 48.307506\n",
      "[Epoch 12] validation: accuracy=0.895933\n",
      "Epoch[13] Batch [49]\tSpeed: 1089.929456 samples/sec\taccuracy=0.943281\n",
      "Epoch[13] Batch [99]\tSpeed: 847.959137 samples/sec\taccuracy=0.944531\n",
      "Epoch[13] Batch [149]\tSpeed: 1098.272846 samples/sec\taccuracy=0.943125\n",
      "Epoch[13] Batch [199]\tSpeed: 1121.010346 samples/sec\taccuracy=0.943516\n",
      "Epoch[13] Batch [249]\tSpeed: 1011.982553 samples/sec\taccuracy=0.943781\n",
      "Epoch[13] Batch [299]\tSpeed: 1089.358873 samples/sec\taccuracy=0.943776\n",
      "Epoch[13] Batch [349]\tSpeed: 848.951383 samples/sec\taccuracy=0.944933\n",
      "[Epoch 13] training: accuracy=0.945433\n",
      "[Epoch 13] time cost: 48.135132\n",
      "[Epoch 13] validation: accuracy=0.895833\n",
      "Epoch[14] Batch [49]\tSpeed: 1050.716424 samples/sec\taccuracy=0.941719\n",
      "Epoch[14] Batch [99]\tSpeed: 1056.256553 samples/sec\taccuracy=0.942109\n",
      "Epoch[14] Batch [149]\tSpeed: 851.389529 samples/sec\taccuracy=0.943177\n",
      "Epoch[14] Batch [199]\tSpeed: 1174.412460 samples/sec\taccuracy=0.944258\n",
      "Epoch[14] Batch [249]\tSpeed: 903.588820 samples/sec\taccuracy=0.944281\n",
      "Epoch[14] Batch [299]\tSpeed: 965.199364 samples/sec\taccuracy=0.945234\n",
      "Epoch[14] Batch [349]\tSpeed: 1055.075430 samples/sec\taccuracy=0.946049\n",
      "[Epoch 14] training: accuracy=0.946391\n",
      "[Epoch 14] time cost: 48.886878\n",
      "[Epoch 14] validation: accuracy=0.895866\n",
      "Epoch[15] Batch [49]\tSpeed: 1112.960320 samples/sec\taccuracy=0.945625\n",
      "Epoch[15] Batch [99]\tSpeed: 1028.638101 samples/sec\taccuracy=0.943594\n",
      "Epoch[15] Batch [149]\tSpeed: 1075.546591 samples/sec\taccuracy=0.943125\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[15] Batch [199]\tSpeed: 1047.945595 samples/sec\taccuracy=0.943320\n",
      "Epoch[15] Batch [249]\tSpeed: 779.633052 samples/sec\taccuracy=0.943312\n",
      "Epoch[15] Batch [299]\tSpeed: 1155.361158 samples/sec\taccuracy=0.944010\n",
      "Epoch[15] Batch [349]\tSpeed: 1056.688977 samples/sec\taccuracy=0.944933\n",
      "[Epoch 15] training: accuracy=0.945352\n",
      "[Epoch 15] time cost: 47.566040\n",
      "[Epoch 15] validation: accuracy=0.896134\n",
      "Epoch[16] Batch [49]\tSpeed: 1096.899568 samples/sec\taccuracy=0.951094\n",
      "Epoch[16] Batch [99]\tSpeed: 1193.534494 samples/sec\taccuracy=0.948828\n",
      "Epoch[16] Batch [149]\tSpeed: 1115.413679 samples/sec\taccuracy=0.948073\n",
      "Epoch[16] Batch [199]\tSpeed: 1149.454064 samples/sec\taccuracy=0.947031\n",
      "Epoch[16] Batch [249]\tSpeed: 1061.750415 samples/sec\taccuracy=0.946500\n",
      "Epoch[16] Batch [299]\tSpeed: 1185.255392 samples/sec\taccuracy=0.946328\n",
      "Epoch[16] Batch [349]\tSpeed: 948.221898 samples/sec\taccuracy=0.947701\n",
      "[Epoch 16] training: accuracy=0.947997\n",
      "[Epoch 16] time cost: 47.947451\n",
      "[Epoch 16] validation: accuracy=0.893329\n",
      "Epoch[17] Batch [49]\tSpeed: 924.405900 samples/sec\taccuracy=0.942813\n",
      "Epoch[17] Batch [99]\tSpeed: 1046.105801 samples/sec\taccuracy=0.942891\n",
      "Epoch[17] Batch [149]\tSpeed: 1056.707696 samples/sec\taccuracy=0.942292\n",
      "Epoch[17] Batch [199]\tSpeed: 1062.258806 samples/sec\taccuracy=0.941445\n",
      "Epoch[17] Batch [249]\tSpeed: 959.333616 samples/sec\taccuracy=0.942219\n",
      "Epoch[17] Batch [299]\tSpeed: 884.435484 samples/sec\taccuracy=0.942760\n",
      "Epoch[17] Batch [349]\tSpeed: 954.125072 samples/sec\taccuracy=0.944196\n",
      "[Epoch 17] training: accuracy=0.944553\n",
      "[Epoch 17] time cost: 48.876224\n",
      "[Epoch 17] validation: accuracy=0.895933\n",
      "Epoch[18] Batch [49]\tSpeed: 1087.313673 samples/sec\taccuracy=0.945000\n",
      "Epoch[18] Batch [99]\tSpeed: 1128.294882 samples/sec\taccuracy=0.948359\n",
      "Epoch[18] Batch [149]\tSpeed: 1123.983905 samples/sec\taccuracy=0.948438\n",
      "Epoch[18] Batch [199]\tSpeed: 1229.021290 samples/sec\taccuracy=0.947656\n",
      "Epoch[18] Batch [249]\tSpeed: 1078.201264 samples/sec\taccuracy=0.946500\n",
      "Epoch[18] Batch [299]\tSpeed: 1002.264340 samples/sec\taccuracy=0.946094\n",
      "Epoch[18] Batch [349]\tSpeed: 1180.060560 samples/sec\taccuracy=0.946496\n",
      "[Epoch 18] training: accuracy=0.947011\n",
      "[Epoch 18] time cost: 47.058622\n",
      "[Epoch 18] validation: accuracy=0.900541\n",
      "Epoch[19] Batch [49]\tSpeed: 1120.402089 samples/sec\taccuracy=0.947187\n",
      "Epoch[19] Batch [99]\tSpeed: 907.153897 samples/sec\taccuracy=0.943828\n",
      "Epoch[19] Batch [149]\tSpeed: 1021.922193 samples/sec\taccuracy=0.944844\n",
      "Epoch[19] Batch [199]\tSpeed: 797.171236 samples/sec\taccuracy=0.944844\n",
      "Epoch[19] Batch [249]\tSpeed: 1122.463474 samples/sec\taccuracy=0.944906\n",
      "Epoch[19] Batch [299]\tSpeed: 836.371289 samples/sec\taccuracy=0.945677\n",
      "Epoch[19] Batch [349]\tSpeed: 1131.655422 samples/sec\taccuracy=0.946496\n",
      "[Epoch 19] training: accuracy=0.946074\n",
      "[Epoch 19] time cost: 47.020878\n",
      "[Epoch 19] validation: accuracy=0.892929\n",
      "Epoch[20] Batch [49]\tSpeed: 1143.176051 samples/sec\taccuracy=0.947500\n",
      "Epoch[20] Batch [99]\tSpeed: 1191.494880 samples/sec\taccuracy=0.944688\n",
      "Epoch[20] Batch [149]\tSpeed: 1062.761373 samples/sec\taccuracy=0.944479\n",
      "Epoch[20] Batch [199]\tSpeed: 1114.075825 samples/sec\taccuracy=0.946016\n",
      "Epoch[20] Batch [249]\tSpeed: 973.075244 samples/sec\taccuracy=0.946875\n",
      "Epoch[20] Batch [299]\tSpeed: 1157.670629 samples/sec\taccuracy=0.947812\n",
      "Epoch[20] Batch [349]\tSpeed: 1141.275671 samples/sec\taccuracy=0.949375\n",
      "[Epoch 20] training: accuracy=0.950368\n",
      "[Epoch 20] time cost: 47.479411\n",
      "[Epoch 20] validation: accuracy=0.903746\n",
      "Epoch[21] Batch [49]\tSpeed: 1226.654067 samples/sec\taccuracy=0.954375\n",
      "Epoch[21] Batch [99]\tSpeed: 1002.172665 samples/sec\taccuracy=0.953047\n",
      "Epoch[21] Batch [149]\tSpeed: 1154.484135 samples/sec\taccuracy=0.952292\n",
      "Epoch[21] Batch [199]\tSpeed: 1114.200679 samples/sec\taccuracy=0.951914\n",
      "Epoch[21] Batch [249]\tSpeed: 1200.379456 samples/sec\taccuracy=0.952094\n",
      "Epoch[21] Batch [299]\tSpeed: 1146.576980 samples/sec\taccuracy=0.952604\n",
      "Epoch[21] Batch [349]\tSpeed: 1208.599808 samples/sec\taccuracy=0.953906\n",
      "[Epoch 21] training: accuracy=0.954708\n",
      "[Epoch 21] time cost: 48.198591\n",
      "[Epoch 21] validation: accuracy=0.903145\n",
      "Epoch[22] Batch [49]\tSpeed: 955.753994 samples/sec\taccuracy=0.953125\n",
      "Epoch[22] Batch [99]\tSpeed: 1140.342380 samples/sec\taccuracy=0.952812\n",
      "Epoch[22] Batch [149]\tSpeed: 822.799497 samples/sec\taccuracy=0.953698\n",
      "Epoch[22] Batch [199]\tSpeed: 1037.497873 samples/sec\taccuracy=0.953750\n",
      "Epoch[22] Batch [249]\tSpeed: 1155.045475 samples/sec\taccuracy=0.953438\n",
      "Epoch[22] Batch [299]\tSpeed: 1137.549236 samples/sec\taccuracy=0.952786\n",
      "Epoch[22] Batch [349]\tSpeed: 1077.210442 samples/sec\taccuracy=0.953906\n",
      "[Epoch 22] training: accuracy=0.954444\n",
      "[Epoch 22] time cost: 47.302513\n",
      "[Epoch 22] validation: accuracy=0.903184\n",
      "Epoch[23] Batch [49]\tSpeed: 1190.168907 samples/sec\taccuracy=0.954063\n",
      "Epoch[23] Batch [99]\tSpeed: 1078.153629 samples/sec\taccuracy=0.952578\n",
      "Epoch[23] Batch [149]\tSpeed: 1092.355563 samples/sec\taccuracy=0.954063\n",
      "Epoch[23] Batch [199]\tSpeed: 1153.442386 samples/sec\taccuracy=0.954688\n",
      "Epoch[23] Batch [249]\tSpeed: 1216.580546 samples/sec\taccuracy=0.955094\n",
      "Epoch[23] Batch [299]\tSpeed: 878.496072 samples/sec\taccuracy=0.955182\n",
      "Epoch[23] Batch [349]\tSpeed: 1069.233934 samples/sec\taccuracy=0.955603\n",
      "[Epoch 23] training: accuracy=0.956202\n",
      "[Epoch 23] time cost: 48.000577\n",
      "[Epoch 23] validation: accuracy=0.902344\n",
      "Epoch[24] Batch [49]\tSpeed: 931.703609 samples/sec\taccuracy=0.952969\n",
      "Epoch[24] Batch [99]\tSpeed: 1050.512883 samples/sec\taccuracy=0.953516\n",
      "Epoch[24] Batch [149]\tSpeed: 1152.313470 samples/sec\taccuracy=0.953490\n",
      "Epoch[24] Batch [199]\tSpeed: 668.888832 samples/sec\taccuracy=0.954258\n",
      "Epoch[24] Batch [249]\tSpeed: 1132.230592 samples/sec\taccuracy=0.954094\n",
      "Epoch[24] Batch [299]\tSpeed: 984.910753 samples/sec\taccuracy=0.954531\n",
      "Epoch[24] Batch [349]\tSpeed: 701.862687 samples/sec\taccuracy=0.954754\n",
      "[Epoch 24] training: accuracy=0.955148\n",
      "[Epoch 24] time cost: 47.801861\n",
      "[Epoch 24] validation: accuracy=0.903045\n",
      "Epoch[25] Batch [49]\tSpeed: 1102.513003 samples/sec\taccuracy=0.954688\n",
      "Epoch[25] Batch [99]\tSpeed: 1046.034463 samples/sec\taccuracy=0.952969\n",
      "Epoch[25] Batch [149]\tSpeed: 1184.063377 samples/sec\taccuracy=0.952604\n",
      "Epoch[25] Batch [199]\tSpeed: 911.970946 samples/sec\taccuracy=0.952227\n",
      "Epoch[25] Batch [249]\tSpeed: 1040.046401 samples/sec\taccuracy=0.952719\n",
      "Epoch[25] Batch [299]\tSpeed: 721.501773 samples/sec\taccuracy=0.952422\n",
      "Epoch[25] Batch [349]\tSpeed: 1073.640902 samples/sec\taccuracy=0.953080\n",
      "[Epoch 25] training: accuracy=0.952945\n",
      "[Epoch 25] time cost: 48.213749\n",
      "[Epoch 25] validation: accuracy=0.904447\n",
      "Epoch[26] Batch [49]\tSpeed: 1065.786791 samples/sec\taccuracy=0.949688\n",
      "Epoch[26] Batch [99]\tSpeed: 1192.595935 samples/sec\taccuracy=0.950391\n",
      "Epoch[26] Batch [149]\tSpeed: 1218.051720 samples/sec\taccuracy=0.951458\n",
      "Epoch[26] Batch [199]\tSpeed: 1197.559941 samples/sec\taccuracy=0.951055\n",
      "Epoch[26] Batch [249]\tSpeed: 1053.628064 samples/sec\taccuracy=0.951844\n",
      "Epoch[26] Batch [299]\tSpeed: 1046.415724 samples/sec\taccuracy=0.951224\n",
      "Epoch[26] Batch [349]\tSpeed: 1124.247520 samples/sec\taccuracy=0.953080\n",
      "[Epoch 26] training: accuracy=0.953345\n",
      "[Epoch 26] time cost: 48.116119\n",
      "[Epoch 26] validation: accuracy=0.900040\n",
      "Epoch[27] Batch [49]\tSpeed: 1209.869028 samples/sec\taccuracy=0.956250\n",
      "Epoch[27] Batch [99]\tSpeed: 1268.161060 samples/sec\taccuracy=0.954688\n",
      "Epoch[27] Batch [149]\tSpeed: 1104.334087 samples/sec\taccuracy=0.954375\n",
      "Epoch[27] Batch [199]\tSpeed: 1067.303847 samples/sec\taccuracy=0.954258\n",
      "Epoch[27] Batch [249]\tSpeed: 1084.984433 samples/sec\taccuracy=0.953344\n",
      "Epoch[27] Batch [299]\tSpeed: 1161.447360 samples/sec\taccuracy=0.953099\n",
      "Epoch[27] Batch [349]\tSpeed: 1122.397767 samples/sec\taccuracy=0.954152\n",
      "[Epoch 27] training: accuracy=0.954487\n",
      "[Epoch 27] time cost: 48.275631\n",
      "[Epoch 27] validation: accuracy=0.905349\n",
      "Epoch[28] Batch [49]\tSpeed: 1015.725609 samples/sec\taccuracy=0.953594\n",
      "Epoch[28] Batch [99]\tSpeed: 1107.710886 samples/sec\taccuracy=0.953438\n",
      "Epoch[28] Batch [149]\tSpeed: 1116.615388 samples/sec\taccuracy=0.954115\n",
      "Epoch[28] Batch [199]\tSpeed: 1033.987088 samples/sec\taccuracy=0.953789\n",
      "Epoch[28] Batch [249]\tSpeed: 845.845635 samples/sec\taccuracy=0.953875\n",
      "Epoch[28] Batch [299]\tSpeed: 976.116508 samples/sec\taccuracy=0.953646\n",
      "Epoch[28] Batch [349]\tSpeed: 1222.428212 samples/sec\taccuracy=0.955045\n",
      "[Epoch 28] training: accuracy=0.955143\n",
      "[Epoch 28] time cost: 48.884224\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Epoch 28] validation: accuracy=0.900441\n",
      "Epoch[29] Batch [49]\tSpeed: 750.140300 samples/sec\taccuracy=0.955156\n",
      "Epoch[29] Batch [99]\tSpeed: 1207.390267 samples/sec\taccuracy=0.950078\n",
      "Epoch[29] Batch [149]\tSpeed: 1092.602325 samples/sec\taccuracy=0.951979\n",
      "Epoch[29] Batch [199]\tSpeed: 1128.612717 samples/sec\taccuracy=0.951094\n",
      "Epoch[29] Batch [249]\tSpeed: 987.579466 samples/sec\taccuracy=0.952000\n",
      "Epoch[29] Batch [299]\tSpeed: 1187.194919 samples/sec\taccuracy=0.953281\n",
      "Epoch[29] Batch [349]\tSpeed: 1133.554986 samples/sec\taccuracy=0.953438\n",
      "[Epoch 29] training: accuracy=0.953726\n",
      "[Epoch 29] time cost: 48.473119\n",
      "[Epoch 29] validation: accuracy=0.899840\n",
      "Epoch[30] Batch [49]\tSpeed: 1127.799510 samples/sec\taccuracy=0.953281\n",
      "Epoch[30] Batch [99]\tSpeed: 1044.894380 samples/sec\taccuracy=0.952109\n",
      "Epoch[30] Batch [149]\tSpeed: 1052.502327 samples/sec\taccuracy=0.952604\n",
      "Epoch[30] Batch [199]\tSpeed: 1088.157738 samples/sec\taccuracy=0.953477\n",
      "Epoch[30] Batch [249]\tSpeed: 1216.649471 samples/sec\taccuracy=0.954937\n",
      "Epoch[30] Batch [299]\tSpeed: 818.489367 samples/sec\taccuracy=0.955052\n",
      "Epoch[30] Batch [349]\tSpeed: 1260.529625 samples/sec\taccuracy=0.956317\n",
      "[Epoch 30] training: accuracy=0.956602\n",
      "[Epoch 30] time cost: 48.168323\n",
      "[Epoch 30] validation: accuracy=0.905558\n",
      "Epoch[31] Batch [49]\tSpeed: 996.375277 samples/sec\taccuracy=0.956562\n",
      "Epoch[31] Batch [99]\tSpeed: 1181.319902 samples/sec\taccuracy=0.955859\n",
      "Epoch[31] Batch [149]\tSpeed: 1218.322604 samples/sec\taccuracy=0.956979\n",
      "Epoch[31] Batch [199]\tSpeed: 1061.057937 samples/sec\taccuracy=0.957109\n",
      "Epoch[31] Batch [249]\tSpeed: 1099.728406 samples/sec\taccuracy=0.957875\n",
      "Epoch[31] Batch [299]\tSpeed: 1054.710625 samples/sec\taccuracy=0.957526\n",
      "Epoch[31] Batch [349]\tSpeed: 1202.632791 samples/sec\taccuracy=0.958415\n",
      "[Epoch 31] training: accuracy=0.958820\n",
      "[Epoch 31] time cost: 47.274505\n",
      "[Epoch 31] validation: accuracy=0.906751\n",
      "Epoch[32] Batch [49]\tSpeed: 909.890553 samples/sec\taccuracy=0.956719\n",
      "Epoch[32] Batch [99]\tSpeed: 1142.511900 samples/sec\taccuracy=0.955781\n",
      "Epoch[32] Batch [149]\tSpeed: 1151.821500 samples/sec\taccuracy=0.956250\n",
      "Epoch[32] Batch [199]\tSpeed: 759.511973 samples/sec\taccuracy=0.956406\n",
      "Epoch[32] Batch [249]\tSpeed: 1110.363844 samples/sec\taccuracy=0.956937\n",
      "Epoch[32] Batch [299]\tSpeed: 1227.515095 samples/sec\taccuracy=0.957708\n",
      "Epoch[32] Batch [349]\tSpeed: 1204.062303 samples/sec\taccuracy=0.958705\n",
      "[Epoch 32] training: accuracy=0.959195\n",
      "[Epoch 32] time cost: 47.985565\n",
      "[Epoch 32] validation: accuracy=0.907051\n",
      "Epoch[33] Batch [49]\tSpeed: 1171.925765 samples/sec\taccuracy=0.955937\n",
      "Epoch[33] Batch [99]\tSpeed: 1152.085974 samples/sec\taccuracy=0.956406\n",
      "Epoch[33] Batch [149]\tSpeed: 1191.264868 samples/sec\taccuracy=0.956823\n",
      "Epoch[33] Batch [199]\tSpeed: 1073.877133 samples/sec\taccuracy=0.955937\n",
      "Epoch[33] Batch [249]\tSpeed: 897.589822 samples/sec\taccuracy=0.956438\n",
      "Epoch[33] Batch [299]\tSpeed: 1059.583667 samples/sec\taccuracy=0.956302\n",
      "Epoch[33] Batch [349]\tSpeed: 1099.516694 samples/sec\taccuracy=0.957612\n",
      "[Epoch 33] training: accuracy=0.957801\n",
      "[Epoch 33] time cost: 48.189878\n",
      "[Epoch 33] validation: accuracy=0.902544\n",
      "Epoch[34] Batch [49]\tSpeed: 1118.276049 samples/sec\taccuracy=0.960938\n",
      "Epoch[34] Batch [99]\tSpeed: 1216.610872 samples/sec\taccuracy=0.958047\n",
      "Epoch[34] Batch [149]\tSpeed: 914.180721 samples/sec\taccuracy=0.958073\n",
      "Epoch[34] Batch [199]\tSpeed: 1107.144370 samples/sec\taccuracy=0.957695\n",
      "Epoch[34] Batch [249]\tSpeed: 872.702569 samples/sec\taccuracy=0.957781\n",
      "Epoch[34] Batch [299]\tSpeed: 1120.778663 samples/sec\taccuracy=0.958073\n",
      "Epoch[34] Batch [349]\tSpeed: 1119.021927 samples/sec\taccuracy=0.958147\n",
      "[Epoch 34] training: accuracy=0.958640\n",
      "[Epoch 34] time cost: 47.464797\n",
      "[Epoch 34] validation: accuracy=0.904748\n",
      "Epoch[35] Batch [49]\tSpeed: 988.532315 samples/sec\taccuracy=0.958125\n",
      "Epoch[35] Batch [99]\tSpeed: 1071.692747 samples/sec\taccuracy=0.957266\n",
      "Epoch[35] Batch [149]\tSpeed: 1204.748596 samples/sec\taccuracy=0.956406\n",
      "Epoch[35] Batch [199]\tSpeed: 987.873853 samples/sec\taccuracy=0.957305\n",
      "Epoch[35] Batch [249]\tSpeed: 990.922503 samples/sec\taccuracy=0.958000\n",
      "Epoch[35] Batch [299]\tSpeed: 1019.889613 samples/sec\taccuracy=0.957969\n",
      "Epoch[35] Batch [349]\tSpeed: 1141.744103 samples/sec\taccuracy=0.958214\n",
      "[Epoch 35] training: accuracy=0.958574\n",
      "[Epoch 35] time cost: 48.849428\n",
      "[Epoch 35] validation: accuracy=0.903946\n",
      "Epoch[36] Batch [49]\tSpeed: 1081.496151 samples/sec\taccuracy=0.955781\n",
      "Epoch[36] Batch [99]\tSpeed: 1148.561411 samples/sec\taccuracy=0.954453\n",
      "Epoch[36] Batch [149]\tSpeed: 977.070472 samples/sec\taccuracy=0.955365\n",
      "Epoch[36] Batch [199]\tSpeed: 1064.444962 samples/sec\taccuracy=0.954570\n",
      "Epoch[36] Batch [249]\tSpeed: 899.382195 samples/sec\taccuracy=0.955219\n",
      "Epoch[36] Batch [299]\tSpeed: 1052.174354 samples/sec\taccuracy=0.955313\n",
      "Epoch[36] Batch [349]\tSpeed: 1010.599563 samples/sec\taccuracy=0.956205\n",
      "[Epoch 36] training: accuracy=0.956382\n",
      "[Epoch 36] time cost: 48.584885\n",
      "[Epoch 36] validation: accuracy=0.904447\n",
      "Epoch[37] Batch [49]\tSpeed: 892.811739 samples/sec\taccuracy=0.955781\n",
      "Epoch[37] Batch [99]\tSpeed: 1162.131199 samples/sec\taccuracy=0.954922\n",
      "Epoch[37] Batch [149]\tSpeed: 1214.771993 samples/sec\taccuracy=0.956667\n",
      "Epoch[37] Batch [199]\tSpeed: 1120.519011 samples/sec\taccuracy=0.956680\n",
      "Epoch[37] Batch [249]\tSpeed: 1083.926396 samples/sec\taccuracy=0.956688\n",
      "Epoch[37] Batch [299]\tSpeed: 1208.961781 samples/sec\taccuracy=0.957240\n",
      "Epoch[37] Batch [349]\tSpeed: 1154.322789 samples/sec\taccuracy=0.957790\n",
      "[Epoch 37] training: accuracy=0.958093\n",
      "[Epoch 37] time cost: 47.316989\n",
      "[Epoch 37] validation: accuracy=0.903846\n",
      "Epoch[38] Batch [49]\tSpeed: 1217.543445 samples/sec\taccuracy=0.955781\n",
      "Epoch[38] Batch [99]\tSpeed: 1222.099858 samples/sec\taccuracy=0.954375\n",
      "Epoch[38] Batch [149]\tSpeed: 1104.872718 samples/sec\taccuracy=0.954792\n",
      "Epoch[38] Batch [199]\tSpeed: 1250.829222 samples/sec\taccuracy=0.954063\n",
      "Epoch[38] Batch [249]\tSpeed: 1117.628875 samples/sec\taccuracy=0.955281\n",
      "Epoch[38] Batch [299]\tSpeed: 1126.878660 samples/sec\taccuracy=0.955182\n",
      "Epoch[38] Batch [349]\tSpeed: 1132.321336 samples/sec\taccuracy=0.956071\n",
      "[Epoch 38] training: accuracy=0.956202\n",
      "[Epoch 38] time cost: 48.173148\n",
      "[Epoch 38] validation: accuracy=0.905360\n",
      "Epoch[39] Batch [49]\tSpeed: 1018.189410 samples/sec\taccuracy=0.956875\n",
      "Epoch[39] Batch [99]\tSpeed: 1151.888225 samples/sec\taccuracy=0.956250\n",
      "Epoch[39] Batch [149]\tSpeed: 1036.624449 samples/sec\taccuracy=0.956458\n",
      "Epoch[39] Batch [199]\tSpeed: 1144.426161 samples/sec\taccuracy=0.956406\n",
      "Epoch[39] Batch [249]\tSpeed: 1099.579749 samples/sec\taccuracy=0.956063\n",
      "Epoch[39] Batch [299]\tSpeed: 1099.791485 samples/sec\taccuracy=0.956354\n",
      "Epoch[39] Batch [349]\tSpeed: 1090.682302 samples/sec\taccuracy=0.956719\n",
      "[Epoch 39] training: accuracy=0.956941\n",
      "[Epoch 39] time cost: 50.768927\n",
      "[Epoch 39] validation: accuracy=0.905148\n"
     ]
    }
   ],
   "source": [
    "initialize_by_key(sp_resnet, 'sp_mode', mx.init.Constant(2), ctx, True)\n",
    "\n",
    "epochs = 40\n",
    "learning_rate = 0.01\n",
    "scheduler = mx.lr_scheduler.MultiFactorScheduler([10*schedule_factor, 20*schedule_factor, 30*schedule_factor, 40*schedule_factor], factor=0.1)\n",
    "\n",
    "train(sp_resnet, learning_rate, momentum, weight_decay, epochs, scheduler, ctx, train_data, val_data, log_interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we fix $W_B$, $W_C$ and only train $\\tilde a$ for another 10 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch[0] Batch [49]\tSpeed: 801.545118 samples/sec\taccuracy=0.953125\n",
      "Epoch[0] Batch [99]\tSpeed: 849.393277 samples/sec\taccuracy=0.954844\n",
      "Epoch[0] Batch [149]\tSpeed: 1027.907432 samples/sec\taccuracy=0.955573\n",
      "Epoch[0] Batch [199]\tSpeed: 1284.460067 samples/sec\taccuracy=0.955156\n",
      "Epoch[0] Batch [249]\tSpeed: 1283.637014 samples/sec\taccuracy=0.955969\n",
      "Epoch[0] Batch [299]\tSpeed: 1266.013885 samples/sec\taccuracy=0.956536\n",
      "Epoch[0] Batch [349]\tSpeed: 1211.157310 samples/sec\taccuracy=0.957790\n",
      "[Epoch 0] training: accuracy=0.958894\n",
      "[Epoch 0] time cost: 52.003801\n",
      "[Epoch 0] validation: accuracy=0.907953\n",
      "Epoch[1] Batch [49]\tSpeed: 1138.062861 samples/sec\taccuracy=0.957344\n",
      "Epoch[1] Batch [99]\tSpeed: 1211.865394 samples/sec\taccuracy=0.956172\n",
      "Epoch[1] Batch [149]\tSpeed: 1195.919335 samples/sec\taccuracy=0.956146\n",
      "Epoch[1] Batch [199]\tSpeed: 1226.970852 samples/sec\taccuracy=0.956211\n",
      "Epoch[1] Batch [249]\tSpeed: 1302.910806 samples/sec\taccuracy=0.956781\n",
      "Epoch[1] Batch [299]\tSpeed: 1214.184067 samples/sec\taccuracy=0.957708\n",
      "Epoch[1] Batch [349]\tSpeed: 1133.162605 samples/sec\taccuracy=0.958594\n",
      "[Epoch 1] training: accuracy=0.958879\n",
      "[Epoch 1] time cost: 43.622476\n",
      "[Epoch 1] validation: accuracy=0.909155\n",
      "Epoch[2] Batch [49]\tSpeed: 1196.537702 samples/sec\taccuracy=0.959688\n",
      "Epoch[2] Batch [99]\tSpeed: 1041.705222 samples/sec\taccuracy=0.957109\n",
      "Epoch[2] Batch [149]\tSpeed: 969.742734 samples/sec\taccuracy=0.957448\n",
      "Epoch[2] Batch [199]\tSpeed: 1148.949779 samples/sec\taccuracy=0.957930\n",
      "Epoch[2] Batch [249]\tSpeed: 1042.400996 samples/sec\taccuracy=0.958469\n",
      "Epoch[2] Batch [299]\tSpeed: 1202.716310 samples/sec\taccuracy=0.958411\n",
      "Epoch[2] Batch [349]\tSpeed: 1226.665277 samples/sec\taccuracy=0.959308\n",
      "[Epoch 2] training: accuracy=0.959759\n",
      "[Epoch 2] time cost: 44.116256\n",
      "[Epoch 2] validation: accuracy=0.909155\n",
      "Epoch[3] Batch [49]\tSpeed: 1212.336085 samples/sec\taccuracy=0.958906\n",
      "Epoch[3] Batch [99]\tSpeed: 1239.176896 samples/sec\taccuracy=0.958516\n",
      "Epoch[3] Batch [149]\tSpeed: 1320.507748 samples/sec\taccuracy=0.958750\n",
      "Epoch[3] Batch [199]\tSpeed: 1173.416517 samples/sec\taccuracy=0.958711\n",
      "Epoch[3] Batch [249]\tSpeed: 1354.267503 samples/sec\taccuracy=0.959469\n",
      "Epoch[3] Batch [299]\tSpeed: 1030.256747 samples/sec\taccuracy=0.959818\n",
      "Epoch[3] Batch [349]\tSpeed: 1085.258588 samples/sec\taccuracy=0.961272\n",
      "[Epoch 3] training: accuracy=0.961679\n",
      "[Epoch 3] time cost: 43.494492\n",
      "[Epoch 3] validation: accuracy=0.910056\n",
      "Epoch[4] Batch [49]\tSpeed: 941.376725 samples/sec\taccuracy=0.960000\n",
      "Epoch[4] Batch [99]\tSpeed: 1075.684510 samples/sec\taccuracy=0.958359\n",
      "Epoch[4] Batch [149]\tSpeed: 1152.617763 samples/sec\taccuracy=0.960417\n",
      "Epoch[4] Batch [199]\tSpeed: 1097.942676 samples/sec\taccuracy=0.959922\n",
      "Epoch[4] Batch [249]\tSpeed: 1017.749327 samples/sec\taccuracy=0.960719\n",
      "Epoch[4] Batch [299]\tSpeed: 1223.537013 samples/sec\taccuracy=0.960182\n",
      "Epoch[4] Batch [349]\tSpeed: 1217.237028 samples/sec\taccuracy=0.960804\n",
      "[Epoch 4] training: accuracy=0.961037\n",
      "[Epoch 4] time cost: 44.864956\n",
      "[Epoch 4] validation: accuracy=0.910056\n",
      "Epoch[5] Batch [49]\tSpeed: 1240.055601 samples/sec\taccuracy=0.958281\n",
      "Epoch[5] Batch [99]\tSpeed: 1079.363147 samples/sec\taccuracy=0.957187\n",
      "Epoch[5] Batch [149]\tSpeed: 1258.074968 samples/sec\taccuracy=0.958281\n",
      "Epoch[5] Batch [199]\tSpeed: 1190.855297 samples/sec\taccuracy=0.957812\n",
      "Epoch[5] Batch [249]\tSpeed: 912.173929 samples/sec\taccuracy=0.958937\n",
      "Epoch[5] Batch [299]\tSpeed: 1185.014705 samples/sec\taccuracy=0.959453\n",
      "Epoch[5] Batch [349]\tSpeed: 1267.074601 samples/sec\taccuracy=0.959911\n",
      "[Epoch 5] training: accuracy=0.960517\n",
      "[Epoch 5] time cost: 44.035769\n",
      "[Epoch 5] validation: accuracy=0.909355\n",
      "Epoch[6] Batch [49]\tSpeed: 1200.626426 samples/sec\taccuracy=0.964063\n",
      "Epoch[6] Batch [99]\tSpeed: 793.321658 samples/sec\taccuracy=0.958438\n",
      "Epoch[6] Batch [149]\tSpeed: 900.483413 samples/sec\taccuracy=0.959115\n",
      "Epoch[6] Batch [199]\tSpeed: 1214.219766 samples/sec\taccuracy=0.958945\n",
      "Epoch[6] Batch [249]\tSpeed: 1160.947561 samples/sec\taccuracy=0.959344\n",
      "Epoch[6] Batch [299]\tSpeed: 1232.907749 samples/sec\taccuracy=0.959609\n",
      "Epoch[6] Batch [349]\tSpeed: 1287.429347 samples/sec\taccuracy=0.960580\n",
      "[Epoch 6] training: accuracy=0.960977\n",
      "[Epoch 6] time cost: 44.125039\n",
      "[Epoch 6] validation: accuracy=0.908722\n",
      "Epoch[7] Batch [49]\tSpeed: 1198.634773 samples/sec\taccuracy=0.955937\n",
      "Epoch[7] Batch [99]\tSpeed: 1235.276456 samples/sec\taccuracy=0.956094\n",
      "Epoch[7] Batch [149]\tSpeed: 994.734075 samples/sec\taccuracy=0.957812\n",
      "Epoch[7] Batch [199]\tSpeed: 1158.767552 samples/sec\taccuracy=0.958320\n",
      "Epoch[7] Batch [249]\tSpeed: 1291.533041 samples/sec\taccuracy=0.959063\n",
      "Epoch[7] Batch [299]\tSpeed: 1021.546906 samples/sec\taccuracy=0.960000\n",
      "Epoch[7] Batch [349]\tSpeed: 1333.420044 samples/sec\taccuracy=0.961339\n",
      "[Epoch 7] training: accuracy=0.961897\n",
      "[Epoch 7] time cost: 43.317396\n",
      "[Epoch 7] validation: accuracy=0.908654\n",
      "Epoch[8] Batch [49]\tSpeed: 1183.353491 samples/sec\taccuracy=0.959688\n",
      "Epoch[8] Batch [99]\tSpeed: 1145.551890 samples/sec\taccuracy=0.958828\n",
      "Epoch[8] Batch [149]\tSpeed: 1225.920202 samples/sec\taccuracy=0.960000\n",
      "Epoch[8] Batch [199]\tSpeed: 1246.950767 samples/sec\taccuracy=0.959961\n",
      "Epoch[8] Batch [249]\tSpeed: 1140.596763 samples/sec\taccuracy=0.960250\n",
      "Epoch[8] Batch [299]\tSpeed: 1280.468311 samples/sec\taccuracy=0.960833\n",
      "Epoch[8] Batch [349]\tSpeed: 1035.219245 samples/sec\taccuracy=0.961808\n",
      "[Epoch 8] training: accuracy=0.961999\n",
      "[Epoch 8] time cost: 43.733970\n",
      "[Epoch 8] validation: accuracy=0.907752\n",
      "Epoch[9] Batch [49]\tSpeed: 1146.559839 samples/sec\taccuracy=0.959844\n",
      "Epoch[9] Batch [99]\tSpeed: 1243.945976 samples/sec\taccuracy=0.958359\n",
      "Epoch[9] Batch [149]\tSpeed: 1005.703941 samples/sec\taccuracy=0.959167\n",
      "Epoch[9] Batch [199]\tSpeed: 1193.967152 samples/sec\taccuracy=0.959844\n",
      "Epoch[9] Batch [249]\tSpeed: 1199.700812 samples/sec\taccuracy=0.960781\n",
      "Epoch[9] Batch [299]\tSpeed: 1252.641591 samples/sec\taccuracy=0.960443\n",
      "Epoch[9] Batch [349]\tSpeed: 1258.641260 samples/sec\taccuracy=0.961496\n",
      "[Epoch 9] training: accuracy=0.962096\n",
      "[Epoch 9] time cost: 43.498626\n",
      "[Epoch 9] validation: accuracy=0.907853\n"
     ]
    }
   ],
   "source": [
    "for k, p in sp_resnet.collect_params().items():\n",
    "    if 'sp_data_weights' in k or 'sp_out_weights' in k:\n",
    "        p.grad_req = 'null'\n",
    "\n",
    "epochs = 10\n",
    "learning_rate = 0.001\n",
    "scheduler = None\n",
    "\n",
    "train(sp_resnet, learning_rate, momentum, weight_decay, epochs, scheduler, ctx, train_data, val_data, log_interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy of the model with quantized $W_A$, $W_B$ should have a validation accuracy slighlty below that of the unquantized model.\n",
    "\n",
    "We finally save the paramters of the quantized model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "sp_resnet.collect_params().save('sp_resnet.params')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Done!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "[1] https://github.com/apache/incubator-mxnet/blob/master/docs/tutorials/gluon/customop.md\n",
    "\n",
    "[2] F. Li, B. Zhang, B. Liu, Ternary Weight Networks, 2016, https://arxiv.org/abs/1605.04711v2\n",
    "\n",
    "[3] K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, 2016, http://arxiv.org/abs/1512.03385\n",
    "\n",
    "[4] https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/gluon/model_zoo/vision/resnet.py\n",
    "\n",
    "[5] https://github.com/apache/incubator-mxnet/blob/master/example/gluon/image_classification.py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}