Synthetic Task(static function learning).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiments based on section 4.1  of arXiv:1808.00508 paper: Simple Function Learning Tasks(static task)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from NALU import NALU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_shape = (10000,100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_examples = x_shape[0]\n",
    "train_columns  = x_shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = np.random.rand(train_examples,train_columns)*10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_x = np.random.rand(train_examples,train_columns)*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Partition the inputs(consistently) to create \"a\" and \"b\"\n",
    "* The paper does not describe if \"a\" and \"b\" are mutually exclusive or not, will have to experiment\n",
    "* In the paper its not clear if for every experiment you  mix (+,-,*,/)   for calculating \"y\", let's experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mutually exclusive(first half is a, second half is b)  and only use \"+\" for calculating \"y\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "partition_index = int(train_columns/2) #half colmns for a,half columns for b\n",
    "\n",
    "a = np.sum(x[:,:partition_index],axis=1)\n",
    "b = np.sum( x[:,partition_index:],axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_a = np.sum(test_x[:,:partition_index],axis=1)\n",
    "test_b = np.sum(test_x[:,partition_index:],axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Let's start simple:  y = a + b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = a + b\n",
    "y = np.expand_dims(y,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_y = test_a + test_b\n",
    "test_y = np.expand_dims(test_y,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "EPOCHS = 50000\n",
    "PRINT_EVERY = 1000\n",
    "LEARNING_RATE = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello NALU world nalu1\n",
      "hello NALU world nalu2\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "\n",
    "X = tf.placeholder(tf.float32,shape = [train_examples,train_columns])\n",
    "Y = tf.placeholder(tf.float32,shape=[train_examples,1])\n",
    "\n",
    "nalu1 = NALU(input_shape=(train_examples,train_columns),size = 2,name = \"nalu1\")\n",
    "nalu1_output = nalu1.NALU_output(X)\n",
    "        \n",
    "nalu2 = NALU(input_shape=(train_examples,2),size=1,name = \"nalu2\")\n",
    "nalu2_output = nalu2.NALU_output(nalu1_output)\n",
    "        \n",
    "loss = tf.losses.mean_squared_error(nalu2_output,Y)\n",
    "adam_optimize = tf.train.AdamOptimizer(learning_rate=LEARNING_RATE).minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 loss 250780.46875\n",
      "Epoch 1000 loss 0.11940136551856995\n",
      "Epoch 2000 loss 5.128748057359189e-07\n",
      "Epoch 3000 loss 6.997026291699626e-10\n",
      "Epoch 4000 loss 6.8414957032914e-10\n",
      "Epoch 5000 loss 6.761401993848892e-10\n",
      "Epoch 6000 loss 7.118098332981049e-10\n",
      "Epoch 7000 loss 2.1796301652443617e-08\n",
      "Epoch 8000 loss 4.0951093183139164e-08\n",
      "Epoch 9000 loss 0.003723412286490202\n",
      "Epoch 10000 loss 9.272247414315871e-10\n",
      "Epoch 11000 loss 6.888993264730914e-10\n",
      "Epoch 12000 loss 5.412106929725269e-06\n",
      "Epoch 13000 loss 0.022368982434272766\n",
      "Epoch 14000 loss 1.6384062767028809\n",
      "Epoch 15000 loss 8.577713828117339e-08\n",
      "Epoch 16000 loss 1.635719115711254e-08\n",
      "Epoch 17000 loss 0.023791154846549034\n",
      "Epoch 18000 loss 0.09449239820241928\n",
      "Epoch 19000 loss 0.5954846739768982\n",
      "Epoch 20000 loss 5.955807846902417e-09\n",
      "Epoch 21000 loss 6.930902518575977e-10\n",
      "Epoch 22000 loss 9.251385790776112e-08\n",
      "Epoch 23000 loss 1.3964753975415078e-07\n",
      "Epoch 24000 loss 0.0003281407698523253\n",
      "Epoch 25000 loss 7.464550644264989e-10\n",
      "Epoch 26000 loss 1.2091361245580856e-09\n",
      "Epoch 27000 loss 6.97467472665636e-10\n",
      "Epoch 28000 loss 8.088536507244726e-10\n",
      "Epoch 29000 loss 4.6279932774950794e-08\n",
      "Epoch 30000 loss 1.4199875719711486e-09\n",
      "Epoch 31000 loss 6.819144138248134e-10\n",
      "Epoch 32000 loss 1.017283701720828e-09\n",
      "Epoch 33000 loss 1.6622245802011548e-09\n",
      "Epoch 34000 loss 7.401872448298263e-09\n",
      "Epoch 35000 loss 2.6396476116019585e-09\n",
      "Epoch 36000 loss 7.058493789457998e-10\n",
      "Epoch 37000 loss 1.2982822816809403e-08\n",
      "Epoch 38000 loss 1.494007847213652e-05\n",
      "Epoch 39000 loss 1.4227637734620657e-07\n",
      "Epoch 40000 loss 5.849189932405352e-08\n",
      "Epoch 41000 loss 9.571202497227205e-09\n",
      "Epoch 42000 loss 3.0487402113976714e-07\n",
      "Epoch 43000 loss 3.070198095755927e-09\n",
      "Epoch 44000 loss 1.68023532864936e-07\n",
      "Epoch 45000 loss 3.8941017521665344e-08\n",
      "Epoch 46000 loss 1.6403073459514417e-05\n",
      "Epoch 47000 loss 9.659677502327213e-10\n",
      "Epoch 48000 loss 1.6589974904945848e-07\n",
      "Epoch 49000 loss 2.217665384662837e-09\n"
     ]
    }
   ],
   "source": [
    "\n",
    "with tf.Session() as session:\n",
    "        \n",
    "    session.run(tf.global_variables_initializer())\n",
    "        \n",
    "    for epoch in range(EPOCHS):\n",
    "        _  = session.run(adam_optimize,feed_dict={X:x,Y:y})\n",
    "            \n",
    "        if epoch % PRINT_EVERY == 0:\n",
    "            batch_loss = session.run(loss,feed_dict={X:x,Y:y})\n",
    "            print(\"Epoch {} loss {}\".format(epoch,batch_loss))\n",
    "                \n",
    "    \n",
    "    test_predictions,test_loss = session.run([nalu2_output,loss],feed_dict={X:test_x,Y:test_y})\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5113.43457031],\n",
       "       [ 4609.26074219],\n",
       "       [ 5120.65966797],\n",
       "       [ 4958.85595703],\n",
       "       [ 5114.78320312],\n",
       "       [ 4690.97900391],\n",
       "       [ 5303.27587891],\n",
       "       [ 5150.89355469],\n",
       "       [ 5120.98632812],\n",
       "       [ 5135.76171875],\n",
       "       [ 5249.66894531],\n",
       "       [ 4974.20556641],\n",
       "       [ 5540.12792969],\n",
       "       [ 4858.22998047],\n",
       "       [ 5154.43457031],\n",
       "       [ 4611.546875  ],\n",
       "       [ 4521.08056641],\n",
       "       [ 5027.15332031],\n",
       "       [ 5280.06689453],\n",
       "       [ 5593.76855469],\n",
       "       [ 5054.38867188],\n",
       "       [ 5286.91894531],\n",
       "       [ 4715.85595703],\n",
       "       [ 5083.53417969],\n",
       "       [ 5015.07421875],\n",
       "       [ 4531.29980469],\n",
       "       [ 5017.35009766],\n",
       "       [ 4883.60302734],\n",
       "       [ 4692.61083984],\n",
       "       [ 5049.95214844]], dtype=float32)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_predictions[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5113.43424518],\n",
       "       [ 4609.26142308],\n",
       "       [ 5120.66000463],\n",
       "       [ 4958.85591925],\n",
       "       [ 5114.78310362],\n",
       "       [ 4690.97927069],\n",
       "       [ 5303.27625141],\n",
       "       [ 5150.89366138],\n",
       "       [ 5120.98663789],\n",
       "       [ 5135.76163751],\n",
       "       [ 5249.66914258],\n",
       "       [ 4974.2058166 ],\n",
       "       [ 5540.12816172],\n",
       "       [ 4858.23002171],\n",
       "       [ 5154.43486268],\n",
       "       [ 4611.54718497],\n",
       "       [ 4521.08107865],\n",
       "       [ 5027.15345423],\n",
       "       [ 5280.06724196],\n",
       "       [ 5593.76835315],\n",
       "       [ 5054.38898575],\n",
       "       [ 5286.91931238],\n",
       "       [ 4715.85604938],\n",
       "       [ 5083.53445873],\n",
       "       [ 5015.07447509],\n",
       "       [ 4531.29996599],\n",
       "       [ 5017.34992426],\n",
       "       [ 4883.60310139],\n",
       "       [ 4692.61090727],\n",
       "       [ 5049.9524399 ]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_y[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0485053e-07"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Almost 0 test error!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##### Let's test substraction:  y = a - b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "EPOCHS = 50000\n",
    "PRINT_EVERY = 1000\n",
    "LEARNING_RATE = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = a - b\n",
    "y = np.expand_dims(y,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_y = test_a - test_b\n",
    "test_y = np.expand_dims(test_y,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello NALU world nalu1\n",
      "hello NALU world nalu2\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "\n",
    "X = tf.placeholder(tf.float32,shape = [train_examples,train_columns])\n",
    "Y = tf.placeholder(tf.float32,shape=[train_examples,1])\n",
    "\n",
    "nalu1 = NALU(input_shape=(train_examples,train_columns),size = 2,name = \"nalu1\")\n",
    "nalu1_output = nalu1.NALU_output(X)\n",
    "        \n",
    "nalu2 = NALU(input_shape=(train_examples,2),size=1,name = \"nalu2\")\n",
    "nalu2_output = nalu2.NALU_output(nalu1_output)\n",
    "        \n",
    "loss = tf.losses.mean_squared_error(nalu2_output,Y)\n",
    "adam_optimize = tf.train.AdamOptimizer(learning_rate=LEARNING_RATE).minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 loss 839.9984741210938\n",
      "Epoch 1000 loss 42.631893157958984\n",
      "Epoch 2000 loss 7.908759593963623\n",
      "Epoch 3000 loss 2.9351465702056885\n",
      "Epoch 4000 loss 0.8166720867156982\n",
      "Epoch 5000 loss 0.0017067322041839361\n",
      "Epoch 6000 loss 2.8314427254372276e-05\n",
      "Epoch 7000 loss 9.774471436685417e-06\n",
      "Epoch 8000 loss 4.690074547397671e-06\n",
      "Epoch 9000 loss 2.526944854253088e-06\n",
      "Epoch 10000 loss 1.6528140349691967e-06\n",
      "Epoch 11000 loss 0.01728249154984951\n",
      "Epoch 12000 loss 1.2513423826021608e-06\n",
      "Epoch 13000 loss 2.911564536134392e-07\n",
      "Epoch 14000 loss 3.714154809131287e-05\n",
      "Epoch 15000 loss 9.203441004501656e-05\n",
      "Epoch 16000 loss 1.798921189788416e-08\n",
      "Epoch 17000 loss 2.059716486257912e-08\n",
      "Epoch 18000 loss 5.751886189564459e-10\n",
      "Epoch 19000 loss 5.688158832839463e-10\n",
      "Epoch 20000 loss 3.8674097257995754e-10\n",
      "Epoch 21000 loss 2.8805799412445765e-10\n",
      "Epoch 22000 loss 4.130210062847084e-10\n",
      "Epoch 23000 loss 4.319617996628722e-09\n",
      "Epoch 24000 loss 1.4198934650266892e-07\n",
      "Epoch 25000 loss 0.00019193232583347708\n",
      "Epoch 26000 loss 8.987980436359067e-06\n",
      "Epoch 27000 loss 2.3133185095502995e-05\n",
      "Epoch 28000 loss 0.00028494521393440664\n",
      "Epoch 29000 loss 1.0950919460128716e-07\n",
      "Epoch 30000 loss 0.0007723443559370935\n",
      "Epoch 31000 loss 2.664052226464264e-05\n",
      "Epoch 32000 loss 2.3969100038101487e-09\n",
      "Epoch 33000 loss 1.1351241058221717e-09\n",
      "Epoch 34000 loss 7.174112970353974e-10\n",
      "Epoch 35000 loss 0.0005128461634740233\n",
      "Epoch 36000 loss 7.246826498885639e-06\n",
      "Epoch 37000 loss 0.004995301831513643\n",
      "Epoch 38000 loss 1.4803216430436805e-09\n",
      "Epoch 39000 loss 2.2800211496587508e-08\n",
      "Epoch 40000 loss 0.0011996246175840497\n",
      "Epoch 41000 loss 5.453765334095806e-05\n",
      "Epoch 42000 loss 9.253266597397669e-08\n",
      "Epoch 43000 loss 4.914726014249027e-05\n",
      "Epoch 44000 loss 2.442955974402139e-06\n",
      "Epoch 45000 loss 2.0811847800672467e-09\n",
      "Epoch 46000 loss 4.162383326899999e-09\n",
      "Epoch 47000 loss 1.1850869174878653e-09\n",
      "Epoch 48000 loss 8.122116867959051e-10\n",
      "Epoch 49000 loss 1.0977406761369934e-09\n"
     ]
    }
   ],
   "source": [
    "\n",
    "with tf.Session() as session:\n",
    "        \n",
    "    session.run(tf.global_variables_initializer())\n",
    "        \n",
    "    for epoch in range(EPOCHS):\n",
    "        _  = session.run(adam_optimize,feed_dict={X:x,Y:y})\n",
    "            \n",
    "        if epoch % PRINT_EVERY == 0:\n",
    "            batch_loss = session.run(loss,feed_dict={X:x,Y:y})\n",
    "            print(\"Epoch {} loss {}\".format(epoch,batch_loss))\n",
    "                \n",
    "    \n",
    "    test_predictions,test_loss = session.run([nalu2_output,loss],feed_dict={X:test_x,Y:test_y})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-339.02444458],\n",
       "       [  87.66676331],\n",
       "       [ 230.49951172],\n",
       "       [ -45.19269562],\n",
       "       [ 386.21716309],\n",
       "       [-148.8387146 ],\n",
       "       [ 216.1870575 ],\n",
       "       [-271.97702026],\n",
       "       [ 156.53552246],\n",
       "       [ 107.4932251 ],\n",
       "       [ 479.77886963],\n",
       "       [  71.05042267],\n",
       "       [-202.28736877],\n",
       "       [-162.69271851],\n",
       "       [  15.70700645],\n",
       "       [-214.10302734],\n",
       "       [ 343.95251465],\n",
       "       [-228.28921509],\n",
       "       [ 243.45069885],\n",
       "       [ 336.44842529],\n",
       "       [ 415.8142395 ],\n",
       "       [-354.76208496],\n",
       "       [ -69.74006653],\n",
       "       [  14.3451252 ],\n",
       "       [-412.66531372],\n",
       "       [ 259.0324707 ],\n",
       "       [ 270.30831909],\n",
       "       [  28.43042374],\n",
       "       [ 725.19287109],\n",
       "       [ 290.15219116]], dtype=float32)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_predictions[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-339.02038609],\n",
       "       [  87.66877671],\n",
       "       [ 230.50130115],\n",
       "       [ -45.19007335],\n",
       "       [ 386.21910168],\n",
       "       [-148.83503545],\n",
       "       [ 216.18949567],\n",
       "       [-271.9737928 ],\n",
       "       [ 156.53768986],\n",
       "       [ 107.49536068],\n",
       "       [ 479.78026317],\n",
       "       [  71.05218225],\n",
       "       [-202.2832789 ],\n",
       "       [-162.688747  ],\n",
       "       [  15.70958759],\n",
       "       [-214.09933957],\n",
       "       [ 343.95412439],\n",
       "       [-228.28538559],\n",
       "       [ 243.45302396],\n",
       "       [ 336.45070865],\n",
       "       [ 415.81587176],\n",
       "       [-354.75806346],\n",
       "       [ -69.73686848],\n",
       "       [  14.34819501],\n",
       "       [-412.66110934],\n",
       "       [ 259.03447667],\n",
       "       [ 270.31056982],\n",
       "       [  28.43300668],\n",
       "       [ 725.19343619],\n",
       "       [ 290.15466975]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_y[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.3306926e-06"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Again almost 0 test error!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Let's test multiplication:  y = a * b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "EPOCHS = 250000\n",
    "PRINT_EVERY = 10000\n",
    "LEARNING_RATE = 0.0001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = a * b\n",
    "y = np.expand_dims(y,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_y = test_a * test_b\n",
    "test_y = np.expand_dims(test_y,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello NALU world nalu1\n",
      "hello NALU world hidden\n",
      "hello NALU world nalu2\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "X = tf.placeholder(tf.float32,shape = [train_examples,train_columns])\n",
    "Y = tf.placeholder(tf.float32,shape=[train_examples,1])\n",
    "\n",
    "nalu1 = NALU(input_shape=(train_examples,train_columns),size = 100,name = \"nalu1\")\n",
    "nalu1_output = nalu1.NALU_output(X)\n",
    "        \n",
    "nalu_h = NALU(input_shape=(train_examples,100),size = 100,name = \"hidden\")\n",
    "naluh_output = nalu_h.NALU_output(nalu1_output)\n",
    "\n",
    "nalu2 = NALU(input_shape=(train_examples,100),size=1,name = \"nalu2\")\n",
    "nalu2_output = nalu2.NALU_output(naluh_output)\n",
    "        \n",
    "loss = tf.losses.mean_squared_error(nalu2_output,Y)\n",
    "adam_optimize = tf.train.AdamOptimizer(learning_rate=LEARNING_RATE).minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 loss 3973484032.0\n",
      "Epoch 10000 loss 14950846.0\n",
      "Epoch 20000 loss 7298551.0\n",
      "Epoch 30000 loss 548227.625\n",
      "Epoch 40000 loss 124180.6875\n",
      "Epoch 50000 loss 48019.03515625\n",
      "Epoch 60000 loss 26755.26953125\n",
      "Epoch 70000 loss 19320.873046875\n",
      "Epoch 80000 loss 13579.779296875\n",
      "Epoch 90000 loss 10281.5234375\n",
      "Epoch 100000 loss 9164.654296875\n",
      "Epoch 110000 loss 30727.833984375\n",
      "Epoch 120000 loss 5449.77587890625\n",
      "Epoch 130000 loss 5677.49267578125\n",
      "Epoch 140000 loss 4365.5048828125\n",
      "Epoch 150000 loss 4343.26708984375\n",
      "Epoch 160000 loss 4014.20751953125\n",
      "Epoch 170000 loss 5540.86865234375\n",
      "Epoch 180000 loss 14034.7294921875\n",
      "Epoch 190000 loss 4980.0087890625\n",
      "Epoch 200000 loss 6113.88232421875\n",
      "Epoch 210000 loss 3672.96044921875\n",
      "Epoch 220000 loss 9633.275390625\n",
      "Epoch 230000 loss 6462.498046875\n",
      "Epoch 240000 loss 3384.115966796875\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "        \n",
    "    session.run(tf.global_variables_initializer())\n",
    "        \n",
    "    for epoch in range(EPOCHS):\n",
    "        _  = session.run(adam_optimize,feed_dict={X:x,Y:y})\n",
    "            \n",
    "        if epoch % PRINT_EVERY == 0:\n",
    "            batch_loss = session.run(loss,feed_dict={X:x,Y:y})\n",
    "            print(\"Epoch {} loss {}\".format(epoch,batch_loss))\n",
    "                \n",
    "    \n",
    "    test_predictions,test_loss = session.run([nalu2_output,loss],feed_dict={X:test_x,Y:test_y})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Not so good results with multiplication :("
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Let's test division:  y = a / b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "EPOCHS = 100000\n",
    "PRINT_EVERY = 10000\n",
    "LEARNING_RATE = 0.0001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = a / b\n",
    "y = np.expand_dims(y,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_y = test_a / test_b\n",
    "test_y = np.expand_dims(test_y,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello NALU world nalu1\n",
      "hello NALU world hidden\n",
      "hello NALU world nalu2\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "X = tf.placeholder(tf.float32,shape = [train_examples,train_columns])\n",
    "Y = tf.placeholder(tf.float32,shape=[train_examples,1])\n",
    "\n",
    "nalu1 = NALU(input_shape=(train_examples,train_columns),size = 100,name = \"nalu1\")\n",
    "nalu1_output = nalu1.NALU_output(X)\n",
    "        \n",
    "nalu_h = NALU(input_shape=(train_examples,100),size = 100,name = \"hidden\")\n",
    "naluh_output = nalu_h.NALU_output(nalu1_output)\n",
    "\n",
    "nalu2 = NALU(input_shape=(train_examples,100),size=1,name = \"nalu2\")\n",
    "nalu2_output = nalu2.NALU_output(naluh_output)\n",
    "        \n",
    "loss = tf.losses.mean_squared_error(nalu2_output,Y)\n",
    "adam_optimize = tf.train.AdamOptimizer(learning_rate=LEARNING_RATE).minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 loss 0.4514217674732208\n",
      "Epoch 10000 loss 0.0030641877092421055\n",
      "Epoch 20000 loss 0.00017290556570515037\n",
      "Epoch 30000 loss 1.2123385204176884e-05\n",
      "Epoch 40000 loss 3.4499701087042922e-06\n",
      "Epoch 50000 loss 1.8503021692595212e-06\n",
      "Epoch 60000 loss 1.4950591094020638e-06\n",
      "Epoch 70000 loss 1.0259531109113595e-06\n",
      "Epoch 80000 loss 1.0278015452058753e-06\n",
      "Epoch 90000 loss 1.3784808743366739e-06\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "        \n",
    "    session.run(tf.global_variables_initializer())\n",
    "        \n",
    "    for epoch in range(EPOCHS):\n",
    "        _  = session.run(adam_optimize,feed_dict={X:x,Y:y})\n",
    "            \n",
    "        if epoch % PRINT_EVERY == 0:\n",
    "            batch_loss = session.run(loss,feed_dict={X:x,Y:y})\n",
    "            print(\"Epoch {} loss {}\".format(epoch,batch_loss))\n",
    "                \n",
    "    \n",
    "    test_predictions,test_loss = session.run([nalu2_output,loss],feed_dict={X:test_x,Y:test_y})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.87564486],\n",
       "       [ 1.03877783],\n",
       "       [ 1.0942715 ],\n",
       "       ..., \n",
       "       [ 1.05664367],\n",
       "       [ 1.12138441],\n",
       "       [ 0.95663732]])"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.76566935],\n",
       "       [ 2.61818552],\n",
       "       [ 3.10014892],\n",
       "       ..., \n",
       "       [ 2.94523335],\n",
       "       [ 2.95017576],\n",
       "       [ 2.35278463]], dtype=float32)"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.6808782"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Conclusions/comments\n",
    "* NALU is very sensitive to parameters initialization, xavier initialization doesnt seemed to help\n",
    "* In all experiments, using 2 units in NALU seem to make it fail, succesfull tests wre using at least 5 units\n",
    "* addition and substraction seems easy to achieve ,multiplication not so much :(\n",
    "* What will happen if batch-norm is added to NALU layers? \n",
    "* Division seems to always predict 2 times the real value. Why ? :("
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}