LSTM Album Name Generator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D7tqLMoKF6uq"
   },
   "source": [
    "# Album Name Generator\n",
    "\n",
    "Training a LSTM character model on a music album dataset parsed from Discogs (https://www.discogs.com). Feel free to contact me to obtain the datset.  \n",
    "The dataset contains images and names of music albums. The focus of this project is building a generative model able to create new album names, so we will be ignoring the album artwork.  \n",
    "The LSTM is trained to predict the next character on a text sequence given past characters.  \n",
    "After being trained, the LSTM is able to generate new album names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "import numpy as np\n",
    "np.random.seed(1)\n",
    "import string\n",
    "import tensorflow as tf\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare data:\n",
    "Read album titles from the image filenames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def read_text(directory):\n",
    "    # Get album cover filenames and remove the .jpg ending\n",
    "    album_titles = [f[:-4].lower() for f in listdir(directory) if isfile(join(directory, f))]\n",
    "    # Shuffle cover names to avoid any bias, since they are initially sorted in alphabetical order\n",
    "    np.random.shuffle(album_titles)\n",
    "    # Add start token > and stop token < before and after each album name\n",
    "    # It would be possible to use a single token for our case, but using both is good practice\n",
    "    return '>' + '<>'.join(album_titles) + '<'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data size 200153 characters\n",
      ">class clown<>lowrell<>you're the one for me<>live on the garden bowl lanes: july 9, 1999<>stockholm disco ep<>people of the sun ep<>chapter 1<>wrecked<>darklands<>book of the bad (volume one)<>forgiveness rock record<>second edition<>combat rock<>changing crisis ep<>shut down<>desolation angels<>mauve<>ashes are burning<>still life (american concert 1981)<>quark, strangeness and charm<>true colours<>vengeance<>extreme aggression<>incense and peppermints<>big city music<>dirty cash<>god of the s\n"
     ]
    }
   ],
   "source": [
    "# Dataset directory\n",
    "directory = '/path_to_files' # Feel free to contact me for the dataset\n",
    "text = read_text(directory)\n",
    "print('Data size %d characters' % len(text))\n",
    "# Print first 500 characters.\n",
    "# Token > indicates the start of an album name, while token < indicates the end\n",
    "print(text[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ga2CYACE-ghb"
   },
   "source": [
    "We select a small validation set to evaluate our objective function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 6184,
     "status": "ok",
     "timestamp": 1445965583138,
     "user": {
      "color": "#1FA15D",
      "displayName": "Vincent Vanhoucke",
      "isAnonymous": false,
      "isMe": true,
      "permissionId": "05076109866853157986",
      "photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
      "sessionId": "6f6f07b359200c46",
      "userId": "102167687554210253930"
     },
     "user_tz": 420
    },
    "id": "w-oBpfFG-j43",
    "outputId": "bdb96002-d021-4379-f6de-a977924f0d02"
   },
   "outputs": [],
   "source": [
    "def train_validation_split(VALID_SIZE):\n",
    "    '''Split text into a training and validation set'''\n",
    "    valid_text = text[:VALID_SIZE]\n",
    "    train_text = text[VALID_SIZE:]\n",
    "    train_size = len(train_text)\n",
    "    return train_text, valid_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "VALID_SIZE = 1009\n",
    "train_text, valid_text = train_validation_split(VALID_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation set:\n",
      " >class clown<>lowrell<>you're the one for me<>live on the garden bowl lanes: july 9, 1999<>stockholm disco ep<>people of the sun ep<>chapter 1<>wrecked<>darklands<>book of the bad (volume one)<>forgiveness rock record<>second edition<>combat rock<>changing crisis ep<>shut down<>desolation angels<>mauve<>ashes are burning<>still life (american concert 1981)<>quark, strangeness and charm<>true colours<>vengeance<>extreme aggression<>incense and peppermints<>big city music<>dirty cash<>god of the serengeti<>the best of both worlds<>pink moon<>bug<>bis zum bitteren ende live!<>desire<>no fuel left for the pilgrims<>z<>impact<>the wörld thät sümmer<>bora bora<>badmotorfinger<>a chorus of storytellers<>makesaracket<>floating coffin<>live killers<>keep on jumpin'<>liquid liquid<>loop-finding-jazz-records<>empty space meditation<>it's gonna be alright (help is on the way)<>pod<>reach out<>new sensations<>mechanical resonance<>an awesome wave<>the many facets of roger<>world destruction<>question 10<\n"
     ]
    }
   ],
   "source": [
    "# Validation set\n",
    "print('Validation set:\\n %s' % valid_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Zdw6i4F8glpp"
   },
   "source": [
    "Utility functions to map characters to vocabulary IDs and back:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6276,
     "status": "ok",
     "timestamp": 1445965583249,
     "user": {
      "color": "#1FA15D",
      "displayName": "Vincent Vanhoucke",
      "isAnonymous": false,
      "isMe": true,
      "permissionId": "05076109866853157986",
      "photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
      "sessionId": "6f6f07b359200c46",
      "userId": "102167687554210253930"
     },
     "user_tz": 420
    },
    "id": "gAL1EECXeZsD",
    "outputId": "88fc9032-feb9-45ff-a9a0-a26759cc1f2e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test mappings:\n",
      "1 26 0 0 27 28\n",
      "a z   > <\n"
     ]
    }
   ],
   "source": [
    "VOCABULARY_SIZE = len(string.ascii_lowercase) + 3 # [a-z] + ' ' + '>' + '<'\n",
    "FIRST_LETTER = ord(string.ascii_lowercase[0])\n",
    "START_TOKEN = 27 # Use 27 for start token\n",
    "STOP_TOKEN = 28 # Use 28 for stop token\n",
    "\n",
    "def char2id(char):\n",
    "    if char in string.ascii_lowercase:\n",
    "        return ord(char) - FIRST_LETTER + 1 # Use abecedary position\n",
    "    elif char == ' ':\n",
    "        return 0 # Use 0 for space\n",
    "    elif char == '>':\n",
    "        return START_TOKEN\n",
    "    elif char == '<':\n",
    "        return STOP_TOKEN\n",
    "    else:\n",
    "        # Return 0 for unexpected character, treat them as spaces\n",
    "        return 0\n",
    "\n",
    "def id2char(dictid):\n",
    "    if (dictid > 0) & (dictid < 27):\n",
    "        return chr(dictid + FIRST_LETTER - 1)\n",
    "    elif dictid == START_TOKEN:\n",
    "        return '>'\n",
    "    elif dictid == STOP_TOKEN:\n",
    "        return '<'\n",
    "    else:\n",
    "        return ' '\n",
    "\n",
    "print('Test mappings:')\n",
    "print(char2id('a'), char2id('z'), char2id(' '), char2id('ï'), char2id('>'), char2id('<'))\n",
    "print(id2char(1), id2char(26), id2char(0), id2char(27), id2char(28))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lFwoyygOmWsL"
   },
   "source": [
    "Generating training batches:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6473,
     "status": "ok",
     "timestamp": 1445965583467,
     "user": {
      "color": "#1FA15D",
      "displayName": "Vincent Vanhoucke",
      "isAnonymous": false,
      "isMe": true,
      "permissionId": "05076109866853157986",
      "photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
      "sessionId": "6f6f07b359200c46",
      "userId": "102167687554210253930"
     },
     "user_tz": 420
    },
    "id": "d9wMtjy5hCj9",
    "outputId": "3dd79c80-454a-4be0-8b71-4a4a357b3367"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['>exile<>on the r', 'mash<>speaking i', ' e p <>cream<>su', 'sey   oracle<>th', 'lyps trak ii<>dr', ' rock steady cre', 'ock konducta  pa', ' thrones and dom', 'obal underground', '<>call for escap', 'tory <>dreamin  ', 'broke my heart s', 'get enough<>emil', 'aging inside me<', 'n of xymox<>dest', 'rs  me   you <>y', 'ht  still<>for l', 'fuck the kids<>t', ' version of me <', 'th it<>under the', ' remix <>laid ba', 'ain<>cloudwalkin', 'itz in moscow<>f', 'rs of the third ', '>various positio', 'one of ya left<>', 'n a dream<>restr', '      <>masters ', '>updating the ex', '<>beautiful frea', 'arkin <>he s com', 'we teach mistake', 'is<>the chemistr', ' on fire<>the op', 'ner   beinhart <', 's <>live at carn', 'ark<>love devoti', ' stories<>aux ar', 'ex pistols<>phan', '<>the grand illu', 'not legalize it ', 'tion picture sou', 'orth<>blood on t', 'on          <>li', ' possible<>you g', 'ssible musics<>i', '>new year s day<', 'h boys today <>u', 'my phone<>there ', ' in my mind<>kil', 'eels<>prince fat', 'h tell me yours<', 'people<>a realit', ' and the beast<>', ' crossing<>lovin', 'time is tight<>p', ' nothing gained ', ' the weak<>bmb  ', 'ten here<>doin  ', '<>king   queen<>', '<>the lick<>ooch', 'c part iv <>what', 'county jail<>the', 'ife in the bush ']\n",
      "['road<>amenity<>t', 'in tongues<>ocea', 'ung tongs<>sick ', 'he soul cages<>t', 'ream police<>thx', 'ew<>pipecarrier ', 'art   <>l a   li', 'minions<>here it', 'd      brazil<>r', 'pe route<>sweet ', ' man live    <>i', 'so i busted your', 'ly s d evolution', '<>fleetwood mac<', 'troy all astro m', 'you really got m', 'lovers  not figh', 'tunes splits the', '<>led zeppelin i', 'e bushes under t', 'ack<>business ca', 'ng<>i am<>took m', 'faith divides us', ' kind  original ', 'ons<>loggins and', '>the return of t', 'riction<>uptight', ' of metal<>kaya<', 'xisting systems<', 'ak<>metal black<', 'ming<>journey in', 'es<>blue kentuck', 'ry of common lif', 'pposite of decem', '<>please hammer ', 'negie hall<>    ', 'ion surrender<>a', 'rmes et c tera<>', 'ntoms<>industry ', 'usion<>chaka<>th', ' <>chilling  thr', 'undtrack <>new h', 'the tracks<>scen', 'ive at birdland<', 'got me burnin <>', 'i against i<>the', '<>some candy tal', 'ubiquity<>two mi', ' goes rhymin  si', 'lling me softly<', 'tty vs mungo s h', '<>sahara effects', 'ty tour<>enforce', '> <>hand springs', 'n  every minute ', 'pay for me<>volu', ' e p <>flying mi', '   don t give wa', ' the best that i', '>the avalanche<>', 'hie wally<>ok ne', 't time is love  ', 'e dangermen sess', ' of ghosts<>cygn']\n",
      "['>c']\n",
      "['cl']\n"
     ]
    }
   ],
   "source": [
    "BATCH_SIZE = 64\n",
    "NUM_UNROLLINGS = 15\n",
    "\n",
    "class BatchGenerator(object):\n",
    "    def __init__(self, text, BATCH_SIZE, NUM_UNROLLINGS):\n",
    "        self._text = text\n",
    "        self._text_size = len(text)\n",
    "        self._BATCH_SIZE = BATCH_SIZE\n",
    "        self._NUM_UNROLLINGS = NUM_UNROLLINGS\n",
    "        segment = self._text_size // BATCH_SIZE\n",
    "        self._cursor = [offset * segment for offset in range(BATCH_SIZE)]\n",
    "        self._last_batch = self._next_batch()\n",
    "  \n",
    "    def _next_batch(self):\n",
    "        \"\"\"Generate a single batch from the current cursor position in the data.\"\"\"\n",
    "        batch = np.zeros(shape=(self._BATCH_SIZE, VOCABULARY_SIZE), dtype=np.float)\n",
    "        for b in range(self._BATCH_SIZE):\n",
    "            batch[b, char2id(self._text[self._cursor[b]])] = 1.0\n",
    "            self._cursor[b] = (self._cursor[b] + 1) % self._text_size\n",
    "        return batch\n",
    "  \n",
    "    def next(self):\n",
    "        \"\"\"Generate the next array of batches from the data. The array consists of\n",
    "        the last batch of the previous array, followed by NUM_UNROLLINGS new ones.\n",
    "        \"\"\"\n",
    "        batches = [self._last_batch]\n",
    "        for step in range(self._NUM_UNROLLINGS):\n",
    "            batches.append(self._next_batch())\n",
    "        self._last_batch = batches[-1]\n",
    "        return batches\n",
    "\n",
    "def characters(probabilities):\n",
    "    \"\"\"Turn a 1-hot encoding or a probability distribution over the possible\n",
    "    characters back into its (most likely) character representation.\"\"\"\n",
    "    return [id2char(c) for c in np.argmax(probabilities, 1)]\n",
    "\n",
    "def batches2string(batches):\n",
    "    \"\"\"Convert a sequence of batches back into their (most likely) string\n",
    "    representation.\"\"\"\n",
    "    s = [''] * batches[0].shape[0]\n",
    "    for b in batches:\n",
    "        s = [''.join(x) for x in zip(s, characters(b))]\n",
    "    return s\n",
    "\n",
    "train_batches = BatchGenerator(train_text, BATCH_SIZE, NUM_UNROLLINGS)\n",
    "valid_batches = BatchGenerator(valid_text, 1, 1)\n",
    "\n",
    "print(batches2string(train_batches.next()))\n",
    "print(batches2string(train_batches.next()))\n",
    "print(batches2string(valid_batches.next()))\n",
    "print(batches2string(valid_batches.next()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utility functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "KyVd8FxT5QBc"
   },
   "outputs": [],
   "source": [
    "def logprob(predictions, labels):\n",
    "    \"\"\"Log-probability of the true labels in a predicted batch.\"\"\"\n",
    "    # To avoid numerical stability issues\n",
    "    predictions[predictions < 1e-10] = 1e-10\n",
    "    return np.sum(np.multiply(labels, -np.log(predictions))) / labels.shape[0]\n",
    "\n",
    "def sample_distribution(distribution):\n",
    "    \"\"\"Sample one element from a distribution assumed to be an array of normalized\n",
    "    probabilities.\n",
    "    \"\"\"\n",
    "    r = random.uniform(0, 1)\n",
    "    s = 0\n",
    "    for i in range(len(distribution)):\n",
    "        s += distribution[i]\n",
    "        if s >= r:\n",
    "            return i\n",
    "    return len(distribution) - 1\n",
    "\n",
    "def sample(prediction):\n",
    "    \"\"\"Turn a (column) prediction into 1-hot encoded samples.\"\"\"\n",
    "    p = np.zeros(shape=[1, VOCABULARY_SIZE], dtype=np.float)\n",
    "    p[0, sample_distribution(prediction[0])] = 1.0\n",
    "    return p\n",
    "\n",
    "def random_distribution():\n",
    "    \"\"\"Generate a random column of probabilities.\"\"\"\n",
    "    b = np.random.uniform(0.0, 1.0, size=[1, VOCABULARY_SIZE])\n",
    "    return b/np.sum(b, 1)[:,None]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "K8f67YXaDr4C"
   },
   "source": [
    "### LSTM Model Definition:\n",
    "We are going to code an LSTM in TensorFlow from scratch. An alternative would be using tf.contrib high level implementation instead. (https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/BasicLSTMCell)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"files/basic_lstm.png\">\n",
    "Image from: https://colah.github.io/posts/2015-08-Understanding-LSTMs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "Q5rxZK6RDuGe"
   },
   "outputs": [],
   "source": [
    "num_nodes = 64\n",
    "\n",
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "  \n",
    "    # Parameters:\n",
    "    # Input gate: input, previous output, and bias\n",
    "    ix = tf.Variable(tf.truncated_normal([VOCABULARY_SIZE, num_nodes], -0.1, 0.1))\n",
    "    im = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1))\n",
    "    ib = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "    # Forget gate: input, previous output, and bias\n",
    "    fx = tf.Variable(tf.truncated_normal([VOCABULARY_SIZE, num_nodes], -0.1, 0.1))\n",
    "    fm = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1))\n",
    "    fb = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "    # Memory cell: input, state and bias                             \n",
    "    cx = tf.Variable(tf.truncated_normal([VOCABULARY_SIZE, num_nodes], -0.1, 0.1))\n",
    "    cm = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1))\n",
    "    cb = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "    # Output gate: input, previous output, and bias\n",
    "    ox = tf.Variable(tf.truncated_normal([VOCABULARY_SIZE, num_nodes], -0.1, 0.1))\n",
    "    om = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1))\n",
    "    ob = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "    # Variables saving state across unrollings, note trainable=False,\n",
    "    # since we do not want to backpropagate through them\n",
    "    saved_output = tf.Variable(tf.zeros([BATCH_SIZE, num_nodes]), trainable=False)\n",
    "    saved_state = tf.Variable(tf.zeros([BATCH_SIZE, num_nodes]), trainable=False)\n",
    "    # Classifier weights and biases\n",
    "    w = tf.Variable(tf.truncated_normal([num_nodes, VOCABULARY_SIZE], -0.1, 0.1))\n",
    "    b = tf.Variable(tf.zeros([VOCABULARY_SIZE]))\n",
    "  \n",
    "    # Definition of the cell computation\n",
    "    def lstm_cell(i, o, state):\n",
    "        \"\"\"Create a LSTM cell. See e.g.: http://arxiv.org/pdf/1402.1128v1.pdf\n",
    "        For simplicity, we omit the various connections between the\n",
    "        previous state and the gates.\"\"\"\n",
    "        input_gate = tf.sigmoid(tf.matmul(i, ix) + tf.matmul(o, im) + ib)\n",
    "        forget_gate = tf.sigmoid(tf.matmul(i, fx) + tf.matmul(o, fm) + fb)\n",
    "        update = tf.matmul(i, cx) + tf.matmul(o, cm) + cb\n",
    "        state = forget_gate * state + input_gate * tf.tanh(update)\n",
    "        output_gate = tf.sigmoid(tf.matmul(i, ox) + tf.matmul(o, om) + ob)\n",
    "        return output_gate * tf.tanh(state), state\n",
    "\n",
    "    # Input data\n",
    "    train_data = []\n",
    "    for _ in range(NUM_UNROLLINGS + 1): # One time step ahead for the last target\n",
    "        train_data.append(\n",
    "            tf.placeholder(tf.float32, shape=[BATCH_SIZE,VOCABULARY_SIZE]))\n",
    "    train_inputs = train_data[:NUM_UNROLLINGS]\n",
    "    train_labels = train_data[1:]  # Labels are inputs shifted by one time step\n",
    "\n",
    "    # Unrolled LSTM loop\n",
    "    outputs = []\n",
    "    output = saved_output\n",
    "    state = saved_state\n",
    "    for i in train_inputs:\n",
    "        output, state = lstm_cell(i, output, state)\n",
    "        outputs.append(output)\n",
    "\n",
    "    # State saving across unrollings\n",
    "    with tf.control_dependencies([saved_output.assign(output),\n",
    "                                saved_state.assign(state)]):\n",
    "        # Classifier\n",
    "        logits = tf.nn.xw_plus_b(tf.concat(outputs, 0), w, b)\n",
    "        # softmax_cross_entropy_with_logits is more efficient than applying a softmax, then compute cross_entropy\n",
    "        loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
    "                              labels=tf.concat(train_labels, 0), logits=logits))\n",
    "\n",
    "    # Optimizer\n",
    "    # Keep track of the training step\n",
    "    global_step = tf.Variable(0)\n",
    "    # Use SGD with exponential decay on the learning rate\n",
    "    learning_rate = tf.train.exponential_decay(10.0, global_step, 5000, 0.1, staircase=True)\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "    # Get gradients\n",
    "    gradients, v = zip(*optimizer.compute_gradients(loss))\n",
    "    # Clip gradients\n",
    "    gradients, _ = tf.clip_by_global_norm(gradients, 1.25)\n",
    "    # Apply gradients\n",
    "    optimizer = optimizer.apply_gradients(zip(gradients, v), global_step=global_step)\n",
    "\n",
    "    # Predictions\n",
    "    train_prediction = tf.nn.softmax(logits)\n",
    "  \n",
    "    # Sampling and validation evaluation: batch 1, no unrolling.\n",
    "    sample_input = tf.placeholder(tf.float32, shape=[1, VOCABULARY_SIZE])\n",
    "    saved_sample_output = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "    saved_sample_state = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "    reset_sample_state = tf.group(saved_sample_output.assign(tf.zeros([1, num_nodes])),\n",
    "                                  saved_sample_state.assign(tf.zeros([1, num_nodes])))\n",
    "    sample_output, sample_state = lstm_cell(sample_input, saved_sample_output, saved_sample_state)\n",
    "    with tf.control_dependencies([saved_sample_output.assign(sample_output),\n",
    "                                  saved_sample_state.assign(sample_state)]):\n",
    "        sample_prediction = tf.nn.softmax(tf.nn.xw_plus_b(sample_output, w, b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the LSTM:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initialized\n",
      "Average loss at step 0: 3.369880 learning rate: 10.000000\n",
      "Minibatch perplexity: 29.08\n",
      "================================================================================\n",
      " l j rudtecd>qejzbsf aopsfbuelcoj orblaopah a<tw<cmeihihy mswxnxtdp>tmba ewgcfol\n",
      "ncrdpjabomordnzuz<idtrnoerbdyxhs e <adwsn eswo hhmhh xert s>gn  nebatits feekfoe\n",
      "fu>q eaa>rdiyghhzecyotghxja tbts s tunaiislxgbk pndppk uijmqbv>nqqcxwq qjzdhyj>a\n",
      "x bpwxjenq>vvbe<qz womih hsojri>kijfmfw ehtbceltfqmto peel kvduap o ck ojfctamse\n",
      "ky oqdmnsecnagv kttdhryebrdlsne>kbu <wegpt >repshrcidzd>faqiei>pn<fsom> qeszgq>n\n",
      "================================================================================\n",
      "Validation set perplexity: 23.58\n",
      "Average loss at step 100: 2.641324 learning rate: 10.000000\n",
      "Minibatch perplexity: 10.84\n",
      "Validation set perplexity: 10.54\n",
      "Average loss at step 200: 2.308523 learning rate: 10.000000\n",
      "Minibatch perplexity: 9.37\n",
      "Validation set perplexity: 9.07\n",
      "Average loss at step 300: 2.197819 learning rate: 10.000000\n",
      "Minibatch perplexity: 9.48\n",
      "Validation set perplexity: 8.27\n",
      "Average loss at step 400: 2.130578 learning rate: 10.000000\n",
      "Minibatch perplexity: 7.67\n",
      "Validation set perplexity: 8.04\n",
      "Average loss at step 500: 2.071974 learning rate: 10.000000\n",
      "Minibatch perplexity: 7.23\n",
      "Validation set perplexity: 7.79\n",
      "Average loss at step 600: 2.035573 learning rate: 10.000000\n",
      "Minibatch perplexity: 7.81\n",
      "Validation set perplexity: 7.64\n",
      "Average loss at step 700: 1.995211 learning rate: 10.000000\n",
      "Minibatch perplexity: 7.13\n",
      "Validation set perplexity: 7.43\n",
      "Average loss at step 800: 1.966857 learning rate: 10.000000\n",
      "Minibatch perplexity: 7.04\n",
      "Validation set perplexity: 7.26\n",
      "Average loss at step 900: 1.939052 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.59\n",
      "Validation set perplexity: 7.23\n",
      "Average loss at step 1000: 1.924375 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.52\n",
      "================================================================================\n",
      " brei bang<>ufferran<>gord<><>on the gang  love  men s pari   a l the fffornen<>\n",
      "darky<>huxas<>undrabodfnar wayla seergroye<>wellomy<>for t jones<>whit pang vol \n",
      "daige<>volo go<>aliverreaale<>heler<>gotel<>the music  <>doning gree<>aca lobing\n",
      "phare dellgel sting  versone<>foromy  a vearn<>petflep<>shagine<>treew<>roct s i\n",
      "red  fraz dack now good the greatem<>theme sainity<>sspips<>vex s   endecioms<>b\n",
      "================================================================================\n",
      "Validation set perplexity: 7.10\n",
      "Average loss at step 1100: 1.900939 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.55\n",
      "Validation set perplexity: 6.88\n",
      "Average loss at step 1200: 1.884600 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.70\n",
      "Validation set perplexity: 6.90\n",
      "Average loss at step 1300: 1.867788 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.95\n",
      "Validation set perplexity: 6.80\n",
      "Average loss at step 1400: 1.854958 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.06\n",
      "Validation set perplexity: 6.88\n",
      "Average loss at step 1500: 1.845574 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.97\n",
      "Validation set perplexity: 6.66\n",
      "Average loss at step 1600: 1.833355 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.16\n",
      "Validation set perplexity: 6.67\n",
      "Average loss at step 1700: 1.828580 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.61\n",
      "Validation set perplexity: 6.55\n",
      "Average loss at step 1800: 1.812317 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.15\n",
      "Validation set perplexity: 6.56\n",
      "Average loss at step 1900: 1.815644 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.87\n",
      "Validation set perplexity: 6.49\n",
      "Average loss at step 2000: 1.796123 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.52\n",
      "================================================================================\n",
      ">watrody no chi<>strotin  for way beat   frenw of new weargreadd<>aby galf chree\n",
      "psass<>chaigs rike tescire<>bulaling daws<>kikfhead crangus <>the patas vanoon<>\n",
      "ges from the sweat of the mash<>   reaming sweat why<>shates   <>signt shasble<>\n",
      "holimed floodind<>a dust   <>crazy atroj mounty<>the pernin  sambind<>nother den\n",
      " it s off<>the brythm in chagan<>the darkstory lius me  latinast marklys of amer\n",
      "================================================================================\n",
      "Validation set perplexity: 6.65\n",
      "Average loss at step 2100: 1.801839 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.56\n",
      "Validation set perplexity: 6.52\n",
      "Average loss at step 2200: 1.784561 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.64\n",
      "Validation set perplexity: 6.50\n",
      "Average loss at step 2300: 1.784042 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.94\n",
      "Validation set perplexity: 6.38\n",
      "Average loss at step 2400: 1.771296 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.34\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 2500: 1.774008 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.10\n",
      "Validation set perplexity: 6.47\n",
      "Average loss at step 2600: 1.761543 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.84\n",
      "Validation set perplexity: 6.47\n",
      "Average loss at step 2700: 1.763231 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.91\n",
      "Validation set perplexity: 6.46\n",
      "Average loss at step 2800: 1.751689 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.19\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 2900: 1.756483 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.60\n",
      "Validation set perplexity: 6.51\n",
      "Average loss at step 3000: 1.743924 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.60\n",
      "================================================================================\n",
      "ents you unowneks<>songs of as changage<>socascit<>bainchy beting<>lonetmire<>ce\n",
      "onnest have we dessa<>dister off r delicor   kuss<>deeperdedy net<>arriad<>sule \n",
      "in enteven<>ss b religion<>oromed dies<>backix<>alection  poh nen whitle to the \n",
      "vol<>takes of super <>wond one <>special<>oodion<>vis tidg one mirts<>prezoults<\n",
      "halal<>get<>jocen two<>what the wollib ibmissent<>our your the out<>intomed comp\n",
      "================================================================================\n",
      "Validation set perplexity: 6.46\n",
      "Average loss at step 3100: 1.743757 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.63\n",
      "Validation set perplexity: 6.42\n",
      "Average loss at step 3200: 1.738333 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.82\n",
      "Validation set perplexity: 6.43\n",
      "Average loss at step 3300: 1.742663 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.47\n",
      "Validation set perplexity: 6.47\n",
      "Average loss at step 3400: 1.732689 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.42\n",
      "Validation set perplexity: 6.48\n",
      "Average loss at step 3500: 1.732121 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.47\n",
      "Validation set perplexity: 6.46\n",
      "Average loss at step 3600: 1.724369 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.51\n",
      "Validation set perplexity: 6.43\n",
      "Average loss at step 3700: 1.727212 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.57\n",
      "Validation set perplexity: 6.43\n",
      "Average loss at step 3800: 1.720301 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.93\n",
      "Validation set perplexity: 6.48\n",
      "Average loss at step 3900: 1.720450 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.03\n",
      "Validation set perplexity: 6.37\n",
      "Average loss at step 4000: 1.715782 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.52\n",
      "================================================================================\n",
      "fly fon pollen<>rocker dufuco pattre<>face thanne bound  t x <>diste two<>empern\n",
      "ffum<>samber is of the stile<>ckt who    makelto   mis<>trina x it firm thises t\n",
      ">yistol<>music jackion<>winfay e p <><>stliken   thing         <>colera for<>fre\n",
      "it frasianas<>the brother<>love stand<>infolyer inconny<>in cond rock tos<>amber\n",
      "s from through in the halling<>at upo spitss   bac<>through chenium<>in the ligh\n",
      "================================================================================\n",
      "Validation set perplexity: 6.50\n",
      "Average loss at step 4100: 1.712107 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.93\n",
      "Validation set perplexity: 6.50\n",
      "Average loss at step 4200: 1.714795 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.24\n",
      "Validation set perplexity: 6.50\n",
      "Average loss at step 4300: 1.705139 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.75\n",
      "Validation set perplexity: 6.47\n",
      "Average loss at step 4400: 1.712735 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.83\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation set perplexity: 6.47\n",
      "Average loss at step 4500: 1.695737 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.55\n",
      "Validation set perplexity: 6.51\n",
      "Average loss at step 4600: 1.712076 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.86\n",
      "Validation set perplexity: 6.35\n",
      "Average loss at step 4700: 1.692828 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.15\n",
      "Validation set perplexity: 6.55\n",
      "Average loss at step 4800: 1.708474 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.83\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 4900: 1.689618 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.21\n",
      "Validation set perplexity: 6.56\n",
      "Average loss at step 5000: 1.701233 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.80\n",
      "================================================================================\n",
      "y for the dop<>introue creet pianka insid  <>feantal so silents<>thicty wig i<>s\n",
      "in<>secordions<> rock sowne<>crysteric  limition    <>e inaverfax alow bubned ag\n",
      "zer remix <>sleesmore<>ame one   siediant<>split byrd remixe<>music from poster<\n",
      "g <>hangy cart c prussenti eir<>handry<>live beach me  veltional to albuched son\n",
      "ness<>dappage<>albagolel   remixes <>godnert      <>carp   back m curtuac johnte\n",
      "================================================================================\n",
      "Validation set perplexity: 6.42\n",
      "Average loss at step 5100: 1.664884 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.88\n",
      "Validation set perplexity: 6.27\n",
      "Average loss at step 5200: 1.641270 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.40\n",
      "Validation set perplexity: 6.25\n",
      "Average loss at step 5300: 1.640427 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.16\n",
      "Validation set perplexity: 6.27\n",
      "Average loss at step 5400: 1.636230 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.77\n",
      "Validation set perplexity: 6.29\n",
      "Average loss at step 5500: 1.635716 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.34\n",
      "Validation set perplexity: 6.26\n",
      "Average loss at step 5600: 1.633636 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.09\n",
      "Validation set perplexity: 6.27\n",
      "Average loss at step 5700: 1.632224 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.21\n",
      "Validation set perplexity: 6.28\n",
      "Average loss at step 5800: 1.630497 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.38\n",
      "Validation set perplexity: 6.28\n",
      "Average loss at step 5900: 1.628706 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.57\n",
      "Validation set perplexity: 6.29\n",
      "Average loss at step 6000: 1.631300 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.78\n",
      "================================================================================\n",
      "fire       s akneath underground<>trurh screechrillam life<>  beat <>live skinso\n",
      ">mustemetch<>now in away<>from themohnanvection<>pentize herus song mitas break<\n",
      "under comp<>more at life<>modestrum<>turn<>the future is alb<>tonight to concera\n",
      "<>the side of hows in the baler<>decion session seast<>twen s piston <>bbd in a \n",
      "harbs<>j shake heading the usacheday do<>the more state<>        original sound \n",
      "================================================================================\n",
      "Validation set perplexity: 6.30\n",
      "Average loss at step 6100: 1.625059 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.87\n",
      "Validation set perplexity: 6.31\n",
      "Average loss at step 6200: 1.633801 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.11\n",
      "Validation set perplexity: 6.30\n",
      "Average loss at step 6300: 1.626005 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.20\n",
      "Validation set perplexity: 6.31\n",
      "Average loss at step 6400: 1.628295 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.82\n",
      "Validation set perplexity: 6.29\n",
      "Average loss at step 6500: 1.623728 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.16\n",
      "Validation set perplexity: 6.31\n",
      "Average loss at step 6600: 1.628759 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.44\n",
      "Validation set perplexity: 6.30\n",
      "Average loss at step 6700: 1.622586 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.82\n",
      "Validation set perplexity: 6.34\n",
      "Average loss at step 6800: 1.623245 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.75\n",
      "Validation set perplexity: 6.34\n",
      "Average loss at step 6900: 1.625974 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.84\n",
      "Validation set perplexity: 6.36\n",
      "Average loss at step 7000: 1.618575 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.14\n",
      "================================================================================\n",
      " king album <>sen up the cullaw<>you annivers project<>a nock ed raine<>the kird\n",
      ">best zo bablloke<>sound of free<>deep randher the golement<>ctrengs to thiem<>a\n",
      "fire    the your      wank on eys<>surperpnowia<>asterdatate presents<>ken magn<\n",
      "ky<>i   ve gallt<>oar best gifffr<>geremon  lyve<>regamon tryckeler<>all the my \n",
      "pet<>too lone moverstern<>the best ot s<>musicie music<>satfum hisider<>the worl\n",
      "================================================================================\n",
      "Validation set perplexity: 6.37\n",
      "Average loss at step 7100: 1.627156 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.91\n",
      "Validation set perplexity: 6.35\n",
      "Average loss at step 7200: 1.617434 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.89\n",
      "Validation set perplexity: 6.35\n",
      "Average loss at step 7300: 1.628506 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.57\n",
      "Validation set perplexity: 6.30\n",
      "Average loss at step 7400: 1.614926 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.51\n",
      "Validation set perplexity: 6.38\n",
      "Average loss at step 7500: 1.628486 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.99\n",
      "Validation set perplexity: 6.34\n",
      "Average loss at step 7600: 1.615525 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.91\n",
      "Validation set perplexity: 6.36\n",
      "Average loss at step 7700: 1.626060 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.95\n",
      "Validation set perplexity: 6.33\n",
      "Average loss at step 7800: 1.616701 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.64\n",
      "Validation set perplexity: 6.36\n",
      "Average loss at step 7900: 1.621500 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.10\n",
      "Validation set perplexity: 6.32\n",
      "Average loss at step 8000: 1.614203 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.80\n",
      "================================================================================\n",
      "fire<>reyorowarn polaft  <>the crowlty rog<>the angel for bloem<>inll dancers  j\n",
      "fire<>what things part s eap<>the basement stop<>the ware dance   sitoris lone s\n",
      "boomiin airicus soul von  limited infelloxor<>crie is bards<>home no stith<>lina\n",
      "phosem  e <>intost a can the precade<>song robot elivio<>a green boufe story out\n",
      "viny hardcore<>avister  ep<>classics ep<>mas hallsweele<>buder<>green a spast  i\n",
      "================================================================================\n",
      "Validation set perplexity: 6.37\n",
      "Average loss at step 8100: 1.620881 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.30\n",
      "Validation set perplexity: 6.35\n",
      "Average loss at step 8200: 1.614797 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.92\n",
      "Validation set perplexity: 6.38\n",
      "Average loss at step 8300: 1.619609 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.97\n",
      "Validation set perplexity: 6.33\n",
      "Average loss at step 8400: 1.613036 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.09\n",
      "Validation set perplexity: 6.34\n",
      "Average loss at step 8500: 1.620552 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.20\n",
      "Validation set perplexity: 6.37\n",
      "Average loss at step 8600: 1.612969 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.10\n",
      "Validation set perplexity: 6.37\n",
      "Average loss at step 8700: 1.614250 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.58\n",
      "Validation set perplexity: 6.35\n",
      "Average loss at step 8800: 1.613208 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.94\n",
      "Validation set perplexity: 6.36\n",
      "Average loss at step 8900: 1.619730 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.30\n",
      "Validation set perplexity: 6.38\n",
      "Average loss at step 9000: 1.612887 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.68\n",
      "================================================================================\n",
      "jeem<>rock under beach<>the flook bass of last boy v <>the boys in traker<>my pe\n",
      "urs ep<>thing it volume   remixes <>la you of dreem<>now a new  original movie b\n",
      "greats<>a choigelion<>cown<>fullio variated bark the da come teferam<>droabled <\n",
      "jear<>the something how<>ctrisoles vol  <>bloodrath<>witk pehenver<>pyunts of de\n",
      "vijuce gold set<>boys roc<>hell  ambore me<>tames sumels<>dance mariey<>the lie \n",
      "================================================================================\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation set perplexity: 6.40\n",
      "Average loss at step 9100: 1.616254 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.32\n",
      "Validation set perplexity: 6.37\n",
      "Average loss at step 9200: 1.610675 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.35\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 9300: 1.615847 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.02\n",
      "Validation set perplexity: 6.34\n",
      "Average loss at step 9400: 1.611845 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.69\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 9500: 1.613635 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.88\n",
      "Validation set perplexity: 6.38\n",
      "Average loss at step 9600: 1.610346 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.46\n",
      "Validation set perplexity: 6.43\n",
      "Average loss at step 9700: 1.611068 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.39\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 9800: 1.614074 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.43\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 9900: 1.607634 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.25\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 10000: 1.616690 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.84\n",
      "================================================================================\n",
      "as<>cocket southern goov<>the tougna<>rocrus<>harding<>rothlife<>eppernies  thei\n",
      "vie<>kig crossing beat   tom original breaved makes in i dun to rey<>stelies   t\n",
      "gold storia<>do the beat<>autions of the triffersions<>true   the ground of the \n",
      " lust volume one<>pirct found<>raituire<>  pres personamted of the sing in colle\n",
      "radinauni<>dikiii<>still  halles<>state dip<>awers light edears   disder ongerso\n",
      "================================================================================\n",
      "Validation set perplexity: 6.37\n",
      "Average loss at step 10100: 1.600374 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.80\n",
      "Validation set perplexity: 6.38\n",
      "Average loss at step 10200: 1.610149 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.97\n",
      "Validation set perplexity: 6.37\n",
      "Average loss at step 10300: 1.598528 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.84\n",
      "Validation set perplexity: 6.38\n",
      "Average loss at step 10400: 1.609790 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.04\n",
      "Validation set perplexity: 6.37\n",
      "Average loss at step 10500: 1.598816 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.09\n",
      "Validation set perplexity: 6.38\n",
      "Average loss at step 10600: 1.605475 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.28\n",
      "Validation set perplexity: 6.38\n",
      "Average loss at step 10700: 1.598464 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.21\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 10800: 1.606684 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.70\n",
      "Validation set perplexity: 6.37\n",
      "Average loss at step 10900: 1.596942 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.50\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 11000: 1.604256 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.77\n",
      "================================================================================\n",
      "xind<>still<>can vear<>love delk<>planet volume      <>i pantly<>don t peavey to\n",
      "fulle<>godenny man bank <>brooutiming howsen right  people<>cheist true cheek   \n",
      " this as originator days<>come deftrn of go<>who di ffendon the picture<>bumbod \n",
      "ver deatrra<>the starserse<>nacion on the taupbe watwrited<>song<>kings get prim\n",
      "us<> as orso<>water to fire<>feeling a never<>lady live  remix    <>sweeter<>cas\n",
      "================================================================================\n",
      "Validation set perplexity: 6.38\n",
      "Average loss at step 11100: 1.599679 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.50\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 11200: 1.603242 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.92\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 11300: 1.599340 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.86\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 11400: 1.602863 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.68\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 11500: 1.597147 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.17\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 11600: 1.604942 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.83\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 11700: 1.597714 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.42\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 11800: 1.606116 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.96\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 11900: 1.598053 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.27\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 12000: 1.603781 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.81\n",
      "================================================================================\n",
      "y<>for the aywar i<>one your with <>rush and water<>my love<>the a music from th\n",
      "quets   to hore<>sume all the bace<>the sestits von  remix <>black   the flyshy<\n",
      "in say <>love on shadow<>strakehis milder<>remixes of blue<>ephooma<>music are a\n",
      "y deaph<>playevous<>non yecre dan the beatles is mile you love  fore<>closer and\n",
      "te priese<>now it formags<>stage lywn mine<>unclaspicafe<>live ald your lives<>c\n",
      "================================================================================\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 12100: 1.598364 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.09\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 12200: 1.603828 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.89\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 12300: 1.598887 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.19\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 12400: 1.600785 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.65\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 12500: 1.603491 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.81\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 12600: 1.597463 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.99\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 12700: 1.602780 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.72\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 12800: 1.596435 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.30\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 12900: 1.607830 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.59\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 13000: 1.593574 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.40\n",
      "================================================================================\n",
      "ess<>electra       <>mis to gration badd<>full birny<>hume hot the live it    <>\n",
      "op<>energe deess<>marly<>adis nyopl  cream vecriclers  and strasher<>one m const\n",
      "mage<>mes bron version dop<>bleas     the beginney gut<>unswones ep <>blue  aliz\n",
      "win in whippy boung<>pretones <>nething ep<>chills<>reft sunvict<>pycks cird ore\n",
      "kwen<>frantars<>now riqut<>presended folloure<>millic  m rad<>with turn the down\n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 13100: 1.608863 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.62\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 13200: 1.596654 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.72\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 13300: 1.606392 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.07\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 13400: 1.596862 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.15\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 13500: 1.603857 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.84\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 13600: 1.595464 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.84\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 13700: 1.602398 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.57\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 13800: 1.598706 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.22\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 13900: 1.601909 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.03\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 14000: 1.595891 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.57\n",
      "================================================================================\n",
      "ressions   the dawn tides teme purple<>madned drige flom <>lang more coundy of t\n",
      "apter<>phactat sesfirs iim<>veffer internigho smeles<>straight out of couttox<>e\n",
      "rhy  the yousbfarthun<>no    addest out and shapour<>take a d and   part  e e me\n",
      ">time<>abariar<>the dradilla  out of wayfarders   come fronts you b   thunderdom\n",
      ">behlike<>too with gold alon  em<>clenca eppeche everition  <>music<>care anglom\n",
      "================================================================================\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation set perplexity: 6.41\n",
      "Average loss at step 14100: 1.604074 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.11\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 14200: 1.596745 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.27\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 14300: 1.600125 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.81\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 14400: 1.596090 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.01\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 14500: 1.606154 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.22\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 14600: 1.598189 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.74\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 14700: 1.601482 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.16\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 14800: 1.596385 learning rate: 0.100000\n",
      "Minibatch perplexity: 5.51\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 14900: 1.602474 learning rate: 0.100000\n",
      "Minibatch perplexity: 4.69\n",
      "Validation set perplexity: 6.40\n",
      "Average loss at step 15000: 1.598110 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.67\n",
      "================================================================================\n",
      "y depaster<>the time at esuraw<>black second will   phaft lovesepues part<>secon\n",
      "in<>the black the beginney<>sound of stulks<>singles vol   <>wither of look   th\n",
      "more are ward the incocan<>the witc donectics<>read dreams<>the the harda vail<>\n",
      "went s <>so king of the my be misard<>by hearmatonis of san suit<>sa raby<>place\n",
      ">hormony three<>a phy rock remix yout the greats<>elekate part   <>bedurmese wit\n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 15100: 1.600058 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.15\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 15200: 1.597331 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.63\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 15300: 1.596661 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.05\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 15400: 1.601433 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.95\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 15500: 1.596215 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.32\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 15600: 1.602755 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.82\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 15700: 1.591707 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.60\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 15800: 1.608244 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.17\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 15900: 1.591640 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.81\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 16000: 1.607708 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.18\n",
      "================================================================================\n",
      "derpat<>the fly<>heartbiead<>telcing e midnide<>the run of you<>original motion \n",
      "king<>eall rahe<>dragesa<>amorrecter<>the fraxica<>life interruck pres do man<>b\n",
      "hyme<>looking and giem<>pitneser<>soul to bie <>clashbicks <>vidi<>new sun<>fran\n",
      "y   mosmonics<>sur e p             <>fever you are are<>for musicce<>the annimia\n",
      "leary<>room      <>diurs in the belim<>just s the pain<>the guration shying stan\n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 16100: 1.592967 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.57\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 16200: 1.603111 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.23\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 16300: 1.593469 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.59\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 16400: 1.603778 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.20\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 16500: 1.594239 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.95\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 16600: 1.600805 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.30\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 16700: 1.595079 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.84\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 16800: 1.600390 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.79\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 16900: 1.596504 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.10\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 17000: 1.598470 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.40\n",
      "================================================================================\n",
      "weat<>the longo<>losyn of       don t set  electre<>iu <>simpline slag<>ton ange\n",
      "ratin i m back r magic<>antroj vivo that <>you daf to a nive   lifece<>dogs psyc\n",
      "y        listenvie<>haibmade<>techiff bornner<>marcode of peobs<>stard with mari\n",
      "ness paris<>action  my histofy<>life ep<>a gu go chall recinot ep<>marks   lyve \n",
      "rined   glight debra<>locks   <>into me e p   sound a g <>intaire<>calip to see \n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 17100: 1.596497 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.69\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 17200: 1.601451 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.94\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 17300: 1.595650 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.51\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 17400: 1.602442 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.98\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 17500: 1.595357 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.35\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 17600: 1.601660 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.76\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 17700: 1.594973 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.33\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 17800: 1.600759 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.62\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 17900: 1.596161 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.85\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 18000: 1.598311 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.69\n",
      "================================================================================\n",
      " wail<>darko jawelins  live<>if obs<>eigh oroh<>magna    <>dance evergiss<>every\n",
      "tens<>album our season<>de wohd<>the grooved<>cobbath<>flaght   s get walk<>burg\n",
      "ar mocky<>a we traugled<>druaklig  elive at the now i <>  <>dox  touch awnoket<>\n",
      "xes<>flumber<>treat<>black to dance in loving nice<>burner and electro <>the vis\n",
      "razision         a rocks<>mindy bir year<>yeafal samber<>decrevelley<>danky flym\n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 18100: 1.601591 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.92\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 18200: 1.594474 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.48\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 18300: 1.602474 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.72\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 18400: 1.593648 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.54\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 18500: 1.605718 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.02\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 18600: 1.591053 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.70\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 18700: 1.608049 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.91\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 18800: 1.592345 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.65\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 18900: 1.604936 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.33\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 19000: 1.594314 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.85\n",
      "================================================================================\n",
      "plip  <>wath calins letons<>messy live<>affover by mars cincopelevallo<>new yoth\n",
      "ness power<>the beach agaztbig sen hapting outs<> i van e p   a slittle searowhe\n",
      "but<>cadi sive ui   soundia nice s knowreman<>your electrons<>picon feles<>carre\n",
      "cares     singles  original swhated<>edit <>this to new resongs  we goos ut time\n",
      "om<>part   <>sinkuif<>lodding save<>spiries<>techmore to bogile and songs<>spase\n",
      "================================================================================\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation set perplexity: 6.41\n",
      "Average loss at step 19100: 1.601579 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.58\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 19200: 1.593281 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.74\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 19300: 1.601084 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.99\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 19400: 1.596061 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.96\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 19500: 1.600720 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.91\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 19600: 1.594728 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.48\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 19700: 1.602067 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.90\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 19800: 1.594454 learning rate: 0.010000\n",
      "Minibatch perplexity: 5.27\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 19900: 1.599826 learning rate: 0.010000\n",
      "Minibatch perplexity: 4.84\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 20000: 1.593193 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.63\n",
      "================================================================================\n",
      "kyres<>tragg  it<>heavy zedges<>handu true<>zathmaner jei  need<>artz  neh out<>\n",
      "priest butter volume   love<>feab foogs builder<>fuck the ring the motion opic<>\n",
      "chazino<>four baby<>pourder sourg      <>miles de ponna i<>venarany live in the \n",
      "<>the ghet stire <>hored<>in davisoop love<>a mark scima a mechu<>love men line<\n",
      "wh one to lele<>rushicaged indarters odcey so heart<>bowith<>get be part  herr>b\n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 20100: 1.604491 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.21\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 20200: 1.596623 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.14\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 20300: 1.600370 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.29\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 20400: 1.595148 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.98\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 20500: 1.601588 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.14\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 20600: 1.595655 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.85\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 20700: 1.597914 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.00\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 20800: 1.599603 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.66\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 20900: 1.594683 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.89\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 21000: 1.601708 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.64\n",
      "================================================================================\n",
      "ky butty<>wild rock melles of ky   <>the ayes iii<>oh birgrats<>collectal  john \n",
      "ney strike<>playerm<>wail toks<>deards<>forpic fills<>frank   of canny liqut<>in\n",
      "dy etural me my den time<>everythidita band<>space storway to night<>itimage   s\n",
      " to the laclienk<>lounderdess<>shain six<>stranger  <>rock dusa<>sabason   the e\n",
      "ildsylar<>pilleye ep<>wend wom b<>the path aftors<>  house stop gails<>si lose r\n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 21100: 1.594115 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.12\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 21200: 1.604322 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.85\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 21300: 1.590305 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.69\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 21400: 1.607506 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.89\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 21500: 1.593205 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.88\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 21600: 1.606811 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.23\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 21700: 1.592200 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.52\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 21800: 1.601910 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.86\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 21900: 1.594040 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.08\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 22000: 1.603496 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.47\n",
      "================================================================================\n",
      "mix everything s fremn<>controm life it    the numballer<>wasking malloddardanch\n",
      "pay<>deve pot me rafelis   phr    <>get redrey sque ep <>you to rla<>basdecle<>t\n",
      "fires<>menter of the grave<>staping fersed<>elliant<>breakdaschine<>drankinseot \n",
      "fire<>baby<>bf d gs everysse<>from flying   donyth un<>live anthlux fechet mursk\n",
      "hinvin <>oligature zuichts<>i kan sbodul<>yor edown<>omiia nevo ravis<>things   \n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 22100: 1.594454 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.55\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 22200: 1.599010 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.26\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 22300: 1.594320 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.07\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 22400: 1.601936 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.86\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 22500: 1.595626 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.81\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 22600: 1.597662 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.90\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 22700: 1.596063 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.39\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 22800: 1.602177 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.92\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 22900: 1.595344 learning rate: 0.001000\n",
      "Minibatch perplexity: 4.70\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 23000: 1.602441 learning rate: 0.001000\n",
      "Minibatch perplexity: 5.06\n",
      "================================================================================\n",
      "<>inside vol  <>band ricor<>tever leans<>you featuring rock<>hearthils egp<>exta\n",
      "<>to the sil diem<>how mook sell<>in the secial silent   live at the baby<>sisth\n",
      "k blues song<>nightmanes deat growny<>these<>the mectron from the majler<>defori\n",
      "future <>in thystew <>volarine beak it i don t desters<>sonalowharl<>blusis to n\n",
      "us<>disco<>spealting starmag<>firil let brothly<>rechar sie<>for ecolation valle\n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 113100: 1.601999 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.91\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 113200: 1.591970 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.91\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 113300: 1.605737 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.08\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 113400: 1.591882 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.13\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 113500: 1.606489 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.83\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 113600: 1.593491 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.74\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 113700: 1.603359 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.18\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 113800: 1.594957 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.14\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 113900: 1.601912 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.72\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 114000: 1.591578 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.47\n",
      "================================================================================\n",
      "te never<>weire thwhileny tal  <>passoning at<>perfected the cabra<>    rulding \n",
      "man in the mix <>comes do it ep<>brass  greatest hits <>tiluwh  original motion \n",
      "kell <>the time junes<>some bugolus of pt  of thanger<>godvaif<>kinds in my chic\n",
      "z <>anniversale<>break  roch<>in the sesting <>live to r bob ries<>come one the \n",
      "night<>zere babie <>the good   volume  <>sorlialinnauch show<>volume <>the room<\n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 155100: 1.589772 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.23\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 155200: 1.607465 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.79\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 155300: 1.593073 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.00\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 155400: 1.605987 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.97\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 155500: 1.593014 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.42\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 155600: 1.601844 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.78\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 155700: 1.594107 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.08\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 155800: 1.602574 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.00\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 155900: 1.594601 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.92\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 156000: 1.600360 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.29\n",
      "================================================================================\n",
      "gooda<>don t you  c lazubes   june<>de black<>chose has up who boeshysing <>quin\n",
      "en hows<>givended of like it <>what a remixes part   <>disco aloh thunder  weirw\n",
      "under the hardcore<>live in and remix  <>where wander roon star eye<> songs of h\n",
      "ken<>the ampty<>dig e p <>out slippe<>black and cerrani<>krope music altatort<>s\n",
      "s of the subbe dj   <>lacester remixes vision<>primite is tolled clabb<>the plet\n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 198100: 1.601079 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.87\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 198200: 1.595316 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.23\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 198300: 1.598254 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.58\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 198400: 1.594715 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.87\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 198500: 1.602996 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.82\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 198600: 1.598689 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.57\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 198700: 1.598217 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.91\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 198800: 1.594140 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.35\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 198900: 1.602243 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.66\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 199000: 1.596949 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.95\n",
      "================================================================================\n",
      "ter piper<>known the gardance<>everything willgle<>hock  i   titifut and derbe d\n",
      "ride<>songs is  <>fum mane and sound<>the fires<>i m want to back<>best of the m\n",
      "watch<>quentay to l e pontase<>the allshupiale<>seeth and when asaire<>blue shal\n",
      "isque<>eurountma<>juchan gone<>stodia<>fly pinrett score sial<>fuck<>the boyon a\n",
      "gand<>the projeg<>the to z     christens  makuctora mirrican<>jore puturas<>sach\n",
      "================================================================================\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 199100: 1.597704 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.01\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 199200: 1.599134 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.48\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 199300: 1.594957 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.85\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 199400: 1.600342 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.49\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 199500: 1.596664 learning rate: 0.000000\n",
      "Minibatch perplexity: 5.38\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 199600: 1.603325 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.48\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 199700: 1.590646 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.79\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 199800: 1.606568 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.64\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 199900: 1.592667 learning rate: 0.000000\n",
      "Minibatch perplexity: 4.28\n",
      "Validation set perplexity: 6.41\n"
     ]
    }
   ],
   "source": [
    "NUM_STEPS = 200000\n",
    "SUMMARY_FREQUENCY = 100\n",
    "\n",
    "# Use interactive session in order to be able to generate album names on the next cell\n",
    "session = tf.InteractiveSession(graph=graph)\n",
    "tf.global_variables_initializer().run()\n",
    "print('Initialized')\n",
    "mean_loss = 0\n",
    "for step in range(NUM_STEPS):\n",
    "    batches = train_batches.next()\n",
    "    feed_dict = dict()\n",
    "    for i in range(NUM_UNROLLINGS + 1):\n",
    "        feed_dict[train_data[i]] = batches[i]\n",
    "    _, l, predictions, lr = session.run([optimizer, loss, train_prediction, learning_rate],\n",
    "                                        feed_dict=feed_dict)\n",
    "    mean_loss += l\n",
    "    if step % SUMMARY_FREQUENCY == 0:\n",
    "        if step > 0:\n",
    "            mean_loss = mean_loss / SUMMARY_FREQUENCY\n",
    "        # The mean loss is an estimate of the loss over the last few batches, and thus more robust.\n",
    "        print('Average loss at step %d: %f learning rate: %f' % (step, mean_loss, lr))\n",
    "        mean_loss = 0\n",
    "        labels = np.concatenate(list(batches)[1:])\n",
    "        print('Minibatch perplexity: %.2f' % float(np.exp(logprob(predictions, labels))))\n",
    "        if step % (SUMMARY_FREQUENCY * 10) == 0:\n",
    "            # Generate some samples.\n",
    "            print('=' * 80)\n",
    "            for _ in range(5):\n",
    "                # Start with a random character\n",
    "                feed = sample(random_distribution())\n",
    "                sentence = characters(feed)[0]\n",
    "                reset_sample_state.run()\n",
    "                for _ in range(79):\n",
    "                    prediction = sample_prediction.eval({sample_input: feed})\n",
    "                    # Sample predicted character from the probability distribution\n",
    "                    feed = sample(prediction)\n",
    "                    # Sampled character is fed to the next time step\n",
    "                    sentence += characters(feed)[0]\n",
    "                print(sentence)\n",
    "            print('=' * 80)\n",
    "        # Measure validation set perplexity.\n",
    "        reset_sample_state.run()\n",
    "        valid_logprob = 0\n",
    "        for _ in range(VALID_SIZE):\n",
    "            b = valid_batches.next()\n",
    "            predictions = sample_prediction.eval({sample_input: b[0]})\n",
    "            valid_logprob = valid_logprob + logprob(predictions, b[1])\n",
    "        print('Validation set perplexity: %.2f' % float(np.exp(valid_logprob / VALID_SIZE)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate album names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "York the all here americaning\n",
      "Cari\n",
      "Lootheries\n",
      "Get ceres party ppick\n",
      "Dim  and voltaie\n",
      "Better sound of the basismials\n",
      "And ther incing and warner\n",
      "Celearh mwnagn the version i capter\n",
      "Sive ey\n",
      "Dendoged naution\n",
      "Pries\n",
      "Get bit\n",
      "Gerord the say\n",
      "Soinatra\n",
      "Circio  mothers\n",
      "Trages the smy\n",
      "I fever piccled\n",
      "Anifucavor\n",
      "Nightsmys\n",
      "I the dig\n",
      "Sky bomend\n",
      "Boy clan\n",
      "Bober\n",
      "Uprese\n",
      "Sabrage factors kill the jazz\n",
      "He song on penfcbody butting s m     de thin dimmass  night of the again\n",
      "Alwoundler\n",
      "Orix anderny papara\n",
      "Pote   song gad at\n",
      "Belle\n",
      "Turcape\n",
      "Sure rock\n",
      "Best songs pecture  nock in melume\n",
      "Adifies\n",
      "Befinhes\n",
      "Mettaripa\n",
      "Best  psyshe\n",
      "I vall messe\n",
      "Construck s de who\n",
      "The jond \n",
      "Jegressey\n",
      "Tele of bytes\n",
      "Cry kin that   original motion picture soundtrack \n",
      "I m watchz\n",
      "Hards ov\n",
      "Bri kpst beage\n",
      "Antug   agoose\n",
      "Sonioust it s greatest\n",
      "For the pt         \n",
      "Markin tudes\n",
      "Bither ep\n",
      "A get dive toninities my music\n",
      "Oh t through \n",
      "Kelbriginal e p \n",
      "A s the stan ame you got\n",
      "Eir store\n",
      "Oh t zervels\n",
      "Sicantr\n",
      "Bibit\n",
      "Care whene\n",
      "Space doine  danceple\n",
      "Ben to right original after the song\n",
      "I you here eduet\n",
      "Oth\n",
      "Hear mess\n",
      "Be   dony\n",
      "Bassa of black on the sonnter\n",
      "Fever b  lemisters  elsen iii\n",
      "Blast food     wake of the night\n",
      "Knowlembed \n",
      "Gape\n",
      "Actis carroor\n",
      "Bohino blues buttination\n",
      "Breata  the beng\n",
      "Pation\n",
      "Nepturess dreams\n",
      "African\n",
      "Not pocelationstion\n",
      "Cockouth the framiscent\n",
      "Prince   dr\n",
      "Mixes and witares   th o ireverolfay out\n",
      "Reen death\n",
      "Reveric\n",
      "A ve ceme teser\n",
      "Scips album\n",
      "A h ghose stund\n",
      "Plidebreading from clug\n",
      "Fatburh in the counter  then say\n",
      "Your finom\n",
      "I knaught best on      \n",
      "Ben to the blanc\n",
      "Vol   \n",
      "Back ep\n",
      "Tustip  vulan man\n",
      "Folets\n",
      "More my shut   nticance\n",
      "Jain bjceur sundred la ex\n",
      "Betale m \n",
      "Cari\n",
      "Muskic worst off remixes \n"
     ]
    }
   ],
   "source": [
    "# Generate num_album album names\n",
    "num_albums = 100\n",
    "for _ in range(num_albums):\n",
    "    # First character is the starting token\n",
    "    feed = np.zeros([1, VOCABULARY_SIZE])\n",
    "    feed[0, START_TOKEN] = 1\n",
    "    character = id2char(START_TOKEN)\n",
    "    sentence = character\n",
    "    reset_sample_state.run()\n",
    "    # Repeat until hitting a STOP TOKEN\n",
    "    while character != id2char(STOP_TOKEN):\n",
    "        prediction = sample_prediction.eval({sample_input: feed})\n",
    "        # Sample predicted character from the probability distribution\n",
    "        feed = sample(prediction)\n",
    "        # Sampled character is fed to the next time step\n",
    "        character = characters(feed)[0]\n",
    "        sentence += character\n",
    "    # Remove START TOKEN\n",
    "    sentence = sentence[1:-1]\n",
    "    # Upper case the first letter of the album, to make it look pretty!\n",
    "    sentence = sentence[0].upper() + sentence[1:]\n",
    "    print(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, it could definitely be improved, we get a lot of nonsense but also some interesting examples:  \n",
    "* **Better sound of the basismials**\n",
    "* **Sabrage factors kill the jazz**\n",
    "* **Muskic worst off remixes** \n",
    "* **Cry kin that   original motion picture soundtrack**\n",
    "\n",
    "It seems like we are indirectly generating movie names too! And I wonder where The Basimials are :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Potential improvements:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Use only English album names.\n",
    "* Deal better with unusual characters.\n",
    "* Add more layers.\n",
    "* Tune Learning rate.\n",
    "* Try other optimizers: RMSProp, AdaGrad."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "default_view": {},
   "name": "6_lstm.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}