Lesson 15 - LSTM Shakespeare generator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM (Long Short Term Memory)\n",
    "\n",
    "There is a branch of Deep Learning that is dedicated to processing time series. These deep Nets are **Recursive Neural Nets (RNNs)**. LSTMs are one of the few types of RNNs that are available. Gated Recurent Units (GRUs) are the other type of popular RNNs.\n",
    "\n",
    "This is an illustration from http://colah.github.io/posts/2015-08-Understanding-LSTMs/ (A highly recommended read)\n",
    "\n",
    "![RNNs](./RNN-unrolled.png)\n",
    "\n",
    "Pros:\n",
    "- Really powerful pattern recognition system for time series\n",
    "\n",
    "Cons:\n",
    "- Cannot deal with missing time steps.\n",
    "- Time steps must be discretised and not continuous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Activation, Dropout, Flatten, Dense, BatchNormalization, LSTM, Embedding, TimeDistributed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def chr2val(ch):\n",
    "    ch = ch.lower()\n",
    "    if ch.isalpha():\n",
    "        return 1 + (ord(ch) - ord('a'))\n",
    "    else:\n",
    "        return 0\n",
    "    \n",
    "def val2chr(v):\n",
    "    if v == 0:\n",
    "        return ' '\n",
    "    else:\n",
    "        return chr(ord('a') + v - 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "THE SONNETS\n",
      "by William Shakespeare\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "I\n",
      "\n",
      "From fairest creatures we desire increase,\n",
      "That thereby be\n",
      "[20  8  5  0 19 15 14 14  5 20 19  0  2 25  0 23  9 12 12  9  1 13  0 19  8\n",
      "  1 11  5 19 16  5  1 18  5  0  0  0  0  0  9  0  0  6 18 15 13  0  6  1  9\n",
      " 18  5 19 20  0  3 18  5  1 20 21 18  5 19  0 23  5  0  4  5 19  9 18  5  0\n",
      "  9 14  3 18  5  1 19  5  0  0 20  8  1 20  0 20  8  5 18  5  2 25  0  2  5]\n"
     ]
    }
   ],
   "source": [
    "with open(\"sonnets.txt\") as f:\n",
    "    text = f.read()\n",
    "    \n",
    "text_num = np.array([chr2val(c) for c in text])\n",
    "print(text[:100])\n",
    "print(text_num[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The range of numbers for the letters are between:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 26]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[min(text_num), max(text_num)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Many to One Model\n",
    "Prepare the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len_vocab = 27\n",
    "sentence_len = 40\n",
    "# n_chars = len(text_num)//sentence_len*sentence_len\n",
    "num_chunks = len(text_num)-sentence_len\n",
    "\n",
    "x = np.zeros((num_chunks, sentence_len))\n",
    "y = np.zeros(num_chunks)\n",
    "for i in range(num_chunks):\n",
    "    x[i,:] = text_num[i:i+sentence_len]\n",
    "    y[i] = text_num[i+sentence_len]\n",
    "\n",
    "# x = np.reshape(x, (num_chunks, sentence_len, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(95610, 40)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, None, 64)          1728      \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 64)                33024     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 27)                1755      \n",
      "=================================================================\n",
      "Total params: 36,507.0\n",
      "Trainable params: 36,507\n",
      "Non-trainable params: 0.0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "# TODO: \n",
    "# 1. Add a Embedding layer https://keras.io/layers/embeddings/ \n",
    "# (the first argument is len_vocab, second the number of hidden units)\n",
    "# 2. Add a LSTM with a suitable number of hidden units\n",
    "# 3. Add a final Dense layer, keep in mind that this is a multiclass classification problem\n",
    "# (how many output classes are there and what is the activation)\n",
    "model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "95610/95610 [==============================] - 100s - loss: 2.4096   \n",
      "esseeas ach co wiwsil  an  wingull   taur to dthuth fe lith fanl  thit no thecives veiss he heag tha\n",
      "--------------------\n",
      "feit  so that other mine thou wilt resto\n",
      "========================================\n",
      "Epoch 1/1\n",
      "95610/95610 [==============================] - 130s - loss: 2.0563   \n",
      "e koume pying copwist love wirnt toll were is my my rate  thoueene glioks of ghich stis arly     kea\n",
      "--------------------\n",
      "s have drain d his blood and fill d his \n",
      "========================================\n",
      "Epoch 1/1\n",
      "95610/95610 [==============================] - 125s - loss: 1.9219   \n",
      "  banch deture  whor all  sweartay i   if liin love theeag   liln heautt s tha lether flove un thou \n",
      "--------------------\n",
      " made  that millions of strange shadows \n",
      "========================================\n",
      "Epoch 1/1\n",
      "95610/95610 [==============================] - 123s - loss: 1.8404   \n",
      "o if yore cand acker a glace in be deach be the with  but tith his ase ade hime that one tooth ate a\n",
      "--------------------\n",
      "e mute    or  if they sing   tis with so\n",
      "========================================\n",
      "Epoch 1/1\n",
      "95610/95610 [==============================] - 133s - loss: 1.7855   \n",
      "t of tipen  knagy alals lacven sight besed thy swicken migh loke gacs  your by were make  which noem\n",
      "--------------------\n",
      "eart   xlvii  betwixt mine eye and heart\n",
      "========================================\n",
      "Epoch 1/1\n",
      "95610/95610 [==============================] - 130s - loss: 1.7439   \n",
      " eypring   fithel  a take wriches eveming alip sim no loves  is wore suskces wost in that faist less\n",
      "--------------------\n",
      "that time    you should live twice   in \n",
      "========================================\n",
      "Epoch 1/1\n",
      "95610/95610 [==============================] - 131s - loss: 1.7112   \n",
      "ir d swipp  a to wen whening my fim brile end   xxxiny dores vearder om tho  my hark  and flendie vi\n",
      "--------------------\n",
      " d  but thy eternal summer shall not fad\n",
      "========================================\n",
      "Epoch 1/1\n",
      "95610/95610 [==============================] - 121s - loss: 1.6844   \n",
      "pent     nxkis nor songu is  al my look  therefuting o  you  lies seat thy fies bolk it seen thine n\n",
      "--------------------\n",
      "itless usurer  why dost thou use so grea\n",
      "========================================\n",
      "Epoch 1/1\n",
      "95610/95610 [==============================] - 114s - loss: 1.6609   \n",
      "tren is allity  and i if the prorom you oft do seellovent  mut    neck tly  fase he ore they beauty \n",
      "--------------------\n",
      "ease find no determination  then you wer\n",
      "========================================\n",
      "Epoch 1/1\n",
      "95610/95610 [==============================] - 112s - loss: 1.6410   \n",
      " pannalter for is doth they ele to retronds and grount impern  my etequer i jead thu pair now bey pa\n",
      "--------------------\n",
      "ise that purpose not to sell   xxii  my \n",
      "========================================\n"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    model.fit(x,y, batch_size=128, epochs=1)\n",
    "    sentence = []\n",
    "    idx = np.random.choice(len(x),1)\n",
    "    x_test = x[idx]\n",
    "    if idx==len(x)-1:\n",
    "        idx -= 1\n",
    "    for i in range(100):\n",
    "        # TODO: \n",
    "        # 1. Given x_test predict the probability of each class\n",
    "        # 2. Given the probability distribution RANDOMLY choose a class \n",
    "        # https://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.random.choice.html\n",
    "        p = \n",
    "        idx2 = \n",
    "        x_test = np.hstack([x_test[:,1:], idx2[None,:]])\n",
    "        sentence.append(val2chr(idx2[0]))\n",
    "\n",
    "    print(''.join(sentence))\n",
    "    print('-'*20)\n",
    "    print(''.join([val2chr(int(v)) for v in x[idx+1,:].tolist()[0]]))\n",
    "    print('='*40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Many to Many LSTM\n",
    "\n",
    "In the previous layer we predicted one time step given the last 40 steps. This time however, we are predicting the 2nd to 41st character given the first 40 characters. Another way of looking at this is that, at each **character input** we are predicting the subsequent character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_5 (Embedding)      (None, None, 64)          1728      \n",
      "_________________________________________________________________\n",
      "lstm_5 (LSTM)                (None, None, 64)          33024     \n",
      "_________________________________________________________________\n",
      "time_distributed_3 (TimeDist (None, None, 27)          1755      \n",
      "=================================================================\n",
      "Total params: 36,507.0\n",
      "Trainable params: 36,507\n",
      "Non-trainable params: 0.0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "# TODO:\n",
    "# 1. Add an Embedding and LSTM layer as before. However, set return_sequences=True in the LSTM\n",
    "# 2. Add a TimeDistributed(Dense) layer instead of just Dense.\n",
    "# See here as to why: https://keras.io/layers/wrappers/#timedistributed\n",
    "\n",
    "model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = np.zeros((num_chunks, sentence_len))\n",
    "y = np.zeros((num_chunks, sentence_len))\n",
    "for i in range(num_chunks):\n",
    "    x[i,:] = text_num[i:i+sentence_len]\n",
    "    y[i,:] = text_num[i+1:i+sentence_len+1]\n",
    "\n",
    "y = np.reshape(y,(y.shape[0], y.shape[1], 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Notes:\n",
    "1. The shape of `y` is now the same as x, as we are not predicting just one character any more.\n",
    "2. In the following cell, it is important to notice that I did not need to use a 40 length character as an input to the predictions. See below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "95610/95610 [==============================] - 137s - loss: 2.2415   \n",
      "thenokinds aunin   wegoub sow ld ad t tn t osth cy wed yat d mieeigr t horumet n ws ghiuy  trw ashar\n",
      "========================================\n",
      "Epoch 1/1\n",
      "31232/95610 [========>.....................] - ETA: 105s - loss: 1.8466"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    model.fit(x,y, batch_size=128, epochs=1)\n",
    "    \n",
    "    sentence = []\n",
    "    letter = np.random.choice(len_vocab,1).reshape((1,1)) #choose a random letter\n",
    "    for i in range(100):\n",
    "        sentence.append(val2chr(letter))\n",
    "        p = model.predict(letter)\n",
    "        letter = np.random.choice(27,1,p=p[0][0])\n",
    "    print(''.join(sentence))\n",
    "    print('='*40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}