GoT_Language_Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a Language Model for Game of Thrones\n",
    "\n",
    "*This notebook is part of the tutorial \"Sequence Modelling with Deep Learning\" presented at the ODSC London Conference in November 2019.*\n",
    "\n",
    "In this notebook, we will build a neural language model that can understand Game of Thrones language and concepts and even write its own passages. The architecture we will use is a **recurrent neural network (RNN)** with **LSTM cells** to boost the model's ability to remember longer-term information within the text. \n",
    "\n",
    "The framework we will use to build the models is `Keras`. Keras is a high-level neural networks API - it acts as a user-friendly layer on top of lower-level frameworks (like Tensorflow or Theano), and allows you to build neural networks in an intuitive, layer-by-layer way. \n",
    "\n",
    "<img src=\"books.jpg\" alt=\"Picture of Game of Thrones books\" width=\"600\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to language models\n",
    "\n",
    "Training a **language model (LM)** is a classic (but difficult) task in the field of **natural language processing (NLP)** - the goal is to predict the next word given the previous words. \n",
    "\n",
    "For example, which word should follow the sequence \"the cat is on the\"? Good guesses are words like \"mat\", \"bed\", \"sofa\", and we would hope that our model would learn to assign high probabilities to these semantically relevant terms. We would hope that words like \"the\", \"hi\", and \"banana\" would be assigned low probabilities. \n",
    "\n",
    "Since LMs learn a deep understanding of the syntax and semantics of a text corpus during training, they have become popular components of many other NLP tasks, via **transfer learning**.\n",
    "\n",
    "Language models are often used **generatively**, as in smartphone keyboard apps, to predict future text based on a **seed sequence**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How are language models trained?\n",
    "\n",
    "All you need to train a language model is a text corpus - **no annotation or labelling of the data is required**. However, language modelling is treated as a **supervised classification task**. The idea is that we extract training data by **sliding a window over the corpus**, and generating input-output pairs that way. More exactly:\n",
    "\n",
    "![Building a dataset for training a language model](lm_data.png)\n",
    "\n",
    "So here, we are sliding a window of some size over the corpus in order to generate sequences of words (here, sequences of 2 words each). Then:\n",
    "+ The initial 2 words in each sequences is our **input** (or **features** or **X values**)\n",
    "+ The final word in each sequence is our **output** (or **label** or **y values**)\n",
    "\n",
    "The model is then trained to use the input words (the **context**) to predict the final word. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Considerations when building the dataset\n",
    "There are a few decisions you have to make with how you will build this dataset. For example:\n",
    "+ Are you going to treat the text as a sequence at the **word level** or the **character level**? \n",
    "\n",
    "    + **The arguments for using words are**: there is a lot of information in words since that's how we structure language. And the length of sequences the model has to deal with and remember will be much shorter, leading to greater coherence. \n",
    "    + **The arguments for using characters are**: the size of the input space is much more manageable (there are fewer characters than words), and you gain the ability to handle unknown words and generate new words.\n",
    "    + **You could also work at the sub-word level**: this is a bit of a happy medium - words are broken down into their components. \n",
    "    \n",
    "+ Are you going to **scrub the text squeaky clean** or do you want the model to learn to deal with **noise**, perhaps at a cost of a hit to performance?\n",
    "+ What sort of **window size** should you be using?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# I. Building a Toy Language Model First"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before launching straight into the Game of Thrones language modelling problem, let's work with a smaller first and understand all of the steps involved. This way, you can more easily understand and track how all of the input, intermediate steps, and output is behaving. \n",
    "\n",
    "Let's use the following poem from Lord of the Rings as our entire corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "tiny_corpus = ['All that is gold does not glitter',\n",
    "               'Not all those who wander are lost;',\n",
    "               'The old that is strong does not wither,',\n",
    "               'Deep roots are not reached by the frost.',\n",
    "               'From the ashes, a fire shall be woken,',\n",
    "               'A light from the shadows shall spring;',\n",
    "               'Renewed shall be blade that was broken,',\n",
    "               'The crownless again shall be king']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### i. Preparing the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get started, the first thing we need to do is **tokenisation** - break the text up into individual units or **tokens**. \n",
    "\n",
    "We can use the text tokeniser from the `Keras` library for this, and specify that we want to treat all text as lowercase, generate tokens by splitting on a space character, and view text at the word level. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "tokeniser = Tokenizer(lower=True, split=' ', char_level=False)\n",
    "tiny_corpus = ' '.join(tiny_corpus)\n",
    "tokeniser.fit_on_texts([tiny_corpus])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tokeniser identifies tokens in the corpus and assigns an index to each word in the vocabulary. We can check which index corresponds to which word like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'the': 1,\n",
       " 'not': 2,\n",
       " 'shall': 3,\n",
       " 'that': 4,\n",
       " 'be': 5,\n",
       " 'all': 6,\n",
       " 'is': 7,\n",
       " 'does': 8,\n",
       " 'are': 9,\n",
       " 'from': 10,\n",
       " 'a': 11,\n",
       " 'gold': 12,\n",
       " 'glitter': 13,\n",
       " 'those': 14,\n",
       " 'who': 15,\n",
       " 'wander': 16,\n",
       " 'lost': 17,\n",
       " 'old': 18,\n",
       " 'strong': 19,\n",
       " 'wither': 20,\n",
       " 'deep': 21,\n",
       " 'roots': 22,\n",
       " 'reached': 23,\n",
       " 'by': 24,\n",
       " 'frost': 25,\n",
       " 'ashes': 26,\n",
       " 'fire': 27,\n",
       " 'woken': 28,\n",
       " 'light': 29,\n",
       " 'shadows': 30,\n",
       " 'spring': 31,\n",
       " 'renewed': 32,\n",
       " 'blade': 33,\n",
       " 'was': 34,\n",
       " 'broken': 35,\n",
       " 'crownless': 36,\n",
       " 'again': 37,\n",
       " 'king': 38}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokeniser.word_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can use this tokeniser to convert (**encode**) our original corpus to a sequence of indices correponding to words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[6, 4, 7, 12, 8, 2, 13]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_corpus = tokeniser.texts_to_sequences([tiny_corpus])[0]\n",
    "encoded_corpus[0:7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can always get back to the words by reversing this process:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['all that is gold does not glitter']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokeniser.sequences_to_texts([encoded_corpus[0:7]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can build a dataset of sequences that we will use for training and evaluating our language model. \n",
    "\n",
    "Let's use a window size of 3 and slide this over the integer-encoded corpus to build our dataset: a **list of lists of length 3**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[6, 4, 7],\n",
       " [4, 7, 12],\n",
       " [7, 12, 8],\n",
       " [12, 8, 2],\n",
       " [8, 2, 13],\n",
       " [2, 13, 2],\n",
       " [13, 2, 6],\n",
       " [2, 6, 14],\n",
       " [6, 14, 15],\n",
       " [14, 15, 16],\n",
       " [15, 16, 9],\n",
       " [16, 9, 17],\n",
       " [9, 17, 1],\n",
       " [17, 1, 18],\n",
       " [1, 18, 4],\n",
       " [18, 4, 7],\n",
       " [4, 7, 19],\n",
       " [7, 19, 8],\n",
       " [19, 8, 2],\n",
       " [8, 2, 20],\n",
       " [2, 20, 21],\n",
       " [20, 21, 22],\n",
       " [21, 22, 9],\n",
       " [22, 9, 2],\n",
       " [9, 2, 23],\n",
       " [2, 23, 24],\n",
       " [23, 24, 1],\n",
       " [24, 1, 25],\n",
       " [1, 25, 10],\n",
       " [25, 10, 1],\n",
       " [10, 1, 26],\n",
       " [1, 26, 11],\n",
       " [26, 11, 27],\n",
       " [11, 27, 3],\n",
       " [27, 3, 5],\n",
       " [3, 5, 28],\n",
       " [5, 28, 11],\n",
       " [28, 11, 29],\n",
       " [11, 29, 10],\n",
       " [29, 10, 1],\n",
       " [10, 1, 30],\n",
       " [1, 30, 3],\n",
       " [30, 3, 31],\n",
       " [3, 31, 32],\n",
       " [31, 32, 3],\n",
       " [32, 3, 5],\n",
       " [3, 5, 33],\n",
       " [5, 33, 4],\n",
       " [33, 4, 34],\n",
       " [4, 34, 35],\n",
       " [34, 35, 1],\n",
       " [35, 1, 36],\n",
       " [1, 36, 37],\n",
       " [36, 37, 3],\n",
       " [37, 3, 5],\n",
       " [3, 5, 38],\n",
       " [5, 38],\n",
       " [38]]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequences = []\n",
    "window_size = 3\n",
    "for i in range(0, len(encoded_corpus)):\n",
    "    sequences.append(encoded_corpus[i:i+window_size])\n",
    "\n",
    "sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that at the end there we have sequences that are not length 3, since we run out of text. We can quickly **pad the sequences with zeroes** to keep the data size consistent: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6,  4,  7],\n",
       "       [ 4,  7, 12],\n",
       "       [ 7, 12,  8],\n",
       "       [12,  8,  2],\n",
       "       [ 8,  2, 13],\n",
       "       [ 2, 13,  2],\n",
       "       [13,  2,  6],\n",
       "       [ 2,  6, 14],\n",
       "       [ 6, 14, 15],\n",
       "       [14, 15, 16],\n",
       "       [15, 16,  9],\n",
       "       [16,  9, 17],\n",
       "       [ 9, 17,  1],\n",
       "       [17,  1, 18],\n",
       "       [ 1, 18,  4],\n",
       "       [18,  4,  7],\n",
       "       [ 4,  7, 19],\n",
       "       [ 7, 19,  8],\n",
       "       [19,  8,  2],\n",
       "       [ 8,  2, 20],\n",
       "       [ 2, 20, 21],\n",
       "       [20, 21, 22],\n",
       "       [21, 22,  9],\n",
       "       [22,  9,  2],\n",
       "       [ 9,  2, 23],\n",
       "       [ 2, 23, 24],\n",
       "       [23, 24,  1],\n",
       "       [24,  1, 25],\n",
       "       [ 1, 25, 10],\n",
       "       [25, 10,  1],\n",
       "       [10,  1, 26],\n",
       "       [ 1, 26, 11],\n",
       "       [26, 11, 27],\n",
       "       [11, 27,  3],\n",
       "       [27,  3,  5],\n",
       "       [ 3,  5, 28],\n",
       "       [ 5, 28, 11],\n",
       "       [28, 11, 29],\n",
       "       [11, 29, 10],\n",
       "       [29, 10,  1],\n",
       "       [10,  1, 30],\n",
       "       [ 1, 30,  3],\n",
       "       [30,  3, 31],\n",
       "       [ 3, 31, 32],\n",
       "       [31, 32,  3],\n",
       "       [32,  3,  5],\n",
       "       [ 3,  5, 33],\n",
       "       [ 5, 33,  4],\n",
       "       [33,  4, 34],\n",
       "       [ 4, 34, 35],\n",
       "       [34, 35,  1],\n",
       "       [35,  1, 36],\n",
       "       [ 1, 36, 37],\n",
       "       [36, 37,  3],\n",
       "       [37,  3,  5],\n",
       "       [ 3,  5, 38],\n",
       "       [ 0,  5, 38],\n",
       "       [ 0,  0, 38]], dtype=int32)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "max_sequence_length = np.max([len(sequence) for sequence in sequences])\n",
    "sequences = pad_sequences(sequences, \n",
    "                          maxlen=max_sequence_length, \n",
    "                          padding='pre')\n",
    "sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks better. \n",
    "\n",
    "Finally, let's break the sequences down into our input data (X; our matrix of features) and our output data (y; our vector of labels):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array([x[0:2] for x in sequences])\n",
    "y = np.array([x[2] for x in sequences])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So for example our input features for the first 5 data points are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6,  4],\n",
       "       [ 4,  7],\n",
       "       [ 7, 12],\n",
       "       [12,  8],\n",
       "       [ 8,  2]], dtype=int32)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And their corresponding labels are: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 7, 12,  8,  2, 13], dtype=int32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final thing we need to do is reformat our label vector y into a **one-hot vector format**. The word index numbers are not actually meaningful (no ordinal relationship) but are discrete classes. We also want to calculate the probabilities of the next word, where a probability of 1 for the correct word is the optimal prediction. \n",
    "\n",
    "We can convert the label vector y to a matrix of one-hot vectors using Keras' `to_categorical` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0.]], dtype=float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.utils import to_categorical\n",
    "\n",
    "vocabulary_size = len(tokeniser.word_index)+1\n",
    "y = to_categorical(y, num_classes=vocabulary_size)\n",
    "y[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To summarise, we have gone from a raw dataset of:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'All that is gold does not glitter Not all those who wander are lost; The old that is strong does not wither, Deep roots are not reached by the frost. From the ashes, a fire shall be woken, A light from the shadows shall spring; Renewed shall be blade that was broken, The crownless again shall be king'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tiny_corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To a formatted dataset ready to be input to a learning algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example features: \n",
      "[6 4]\n",
      "[4 7]\n",
      "[ 7 12]\n",
      "[12  8]\n",
      "[8 2]\n",
      "Example labels: \n",
      "[0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "[0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "[0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "print('Example features: ', *X[0:5], sep='\\n')\n",
    "print('Example labels: ', *y[0:5], sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ii. Setting up the language model architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the dataset sorted out, it's time to think about how we want to approach the modelling problem.\n",
    "\n",
    "Let's build this small recurrent neural network with LSTM units:\n",
    "\n",
    "![tiny_network](small_network.png)\n",
    "\n",
    "To explain this network:\n",
    "+ Our **input layer** represents input into the network. The size of the input layer is the size of the vocabulary of our corpus (+1).\n",
    "+ We then have an **embedding layer** immediately after the input layer, which will learn **word embeddings** for us (continuous representation of the discrete words in our vocabulary; see my explanatory blog post on embeddings [here](https://towardsdatascience.com/why-do-we-use-embeddings-in-nlp-2f20e1b632d2). An embedding layer first changes your integer-encoded input to a one-hot vector format, followed by a fully-connected layer, and the learned weight matrix of this layer functions as our word embeddings.\n",
    "+ Our LSTM layer's job is to actually learn something - it takes as input the embeddings of the current word (at time step $t$) and also the hidden state from the previous word (at time step $t-1$), and tries to make predictions of the next word based on this information.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `Keras` code, we would build this network like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding\n",
    "from keras.layers import LSTM\n",
    "from keras.layers import Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(input_dim=vocabulary_size, output_dim=10, input_length=max_sequence_length-1))\n",
    "model.add(LSTM(units=50))\n",
    "model.add(Dense(units=vocabulary_size, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An explanation of this code block:\n",
    "+ Keras allows the sequential layer-by-layer building of neural network models using its `Sequential` API.\n",
    "+ The input layer is assumed, we don't need to explicitly build it.\n",
    "+ The first layer we add is our `Embedding` layer. The input dimensionality is our vocabulary size (the size of our input layer), and let's give this embedding layer a small size of 10 neurons. This means each word will get represented as a real-valued vector of length 10. We state that the length of inputs the network should expect is 2. \n",
    "+ Next, we add the workhorse of the network - our layer of `LSTM` neurons. Let's make the layer have 50 of these neurons (which is not a lot). We leave all other options to the default (activation functions, initialisation,etc.)\n",
    "+ Finally, as our output layer, we add a `Dense` fully-connected layer and softmax it. This means that the output of the network will be a vector of probabilities (summing to 1) spread across all the words of our vocabulary (see example below). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can examine our model so far using Keras' `model.summary()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 2, 10)             390       \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 50)                12200     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 39)                1989      \n",
      "=================================================================\n",
      "Total params: 14,579\n",
      "Trainable params: 14,579\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This summarises the number of parameters in our model and where they are.\n",
    "+ **390 parameters from $39*10$**: the number of input neurons times the number of neurons in the embedding layer (it is fully connected - meaning there is a connection between every neuron)\n",
    "+ **12200 LSTM parameters from $4*((10*50) + (50*50) + 50)$**: 10x50 \"normal\" weights, 50x50 weights between the previous time step's hidden layer and the current time steps hidden layer, 50 parameters from 1 bias teach, and all times 4 for the 4 gates.  \n",
    "+ **1989 parameters from $50*39 + 39$**: the number of previous layer neurons (50) times the number of output neurons (39), plus 39 bias parameters from the output layer\n",
    "\n",
    "Now that we have defined the network, we need to do a `model.compile()` to signify that we have finished building the network and want to define how training should proceed. Specifically, we need to provide:\n",
    "+ Which loss function we want to use (i.e. what is the goal the model is optimising for as it trains, or what signal is it following in order to improve)\n",
    "+ Which optimiser we want to use to do our gradient updates (Adam, Adagrad, RMSProp, Nesterov momentum, etc.)\n",
    "+ Any metrics we want to calculate and output during training in order to keep track of progress. Let's keep track of accuracy, which is just the percentage of predictions that the model gets right. \n",
    "\n",
    "We can just use reasonable defaults for now. Since our task is a multiclass classification task, a sensible loss metric to use is **categorical cross-entropy**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll avoid dumping equations on you and just say that:\n",
    "+ The model's categorical cross-entropy loss will be **low** when the network generally predicts the next words correctly. This means it tends to assign higher probability to the correct word.\n",
    "+ A training loss of zero means that the network always assigns a probability of 1 to the correct word and 0 to all other words - its predictions are perfect (in the training set).\n",
    "+ The model's categorical cross-entropy loss will be **high** when the network generally doesn't predict the next words well. This means it tends to incorrectly assign high probabilities to incorrect words.\n",
    "\n",
    "During the training process, the model optimises its internal parameters such that training loss is minimised (for an explanation of how this happens, read about backpropagation and gradient descent [here](http://neuralnetworksanddeeplearning.com/chap1.html))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### iii. Training the language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the network is compiled, we can begin training it for some time (for some number of **epochs** - which is the number of times the network sees your training data). \n",
    "\n",
    "Hopefully, as model training proceeds, we will see that the training loss steadily decreases and the accuracy increases: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x14cf75f90>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# model.fit(X, y, epochs=50, verbose=0)\n",
    "model.fit(X, y, epochs=50, verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's the model trained! Training is very fast because our dataset is tiny and the network is small. The accuracy doesn't look that bad either (though of course the model is likely to be **overfitted**; see later section). \n",
    "\n",
    "There's a few different things you can do with a trained Keras sequence model. \n",
    "\n",
    "You can see all the options by typing `model.` followed by a `tab` in a cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 3.15541029e-01,  2.03606471e-01,  2.80016929e-01,\n",
       "         1.27926588e-01, -3.79317701e-01, -9.04322043e-02,\n",
       "         2.27908164e-01,  3.21692497e-01,  7.95110017e-02,\n",
       "        -1.13509171e-01],\n",
       "       [-1.72527492e-01,  3.44863534e-01,  3.81999835e-02,\n",
       "         3.60623091e-01,  3.40169400e-01, -2.72232652e-01,\n",
       "         4.05081093e-01, -7.49856979e-02,  1.09128311e-01,\n",
       "        -3.01918000e-01],\n",
       "       [ 2.47175246e-01, -2.75743902e-01, -2.80102402e-01,\n",
       "        -1.90534458e-01,  1.43431589e-01,  1.39929637e-01,\n",
       "        -4.07428294e-01, -3.12202513e-01, -2.68854462e-02,\n",
       "         1.76154733e-01],\n",
       "       [-1.93544433e-01, -1.22530438e-01, -1.88495405e-02,\n",
       "         3.50918263e-01, -3.15138280e-01, -1.28867984e-01,\n",
       "         2.45830148e-01,  1.23829342e-01, -2.26835892e-01,\n",
       "         1.60934970e-01],\n",
       "       [ 3.56040895e-01, -3.13914686e-01,  3.78356546e-01,\n",
       "         2.19874486e-01, -2.96347409e-01,  3.83741379e-01,\n",
       "        -4.56347942e-01, -1.03706792e-01, -4.22252208e-01,\n",
       "         3.95310014e-01],\n",
       "       [ 2.47793809e-01,  4.00860131e-01, -2.12640408e-02,\n",
       "         2.97161728e-01,  3.22497077e-02, -3.14674199e-01,\n",
       "         3.54979515e-01,  3.70940953e-01,  2.11613193e-01,\n",
       "        -4.18641269e-01],\n",
       "       [ 8.88367221e-02,  2.66295731e-01,  3.39539289e-01,\n",
       "        -3.38406533e-01,  1.98719576e-01,  1.85448140e-01,\n",
       "        -1.67292193e-01,  7.22297356e-02, -9.91282761e-02,\n",
       "         1.48927599e-01],\n",
       "       [ 3.40956785e-02, -2.88842231e-01,  3.44839841e-01,\n",
       "         2.84668505e-01, -3.25176686e-01, -2.09063768e-01,\n",
       "        -2.52692163e-01,  3.47878456e-01, -3.20791185e-01,\n",
       "         3.64716798e-01],\n",
       "       [ 2.80208915e-01, -3.68876666e-01, -1.60457864e-01,\n",
       "        -3.42754632e-01, -1.32390246e-01, -2.12022271e-02,\n",
       "        -3.71442139e-01, -6.81050494e-02,  1.98423773e-01,\n",
       "        -1.49220571e-01],\n",
       "       [ 2.00558886e-01,  8.43887478e-02, -7.25398734e-02,\n",
       "        -2.48696521e-01,  3.24698031e-01,  2.94449199e-02,\n",
       "        -2.11172968e-01,  7.42750019e-02,  3.35146040e-01,\n",
       "        -3.15184683e-01],\n",
       "       [ 1.78098097e-01,  2.93547839e-01, -2.78206617e-01,\n",
       "        -1.91922396e-01,  2.73050696e-01,  3.36493313e-01,\n",
       "        -2.73830533e-01,  8.66670534e-03,  2.07513392e-01,\n",
       "        -1.56434849e-01],\n",
       "       [-3.19854528e-01,  2.92544633e-01,  8.80934298e-02,\n",
       "         3.35451365e-01,  3.47606510e-01, -2.20455423e-01,\n",
       "         2.61545330e-01, -1.51953310e-01, -2.49599162e-02,\n",
       "        -1.60332192e-02],\n",
       "       [ 2.88714081e-01, -2.47285679e-01,  3.16005111e-01,\n",
       "        -1.29828766e-01, -2.93719292e-01, -2.38989845e-01,\n",
       "        -9.04715583e-02,  2.57032424e-01,  1.27137840e-01,\n",
       "        -1.35510206e-01],\n",
       "       [ 4.10577923e-01, -2.42239311e-01, -1.51165947e-01,\n",
       "        -3.54308814e-01,  1.22696407e-01,  2.28801537e-02,\n",
       "        -3.44336122e-01, -9.43425819e-02,  2.79674619e-01,\n",
       "        -1.58249438e-01],\n",
       "       [-3.51372212e-01,  3.70013982e-01,  1.63637549e-01,\n",
       "        -1.01301037e-01, -5.11572585e-02,  1.93408042e-01,\n",
       "         1.24576911e-01, -2.77272552e-01, -6.42187148e-02,\n",
       "         3.81780446e-01],\n",
       "       [-2.17601225e-01, -1.69383869e-01,  2.54127771e-01,\n",
       "        -2.49165192e-01, -3.35291088e-01, -1.34573072e-01,\n",
       "        -1.94005147e-01, -1.97682649e-01, -1.85851604e-01,\n",
       "         3.36478770e-01],\n",
       "       [-8.09390619e-02, -3.32207292e-01,  1.09145679e-02,\n",
       "        -2.23076761e-01, -3.23500223e-02,  3.15826654e-01,\n",
       "        -1.68745875e-01,  1.04888871e-01,  3.60645950e-01,\n",
       "        -1.42324835e-01],\n",
       "       [ 3.16392750e-01,  2.56886721e-01, -1.72499329e-01,\n",
       "        -1.77069619e-01,  2.30184585e-01,  3.05743396e-01,\n",
       "        -3.27665269e-01, -1.43605724e-01,  1.81701064e-01,\n",
       "        -5.49116097e-02],\n",
       "       [ 1.14615716e-01, -5.27771004e-02,  2.28099480e-01,\n",
       "        -7.49025643e-02,  1.74155414e-01, -1.11714110e-01,\n",
       "         8.81427824e-02,  1.67466924e-01,  1.50562525e-01,\n",
       "        -7.52063096e-02],\n",
       "       [ 2.94770122e-01, -2.74756521e-01,  3.26041490e-01,\n",
       "        -1.41754553e-01, -2.14757606e-01, -2.91940182e-01,\n",
       "        -8.04766342e-02,  3.39729071e-01,  1.33409485e-01,\n",
       "        -8.48367065e-02],\n",
       "       [ 8.60327929e-02, -2.31971517e-01, -2.29909301e-01,\n",
       "         1.15025051e-01,  1.74323231e-01,  3.17028403e-01,\n",
       "        -2.99564153e-01, -2.77872860e-01, -3.31115603e-01,\n",
       "         3.47480804e-01],\n",
       "       [-8.30457807e-02, -2.71603733e-01,  2.88642291e-02,\n",
       "        -3.24859381e-01, -2.88815677e-01,  8.70091692e-02,\n",
       "        -3.39641154e-01, -1.06064409e-01, -2.04691753e-01,\n",
       "         3.31194729e-01],\n",
       "       [ 2.82985806e-01, -3.29395324e-01,  3.38947505e-01,\n",
       "        -3.11055183e-01, -3.70540559e-01, -1.78508788e-01,\n",
       "        -1.54852137e-01,  2.76971817e-01,  1.49963319e-01,\n",
       "        -1.12355627e-01],\n",
       "       [ 2.46849895e-01,  3.02497029e-01, -2.66649961e-01,\n",
       "        -2.66891867e-01,  2.22506493e-01, -2.09500611e-01,\n",
       "        -8.27855617e-02,  3.33098382e-01,  3.01658511e-01,\n",
       "        -2.21573904e-01],\n",
       "       [ 2.90179551e-01,  9.74220484e-02, -1.55022904e-01,\n",
       "        -2.04011261e-01, -5.68235107e-02,  2.41916478e-01,\n",
       "        -3.03888381e-01,  1.31691545e-01,  2.45612204e-01,\n",
       "        -5.03743440e-02],\n",
       "       [ 3.04310739e-01,  3.02584529e-01, -2.07236826e-01,\n",
       "        -2.14831069e-01,  2.83117175e-01, -3.10667455e-01,\n",
       "        -1.67438518e-02,  2.61795998e-01,  2.44940192e-01,\n",
       "        -2.81453431e-01],\n",
       "       [-2.63507068e-01,  2.35787570e-01,  2.35560641e-01,\n",
       "         2.79044569e-01,  3.70300055e-01,  2.42498279e-01,\n",
       "        -4.01051074e-01, -3.43128234e-01, -3.44171375e-01,\n",
       "         5.41379629e-03],\n",
       "       [-3.33778560e-01, -2.54142970e-01,  3.39852512e-01,\n",
       "         1.58262655e-01, -2.08767936e-01, -3.13552678e-01,\n",
       "         2.95217305e-01,  2.84854919e-01, -3.19489360e-01,\n",
       "         2.93755203e-01],\n",
       "       [-4.45284098e-02,  3.57496351e-01, -1.37664720e-01,\n",
       "         3.08985978e-01,  3.44591230e-01,  3.59153539e-01,\n",
       "        -2.75231510e-01, -3.61187100e-01, -3.04779053e-01,\n",
       "        -5.11289984e-02],\n",
       "       [ 2.54620135e-01,  2.87434578e-01, -2.26596668e-01,\n",
       "        -2.52661169e-01,  2.75516689e-01, -3.20568383e-01,\n",
       "        -1.25848100e-01,  2.18799129e-01,  2.64874965e-01,\n",
       "        -3.42043221e-01],\n",
       "       [-3.76313657e-01, -3.26550901e-01,  2.85376757e-01,\n",
       "         2.11465508e-01, -2.81778961e-01, -2.08223581e-01,\n",
       "         1.36534944e-01, -6.04284219e-02, -3.88719946e-01,\n",
       "         3.17460060e-01],\n",
       "       [-2.21077070e-01,  6.53859749e-02,  2.22808078e-01,\n",
       "         2.37585530e-01, -5.62070869e-02, -2.56810069e-01,\n",
       "         3.18910778e-01,  2.88306803e-01,  8.11624154e-02,\n",
       "        -3.03640097e-01],\n",
       "       [-2.94776827e-01, -2.46593684e-01,  3.54912788e-01,\n",
       "         2.24729747e-01, -2.18741819e-01, -3.42155606e-01,\n",
       "         2.88599670e-01,  2.01231807e-01, -2.94590265e-01,\n",
       "         2.85064340e-01],\n",
       "       [-2.25475743e-01, -2.28161365e-01, -3.29921514e-01,\n",
       "         2.16007695e-01, -1.65900320e-01, -1.76056057e-01,\n",
       "         1.36633351e-01,  2.01150641e-01, -4.23298217e-04,\n",
       "         4.54828553e-02],\n",
       "       [ 2.89333910e-01,  2.93016434e-01, -5.90687804e-02,\n",
       "        -2.52628416e-01,  2.92143703e-01, -8.78243074e-02,\n",
       "        -1.46283448e-01,  3.55762303e-01,  2.24023923e-01,\n",
       "        -3.09020251e-01],\n",
       "       [ 3.32005471e-01,  8.46400484e-02, -2.46884599e-01,\n",
       "        -1.36407018e-01,  1.73842922e-01,  3.24961096e-01,\n",
       "        -3.60125810e-01, -1.14667788e-03,  1.61950752e-01,\n",
       "         5.39873466e-02],\n",
       "       [-3.57718736e-01, -7.56635591e-02,  3.05204183e-01,\n",
       "        -2.77873814e-01,  7.75808692e-02, -2.82032549e-01,\n",
       "         3.70199114e-01,  2.13330626e-01, -1.35494828e-01,\n",
       "        -2.29550257e-01],\n",
       "       [-3.61090302e-01, -2.24134445e-01,  2.83489138e-01,\n",
       "         1.35124147e-01, -1.87137797e-01, -3.33705008e-01,\n",
       "         2.69095480e-01,  2.82646686e-01, -3.12653720e-01,\n",
       "         2.38916934e-01],\n",
       "       [ 3.27349789e-02, -1.15534663e-02, -2.42078900e-02,\n",
       "        -1.35026574e-02, -9.86969471e-03, -1.06306449e-02,\n",
       "         4.67628278e-02,  6.51412085e-03, -1.97892077e-02,\n",
       "        -2.58724336e-02]], dtype=float32)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_weights()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<keras.layers.embeddings.Embedding at 0x14a3b0890>,\n",
       " <keras.layers.recurrent.LSTM at 0x149de8fd0>,\n",
       " <keras.layers.core.Dense at 0x149503850>]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### iv. Using the trained model to make predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Probably the most interesting thing to do now is use the trained model to make new predictions. For this, we can use the `model.predict_classes()` method. \n",
    "\n",
    "We hope that the model will predict the next word given a seed sequence well, i.e. that it learned about word structure from our poem corpus. For instance, given the seed sequence \"shall be\", we hope the model predicts the correct, observed next words like \"king\", \"broken\", and \"blade\".\n",
    "\n",
    "However, we can't just run `model.predict_classes()` on raw text data like \"shall be\", since the text data has to first be tokenised, assigned to an integer index, and reshaped into the correct array dimensions:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded seed sequence: [3, 5]\n",
      "Formatted encoded seed sequence: [[3 5]]\n"
     ]
    }
   ],
   "source": [
    "seed_sequence = 'shall be'\n",
    "seed_sequence_encoded = tokeniser.texts_to_sequences([seed_sequence])[0]\n",
    "print('Encoded seed sequence: %s' % seed_sequence_encoded)\n",
    "seed_sequence_encoded = np.array(seed_sequence_encoded).reshape(-1,2)\n",
    "print('Formatted encoded seed sequence: %s' % seed_sequence_encoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use the trained model to make a prediction for the next word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction for the next word index: [38]\n",
      "This index corresponds to word: ['king']\n"
     ]
    }
   ],
   "source": [
    "prediction_index = model.predict_classes(seed_sequence_encoded)\n",
    "print('Prediction for the next word index: %s' % prediction_index)\n",
    "print('This index corresponds to word: %s' % tokeniser.sequences_to_texts([prediction_index]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, that looks like a decent prediction for the next word!\n",
    "\n",
    "Rather than just have the 1 best prediction, it would be interesting to see the probabilities assigned to each possible next word. With a bit of manoeuvring we can get these scores out: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>word</th>\n",
       "      <th>probability</th>\n",
       "      <th>rounded_probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>38</td>\n",
       "      <td>38</td>\n",
       "      <td>king</td>\n",
       "      <td>0.253669</td>\n",
       "      <td>0.25367</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33</td>\n",
       "      <td>33</td>\n",
       "      <td>blade</td>\n",
       "      <td>0.168580</td>\n",
       "      <td>0.16858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>28</td>\n",
       "      <td>woken</td>\n",
       "      <td>0.164102</td>\n",
       "      <td>0.16410</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>renewed</td>\n",
       "      <td>0.123998</td>\n",
       "      <td>0.12400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>be</td>\n",
       "      <td>0.071628</td>\n",
       "      <td>0.07163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>shall</td>\n",
       "      <td>0.061355</td>\n",
       "      <td>0.06135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>does</td>\n",
       "      <td>0.039532</td>\n",
       "      <td>0.03953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>that</td>\n",
       "      <td>0.022705</td>\n",
       "      <td>0.02270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31</td>\n",
       "      <td>31</td>\n",
       "      <td>spring</td>\n",
       "      <td>0.021434</td>\n",
       "      <td>0.02143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>37</td>\n",
       "      <td>37</td>\n",
       "      <td>again</td>\n",
       "      <td>0.018060</td>\n",
       "      <td>0.01806</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    index     word  probability  rounded_probability\n",
       "38     38     king     0.253669              0.25367\n",
       "33     33    blade     0.168580              0.16858\n",
       "28     28    woken     0.164102              0.16410\n",
       "32     32  renewed     0.123998              0.12400\n",
       "5       5       be     0.071628              0.07163\n",
       "3       3    shall     0.061355              0.06135\n",
       "8       8     does     0.039532              0.03953\n",
       "4       4     that     0.022705              0.02270\n",
       "31     31   spring     0.021434              0.02143\n",
       "37     37    again     0.018060              0.01806"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "class_indices = list(range(0, vocabulary_size+1))\n",
    "\n",
    "df = pd.DataFrame(list(zip(class_indices, \n",
    "                      [tokeniser.sequences_to_texts([[index]])[0] for index in class_indices],\n",
    "                       model.predict(seed_sequence_encoded)[0],\n",
    "                       np.round(model.predict(seed_sequence_encoded)[0],5))),\n",
    "                  columns=['index', 'word', 'probability', 'rounded_probability'])\n",
    "\n",
    "df.sort_values('probability', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool, it looks like the network does indeed assign the highest probabilities to the 3 words that actually occur in the corpus! It's fun to see that such a small network can produce sensible results on such a small dataset. \n",
    "\n",
    "Let's try another example with a different seed sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded seed sequence: [8, 2]\n",
      "Formatted encoded seed sequence: [[8 2]]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>word</th>\n",
       "      <th>probability</th>\n",
       "      <th>rounded_probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>wither</td>\n",
       "      <td>0.199929</td>\n",
       "      <td>0.19993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>13</td>\n",
       "      <td>glitter</td>\n",
       "      <td>0.165880</td>\n",
       "      <td>0.16588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>not</td>\n",
       "      <td>0.157340</td>\n",
       "      <td>0.15734</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>all</td>\n",
       "      <td>0.134247</td>\n",
       "      <td>0.13425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>those</td>\n",
       "      <td>0.048063</td>\n",
       "      <td>0.04806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>23</td>\n",
       "      <td>reached</td>\n",
       "      <td>0.045011</td>\n",
       "      <td>0.04501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "      <td>by</td>\n",
       "      <td>0.040465</td>\n",
       "      <td>0.04047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>the</td>\n",
       "      <td>0.039344</td>\n",
       "      <td>0.03934</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>deep</td>\n",
       "      <td>0.029354</td>\n",
       "      <td>0.02935</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>17</td>\n",
       "      <td>lost</td>\n",
       "      <td>0.028293</td>\n",
       "      <td>0.02829</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    index     word  probability  rounded_probability\n",
       "20     20   wither     0.199929              0.19993\n",
       "13     13  glitter     0.165880              0.16588\n",
       "2       2      not     0.157340              0.15734\n",
       "6       6      all     0.134247              0.13425\n",
       "14     14    those     0.048063              0.04806\n",
       "23     23  reached     0.045011              0.04501\n",
       "24     24       by     0.040465              0.04047\n",
       "1       1      the     0.039344              0.03934\n",
       "21     21     deep     0.029354              0.02935\n",
       "17     17     lost     0.028293              0.02829"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seed_sequence = 'does not'\n",
    "seed_sequence_encoded = tokeniser.texts_to_sequences([seed_sequence])[0]\n",
    "print('Encoded seed sequence: %s' % seed_sequence_encoded)\n",
    "seed_sequence_encoded = np.array(seed_sequence_encoded).reshape(-1,2)\n",
    "print('Formatted encoded seed sequence: %s' % seed_sequence_encoded)\n",
    "df = pd.DataFrame(list(zip(class_indices, \n",
    "                      [tokeniser.sequences_to_texts([[index]])[0] for index in class_indices],\n",
    "                       model.predict(seed_sequence_encoded)[0],\n",
    "                       np.round(model.predict(seed_sequence_encoded)[0],5))),\n",
    "                  columns=['index', 'word', 'probability', 'rounded_probability'])\n",
    "df.sort_values('probability', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, that also looks correct.\n",
    "\n",
    "Rather than predicting just the next 1 word, would be nice to just let the network write continuous text for us, given some seed sequence starting point. Let's package up the above code into a function that lets us do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def write_text_sequence(seed_sequence,\n",
    "                        length_to_write,\n",
    "                        model, \n",
    "                        tokeniser, \n",
    "                        input_length,\n",
    "                        verbose=True):\n",
    "    \"\"\"\n",
    "    Generates text using a trained language\n",
    "    model and seed sequence.\n",
    "    \"\"\"\n",
    "\n",
    "    print('Using seed sequence: \"%s\"' % seed_sequence)\n",
    "    sequence = seed_sequence\n",
    "    \n",
    "    for i in range(length_to_write):\n",
    "        \n",
    "        # tokenise and encode the seed sequence\n",
    "        encoded_sequence = tokeniser.texts_to_sequences([sequence])[0]\n",
    "        assert len(encoded_sequence)>=input_length, \\\n",
    "            'ERROR: seed sequence must be at least %s words.' % input_length\n",
    "        encoded_sequence = encoded_sequence[-input_length:]\n",
    "        encoded_sequence = np.array(encoded_sequence).reshape(-1,input_length)\n",
    "\n",
    "        # predict the next word index and corresponding word\n",
    "        prediction_index = model.predict_classes(encoded_sequence)\n",
    "        prediction = tokeniser.sequences_to_texts([prediction_index])\n",
    "        \n",
    "        if verbose:\n",
    "            print('Sequence so far: %s' % sequence)\n",
    "            print('Seed sequence encoded: %s' % encoded_sequence)\n",
    "            print('Most likely next word is {0} (index {1})'.format(prediction, prediction_index[0]))\n",
    "\n",
    "        sequence += ' ' + prediction[0]\n",
    "    \n",
    "    print('Output:\\n' + sequence)\n",
    "    \n",
    "#     return sequence\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using seed sequence: \"all that\"\n",
      "Sequence so far: all that\n",
      "Seed sequence encoded: [[6 4]]\n",
      "Most likely next word is ['is'] (index 7)\n",
      "Sequence so far: all that is\n",
      "Seed sequence encoded: [[4 7]]\n",
      "Most likely next word is ['gold'] (index 12)\n",
      "Sequence so far: all that is gold\n",
      "Seed sequence encoded: [[ 7 12]]\n",
      "Most likely next word is ['does'] (index 8)\n",
      "Sequence so far: all that is gold does\n",
      "Seed sequence encoded: [[12  8]]\n",
      "Most likely next word is ['not'] (index 2)\n",
      "Sequence so far: all that is gold does not\n",
      "Seed sequence encoded: [[8 2]]\n",
      "Most likely next word is ['wither'] (index 20)\n",
      "Output:\n",
      "all that is gold does not wither\n"
     ]
    }
   ],
   "source": [
    "write_text_sequence(\"all that\", 5,\n",
    "                    model, tokeniser, \n",
    "                    max_sequence_length-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool, let's write some more text, but let's turn off the verbosity of the function so we just get the final result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using seed sequence: \"the light\"\n",
      "Output:\n",
      "the light from the shadows shall be king\n"
     ]
    }
   ],
   "source": [
    "write_text_sequence(\"the light\", 6,\n",
    "                    model, tokeniser, \n",
    "                    max_sequence_length-1,\n",
    "                    verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's kind of artsy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And again, writing a longer passage this time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using seed sequence: \"ashes are\"\n",
      "Output:\n",
      "ashes are fire the be that a strong shall king be king\n"
     ]
    }
   ],
   "source": [
    "write_text_sequence(\"ashes are\", 10,\n",
    "                    model, tokeniser, \n",
    "                    max_sequence_length-1,\n",
    "                    verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our tiny model only knows the few words in the poem so this is a bit gibberish :) But it's still interesting to see.\n",
    "\n",
    "This is pretty much all there is to a basic language model. Now, let's tackle a real corpus (Game of Thrones) and build a bigger, more powerful model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# II. Building a language model for Game of Thrones text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The technical approach we'll take to building a GoT language model is pretty similar, with the major difference being the dataset. We are going to need access to a lot of GoT text - preferably, both the books and the subtitles from the HBO show. \n",
    "\n",
    "### i. Identifying some datasets\n",
    "\n",
    "Interestingly, there seems to already be a rich ecosystem of technical work surrounding GoT content. \n",
    "\n",
    "Check out projects like:\n",
    "+ The [Network of Thrones](https://networkofthrones.wordpress.com/) blog for network analyses of characters (e.g. which character is the most 'central' to the story?)\n",
    "+ An [API of Ice and Fire](https://anapioficeandfire.com) for grabbing various structured data about the universe\n",
    "+ And [this Reddit post](https://www.reddit.com/r/datasets/comments/769nhw/game_of_thrones_datasets_xpost_from_rfreefolk/) for a list of various datasets compiled about GoT.\n",
    "\n",
    "Maybe it's just me, but even despite these resources, I still couldn't actually find the raw text from the books and TV show. \n",
    "\n",
    "I did eventually come across 2 Kaggle datasets that contained exactly what I wanted:\n",
    "1. [Plain text files of all the books](https://www.kaggle.com/muhammedfathi/game-of-thrones-book-files/download) \n",
    "2. [Subtitle data for the episodes](https://filmora.wondershare.com/video-editing-tips/game-of-thrones-subtitles.html)\n",
    "    \n",
    "A bit of initial manual + regex clean up later, and you get the files included in this repo. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ii. Grabbing all text data from the Game of Thrones books"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we've got a few books in our current directory in .txt format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found these .txt files in the current directory:\n",
      "Book_1_A_Game_of_Thrones.txt\n",
      "Book_2_A_Clash_of_Kings.txt\n",
      "Book_3_A_Storm_of_Swords.txt\n",
      "Book_4_A_Feast_for_Crows.txt\n",
      "Book_5_A_Dance_with_Dragons.txt\n",
      "requirements.txt\n"
     ]
    }
   ],
   "source": [
    "import glob\n",
    "book_txt_files = sorted(glob.glob('*.txt'))\n",
    "print('Found these .txt files in the current directory:', *book_txt_files, sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can write a function to extract all of the text in these files, glue it together, and flatten the resulting list of lists into a single mega GoT list of text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from iteration_utilities import flatten\n",
    "\n",
    "def grab_book_data(txt_files):\n",
    "    \"\"\"\n",
    "    Grabb text data from a set of text files.\n",
    "    \"\"\"\n",
    "\n",
    "    # keep all text segments in this list\n",
    "    all_text_segments = []   \n",
    "    \n",
    "    # iterate over each book file\n",
    "    for txt_file in txt_files:\n",
    "    \n",
    "        print('Extracting text from file \"%s\"...' % txt_file)\n",
    "        # open file\n",
    "        with open(txt_file, 'r') as file:\n",
    "            data = file.read()\n",
    "            print('Found {0} lines of text in this book.'.format(len(data.split('\\n'))))\n",
    "            print('First few lines:\\n %s\\n' % ' '.join(data.split('\\n')[0:5]))  \n",
    "            all_text_segments.append(data)\n",
    "            \n",
    "    return ''.join(list(flatten(all_text_segments)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And use it to put all the book text data in one place:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting text from file \"Book_1_A_Game_of_Thrones.txt\"...\n",
      "Found 14002 lines of text in this book.\n",
      "First few lines:\n",
      " A GAME OF THRONES  PROLOGUE  “We should start back,” Gared urged as the woods began to grow dark around them.\n",
      "\n",
      "Extracting text from file \"Book_2_A_Clash_of_Kings.txt\"...\n",
      "Found 15765 lines of text in this book.\n",
      "First few lines:\n",
      " A CLASH OF KINGS  PROLOGUE  The comet’s tail spread across the dawn, a red slash that bled above the crags of Dragonstone like a wound in the pink and purple sky.\n",
      "\n",
      "Extracting text from file \"Book_3_A_Storm_of_Swords.txt\"...\n",
      "Found 19641 lines of text in this book.\n",
      "First few lines:\n",
      " A STORM OF SWORDS  PROLOGUE  The day was grey and bitter cold, and the dogs would not take the scent.\n",
      "\n",
      "Extracting text from file \"Book_4_A_Feast_for_Crows.txt\"...\n",
      "Found 16225 lines of text in this book.\n",
      "First few lines:\n",
      " A FEAST FOR CROWS  PROLOGUE  Dragons,” said Mollander. He snatched a withered apple off the ground and tossed it hand to hand.\n",
      "\n",
      "Extracting text from file \"Book_5_A_Dance_with_Dragons.txt\"...\n",
      "Found 18889 lines of text in this book.\n",
      "First few lines:\n",
      " A DANCE WITH DRAGONS  PROLOGUE  The night was rank with the smell of man.\n",
      "\n",
      "Extracting text from file \"requirements.txt\"...\n",
      "Found 62 lines of text in this book.\n",
      "First few lines:\n",
      " absl-py==0.8.1 appnope==0.1.0 astor==0.8.0 backcall==0.1.0 cachetools==3.1.1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "book_data = grab_book_data(book_txt_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's quickly summarise the amount of data we're working with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of lines in this corpus: 84579\n",
      "The number of words in this corpus: 1724951\n"
     ]
    }
   ],
   "source": [
    "# count lines and words\n",
    "print('The number of lines in this corpus: {0}\\n'\n",
    "      'The number of words in this corpus: {1}'.format(len(book_data.split('\\n')),\n",
    "                                                       len(book_data.split(' '))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## iii. Grabbing all text data from the Game of Thrones show"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The subtitle data is a bit more complicated to grab because it's in JSON file format, and also frankly the text is a bit messy - there's markup tags, music note symbols, and various other odd non-textual things. \n",
    "\n",
    "We have the following `.json` subtitle files in our current directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found these .json files in the current directory:\n",
      "Season_1_Subtitles.json\n",
      "Season_2_Subtitles.json\n",
      "Season_3_Subtitles.json\n",
      "Season_4_Subtitles.json\n",
      "Season_5_Subtitles.json\n",
      "Season_6_Subtitles.json\n",
      "Season_7_Subtitles.json\n"
     ]
    }
   ],
   "source": [
    "subtitle_json_files = sorted(glob.glob(\"*.json\"))\n",
    "print('Found these .json files in the current directory:', *subtitle_json_files, sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need to write a function to get the data out. The function below will:\n",
    "+ **Iterate** over a given list of json subtitle files, **open** each file and **parse** the json\n",
    "+ **Sort** the subtitles by index. At the moment, the indices are sorted as strings (so, e.g. '1' is followed by '11') so we need to convert the indices to integers and sort them numerically. This is important to get right because otherwise the subtitles are jumbled out of order! \n",
    "+ And finally we **extract** the subtitle text and **append** to a master list (which we reformat by flattening) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "def grab_subtitle_data(subtitle_json_files, verbose=True):\n",
    "    \"\"\"\n",
    "    Grabbing GoT subtitle data from json files.\n",
    "    \"\"\"\n",
    "\n",
    "    # keep all text segments in this list\n",
    "    all_text_segments = []\n",
    "\n",
    "    # iterate over each subtitles file\n",
    "    for season, subtitles_file in enumerate(subtitle_json_files):\n",
    "\n",
    "        # open subtitle file\n",
    "        with open(subtitles_file, 'r') as file:\n",
    "            data = json.load(file)\n",
    "\n",
    "        # iterate over episodes in the season\n",
    "        for episode in data.keys():\n",
    "            episode_data = {int(key):value for key,value in data[episode].items()}\n",
    "            episode_data = sorted(episode_data.items()) # deal with sorting by line (as integer) s\n",
    "            episode_text_segments = list(dict(episode_data).values())\n",
    "            print('Found {0} text segments in Season {1} '\n",
    "                  'Episode \"{2}\".'.format(len(episode_text_segments), \n",
    "                                          season, \n",
    "                                          episode.split('.')[0]))\n",
    "            if verbose:\n",
    "                print('First few segments:\\n%s' % '\\n'.join(episode_text_segments[0:5]))            \n",
    "            all_text_segments.append(episode_text_segments)\n",
    "            \n",
    "    return list(flatten(all_text_segments))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 559 text segments in Season 0 Episode \"Game Of Thrones S01E01 Winter Is Coming\".\n",
      "Found 571 text segments in Season 0 Episode \"Game Of Thrones S01E02 The Kingsroad\".\n",
      "Found 740 text segments in Season 0 Episode \"Game Of Thrones S01E03 Lord Snow\".\n",
      "Found 754 text segments in Season 0 Episode \"Game Of Thrones S01E04 Cripples, Bastards, And Broken Things\".\n",
      "Found 741 text segments in Season 0 Episode \"Game Of Thrones S01E05 The Wolf And The Lion\".\n",
      "Found 583 text segments in Season 0 Episode \"Game Of Thrones S01E06 A Golden Crown\".\n",
      "Found 775 text segments in Season 0 Episode \"Game Of Thrones S01E07 You Win Or You Die\".\n",
      "Found 666 text segments in Season 0 Episode \"Game Of Thrones S01E08 The Pointy End\".\n",
      "Found 679 text segments in Season 0 Episode \"Game Of Thrones S01E09 Baelor\".\n",
      "Found 590 text segments in Season 0 Episode \"Game Of Thrones S01E10 Fire And Blood\".\n",
      "Found 700 text segments in Season 1 Episode \"Game Of Thrones S02E01 The North Remembers\".\n",
      "Found 755 text segments in Season 1 Episode \"Game Of Thrones S02E02 The Night Lands\".\n",
      "Found 654 text segments in Season 1 Episode \"Game Of Thrones S02E03 What Is Dead May Never Die\".\n",
      "Found 619 text segments in Season 1 Episode \"Game Of Thrones S02E04 Garden Of Bones\".\n",
      "Found 781 text segments in Season 1 Episode \"Game Of Thrones S02E05 The Ghost Of Harrenhal\".\n",
      "Found 730 text segments in Season 1 Episode \"Game Of Thrones S02E06 The Old Gods And The New\".\n",
      "Found 762 text segments in Season 1 Episode \"Game Of Thrones S02E07 A Man Without Honor\".\n",
      "Found 775 text segments in Season 1 Episode \"Game Of Thrones S02E08 The Prince Of Winterfell\".\n",
      "Found 640 text segments in Season 1 Episode \"Game Of Thrones S02E09 Blackwater\".\n",
      "Found 641 text segments in Season 1 Episode \"Game Of Thrones S02E10 Valar Morghulis\".\n",
      "Found 637 text segments in Season 2 Episode \"Game Of Thrones S03E01 Valar Dohaeris\".\n",
      "Found 778 text segments in Season 2 Episode \"Game Of Thrones S03E02 Dark Wings, Dark Words\".\n",
      "Found 661 text segments in Season 2 Episode \"Game Of Thrones S03E03 Walk Of Punishment\".\n",
      "Found 703 text segments in Season 2 Episode \"Game Of Thrones S03E04 And Now His Watch Is Ended\".\n",
      "Found 821 text segments in Season 2 Episode \"Game Of Thrones S03E05 Kissed By Fire\".\n",
      "Found 652 text segments in Season 2 Episode \"Game Of Thrones S03E06 The Climb\".\n",
      "Found 714 text segments in Season 2 Episode \"Game Of Thrones S03E07 The Bear And The Maiden Fair\".\n",
      "Found 576 text segments in Season 2 Episode \"Game Of Thrones S03E08 Second Sons\".\n",
      "Found 524 text segments in Season 2 Episode \"Game Of Thrones S03E09 The Rains Of Castamere\".\n",
      "Found 785 text segments in Season 2 Episode \"Game Of Thrones S03E10 Mhysa\".\n",
      "Found 756 text segments in Season 3 Episode \"Game Of Thrones S04E01 Two Swords\".\n",
      "Found 636 text segments in Season 3 Episode \"Game Of Thrones S04E02 The Lion And The Rose\".\n",
      "Found 753 text segments in Season 3 Episode \"Game Of Thrones S04E03 Breaker Of Chains\".\n",
      "Found 634 text segments in Season 3 Episode \"Game Of Thrones S04E04 Oathkeeper\".\n",
      "Found 664 text segments in Season 3 Episode \"Game Of Thrones S04E05 First Of His Name\".\n",
      "Found 622 text segments in Season 3 Episode \"Game Of Thrones S04E06 The Laws Of Gods And Men\".\n",
      "Found 695 text segments in Season 3 Episode \"Game Of Thrones S04E07 Mockingbird\".\n",
      "Found 662 text segments in Season 3 Episode \"Game Of Thrones S04E08 The Mountain And The Viper\".\n",
      "Found 451 text segments in Season 3 Episode \"Game Of Thrones S04E09 The Watchers On The Wall\".\n",
      "Found 613 text segments in Season 3 Episode \"Game Of Thrones S04E10 The Children\".\n",
      "Found 0 text segments in Season 3 Episode \"season4\".\n",
      "Found 631 text segments in Season 4 Episode \"Game Of Thrones S05E01 The Wars To Come\".\n",
      "Found 748 text segments in Season 4 Episode \"Game Of Thrones S05E02 The House Of Black And White\".\n",
      "Found 782 text segments in Season 4 Episode \"Game Of Thrones S05E03 High Sparrow\".\n",
      "Found 601 text segments in Season 4 Episode \"Game Of Thrones S05E04 Sons Of The Harpy\".\n",
      "Found 647 text segments in Season 4 Episode \"Game Of Thrones S05E05 Kill The Boy\".\n",
      "Found 628 text segments in Season 4 Episode \"Game Of Thrones S05E06 Unbowed, Unbent, Unbroken\".\n",
      "Found 700 text segments in Season 4 Episode \"Game Of Thrones S05E07 The Gift\".\n",
      "Found 658 text segments in Season 4 Episode \"Game Of Thrones S05E08 Hardhome\".\n",
      "Found 481 text segments in Season 4 Episode \"Game Of Thrones S05E09 The Dance Of Dragons\".\n",
      "Found 563 text segments in Season 4 Episode \"Game Of Thrones S05E10 Mother's Mercy\".\n",
      "Found 385 text segments in Season 5 Episode \"Game Of Thrones S06E01 The Red Woman\".\n",
      "Found 480 text segments in Season 5 Episode \"Game Of Thrones S06E02 Home\".\n",
      "Found 598 text segments in Season 5 Episode \"Game Of Thrones S06E03 Oathbreaker\".\n",
      "Found 640 text segments in Season 5 Episode \"Game Of Thrones S06E04 Book of the Stranger\".\n",
      "Found 644 text segments in Season 5 Episode \"Game Of Thrones S06E05 The Door\".\n",
      "Found 605 text segments in Season 5 Episode \"Game Of Thrones S06E06 Blood of My Blood\".\n",
      "Found 615 text segments in Season 5 Episode \"Game Of Thrones S06E07 The Broken Man\".\n",
      "Found 661 text segments in Season 5 Episode \"Game Of Thrones S06E08 No One\".\n",
      "Found 446 text segments in Season 5 Episode \"Game Of Thrones S06E09 Battle of the Bastards\".\n",
      "Found 605 text segments in Season 5 Episode \"Game Of Thrones S06E10 The Winds of Winter\".\n",
      "Found 753 text segments in Season 6 Episode \"Game Of Thrones S07E01 Dragonstone\".\n",
      "Found 812 text segments in Season 6 Episode \"Game Of Thrones S07E02 Stormborn\".\n",
      "Found 983 text segments in Season 6 Episode \"Game Of Thrones S07E03 The Queen's Justice\".\n",
      "Found 608 text segments in Season 6 Episode \"Game Of Thrones S07E04 The Spoils Of War\".\n",
      "Found 764 text segments in Season 6 Episode \"Game Of Thrones S07E05 Eastwatch\".\n",
      "Found 796 text segments in Season 6 Episode \"Game Of Thrones S07E06 Beyond The Wall\".\n",
      "Found 958 text segments in Season 6 Episode \"Game Of Thrones S07E07 The Dragon And The Wolf\".\n"
     ]
    }
   ],
   "source": [
    "subtitle_data = grab_subtitle_data(subtitle_json_files, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final array of subtitle data looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Easy, boy.',\n",
       " \"What do you expect? They're savages.\",\n",
       " 'One lot steals a goat from another lot,',\n",
       " \"before you know it they're ripping each other to pieces.\",\n",
       " \"I've never seen wildlings do a thing like this.\"]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subtitle_data[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can summarise the dataset size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of text segments in this corpus: 44844\n",
      "The number of words in this corpus: 244447\n"
     ]
    }
   ],
   "source": [
    "# count lines and words\n",
    "all_subtitle_text = '\\n'.join(subtitle_data)\n",
    "print('The number of text segments in this corpus: {0}\\n'\n",
    "      'The number of words in this corpus: {1}'.format(len(all_subtitle_text.split('\\n')),\n",
    "                                                       len(all_subtitle_text.split(' '))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### iv. Combining the book and subtitle datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can put the book and subtitle data together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "got_data = book_data + all_subtitle_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And report on the size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of lines in the final corpus: 129422\n",
      "The number of words in the final corpus: 1969397\n"
     ]
    }
   ],
   "source": [
    "print('The number of lines in the final corpus: {0}\\n'\n",
    "      'The number of words in the final corpus: {1}'.format(len(got_data.split('\\n')),\n",
    "                                                            len(got_data.split(' '))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's almost 2 million words to play with, which should help our language model tremendously. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### v. Preparing the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The process to make the sequence datasets is the same as before. The only difference is that we'll use longer sequences as our input (`window_size` is now 10), so we're taking into account more text before making our prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The vocabulary size for this corpus is: 30416\n",
      "Generated 2095106 sequences each of length 10.\n",
      "Shape of X matrix: (2095106, 9) and y vector: (2095106,)\n",
      "Shape of X matrix: (2095106, 9) and categorical y matrix: (2095106, 30416)\n"
     ]
    }
   ],
   "source": [
    "# tokenise the data\n",
    "tokeniser = Tokenizer(lower=True, split=' ', char_level=False)\n",
    "tokeniser.fit_on_texts([got_data])\n",
    "vocabulary_size = len(tokeniser.word_index)+1\n",
    "print('The vocabulary size for this corpus is: %s' % vocabulary_size)\n",
    "\n",
    "# encode the corpus using the fitted tokeniser\n",
    "encoded_corpus = tokeniser.texts_to_sequences([got_data])[0]\n",
    "\n",
    "# generate sequences\n",
    "sequences = []\n",
    "window_size = 10\n",
    "for i in range(0, len(encoded_corpus)):\n",
    "    sequences.append(encoded_corpus[i:i+window_size])\n",
    "print('Generated {0} sequences each of length {1}.'.format(len(sequences), window_size))\n",
    "\n",
    "# pad the sequences at the end so each sequence is the same length\n",
    "max_sequence_length = np.max([len(sequence) for sequence in sequences])\n",
    "sequences = pad_sequences(sequences, \n",
    "                          maxlen=max_sequence_length, \n",
    "                          padding='pre')\n",
    "\n",
    "# separate sequences into input arrays X \n",
    "# and the output label vector y\n",
    "X = np.array([seq[0:window_size-1] for seq in sequences])\n",
    "y = np.array([seq[window_size-1] for seq in sequences])\n",
    "print(\"Shape of X matrix: {0} and y vector: {1}\".format(X.shape, y.shape))\n",
    "y = to_categorical(y, num_classes=vocabulary_size)\n",
    "print(\"Shape of X matrix: {0} and categorical y matrix: {1}\".format(X.shape, y.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, our features look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[    5,   972,     6,  3798, 12148,   322,   122,  1131,    62],\n",
       "       [  972,     6,  3798, 12148,   322,   122,  1131,    62,     4],\n",
       "       [    6,  3798, 12148,   322,   122,  1131,    62,     4,  4583],\n",
       "       [ 3798, 12148,   322,   122,  1131,    62,     4,  4583,  1623],\n",
       "       [12148,   322,   122,  1131,    62,     4,  4583,  1623,    17]],\n",
       "      dtype=int32)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And our labels look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 1., 0., ..., 0., 0., 0.]], dtype=float32)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One useful extra step: we should **split the dataset into a train and test set**. The main reason for this is that it will help us get a better estimate of the model's true \"in the wild\" performance, since we can evaluate its performance on data that *wasn't* used in training. We will also use a validation set during training to keep track of progress as the model learns. \n",
    "\n",
    "Evaluating a model on data that was used for training is cheating, since it's already seen that data before, and hence will do unrealistically well when making predictions on it because it has to some extent **overfit** to the training data.\n",
    "\n",
    "We will also shuffle the entries, since otherwise our dataset first contains Book 1, then Book 2, ..., Book 5 then finally the subtitle data, whereas we want the model to learn from each source simultaneously. \n",
    "\n",
    "I would normally do this using sklearn's `train_test_split`, but because our dataset is so large, I wrote a function to do this splitting in batches and using scipy's sparse matrix utilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from scipy import sparse, hstack\n",
    "\n",
    "def batch_train_test_split(X, y, batch_size=10000):\n",
    "    \n",
    "    X_train = []\n",
    "    X_test = []\n",
    "    y_train = []\n",
    "    y_test = []\n",
    "\n",
    "    n_batches = int(np.ceil(X.shape[0]/batch_size))\n",
    "\n",
    "    for batch_index in range(0, n_batches):\n",
    "        print('On batch {0} of {1}...'.format(str(batch_index), str(n_batches)))\n",
    "        start_index = batch_index*batch_size\n",
    "        end_index = start_index+batch_size\n",
    "\n",
    "        # grab the small batch\n",
    "        small_X = X[start_index:end_index]\n",
    "        small_y = y[start_index:end_index]\n",
    "\n",
    "        # do the train test split on this small batch\n",
    "        small_X_train, small_X_test, \\\n",
    "        small_y_train, small_y_test = train_test_split(small_X, small_y, test_size=0.1, shuffle=True)\n",
    "\n",
    "        # append\n",
    "        X_train.append(small_X_train)\n",
    "        X_test.append(small_X_test)\n",
    "        y_train.append(sparse.csr_matrix(small_y_train))\n",
    "        y_test.append(sparse.csr_matrix(small_y_test))\n",
    "\n",
    "    # reformat results\n",
    "    X_train = np.array(list(flatten(X_train)))\n",
    "    X_test = np.array(list(flatten(X_test)))\n",
    "    y_train = sparse.vstack(y_train)\n",
    "    y_test = sparse.vstack(y_test)\n",
    "    \n",
    "    return X_train, X_test, y_train, y_test\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "On batch 0 of 100...\n",
      "On batch 1 of 100...\n",
      "On batch 2 of 100...\n",
      "On batch 3 of 100...\n",
      "On batch 4 of 100...\n",
      "On batch 5 of 100...\n",
      "On batch 6 of 100...\n",
      "On batch 7 of 100...\n",
      "On batch 8 of 100...\n",
      "On batch 9 of 100...\n",
      "On batch 10 of 100...\n",
      "On batch 11 of 100...\n",
      "On batch 12 of 100...\n",
      "On batch 13 of 100...\n",
      "On batch 14 of 100...\n",
      "On batch 15 of 100...\n",
      "On batch 16 of 100...\n",
      "On batch 17 of 100...\n",
      "On batch 18 of 100...\n",
      "On batch 19 of 100...\n",
      "On batch 20 of 100...\n",
      "On batch 21 of 100...\n",
      "On batch 22 of 100...\n",
      "On batch 23 of 100...\n",
      "On batch 24 of 100...\n",
      "On batch 25 of 100...\n",
      "On batch 26 of 100...\n",
      "On batch 27 of 100...\n",
      "On batch 28 of 100...\n",
      "On batch 29 of 100...\n",
      "On batch 30 of 100...\n",
      "On batch 31 of 100...\n",
      "On batch 32 of 100...\n",
      "On batch 33 of 100...\n",
      "On batch 34 of 100...\n",
      "On batch 35 of 100...\n",
      "On batch 36 of 100...\n",
      "On batch 37 of 100...\n",
      "On batch 38 of 100...\n",
      "On batch 39 of 100...\n",
      "On batch 40 of 100...\n",
      "On batch 41 of 100...\n",
      "On batch 42 of 100...\n",
      "On batch 43 of 100...\n",
      "On batch 44 of 100...\n",
      "On batch 45 of 100...\n",
      "On batch 46 of 100...\n",
      "On batch 47 of 100...\n",
      "On batch 48 of 100...\n",
      "On batch 49 of 100...\n",
      "On batch 50 of 100...\n",
      "On batch 51 of 100...\n",
      "On batch 52 of 100...\n",
      "On batch 53 of 100...\n",
      "On batch 54 of 100...\n",
      "On batch 55 of 100...\n",
      "On batch 56 of 100...\n",
      "On batch 57 of 100...\n",
      "On batch 58 of 100...\n",
      "On batch 59 of 100...\n",
      "On batch 60 of 100...\n",
      "On batch 61 of 100...\n",
      "On batch 62 of 100...\n",
      "On batch 63 of 100...\n",
      "On batch 64 of 100...\n",
      "On batch 65 of 100...\n",
      "On batch 66 of 100...\n",
      "On batch 67 of 100...\n",
      "On batch 68 of 100...\n",
      "On batch 69 of 100...\n",
      "On batch 70 of 100...\n",
      "On batch 71 of 100...\n",
      "On batch 72 of 100...\n",
      "On batch 73 of 100...\n",
      "On batch 74 of 100...\n",
      "On batch 75 of 100...\n",
      "On batch 76 of 100...\n",
      "On batch 77 of 100...\n",
      "On batch 78 of 100...\n",
      "On batch 79 of 100...\n",
      "On batch 80 of 100...\n",
      "On batch 81 of 100...\n",
      "On batch 82 of 100...\n",
      "On batch 83 of 100...\n",
      "On batch 84 of 100...\n",
      "On batch 85 of 100...\n",
      "On batch 86 of 100...\n",
      "On batch 87 of 100...\n",
      "On batch 88 of 100...\n",
      "On batch 89 of 100...\n",
      "On batch 90 of 100...\n",
      "On batch 91 of 100...\n",
      "On batch 92 of 100...\n",
      "On batch 93 of 100...\n",
      "On batch 94 of 100...\n",
      "On batch 95 of 100...\n",
      "On batch 96 of 100...\n",
      "On batch 97 of 100...\n",
      "On batch 98 of 100...\n",
      "On batch 99 of 100...\n"
     ]
    }
   ],
   "source": [
    "# X_train, X_test, y_train, y_test = batch_train_test_split(X, y, batch_size=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### vi. Setting up the language model architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, let's build a slightly larger network:\n",
    "\n",
    "![Larger RNN language model](bigger_network.png)\n",
    "\n",
    "The main differences here are:\n",
    "+ Our vocabulary is much larger\n",
    "+ Our word embeddings are bigger (100 rather than 50 dimensions), which should allow for richer representations of word meaning\n",
    "+ We have 2 LSTM layers instead of 1. This should allow the model to learn more complex, hierarchical representations of the text.\n",
    "+ We have added a dense (fully-connected) layer after the LSTM layers for some additional processing capacity (perhaps, again, allowing for higher-level conceptual representations)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `Keras` code, we would build the network as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Embedding(vocabulary_size, 50, input_length=max_sequence_length-1))\n",
    "model.add(LSTM(100, return_sequences=True))\n",
    "model.add(LSTM(100))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(vocabulary_size, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is very similar code to before, but we have reason to think that this network will be much more complex and nuanced than the previous one:\n",
    "+ The dataset we are using is much larger and richer than the toy dataset\n",
    "+ The network we are training is larger and deeper, and should have more expressive power\n",
    "\n",
    "We can summarise the **model structure and parameters**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 9, 50)             1520800   \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 9, 100)            60400     \n",
      "_________________________________________________________________\n",
      "lstm_3 (LSTM)                (None, 100)               80400     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 100)               10100     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 30416)             3072016   \n",
      "=================================================================\n",
      "Total params: 4,743,716\n",
      "Trainable params: 4,743,716\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And compile the finished model and specify some **training settings**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### vii. Training the language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can start the training run by passing the training data to the model. This would take a reasonably long time to train - it would be helpful to have access to a **GPU** to run this on (e.g. via Google Colab, AWS/GCP, your own GPU) to make use of computation **parallelisation** and drastically reduce training time.\n",
    "\n",
    "Since it's a longer training run, we would also ideally want to save some intermediate results while training is happening. One way to do this is using Keras' `ModelCheckpoint` utility. To save some disc space, you can specify that you only want to save a new checkpoint file when something about the model has improved (commonly, validation accuracy or validation loss). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1337429 samples, validate on 148604 samples\n",
      "Epoch 1/50\n",
      "1337429/1337429 [==============================] - 2557s 2ms/step - loss: 6.2683 - accuracy: 0.0981 - val_loss: 6.1418 - val_accuracy: 0.1140\n",
      "\n",
      "Epoch 00001: val_accuracy improved from -inf to 0.11401, saving model to GoT_Language_Model_01_0.114.hdf5\n",
      "Epoch 2/50\n",
      "1337429/1337429 [==============================] - 2569s 2ms/step - loss: 5.8256 - accuracy: 0.1275 - val_loss: 6.0192 - val_accuracy: 0.1246\n",
      "\n",
      "Epoch 00002: val_accuracy improved from 0.11401 to 0.12455, saving model to GoT_Language_Model_02_0.125.hdf5\n",
      "Epoch 3/50\n",
      "1337429/1337429 [==============================] - 2691s 2ms/step - loss: 5.6954 - accuracy: 0.1362 - val_loss: 5.9612 - val_accuracy: 0.1285\n",
      "\n",
      "Epoch 00003: val_accuracy improved from 0.12455 to 0.12846, saving model to GoT_Language_Model_03_0.128.hdf5\n",
      "Epoch 4/50\n",
      "1337429/1337429 [==============================] - 2715s 2ms/step - loss: 5.6139 - accuracy: 0.1421 - val_loss: 5.9227 - val_accuracy: 0.1313\n",
      "\n",
      "Epoch 00004: val_accuracy improved from 0.12846 to 0.13133, saving model to GoT_Language_Model_04_0.131.hdf5\n",
      "Epoch 5/50\n",
      "1337429/1337429 [==============================] - 2727s 2ms/step - loss: 5.5509 - accuracy: 0.1467 - val_loss: 5.9034 - val_accuracy: 0.1345\n",
      "\n",
      "Epoch 00005: val_accuracy improved from 0.13133 to 0.13449, saving model to GoT_Language_Model_05_0.134.hdf5\n",
      "Epoch 6/50\n",
      "1337429/1337429 [==============================] - 2769s 2ms/step - loss: 5.4978 - accuracy: 0.1506 - val_loss: 5.8769 - val_accuracy: 0.1373\n",
      "\n",
      "Epoch 00006: val_accuracy improved from 0.13449 to 0.13726, saving model to GoT_Language_Model_06_0.137.hdf5\n",
      "Epoch 7/50\n",
      "1337429/1337429 [==============================] - 2773s 2ms/step - loss: 5.4490 - accuracy: 0.1541 - val_loss: 5.8643 - val_accuracy: 0.1386\n",
      "\n",
      "Epoch 00007: val_accuracy improved from 0.13726 to 0.13857, saving model to GoT_Language_Model_07_0.139.hdf5\n",
      "Epoch 8/50\n",
      "1337429/1337429 [==============================] - 2750s 2ms/step - loss: 5.4065 - accuracy: 0.1571 - val_loss: 5.8659 - val_accuracy: 0.1395\n",
      "\n",
      "Epoch 00008: val_accuracy improved from 0.13857 to 0.13953, saving model to GoT_Language_Model_08_0.140.hdf5\n",
      "Epoch 9/50\n",
      "1337429/1337429 [==============================] - 2781s 2ms/step - loss: 5.3721 - accuracy: 0.1600 - val_loss: 5.8758 - val_accuracy: 0.1412\n",
      "\n",
      "Epoch 00009: val_accuracy improved from 0.13953 to 0.14123, saving model to GoT_Language_Model_09_0.141.hdf5\n",
      "Epoch 10/50\n",
      "1337429/1337429 [==============================] - 2811s 2ms/step - loss: 5.3443 - accuracy: 0.1626 - val_loss: 5.8995 - val_accuracy: 0.1414\n",
      "\n",
      "Epoch 00010: val_accuracy improved from 0.14123 to 0.14144, saving model to GoT_Language_Model_10_0.141.hdf5\n",
      "Epoch 11/50\n",
      "1337429/1337429 [==============================] - 2784s 2ms/step - loss: 5.3247 - accuracy: 0.1648 - val_loss: 5.9275 - val_accuracy: 0.1423\n",
      "\n",
      "Epoch 00011: val_accuracy improved from 0.14144 to 0.14226, saving model to GoT_Language_Model_11_0.142.hdf5\n",
      "Epoch 12/50\n",
      "1337429/1337429 [==============================] - 2816s 2ms/step - loss: 5.3093 - accuracy: 0.1666 - val_loss: 5.9700 - val_accuracy: 0.1421\n",
      "\n",
      "Epoch 00012: val_accuracy did not improve from 0.14226\n",
      "Epoch 13/50\n",
      "1337429/1337429 [==============================] - 2798s 2ms/step - loss: 5.2988 - accuracy: 0.1680 - val_loss: 6.0055 - val_accuracy: 0.1428\n",
      "\n",
      "Epoch 00013: val_accuracy improved from 0.14226 to 0.14277, saving model to GoT_Language_Model_13_0.143.hdf5\n",
      "Epoch 14/50\n",
      "1337429/1337429 [==============================] - 2831s 2ms/step - loss: 5.2899 - accuracy: 0.1694 - val_loss: 6.0438 - val_accuracy: 0.1425\n",
      "\n",
      "Epoch 00014: val_accuracy did not improve from 0.14277\n",
      "Epoch 15/50\n",
      "1337429/1337429 [==============================] - 2803s 2ms/step - loss: 5.2809 - accuracy: 0.1709 - val_loss: 6.0922 - val_accuracy: 0.1424\n",
      "\n",
      "Epoch 00015: val_accuracy did not improve from 0.14277\n",
      "Epoch 16/50\n",
      "1337429/1337429 [==============================] - 2810s 2ms/step - loss: 5.2721 - accuracy: 0.1717 - val_loss: 6.1428 - val_accuracy: 0.1427\n",
      "\n",
      "Epoch 00016: val_accuracy did not improve from 0.14277\n",
      "Epoch 17/50\n",
      "1337429/1337429 [==============================] - 2829s 2ms/step - loss: 5.2664 - accuracy: 0.1729 - val_loss: 6.1566 - val_accuracy: 0.1413\n",
      "\n",
      "Epoch 00017: val_accuracy did not improve from 0.14277\n",
      "Epoch 18/50\n",
      "1337429/1337429 [==============================] - 2824s 2ms/step - loss: 5.2586 - accuracy: 0.1736 - val_loss: 6.1786 - val_accuracy: 0.1419\n",
      "\n",
      "Epoch 00018: val_accuracy did not improve from 0.14277\n",
      "Epoch 19/50\n",
      "1337429/1337429 [==============================] - 2803s 2ms/step - loss: 5.2501 - accuracy: 0.1746 - val_loss: 6.1953 - val_accuracy: 0.1426\n",
      "\n",
      "Epoch 00019: val_accuracy did not improve from 0.14277\n",
      "Epoch 20/50\n",
      "1337429/1337429 [==============================] - 2802s 2ms/step - loss: 5.2421 - accuracy: 0.1750 - val_loss: 6.2369 - val_accuracy: 0.1426\n",
      "\n",
      "Epoch 00020: val_accuracy did not improve from 0.14277\n",
      "Epoch 21/50\n",
      "1337429/1337429 [==============================] - 2810s 2ms/step - loss: 5.2285 - accuracy: 0.1757 - val_loss: 6.2274 - val_accuracy: 0.1426\n",
      "\n",
      "Epoch 00021: val_accuracy did not improve from 0.14277\n",
      "Epoch 22/50\n",
      "1337429/1337429 [==============================] - 2844s 2ms/step - loss: 5.2184 - accuracy: 0.1763 - val_loss: 6.2459 - val_accuracy: 0.1412\n",
      "\n",
      "Epoch 00022: val_accuracy did not improve from 0.14277\n",
      "Epoch 23/50\n",
      "1337429/1337429 [==============================] - 2832s 2ms/step - loss: 5.2101 - accuracy: 0.1767 - val_loss: 6.2533 - val_accuracy: 0.1414\n",
      "\n",
      "Epoch 00023: val_accuracy did not improve from 0.14277\n",
      "Epoch 24/50\n",
      "1337429/1337429 [==============================] - 2840s 2ms/step - loss: 5.2068 - accuracy: 0.1771 - val_loss: 6.3367 - val_accuracy: 0.1417\n",
      "\n",
      "Epoch 00024: val_accuracy did not improve from 0.14277\n",
      "Epoch 25/50\n",
      "1337429/1337429 [==============================] - 2912s 2ms/step - loss: 5.2045 - accuracy: 0.1777 - val_loss: 6.3325 - val_accuracy: 0.1406\n",
      "\n",
      "Epoch 00025: val_accuracy did not improve from 0.14277\n",
      "Epoch 26/50\n",
      "1337429/1337429 [==============================] - 2964s 2ms/step - loss: 5.1926 - accuracy: 0.1780 - val_loss: 6.3063 - val_accuracy: 0.1414\n",
      "\n",
      "Epoch 00026: val_accuracy did not improve from 0.14277\n",
      "Epoch 27/50\n",
      "1337429/1337429 [==============================] - 2920s 2ms/step - loss: 5.1883 - accuracy: 0.1785 - val_loss: 6.3196 - val_accuracy: 0.1404\n",
      "\n",
      "Epoch 00027: val_accuracy did not improve from 0.14277\n",
      "Epoch 28/50\n",
      "1337429/1337429 [==============================] - 2934s 2ms/step - loss: 5.1897 - accuracy: 0.1789 - val_loss: 6.3803 - val_accuracy: 0.1424\n",
      "\n",
      "Epoch 00028: val_accuracy did not improve from 0.14277\n",
      "Epoch 29/50\n",
      "1337429/1337429 [==============================] - 2777s 2ms/step - loss: 5.1902 - accuracy: 0.1793 - val_loss: 6.4110 - val_accuracy: 0.1399\n",
      "\n",
      "Epoch 00029: val_accuracy did not improve from 0.14277\n",
      "Epoch 30/50\n",
      "1337429/1337429 [==============================] - 2795s 2ms/step - loss: 5.1924 - accuracy: 0.1797 - val_loss: 6.4658 - val_accuracy: 0.1405\n",
      "\n",
      "Epoch 00030: val_accuracy did not improve from 0.14277\n",
      "Epoch 31/50\n",
      "1337429/1337429 [==============================] - 2787s 2ms/step - loss: 5.1866 - accuracy: 0.1800 - val_loss: 6.4761 - val_accuracy: 0.1414\n",
      "\n",
      "Epoch 00031: val_accuracy did not improve from 0.14277\n",
      "Epoch 32/50\n",
      "1337429/1337429 [==============================] - 2825s 2ms/step - loss: 5.1808 - accuracy: 0.1807 - val_loss: 6.4567 - val_accuracy: 0.1417\n",
      "\n",
      "Epoch 00032: val_accuracy did not improve from 0.14277\n",
      "Epoch 33/50\n",
      "1337429/1337429 [==============================] - 2845s 2ms/step - loss: 5.1734 - accuracy: 0.1806 - val_loss: 6.4617 - val_accuracy: 0.1410\n",
      "\n",
      "Epoch 00033: val_accuracy did not improve from 0.14277\n",
      "Epoch 34/50\n",
      "1337429/1337429 [==============================] - 2875s 2ms/step - loss: 5.1643 - accuracy: 0.1810 - val_loss: 6.4347 - val_accuracy: 0.1408\n",
      "\n",
      "Epoch 00034: val_accuracy did not improve from 0.14277\n",
      "Epoch 35/50\n",
      "1337429/1337429 [==============================] - 2890s 2ms/step - loss: 5.1543 - accuracy: 0.1813 - val_loss: 6.4012 - val_accuracy: 0.1405\n",
      "\n",
      "Epoch 00035: val_accuracy did not improve from 0.14277\n",
      "Epoch 36/50\n",
      "1337429/1337429 [==============================] - 2893s 2ms/step - loss: 5.1467 - accuracy: 0.1816 - val_loss: 6.3653 - val_accuracy: 0.1393\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch 00036: val_accuracy did not improve from 0.14277\n",
      "Epoch 37/50\n",
      "1337429/1337429 [==============================] - 2904s 2ms/step - loss: 5.1432 - accuracy: 0.1819 - val_loss: 6.4026 - val_accuracy: 0.1386\n",
      "\n",
      "Epoch 00037: val_accuracy did not improve from 0.14277\n",
      "Epoch 38/50\n",
      "1337429/1337429 [==============================] - 2896s 2ms/step - loss: 5.1429 - accuracy: 0.1820 - val_loss: 6.4595 - val_accuracy: 0.1399\n",
      "\n",
      "Epoch 00038: val_accuracy did not improve from 0.14277\n",
      "Epoch 39/50\n",
      "1337429/1337429 [==============================] - 2931s 2ms/step - loss: 5.1449 - accuracy: 0.1823 - val_loss: 6.4318 - val_accuracy: 0.1395\n",
      "\n",
      "Epoch 00039: val_accuracy did not improve from 0.14277\n",
      "Epoch 40/50\n",
      "1337429/1337429 [==============================] - 2884s 2ms/step - loss: 5.1427 - accuracy: 0.1825 - val_loss: 6.4437 - val_accuracy: 0.1370\n",
      "\n",
      "Epoch 00040: val_accuracy did not improve from 0.14277\n",
      "Epoch 41/50\n",
      "1337429/1337429 [==============================] - 2879s 2ms/step - loss: 5.1444 - accuracy: 0.1829 - val_loss: 6.4307 - val_accuracy: 0.1374\n",
      "\n",
      "Epoch 00041: val_accuracy did not improve from 0.14277\n",
      "Epoch 42/50\n",
      "1337429/1337429 [==============================] - 2924s 2ms/step - loss: 5.1438 - accuracy: 0.1828 - val_loss: 6.4420 - val_accuracy: 0.1397\n",
      "\n",
      "Epoch 00042: val_accuracy did not improve from 0.14277\n",
      "Epoch 43/50\n",
      "1337429/1337429 [==============================] - 2915s 2ms/step - loss: 5.1407 - accuracy: 0.1831 - val_loss: 6.4679 - val_accuracy: 0.1384\n",
      "\n",
      "Epoch 00043: val_accuracy did not improve from 0.14277\n",
      "Epoch 44/50\n",
      "1337429/1337429 [==============================] - 2974s 2ms/step - loss: 5.1433 - accuracy: 0.1834 - val_loss: 6.4883 - val_accuracy: 0.1396\n",
      "\n",
      "Epoch 00044: val_accuracy did not improve from 0.14277\n",
      "Epoch 45/50\n",
      "1337429/1337429 [==============================] - 2980s 2ms/step - loss: 5.1463 - accuracy: 0.1832 - val_loss: 6.5102 - val_accuracy: 0.1396\n",
      "\n",
      "Epoch 00045: val_accuracy did not improve from 0.14277\n",
      "Epoch 46/50\n",
      "1337429/1337429 [==============================] - 2958s 2ms/step - loss: 5.1462 - accuracy: 0.1834 - val_loss: 6.5130 - val_accuracy: 0.1374\n",
      "\n",
      "Epoch 00046: val_accuracy did not improve from 0.14277\n",
      "Epoch 47/50\n",
      "1337429/1337429 [==============================] - 2960s 2ms/step - loss: 5.1469 - accuracy: 0.1834 - val_loss: 6.5486 - val_accuracy: 0.1391\n",
      "\n",
      "Epoch 00047: val_accuracy did not improve from 0.14277\n",
      "Epoch 48/50\n",
      "1337429/1337429 [==============================] - 2982s 2ms/step - loss: 5.1512 - accuracy: 0.1836 - val_loss: 6.5586 - val_accuracy: 0.1397\n",
      "\n",
      "Epoch 00048: val_accuracy did not improve from 0.14277\n",
      "Epoch 49/50\n",
      "1337429/1337429 [==============================] - 3000s 2ms/step - loss: 5.1561 - accuracy: 0.1837 - val_loss: 6.5449 - val_accuracy: 0.1384\n",
      "\n",
      "Epoch 00049: val_accuracy did not improve from 0.14277\n",
      "Epoch 50/50\n",
      "1337429/1337429 [==============================] - 3011s 2ms/step - loss: 5.1568 - accuracy: 0.1841 - val_loss: 6.5513 - val_accuracy: 0.1391\n",
      "\n",
      "Epoch 00050: val_accuracy did not improve from 0.14277\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x16bab2ad0>"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.callbacks import ModelCheckpoint\n",
    "\n",
    "checkpoint_filename=\"GoT_Language_Model_{epoch:02d}_{val_accuracy:.3f}.hdf5\"\n",
    "checkpoint = ModelCheckpoint(checkpoint_filename, \n",
    "                             monitor='val_accuracy', \n",
    "                             save_best_only=True, \n",
    "                             mode='max',  # 'best' file maximises validation_accuracy \n",
    "                             verbose=1, )\n",
    "model.fit(X_train, y_train, epochs=50, validation_split=0.1, callbacks=[checkpoint])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"final_trained_GoT_language_model.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now, to save time, I will just **load a model** that I already trained. \n",
    "\n",
    "For reference, this model was really accessible to train - it was trained overnight on my MacBook, so there's no special GPU supercomputer involved. The model was still improving quite rapidly at that point, so we would see even better performance if the model were given enough time to reach **convergence** (\"finish\" learning, or at least hit serious diminishing returns)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    }
   ],
   "source": [
    "from keras.models import load_model\n",
    "\n",
    "loaded_model = load_model('final_trained_GoT_language_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### viii. Exploring our Game of Thrones language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's see if we can have the language model write some Game of Thrones text for us.\n",
    "\n",
    "We can use the same function as before to continuously feed in a seed sequence to the model, generate one word, and then append the generated word to the seed sequence. In this way, the model uses its own previous output as input to itself in the future. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using seed sequence: \"You would not believe the start of the story\"\n",
      "Output:\n",
      "You would not believe the start of the story ” the king said “i am a man of the night’s watch ” he said “i am a man of the andals and the rhoynar and the first men\n"
     ]
    }
   ],
   "source": [
    "write_text_sequence(\"You would not believe the start of the story\", 29,\n",
    "                    loaded_model, tokeniser, \n",
    "                    max_sequence_length-1,\n",
    "                    verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using seed sequence: \"A dragon, the dead, and Tyrion walk into a\"\n",
      "Output:\n",
      "A dragon, the dead, and Tyrion walk into a great roast of rubies the wind was the same and the way he had been a man of the harpy\n"
     ]
    }
   ],
   "source": [
    "write_text_sequence(\"A dragon, the dead, and Tyrion walk into a\", 20,\n",
    "                    loaded_model, tokeniser, \n",
    "                    max_sequence_length-1,\n",
    "                    verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using seed sequence: \"Daenerys of the House Targaryen, the First of Her Name, The Unburnt\"\n",
      "Output:\n",
      "Daenerys of the House Targaryen, the First of Her Name, The Unburnt bastard of the night’s watch had been seated in the middle of the night the wind had been a dozen times\n"
     ]
    }
   ],
   "source": [
    "write_text_sequence(\"Daenerys of the House Targaryen, the First of Her Name, The Unburnt\", 21,\n",
    "                    loaded_model, tokeniser, \n",
    "                    max_sequence_length-1,\n",
    "                    verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uhh.. it's clear from these samples that the model has clearly learned something about both English language structure and GoT content, but admittedly it's still a bit clunky. \n",
    "\n",
    "Check out the section at the bottom for suggestions on how to improve the model. \n",
    "\n",
    "This all looks like a decent start - I especially like that the Lannisters are originally Northerners. Other segments sound like weird GoT beat poetry, and I can almost hear the soft accompanying bongo beats.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ix. Generating more creative output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you run the model generatively and write longer stories, you'll see that it can sometimes get stuck in a loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using seed sequence: \"I would not have expected that Ned and Jon\"\n",
      "Output:\n",
      "I would not have expected that Ned and Jon man within to he my the ser her man a torn ” the beard and the off of the wasn’t her man people ” the wasn’t and the off her man she’d of the polished and the swayed and the think there of the polished and the polished and the swayed and the think how the sansa king his a can of the “were and the dish of the live hair of the insist the great rodrik of while and the forgiveness wrath of the deeply blood the blood of the braid hot his a fighting there and the handful\n"
     ]
    }
   ],
   "source": [
    "write_text_sequence(\"I would not have expected that Ned and Jon\", 100,\n",
    "                    loaded_model, tokeniser, \n",
    "                    max_sequence_length-1,\n",
    "                    verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It really likes the polished and the swayed!\n",
    "\n",
    "This is because our function `write_text_sequences` will always greedily choose the most probable next word as the next token as it generates text. This is the cause of these repetitive loops. \n",
    "\n",
    "Ideally, we want to give the model a bit more space to be creative than this. The easiest way to do this is instead of using the **most probable next word** as our prediction, we can **sample from all possible words proportionally to their probability**. This will help introduce some fun linguistic variety into our generated text. \n",
    "\n",
    "The most probable words are still, of course, most likely to be chosen, but there is now space for less probable words to be used as well. We are trading off (potentially rigid) local correctness for (potentially noisy) creativity. \n",
    "\n",
    "We can modify our `write_text_sequences` to have an option to use this probabilistic sampling approach:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def write_text_sequence(seed_sequence,\n",
    "                        length_to_write,\n",
    "                        model, \n",
    "                        tokeniser, \n",
    "                        input_length,\n",
    "                        verbose=True,\n",
    "                        use_sampling=True):\n",
    "    \"\"\"\n",
    "    Generates text using a trained language\n",
    "    model and seed sequence.\n",
    "    \"\"\"\n",
    "\n",
    "    print('Using seed sequence: \"%s\"' % seed_sequence)\n",
    "    sequence = seed_sequence\n",
    "    \n",
    "    for i in range(length_to_write):\n",
    "        \n",
    "        # tokenise and encode the seed sequence\n",
    "        encoded_sequence = tokeniser.texts_to_sequences([sequence])[0]\n",
    "        assert len(encoded_sequence)>=input_length, \\\n",
    "            'ERROR: seed sequence must be at least %s words.' % input_length\n",
    "        encoded_sequence = encoded_sequence[-input_length:]\n",
    "        encoded_sequence = np.array(encoded_sequence).reshape(-1,input_length)\n",
    "\n",
    "        # predict the next word index and corresponding word\n",
    "        if use_sampling:\n",
    "            next_word_probabilities = model.predict_proba(encoded_sequence)[0]\n",
    "            next_word_indices = range(0, vocabulary_size)\n",
    "            prediction_index = np.random.choice(next_word_indices, size=1, p=next_word_probabilities)             \n",
    "        else: \n",
    "            prediction_index = model.predict_classes(encoded_sequence)\n",
    "        \n",
    "        # convert prediction index to actual word\n",
    "        prediction = tokeniser.sequences_to_texts([prediction_index])\n",
    "        \n",
    "        if verbose:\n",
    "            print('Sequence so far: %s' % sequence)\n",
    "            print('Seed sequence encoded: %s' % encoded_sequence)\n",
    "            print('Most likely next word is {0} (index {1})'.format(prediction, prediction_index[0]))\n",
    "\n",
    "        # append prediction to the sequence\n",
    "        sequence += ' ' + prediction[0]\n",
    "    \n",
    "    print('Output:\\n' + sequence)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if this addition helps us get out of the infinite loop of the Andals "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using seed sequence: \"I would not have expected that Ned and Jon\"\n",
      "Output:\n",
      "I would not have expected that Ned and Jon snow the three who would drive them on the canals “you call him your fight for an instant i dreamed so near the blade in astonishment ” meera leaned back to the door and found a tyrion nails obvious softly her face covered as if he’d told jon snow and some to cold now another handed of miles south warm off the kitchens and three headed dragon out of the lesser but the seven eyed road was their ordinary mouse galloping up beneath it he’d done as much to land a flagon of laughing catelyn thought the cook thought “the steward will have the duty his brothers are enough if you find much to bring our hands away ” margaery could not go out these galleys above the sword until they didn’t you what old bolts of feathers “you have only the outriders shall you go on off ” she walked forward on the laces in the wet earth of slaughter but he served worse than far to the great hall of the old maester through wind “blood of your stump ” tap illyrio told ser gregor “hodor hodor hodor hodor hodor hodor hodor will send more same in the\n"
     ]
    }
   ],
   "source": [
    "write_text_sequence(\"I would not have expected that Ned and Jon\", 200,\n",
    "                    loaded_model, tokeniser, \n",
    "                    max_sequence_length-1,\n",
    "                    verbose=False, \n",
    "                    use_sampling=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well.. we're definitely not trapped anymore! And we get a nice hodor sequence at the end (which is really what we're all here for). But now the text sounds absolutely bonkers. Let's see if we can put some breaks on this thing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### x. Generating more creative (but controlled) output \n",
    "\n",
    "The main way of controlling how creative or random these sampling-based predictions are is by using a hyperparameter called `temperature`. Essentially:\n",
    "+ **Higher temperature** will emphasise the least likely predictions in a distribution - less likely predictions will have their probabilities increased. To remember this, think of \"hot\" = more randomness, just like with higher physical temperature leading to more random molecular motion. \n",
    "+ **Lower temperature** will downplay the less likely predictions. At the lowest temperatures, we are only ever considering the most likely prediction (our sampling starts to function like an `argmax` and we go back to the greedy approach)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can write a function to do this scaling of probabilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_temp_to_softmax_probs(probs, temp, verbose=False):\n",
    "    \"\"\"\n",
    "    Rescales softmax probabilities using some given temperature.\n",
    "    \"\"\"\n",
    "    \n",
    "    # add a very small number to probabilities\n",
    "    # to avoid taking log of zero later (undefined) \n",
    "    epsilon = 10e-16 \n",
    "    probs = probs + epsilon\n",
    "\n",
    "    # take logs of probabilities\n",
    "    log_probs = np.log(probs)\n",
    "    \n",
    "    # the crucial step - divide the log probabilities by temperature\n",
    "    scaled_log_probs = log_probs / temp \n",
    "\n",
    "    # undo logging to get back to probabilities\n",
    "    new_probs = np.exp(scaled_log_probs) \n",
    "\n",
    "    # and renormalise so that probabilities sum to 1\n",
    "    normalised_probs = new_probs / np.sum(new_probs)\n",
    "    \n",
    "    if verbose:\n",
    "        print('1. Original probabilities:\\n%s\\n' % probs)\n",
    "        print('2. Log of probabilities:\\n%s\\n' % log_probs)\n",
    "        print('3. Temperature scaled log of probabilities:\\n%s\\n' % scaled_log_probs)\n",
    "        print('4. Back to pseudo-probabilities by undoing logging:\\n%s\\n' % new_probs)\n",
    "        print('5. Final normalised probabilities:\\n%s\\n' % normalised_probs)\n",
    "\n",
    "    return normalised_probs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check out how temperature scaling of probability arrays happens by testing out this function in verbose mode. Let's scale the array `np.array([0.8, 0.1, 0.05, 0.05])` using different temperatures:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### temperature=1 (should do nothing at all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Original probabilities:\n",
      "[0.8  0.1  0.05 0.05]\n",
      "\n",
      "2. Log of probabilities:\n",
      "[-0.22314355 -2.30258509 -2.99573227 -2.99573227]\n",
      "\n",
      "3. Temperature scaled log of probabilities:\n",
      "[-0.22314355 -2.30258509 -2.99573227 -2.99573227]\n",
      "\n",
      "4. Back to pseudo-probabilities by undoing logging:\n",
      "[0.8  0.1  0.05 0.05]\n",
      "\n",
      "5. Final normalised probabilities:\n",
      "[0.8  0.1  0.05 0.05]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "_ = apply_temp_to_softmax_probs(np.array([0.8, 0.1, 0.05, 0.05]), \n",
    "                            temp=1, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, it's good to know that using a temperature of 1 does nothing at all to the probabilities. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### temperature=10 (should boost low probabilities and introduce more randomness)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 285,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Original probabilities:\n",
      "[0.8  0.1  0.05 0.05]\n",
      "\n",
      "2. Log of probabilities:\n",
      "[-0.22314355 -2.30258509 -2.99573227 -2.99573227]\n",
      "\n",
      "3. Temperature scaled log of probabilities:\n",
      "[-0.02231436 -0.23025851 -0.29957323 -0.29957323]\n",
      "\n",
      "4. Back to pseudo-probabilities by undoing logging:\n",
      "[0.97793277 0.79432823 0.74113445 0.74113445]\n",
      "\n",
      "5. Final normalised probabilities:\n",
      "[0.30048357 0.2440685  0.22772396 0.22772396]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "_ = apply_temp_to_softmax_probs(np.array([0.8, 0.1, 0.05, 0.05]), \n",
    "                                temp=10, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A high temperature of 10 really amplifies those low probabilities!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### temperature=0.1 (should dampen out lower probabilities and boost already high probabilities)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 286,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Original probabilities:\n",
      "[0.8  0.1  0.05 0.05]\n",
      "\n",
      "2. Log of probabilities:\n",
      "[-0.22314355 -2.30258509 -2.99573227 -2.99573227]\n",
      "\n",
      "3. Temperature scaled log of probabilities:\n",
      "[ -2.23143551 -23.02585093 -29.95732274 -29.95732274]\n",
      "\n",
      "4. Back to pseudo-probabilities by undoing logging:\n",
      "[1.07374182e-01 1.00000000e-10 9.76562500e-14 9.76562500e-14]\n",
      "\n",
      "5. Final normalised probabilities:\n",
      "[9.99999999e-01 9.31322574e-10 9.09494701e-13 9.09494701e-13]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "_ = apply_temp_to_softmax_probs(np.array([0.8, 0.1, 0.05, 0.05]), \n",
    "                                temp=0.1, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a low temperature of 0.1 really freezes down those low probabilities, they are practically 0. \n",
    "\n",
    "How come the maths works? Essentially:\n",
    "+ Probabilities that are already big don't have big (negative) logarithms, so scaling them by multiplying/dividing by temperature won't make that much of a difference.\n",
    "+ But small probabilities have very big (negative) logarithms, so scaling them by multiplying/dividing by temperature can hugely change their values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add 1 line to our `write_text_sequence` function to make use of temperature (line 29):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [],
   "source": [
    "def write_text_sequence(seed_sequence,\n",
    "                        length_to_write,\n",
    "                        model, \n",
    "                        tokeniser, \n",
    "                        input_length,\n",
    "                        verbose=True,\n",
    "                        use_sampling=True, \n",
    "                        temperature=1):\n",
    "    \"\"\"\n",
    "    Generates text using a trained language\n",
    "    model and seed sequence.\n",
    "    \"\"\"\n",
    "\n",
    "    print('Using seed sequence: \"%s\"' % seed_sequence)\n",
    "    sequence = seed_sequence\n",
    "    \n",
    "    for i in range(length_to_write):\n",
    "        \n",
    "        # tokenise and encode the seed sequence\n",
    "        encoded_sequence = tokeniser.texts_to_sequences([sequence])[0]\n",
    "        assert len(encoded_sequence)>=input_length, \\\n",
    "            'ERROR: seed sequence must be at least %s words.' % input_length\n",
    "        encoded_sequence = encoded_sequence[-input_length:]\n",
    "        encoded_sequence = np.array(encoded_sequence).reshape(-1,input_length)\n",
    "\n",
    "        # predict the next word index and corresponding word\n",
    "        if use_sampling:\n",
    "            next_word_probabilities = model.predict_proba(encoded_sequence)[0]\n",
    "            next_word_probabilities = apply_temp_to_softmax_probs(next_word_probabilities, temperature)\n",
    "            next_word_indices = range(0, vocabulary_size)\n",
    "            prediction_index = np.random.choice(next_word_indices, size=1, p=next_word_probabilities)             \n",
    "        else: \n",
    "            prediction_index = model.predict_classes(encoded_sequence)\n",
    "        \n",
    "        # convert prediction index to actual word\n",
    "        prediction = tokeniser.sequences_to_texts([prediction_index])\n",
    "        \n",
    "        if verbose:\n",
    "            print('Sequence so far: %s' % sequence)\n",
    "            print('Seed sequence encoded: %s' % encoded_sequence)\n",
    "            print('Most likely next word is {0} (index {1})'.format(prediction, prediction_index[0]))\n",
    "\n",
    "        # append prediction to the sequence\n",
    "        sequence += ' ' + prediction[0]\n",
    "    \n",
    "    print('Output:\\n' + sequence)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can control the creativity level of the text generation by changing the value of one argument:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predictable text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using seed sequence: \"A new adventure starring Varys, The Hound, and Ygritte started by\"\n",
      "Output:\n",
      "A new adventure starring Varys, The Hound, and Ygritte started by the power of his blood and growled the girl had been a man grown through the womb of the city and he was and see your sister ” he explained\n"
     ]
    }
   ],
   "source": [
    "write_text_sequence(\"A new adventure starring Varys, The Hound, and Ygritte started by\", 30,\n",
    "                    loaded_model, tokeniser, \n",
    "                    max_sequence_length-1,\n",
    "                    verbose=False, \n",
    "                    use_sampling=True, \n",
    "                    temperature=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Normal text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using seed sequence: \"A new adventure starring Varys, The Hound, and Ygritte started by\"\n",
      "Output:\n",
      "A new adventure starring Varys, The Hound, and Ygritte started by the rain in the arm of ned’s chair the very hand of one of the hungry man before faint squeezing every of the days himself lord blackwood waited tonight itself\n"
     ]
    }
   ],
   "source": [
    "write_text_sequence(\"A new adventure starring Varys, The Hound, and Ygritte started by\", 30,\n",
    "                    loaded_model, tokeniser, \n",
    "                    max_sequence_length-1,\n",
    "                    verbose=False, \n",
    "                    use_sampling=True, \n",
    "                    temperature=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mental text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using seed sequence: \"A new adventure starring Varys, The Hound, and Ygritte started by\"\n",
      "Output:\n",
      "A new adventure starring Varys, The Hound, and Ygritte started by robert’s snow asking when which setting oznak zo weirwoods shook down part suffice north city’s castle blare a true purpose wizard “always maegi conquer vargo hoat’s sam pleased maddy miss\n"
     ]
    }
   ],
   "source": [
    "write_text_sequence(\"A new adventure starring Varys, The Hound, and Ygritte started by\", 30,\n",
    "                    model, tokeniser, \n",
    "                    max_sequence_length-1,\n",
    "                    verbose=False, \n",
    "                    use_sampling=True, \n",
    "                    temperature=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### xi. Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I hope this project gave you a taste of language models, have a play around with the language model yourself! \n",
    "\n",
    "That's all for this tutorial for now. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# III. Suggested Extensions\n",
    "\n",
    "Here are some suggestions for extending this work in order to build a more serious and performant Game of Thrones language model:\n",
    "\n",
    "1. **Data**: Spend more time cleaning up the text corpus, there is definitely some weird stuff in there (e.g. I saw markup tags in the subtitle data)\n",
    "2. **Data**: Use all of the data, and perhaps think about grabbing more data (maybe by scraping some of the fan Wikis)\n",
    "3. **Data**: Here, we used lower case words and ommitted punctuation entirely. This makes the modelling task easier for the model (fewer tokens to deal with), but the output doesn't read as nicely. Try to complete the task by sticking to as natural of language as possible, with little clean up or omission. \n",
    "3. **Representation**: Use **pre-trained word embeddings** (e.g. FastText, GloVe, Word2Vec) and possibly update them during training\n",
    "4. **Representation**: Think about using **sub-word tokenisation** rather than word-based tokenisation\n",
    "8. **Modelling**: **Train for longer**, until convergence :) Monitor for overfitting using a validation set to early stop. \n",
    "5. **Modelling**: Look into using **regularisation techniques** (dropout, weight penalties) to improve model performance and generalisability\n",
    "6. **Modelling**: Experiment with different numbers of layers, sizes, activation functions, initialisation approaches, etc.\n",
    "7. **Modelling**: Optimise some of the **hyperparameters** in the model (learning rate, momentum, batch sizes)\n",
    "9. **Modelling**: Forget RNNs for language modelling completely and jump on the **Transformer hype train** ([choo](https://paperswithcode.com/task/language-modelling) [choo!](https://arxiv.org/abs/1904.09408)). \n",
    "10. **Modelling**: Try downloading a **pre-trained language model** (like **Google AI's BERT** or **OpenAI's GPT models** or **Carnegie Mellon/Google Brain's XLNet**) and fine-tuning it to Game of Thrones text. This is likely to give the easiest, biggest gains, since these models are pre-trained on massive corpora with a huge amount of GPUs. \n",
    "10. **Visualisation**: Try using **Tensorboard** to visualise the progression of model training and diagnose any weird behaviour. \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}