Ch10_Recurrent_Neural_Networks/Deep_Recurrent_Neural_Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Recurrent Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Up to now, we only discussed recurrent neural networks with a single unidirectional hidden layer. In it the\n",
    "specific functional form of how latent variables and observations interact was rather arbitrary. This isn’t\n",
    "a big problem as long as we have enough flexibility to model different types of interactions. With a single\n",
    "layer, however, this can be quite challenging. In the case of the perceptron we fixed this problem by adding\n",
    "more layers. Within RNNs this is a bit more tricky, since we first need to decide how and where to add extra\n",
    "nonlinearity. Our discussion below focuses primarily on LSTMs but it applies to other sequence models, too.\n",
    "   \n",
    "   • We could add extra nonlinearity to the gating mechansims. That is, instead of using a single perceptron\n",
    "     we could use multiple layers. This leaves the mechanism of the LSTM unchanged. Instead it makes\n",
    "     it more sophisticated. This would make sense if we were led to believe that the LSTM mechanism\n",
    "     describes some form of universal truth of how latent variable autoregressive models work.\n",
    "   \n",
    "   • We could stack multiple layers of LSTMs on top of each other. This results in a mechanism that is\n",
    "     more flexible, due to the combination of several simple layers. In particular, data might be relevant\n",
    "     at different levels of the stack. For instance, we might want to keep high-level data about financial\n",
    "     market conditions (bear or bull market) available at a high level, whereas at a lower level we only\n",
    "     record shorter-term temporal dynamics.\n",
    "    \n",
    "Beyond all this abstract discussion it is probably easiest to understand the family of models we are interested\n",
    "in by reviewing the diagram below. It describes a deep recurrent neural network with L hidden layers. Each\n",
    "hidden state is continuously passed to the next time step of the current layer and the next layer of the current\n",
    "time step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"227pt\" version=\"1.1\" viewBox=\"0 0 221 227\" width=\"221pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M 6.25 -6.015625 L 4.421875 -6.015625 L 4.421875 -5.796875 L 4.609375 -5.78125 C 4.828125 -5.765625 4.90625 -5.703125 4.90625 -5.5625 C 4.90625 -5.40625 4.75 -5.171875 4.28125 -4.6875 L 3.4375 -3.78125 L 3.28125 -4.28125 C 3.078125 -4.9375 2.96875 -5.34375 2.96875 -5.46875 C 2.96875 -5.671875 3.078125 -5.734375 3.375 -5.765625 L 3.640625 -5.796875 L 3.640625 -6.015625 L 0.890625 -6.015625 L 0.890625 -5.796875 C 1.4375 -5.734375 1.578125 -5.703125 1.734375 -5.203125 L 2.46875 -2.78125 L 1.859375 -2.109375 C 0.515625 -0.640625 0.28125 -0.3125 -0.21875 -0.21875 L -0.21875 0 L 1.703125 0 L 1.703125 -0.21875 C 1.3125 -0.265625 1.140625 -0.375 1.140625 -0.578125 C 1.140625 -0.703125 1.21875 -0.859375 1.4375 -1.078125 L 2.59375 -2.375 L 2.78125 -1.84375 C 3.015625 -1.140625 3.109375 -0.765625 3.109375 -0.5625 C 3.109375 -0.328125 3.03125 -0.265625 2.515625 -0.21875 L 2.515625 0 L 5.28125 0 L 5.28125 -0.21875 C 4.625 -0.296875 4.484375 -0.5 4.234375 -1.265625 L 3.5625 -3.40625 L 5.359375 -5.25 C 5.734375 -5.65625 5.921875 -5.75 6.25 -5.796875 Z M 6.25 -6.015625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-2\" overflow=\"visible\">\n",
       "<path d=\"M 7.1875 -6.015625 L 4.515625 -6.015625 L 4.515625 -5.796875 L 4.796875 -5.765625 C 5.078125 -5.71875 5.171875 -5.640625 5.171875 -5.46875 C 5.171875 -5.359375 5.125 -5.078125 5.078125 -4.90625 L 4.65625 -3.359375 L 2.5 -3.359375 L 2.96875 -5.078125 C 3.125 -5.65625 3.296875 -5.78125 3.890625 -5.796875 L 3.890625 -6.015625 L 1.046875 -6.015625 L 1.046875 -5.796875 L 1.328125 -5.765625 C 1.609375 -5.71875 1.703125 -5.640625 1.703125 -5.4375 C 1.703125 -5.328125 1.671875 -5.171875 1.609375 -4.90625 L 0.515625 -0.890625 C 0.375 -0.375 0.234375 -0.28125 -0.21875 -0.21875 L -0.21875 0 L 2.4375 0 L 2.4375 -0.21875 C 1.859375 -0.265625 1.796875 -0.453125 1.796875 -0.578125 C 1.796875 -0.71875 1.8125 -0.859375 1.90625 -1.1875 L 2.390625 -2.96875 L 4.546875 -2.96875 L 3.984375 -0.9375 C 3.828125 -0.375 3.671875 -0.265625 3.0625 -0.21875 L 3.0625 0 L 5.90625 0 L 5.90625 -0.21875 C 5.421875 -0.25 5.25 -0.328125 5.25 -0.59375 C 5.25 -0.734375 5.3125 -0.9375 5.453125 -1.453125 L 6.453125 -5.125 C 6.578125 -5.609375 6.75 -5.796875 7.1875 -5.796875 Z M 7.1875 -6.015625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-3\" overflow=\"visible\">\n",
       "<path d=\"M 6.21875 -4 C 6.21875 -5.3125 5.28125 -6.171875 4.046875 -6.171875 C 3.28125 -6.171875 2.5625 -5.859375 1.921875 -5.34375 C 0.921875 -4.53125 0.25 -3.203125 0.25 -1.984375 C 0.25 -0.59375 1.09375 0.15625 2.359375 0.15625 C 3.203125 0.15625 3.96875 -0.140625 4.59375 -0.703125 C 5.5625 -1.5625 6.21875 -2.84375 6.21875 -4 Z M 4.921875 -4.75 C 4.921875 -3.734375 4.46875 -2.0625 3.875 -1.171875 C 3.421875 -0.515625 3.03125 -0.140625 2.453125 -0.140625 C 1.890625 -0.140625 1.53125 -0.609375 1.53125 -1.28125 C 1.53125 -2.359375 2.015625 -3.875 2.5625 -4.78125 C 3 -5.53125 3.453125 -5.859375 4.015625 -5.859375 C 4.546875 -5.859375 4.921875 -5.5 4.921875 -4.75 Z M 4.921875 -4.75 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-1\" overflow=\"visible\">\n",
       "<path d=\"M 4.4375 -4.578125 L 0.703125 -4.578125 L 0.40625 -3.484375 L 0.53125 -3.453125 C 0.875 -4.21875 1.09375 -4.328125 2.203125 -4.328125 L 1.203125 -0.640625 C 1.09375 -0.265625 0.9375 -0.15625 0.453125 -0.109375 L 0.453125 0 L 2.46875 0 L 2.46875 -0.109375 L 2.21875 -0.140625 C 1.9375 -0.15625 1.859375 -0.21875 1.859375 -0.4375 C 1.859375 -0.59375 1.90625 -0.75 1.953125 -0.890625 L 2.90625 -4.328125 L 3.3125 -4.328125 C 3.765625 -4.328125 4.03125 -4.125 4.03125 -3.765625 C 4.03125 -3.671875 4.015625 -3.5625 4 -3.4375 L 4.125 -3.421875 Z M 4.4375 -4.578125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.703125 -3.546875 C 0.953125 -4.015625 1.3125 -4.21875 1.734375 -4.21875 C 2.28125 -4.21875 2.5625 -3.859375 2.5625 -3.375 C 2.5625 -3.21875 2.546875 -3.09375 2.5 -3 C 2.328125 -2.625 2.015625 -2.171875 1.546875 -1.6875 L 0.078125 -0.125 L 0.078125 0 L 2.5625 0 L 2.90625 -0.9375 L 2.796875 -0.984375 C 2.625 -0.640625 2.484375 -0.53125 2.078125 -0.53125 L 0.8125 -0.53125 L 0.8125 -0.5625 L 2.125 -1.90625 C 2.953125 -2.765625 3.15625 -3.078125 3.15625 -3.546875 C 3.15625 -4.21875 2.703125 -4.734375 2.03125 -4.734375 C 1.703125 -4.734375 1.40625 -4.640625 1.140625 -4.4375 C 0.890625 -4.25 0.734375 -4.015625 0.5625 -3.59375 Z M 0.703125 -3.546875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-3\" overflow=\"visible\">\n",
       "<path d=\"M 2.8125 -4.734375 C 2.265625 -4.625 2.234375 -4.640625 1.625 -4.5 L 1.46875 -4.46875 L 1.46875 -4.359375 C 1.59375 -4.375 1.75 -4.375 1.875 -4.375 L 1.890625 -4.375 C 2.046875 -4.375 2.140625 -4.296875 2.140625 -4.15625 C 2.140625 -4.109375 2.125 -4.03125 2.09375 -3.90625 L 1.140625 -0.484375 C 1.0625 -0.21875 0.90625 -0.140625 0.34375 -0.109375 L 0.34375 0 L 2.296875 0 L 2.296875 -0.109375 L 2.15625 -0.109375 C 1.84375 -0.109375 1.6875 -0.171875 1.6875 -0.328125 C 1.6875 -0.40625 1.765625 -0.671875 1.8125 -0.84375 L 2.828125 -4.484375 C 2.84375 -4.578125 2.859375 -4.640625 2.859375 -4.6875 C 2.859375 -4.71875 2.84375 -4.734375 2.828125 -4.734375 Z M 2.8125 -4.734375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-4\" overflow=\"visible\">\n",
       "<path d=\"M 2.109375 -4.6875 C 1.46875 -4.09375 1.078125 -3.734375 0.8125 -3.296875 C 0.453125 -2.6875 0.296875 -1.984375 0.296875 -1.265625 C 0.296875 -0.359375 0.53125 0.546875 0.984375 1.265625 L 1.109375 1.21875 C 0.859375 0.515625 0.78125 0.078125 0.78125 -0.59375 C 0.78125 -2.015625 1.140625 -3.5625 2.203125 -4.578125 Z M 2.109375 -4.6875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-5\" overflow=\"visible\">\n",
       "<path d=\"M 1.203125 -4.640625 C 1.4375 -4.078125 1.546875 -3.515625 1.546875 -2.8125 C 1.546875 -2.15625 1.40625 -1.34375 1.203125 -0.671875 C 0.953125 0.0625 0.703125 0.53125 0.109375 1.15625 L 0.203125 1.265625 C 0.84375 0.703125 1.203125 0.3125 1.484375 -0.15625 C 1.828125 -0.75 2.015625 -1.421875 2.015625 -2.125 C 2.015625 -2.5 1.96875 -2.90625 1.890625 -3.296875 C 1.765625 -3.828125 1.65625 -4.09375 1.328125 -4.6875 Z M 1.203125 -4.640625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-6\" overflow=\"visible\">\n",
       "<path d=\"M 3.90625 -1.265625 L 3.765625 -1.296875 C 3.578125 -0.890625 3.421875 -0.671875 3.203125 -0.515625 C 2.859375 -0.28125 2.640625 -0.25 1.9375 -0.25 C 1.359375 -0.25 1.203125 -0.328125 1.203125 -0.53125 C 1.203125 -0.640625 1.234375 -0.75 1.28125 -0.921875 L 2.109375 -3.921875 C 2.21875 -4.3125 2.40625 -4.4375 2.828125 -4.453125 L 2.828125 -4.578125 L 0.90625 -4.578125 L 0.90625 -4.453125 C 1.34375 -4.4375 1.453125 -4.359375 1.453125 -4.15625 C 1.453125 -4.046875 1.421875 -3.90625 1.375 -3.71875 L 0.515625 -0.65625 C 0.390625 -0.21875 0.34375 -0.15625 -0.0625 -0.109375 L -0.0625 0 L 3.5 0 Z M 3.90625 -1.265625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-7\" overflow=\"visible\">\n",
       "<path d=\"M 1.25 -3.96875 C 1.5 -4.296875 1.734375 -4.4375 2.03125 -4.4375 C 2.40625 -4.4375 2.640625 -4.1875 2.640625 -3.796875 C 2.640625 -3.5 2.5 -3.265625 2.21875 -3.0625 C 1.90625 -2.84375 1.609375 -2.75 0.984375 -2.640625 L 0.984375 -2.53125 L 1.15625 -2.53125 C 1.828125 -2.53125 2.28125 -2.171875 2.28125 -1.421875 C 2.28125 -1.09375 2.1875 -0.75 2.015625 -0.53125 C 1.828125 -0.34375 1.59375 -0.21875 1.359375 -0.21875 C 1.21875 -0.21875 1.046875 -0.28125 0.859375 -0.4375 C 0.671875 -0.609375 0.546875 -0.65625 0.40625 -0.65625 C 0.234375 -0.65625 0.109375 -0.53125 0.109375 -0.34375 C 0.109375 -0.125 0.390625 0.046875 0.890625 0.046875 C 1.390625 0.046875 1.84375 -0.078125 2.1875 -0.390625 C 2.578125 -0.734375 2.828125 -1.21875 2.828125 -1.78125 C 2.828125 -2.28125 2.640625 -2.609375 2.171875 -2.78125 L 2.171875 -2.8125 C 2.96875 -3.09375 3.25 -3.359375 3.25 -3.875 C 3.25 -4.375 2.875 -4.734375 2.328125 -4.734375 C 1.8125 -4.734375 1.359375 -4.515625 1.140625 -4 Z M 1.25 -3.96875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-1\" overflow=\"visible\">\n",
       "<path d=\"M 2.765625 0 L 2.765625 -0.109375 C 2.234375 -0.109375 2.09375 -0.234375 2.09375 -0.53125 L 2.09375 -4.71875 L 2.03125 -4.734375 L 0.78125 -4.09375 L 0.78125 -4 L 0.96875 -4.0625 C 1.09375 -4.109375 1.21875 -4.15625 1.28125 -4.15625 C 1.421875 -4.15625 1.484375 -4.046875 1.484375 -3.8125 L 1.484375 -0.671875 C 1.484375 -0.28125 1.34375 -0.140625 0.828125 -0.109375 L 0.828125 0 Z M 2.765625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-2\" overflow=\"visible\">\n",
       "<path d=\"M 3.3125 -0.953125 L 3.21875 -1 C 2.984375 -0.609375 2.84375 -0.53125 2.546875 -0.53125 L 0.90625 -0.53125 L 2.0625 -1.765625 C 2.6875 -2.421875 2.96875 -2.953125 2.96875 -3.5 C 2.96875 -4.1875 2.453125 -4.734375 1.671875 -4.734375 C 0.8125 -4.734375 0.359375 -4.15625 0.203125 -3.34375 L 0.359375 -3.296875 C 0.640625 -3.984375 0.875 -4.21875 1.390625 -4.21875 C 1.984375 -4.21875 2.359375 -3.859375 2.359375 -3.234375 C 2.359375 -2.625 2.109375 -2.09375 1.453125 -1.40625 L 0.203125 -0.078125 L 0.203125 0 L 2.9375 0 Z M 3.3125 -0.953125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-3\" overflow=\"visible\">\n",
       "<path d=\"M 4.15625 -3.4375 L 4.109375 -4.640625 L 0.171875 -4.640625 L 0.125 -3.4375 L 0.28125 -3.4375 C 0.421875 -4.21875 0.609375 -4.34375 1.359375 -4.34375 L 1.78125 -4.34375 L 1.78125 -0.859375 C 1.78125 -0.25 1.703125 -0.171875 1.125 -0.140625 L 1.125 0 L 3.15625 0 L 3.15625 -0.140625 C 2.59375 -0.15625 2.484375 -0.25 2.484375 -0.78125 L 2.484375 -4.34375 L 2.90625 -4.34375 C 3.640625 -4.34375 3.828125 -4.21875 3.984375 -3.4375 Z M 4.15625 -3.4375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-4\" overflow=\"visible\">\n",
       "<path d=\"M 0.421875 -3.5625 C 0.71875 -4.09375 1.03125 -4.3125 1.46875 -4.3125 C 1.9375 -4.3125 2.21875 -4.078125 2.21875 -3.59375 C 2.21875 -3.171875 2 -2.859375 1.65625 -2.65625 C 1.515625 -2.578125 1.34375 -2.5 1.078125 -2.40625 L 1.078125 -2.3125 C 1.46875 -2.3125 1.625 -2.28125 1.78125 -2.234375 C 2.265625 -2.09375 2.515625 -1.75 2.515625 -1.21875 C 2.515625 -0.625 2.125 -0.15625 1.609375 -0.15625 C 1.40625 -0.15625 1.265625 -0.1875 1 -0.375 C 0.8125 -0.515625 0.6875 -0.546875 0.578125 -0.546875 C 0.40625 -0.546875 0.28125 -0.453125 0.28125 -0.296875 C 0.28125 -0.046875 0.5625 0.09375 1.078125 0.09375 C 1.6875 0.09375 2.359375 -0.109375 2.6875 -0.546875 C 2.890625 -0.8125 3.015625 -1.15625 3.015625 -1.53125 C 3.015625 -1.890625 2.90625 -2.21875 2.703125 -2.4375 C 2.5625 -2.59375 2.4375 -2.671875 2.125 -2.8125 C 2.59375 -3.09375 2.78125 -3.4375 2.78125 -3.765625 C 2.78125 -4.34375 2.34375 -4.734375 1.6875 -4.734375 C 0.953125 -4.734375 0.515625 -4.265625 0.3125 -3.59375 Z M 0.421875 -3.5625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-1\" overflow=\"visible\">\n",
       "<path d=\"M 8.875 -0.4375 C 8.875 -0.734375 8.625 -1 8.328125 -1 C 8.015625 -1 7.765625 -0.75 7.765625 -0.4375 C 7.765625 -0.125 8.015625 0.109375 8.3125 0.109375 C 8.625 0.109375 8.875 -0.140625 8.875 -0.4375 Z M 5.546875 -0.4375 C 5.546875 -0.734375 5.296875 -1 5 -1 C 4.6875 -1 4.4375 -0.75 4.4375 -0.4375 C 4.4375 -0.125 4.6875 0.109375 4.984375 0.109375 C 5.296875 0.109375 5.546875 -0.140625 5.546875 -0.4375 Z M 2.21875 -0.4375 C 2.21875 -0.734375 1.953125 -1 1.671875 -1 C 1.359375 -1 1.109375 -0.75 1.109375 -0.4375 C 1.109375 -0.125 1.34375 0.109375 1.65625 0.109375 C 1.953125 0.109375 2.21875 -0.140625 2.21875 -0.4375 Z M 2.21875 -0.4375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<path d=\"M 230.898438 472.691406 L 254.898438 472.691406 L 254.898438 496.691406 L 230.898438 496.691406 Z M 230.898438 472.691406 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"181.9525\" xlink:href=\"#glyph0-1\" y=\"214.6925\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"187.9555\" xlink:href=\"#glyph1-1\" y=\"218.6925\"/>\n",
       "</g>\n",
       "<path d=\"M 102.101562 472.691406 L 126.101562 472.691406 L 126.101562 496.691406 L 102.101562 496.691406 Z M 102.101562 472.691406 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"53.3485\" xlink:href=\"#glyph0-1\" y=\"214.6925\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"59.3515\" xlink:href=\"#glyph1-2\" y=\"218.6925\"/>\n",
       "</g>\n",
       "<path d=\"M 57.101562 472.691406 L 81.101562 472.691406 L 81.101562 496.691406 L 57.101562 496.691406 Z M 57.101562 472.691406 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"8.3485\" xlink:href=\"#glyph0-1\" y=\"214.6925\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"14.3515\" xlink:href=\"#glyph1-3\" y=\"218.6925\"/>\n",
       "</g>\n",
       "<path d=\"M 57.101562 384.125 L 81.101562 384.125 L 81.101562 408.125 L 57.101562 408.125 Z M 57.101562 384.125 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"6.1\" xlink:href=\"#glyph0-2\" y=\"128.1238\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"13.102\" xlink:href=\"#glyph2-1\" y=\"132.1238\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"13.74209\" xlink:href=\"#glyph1-4\" y=\"122.2202\"/>\n",
       "  <use x=\"16.07309\" xlink:href=\"#glyph1-2\" y=\"122.2202\"/>\n",
       "  <use x=\"19.57309\" xlink:href=\"#glyph1-5\" y=\"122.2202\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"142.9\" xlink:href=\"#glyph3-1\" y=\"172.4081\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"186.4\" xlink:href=\"#glyph3-1\" y=\"127.1238\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"144.9\" xlink:href=\"#glyph3-1\" y=\"128.1238\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"144.9\" xlink:href=\"#glyph3-1\" y=\"61.3976\"/>\n",
       "</g>\n",
       "<path d=\"M 230.898438 272.832031 L 254.898438 272.832031 L 254.898438 296.832031 L 230.898438 296.832031 Z M 230.898438 272.832031 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"181.705\" xlink:href=\"#glyph0-3\" y=\"14.8326\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"188.203\" xlink:href=\"#glyph1-1\" y=\"18.8326\"/>\n",
       "</g>\n",
       "<path d=\"M 102.101562 272.832031 L 126.101562 272.832031 L 126.101562 296.832031 L 102.101562 296.832031 Z M 102.101562 272.832031 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"53.101\" xlink:href=\"#glyph0-3\" y=\"14.8326\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"59.599\" xlink:href=\"#glyph1-2\" y=\"18.8326\"/>\n",
       "</g>\n",
       "<path d=\"M 57.101562 272.832031 L 81.101562 272.832031 L 81.101562 296.832031 L 57.101562 296.832031 Z M 57.101562 272.832031 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"8.101\" xlink:href=\"#glyph0-3\" y=\"14.8326\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"14.599\" xlink:href=\"#glyph1-3\" y=\"18.8326\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"8.1\" xlink:href=\"#glyph3-1\" y=\"94.0619\"/>\n",
       "</g>\n",
       "<path d=\"M 114.101562 472.691406 L 114.101562 458.308594 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 114.101562 454.308594 L 114.101562 458.308594 M 112.601562 458.308594 L 114.101562 454.308594 L 115.601562 458.308594 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 57.101562 428.40625 L 81.101562 428.40625 L 81.101562 452.40625 L 57.101562 452.40625 Z M 57.101562 428.40625 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"6.1\" xlink:href=\"#glyph0-2\" y=\"172.4081\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"13.102\" xlink:href=\"#glyph1-3\" y=\"176.4081\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"13.74209\" xlink:href=\"#glyph1-4\" y=\"166.5045\"/>\n",
       "  <use x=\"16.07309\" xlink:href=\"#glyph1-3\" y=\"166.5045\"/>\n",
       "  <use x=\"19.57309\" xlink:href=\"#glyph1-5\" y=\"166.5045\"/>\n",
       "</g>\n",
       "<path d=\"M 57.101562 317.398438 L 81.101562 317.398438 L 81.101562 341.398438 L 57.101562 341.398438 Z M 57.101562 317.398438 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"6.1\" xlink:href=\"#glyph0-2\" y=\"61.3976\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"13.102\" xlink:href=\"#glyph1-3\" y=\"65.3976\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"13.54609\" xlink:href=\"#glyph1-4\" y=\"55.4939\"/>\n",
       "  <use x=\"15.87709\" xlink:href=\"#glyph1-6\" y=\"55.4939\"/>\n",
       "  <use x=\"19.76909\" xlink:href=\"#glyph1-5\" y=\"55.4939\"/>\n",
       "</g>\n",
       "<path d=\"M 102.101562 428.40625 L 126.101562 428.40625 L 126.101562 452.40625 L 102.101562 452.40625 Z M 102.101562 428.40625 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"51.1\" xlink:href=\"#glyph0-2\" y=\"172.4081\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"58.102\" xlink:href=\"#glyph1-2\" y=\"176.4081\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"58.7421\" xlink:href=\"#glyph1-4\" y=\"166.5045\"/>\n",
       "  <use x=\"61.0731\" xlink:href=\"#glyph1-3\" y=\"166.5045\"/>\n",
       "  <use x=\"64.5731\" xlink:href=\"#glyph1-5\" y=\"166.5045\"/>\n",
       "</g>\n",
       "<path d=\"M 230.898438 428.40625 L 254.898438 428.40625 L 254.898438 452.40625 L 230.898438 452.40625 Z M 230.898438 428.40625 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"179.9\" xlink:href=\"#glyph0-2\" y=\"172.4081\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"186.902\" xlink:href=\"#glyph1-1\" y=\"176.4081\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"187.5421\" xlink:href=\"#glyph1-4\" y=\"166.5045\"/>\n",
       "  <use x=\"189.8731\" xlink:href=\"#glyph1-3\" y=\"166.5045\"/>\n",
       "  <use x=\"193.3731\" xlink:href=\"#glyph1-5\" y=\"166.5045\"/>\n",
       "</g>\n",
       "<path d=\"M 102.101562 384.125 L 126.101562 384.125 L 126.101562 408.125 L 102.101562 408.125 Z M 102.101562 384.125 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"51.1\" xlink:href=\"#glyph0-2\" y=\"128.1238\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"58.102\" xlink:href=\"#glyph2-2\" y=\"132.1238\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"58.7421\" xlink:href=\"#glyph1-4\" y=\"122.2202\"/>\n",
       "  <use x=\"61.0731\" xlink:href=\"#glyph1-2\" y=\"122.2202\"/>\n",
       "  <use x=\"64.5731\" xlink:href=\"#glyph1-5\" y=\"122.2202\"/>\n",
       "</g>\n",
       "<path d=\"M 230.898438 384.125 L 254.898438 384.125 L 254.898438 408.125 L 230.898438 408.125 Z M 230.898438 384.125 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"179.9\" xlink:href=\"#glyph0-2\" y=\"128.1238\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"186.902\" xlink:href=\"#glyph2-3\" y=\"132.1238\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"187.5421\" xlink:href=\"#glyph1-4\" y=\"122.2202\"/>\n",
       "  <use x=\"189.8731\" xlink:href=\"#glyph1-2\" y=\"122.2202\"/>\n",
       "  <use x=\"193.3731\" xlink:href=\"#glyph1-5\" y=\"122.2202\"/>\n",
       "</g>\n",
       "<path d=\"M 102.101562 317.398438 L 126.101562 317.398438 L 126.101562 341.398438 L 102.101562 341.398438 Z M 102.101562 317.398438 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"51.1\" xlink:href=\"#glyph0-2\" y=\"61.3976\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"58.102\" xlink:href=\"#glyph1-2\" y=\"65.3976\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"58.5461\" xlink:href=\"#glyph1-4\" y=\"55.4939\"/>\n",
       "  <use x=\"60.8771\" xlink:href=\"#glyph1-6\" y=\"55.4939\"/>\n",
       "  <use x=\"64.7691\" xlink:href=\"#glyph1-5\" y=\"55.4939\"/>\n",
       "</g>\n",
       "<path d=\"M 230.898438 317.355469 L 254.898438 317.355469 L 254.898438 341.355469 L 230.898438 341.355469 Z M 230.898438 317.355469 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"179.9\" xlink:href=\"#glyph0-2\" y=\"61.3547\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"186.902\" xlink:href=\"#glyph1-1\" y=\"65.3547\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"187.3461\" xlink:href=\"#glyph1-4\" y=\"55.4511\"/>\n",
       "  <use x=\"189.6771\" xlink:href=\"#glyph1-6\" y=\"55.4511\"/>\n",
       "  <use x=\"193.5691\" xlink:href=\"#glyph1-5\" y=\"55.4511\"/>\n",
       "</g>\n",
       "<path d=\"M 69.101562 472.691406 L 69.101562 458.308594 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 69.101562 454.308594 L 69.101562 458.308594 M 67.601562 458.308594 L 69.101562 454.308594 L 70.601562 458.308594 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 81.101562 440.40625 L 96.199219 440.40625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 100.199219 440.40625 L 96.199219 440.40625 M 96.199219 438.90625 L 100.199219 440.40625 L 96.199219 441.90625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 126.101562 440.40625 L 190 440.40625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 194 440.40625 L 190 440.40625 M 190 438.90625 L 194 440.40625 L 190 441.90625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 211.898438 440.40625 L 225 440.40625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 229 440.40625 L 225 440.40625 M 225 438.90625 L 229 440.40625 L 225 441.90625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 69.101562 317.398438 L 69.101562 302.734375 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 69.101562 298.734375 L 69.101562 302.734375 M 67.601562 302.734375 L 69.101562 298.734375 L 70.601562 302.734375 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 242.898438 472.691406 L 242.898438 458.308594 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 242.898438 454.308594 L 242.898438 458.308594 M 241.398438 458.308594 L 242.898438 454.308594 L 244.398438 458.308594 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 81.101562 329.398438 L 96.199219 329.398438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 100.199219 329.398438 L 96.199219 329.398438 M 96.199219 327.898438 L 100.199219 329.398438 L 96.199219 330.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 126.101562 329.398438 L 192 329.398438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 196 329.398438 L 192 329.398438 M 192 327.898438 L 196 329.398438 L 192 330.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 213.898438 329.386719 L 225 329.375 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 229 329.371094 L 225 329.375 M 225 327.875 L 229 329.371094 L 225 330.875 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 242.898438 317.355469 L 242.898438 302.734375 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 242.898438 298.734375 L 242.898438 302.734375 M 241.398438 302.734375 L 242.898438 298.734375 L 244.398438 302.734375 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 81.101562 396.125 L 96.199219 396.125 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 100.199219 396.125 L 96.199219 396.125 M 96.199219 394.625 L 100.199219 396.125 L 96.199219 397.625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 126.101562 396.125 L 192 396.125 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 196 396.125 L 192 396.125 M 192 394.625 L 196 396.125 L 192 397.625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 213.898438 396.125 L 225 396.125 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 229 396.125 L 225 396.125 M 225 394.625 L 229 396.125 L 225 397.625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 114.101562 317.398438 L 114.101562 302.734375 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 114.101562 298.734375 L 114.101562 302.734375 M 112.601562 302.734375 L 114.101562 298.734375 L 115.601562 302.734375 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 69.101562 428.40625 L 69.101562 414.023438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 69.101562 410.023438 L 69.101562 414.023438 M 67.601562 414.023438 L 69.101562 410.023438 L 70.601562 414.023438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 69.101562 384.125 L 69.101562 376.960938 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 69.101562 372.960938 L 69.101562 376.960938 M 67.601562 376.960938 L 69.101562 372.960938 L 70.601562 376.960938 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 69.101562 353.0625 L 69.101562 347.296875 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 69.101562 343.296875 L 69.101562 347.296875 M 67.601562 347.296875 L 69.101562 343.296875 L 70.601562 347.296875 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 114.101562 428.40625 L 114.101562 414.023438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 114.101562 410.023438 L 114.101562 414.023438 M 112.601562 414.023438 L 114.101562 410.023438 L 115.601562 414.023438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 242.898438 428.40625 L 242.898438 414.023438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 242.898438 410.023438 L 242.898438 414.023438 M 241.398438 414.023438 L 242.898438 410.023438 L 244.398438 414.023438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 81.101562 284.832031 L 96.199219 284.832031 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 100.199219 284.832031 L 96.199219 284.832031 M 96.199219 283.332031 L 100.199219 284.832031 L 96.199219 286.332031 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"53.1\" xlink:href=\"#glyph3-1\" y=\"95.4619\"/>\n",
       "</g>\n",
       "<path d=\"M 114.101562 384.125 L 114.101562 378.363281 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 114.101562 374.363281 L 114.101562 378.363281 M 112.601562 378.363281 L 114.101562 374.363281 L 115.601562 378.363281 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 114.101562 354.460938 L 114.101562 347.296875 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 114.101562 343.296875 L 114.101562 347.296875 M 112.601562 347.296875 L 114.101562 343.296875 L 115.601562 347.296875 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"181.9\" xlink:href=\"#glyph3-1\" y=\"94.4619\"/>\n",
       "</g>\n",
       "<path d=\"M 242.898438 384.125 L 242.898438 377.363281 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 242.898438 373.363281 L 242.898438 377.363281 M 241.398438 377.363281 L 242.898438 373.363281 L 244.398438 377.363281 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 242.898438 353.460938 L 242.898438 347.253906 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 242.898438 343.253906 L 242.898438 347.253906 M 241.398438 347.253906 L 242.898438 343.253906 L 244.398438 347.253906 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"144.9\" xlink:href=\"#glyph3-1\" y=\"16.8326\"/>\n",
       "</g>\n",
       "<path d=\"M 126.101562 284.832031 L 192 284.832031 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 196 284.832031 L 192 284.832031 M 192 283.332031 L 196 284.832031 L 192 286.332031 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 213.898438 284.832031 L 225 284.832031 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 229 284.832031 L 225 284.832031 M 225 283.332031 L 229 284.832031 L 225 286.332031 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 254.898438 440.421875 L 270.5 440.441406 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 274.5 440.445312 L 270.5 440.441406 M 270.5 438.941406 L 274.5 440.445312 L 270.5 441.941406 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 254.898438 329.371094 L 270.5 329.386719 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 274.5 329.394531 L 270.5 329.386719 M 270.5 327.886719 L 274.5 329.394531 L 270.5 330.886719 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 254.898438 396.136719 L 270.5 396.15625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 274.5 396.160156 L 270.5 396.15625 M 270.5 394.65625 L 274.5 396.160156 L 270.5 397.65625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 254.898438 284.847656 L 270.5 284.867188 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 274.5 284.871094 L 270.5 284.867188 M 270.5 283.367188 L 274.5 284.871094 L 270.5 286.367188 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 149 472.691406 L 173 472.691406 L 173 496.691406 L 149 496.691406 Z M 149 472.691406 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"100.2485\" xlink:href=\"#glyph0-1\" y=\"214.6925\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"106.2515\" xlink:href=\"#glyph1-7\" y=\"218.6925\"/>\n",
       "</g>\n",
       "<path d=\"M 149 272.832031 L 173 272.832031 L 173 296.832031 L 149 296.832031 Z M 149 272.832031 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"100.001\" xlink:href=\"#glyph0-3\" y=\"14.8326\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"106.499\" xlink:href=\"#glyph1-7\" y=\"18.8326\"/>\n",
       "</g>\n",
       "<path d=\"M 161 472.691406 L 161 458.308594 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 161 454.308594 L 161 458.308594 M 159.5 458.308594 L 161 454.308594 L 162.5 458.308594 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 149 428.40625 L 173 428.40625 L 173 452.40625 L 149 452.40625 Z M 149 428.40625 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"98\" xlink:href=\"#glyph0-2\" y=\"172.4081\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"105.002\" xlink:href=\"#glyph1-7\" y=\"176.4081\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"105.6421\" xlink:href=\"#glyph1-4\" y=\"166.5045\"/>\n",
       "  <use x=\"107.9731\" xlink:href=\"#glyph1-3\" y=\"166.5045\"/>\n",
       "  <use x=\"111.4731\" xlink:href=\"#glyph1-5\" y=\"166.5045\"/>\n",
       "</g>\n",
       "<path d=\"M 149 384.125 L 173 384.125 L 173 408.125 L 149 408.125 Z M 149 384.125 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"98\" xlink:href=\"#glyph0-2\" y=\"128.1238\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"105.002\" xlink:href=\"#glyph2-4\" y=\"132.1238\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"105.6421\" xlink:href=\"#glyph1-4\" y=\"122.2202\"/>\n",
       "  <use x=\"107.9731\" xlink:href=\"#glyph1-2\" y=\"122.2202\"/>\n",
       "  <use x=\"111.4731\" xlink:href=\"#glyph1-5\" y=\"122.2202\"/>\n",
       "</g>\n",
       "<path d=\"M 149 317.398438 L 173 317.398438 L 173 341.398438 L 149 341.398438 Z M 149 317.398438 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"98\" xlink:href=\"#glyph0-2\" y=\"61.3976\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"105.002\" xlink:href=\"#glyph1-7\" y=\"65.3976\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"105.4461\" xlink:href=\"#glyph1-4\" y=\"55.4939\"/>\n",
       "  <use x=\"107.7771\" xlink:href=\"#glyph1-6\" y=\"55.4939\"/>\n",
       "  <use x=\"111.6691\" xlink:href=\"#glyph1-5\" y=\"55.4939\"/>\n",
       "</g>\n",
       "<path d=\"M 126.101562 440.40625 L 143.101562 440.40625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 147.101562 440.40625 L 143.101562 440.40625 M 143.101562 438.90625 L 147.101562 440.40625 L 143.101562 441.90625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 126.101562 329.398438 L 143.101562 329.398438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 147.101562 329.398438 L 143.101562 329.398438 M 143.101562 327.898438 L 147.101562 329.398438 L 143.101562 330.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 126.101562 396.125 L 143.101562 396.125 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 147.101562 396.125 L 143.101562 396.125 M 143.101562 394.625 L 147.101562 396.125 L 143.101562 397.625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 161 317.398438 L 161 302.734375 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 161 298.734375 L 161 302.734375 M 159.5 302.734375 L 161 298.734375 L 162.5 302.734375 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 161 428.40625 L 161 414.023438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 161 410.023438 L 161 414.023438 M 159.5 414.023438 L 161 410.023438 L 162.5 414.023438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 126.101562 284.832031 L 143.101562 284.832031 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 147.101562 284.832031 L 143.101562 284.832031 M 143.101562 283.332031 L 147.101562 284.832031 L 143.101562 286.332031 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"100\" xlink:href=\"#glyph3-1\" y=\"95.4619\"/>\n",
       "</g>\n",
       "<path d=\"M 161 384.125 L 161 378.363281 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 161 374.363281 L 161 378.363281 M 159.5 378.363281 L 161 374.363281 L 162.5 378.363281 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 161 354.460938 L 161 347.296875 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "<path d=\"M 161 343.296875 L 161 347.296875 M 159.5 347.296875 L 161 343.296875 L 162.5 347.296875 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-56,-271)\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "SVG('img/deep-rnn.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functional Dependencies "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At time step t we assume that we have a minibatch $X_t ∈ R^{n×d}$ (number of examples: $n$, number of inputs:\n",
    "$d$). The hidden state of hidden layer $l (l = 1, . . . , T )$ is $H_t^{(l)} ∈ R^{n×h}$ (number of hidden units: $h$), the output\n",
    "layer variable is $O_t ∈ R^{(n×q)}$\n",
    "(number of outputs: $q$) and a hidden layer activation function $f_l$ for layer $l$. We\n",
    "compute the hidden state of layer 1 as before, using $X_t$ as input. For all subsequent layers the hidden state\n",
    "of the previous layer is used in its place.\n",
    "$$H_t^{(1)} = f_1(X_t , H_{t−1}^{(1)})$$\n",
    "$$H_t^{(1)} = f_1(H_{t}^{(1-1)} , H_{t−1}^{(1)})$$\n",
    "Finally, the output of the output layer is only based on the hidden state of hidden layer $L$. We use the\n",
    "output function $g$ to address this:\n",
    "$$O_t = g(H_t^{(L)})$$\n",
    "Just as with multilayer perceptrons, the number of hidden layers $L$ and number of hidden units $h$ are hyper\n",
    "parameters. In particular, we can pick a regular RNN, a GRU or an LSTM to implement the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concise Implementation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fortunately many of the logistical details required to implement multiple layers of an RNN are readily avail-\n",
    "able in Gluon. To keep things simple we only illustrate the implementation using such built-in functionality.\n",
    "The code is very similar to the one we used previously for LSTMs. In fact, the only difference is that\n",
    "we specify the number of layers explicitly rather than picking the default of a single layer. Let’s begin by\n",
    "importing the appropriate modules and data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k9GDjrishzdN"
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')\n",
    "\n",
    "import d2l\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from d2l import RNNModel \n",
    "from d2l import load_data_time_machine\n",
    "from d2l import train_and_predict_rnn_nn\n",
    "torch.set_default_tensor_type('torch.cuda.FloatTensor')\n",
    "\n",
    "corpus_indices, vocab = load_data_time_machine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The architectural decisions (parameters, etc.) are very similar to those of previous sections. We pick the\n",
    "same number of inputs and outputs as we have distinct tokens, i.e. vocab_size. The number of hidden\n",
    "units is still 256 and we retain a learning rate of 100. The only difference is that we now select a nontrivial\n",
    "number of layers num_layers = 2. Since the model is somewhat slower to train we use 3000 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "bcuj0nqVh5rx",
    "outputId": "d615d296-7ab6-4ba1-db0d-23016d1ad35f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cuda:0\n"
     ]
    }
   ],
   "source": [
    "num_inputs, num_hiddens, num_layers, num_outputs = len(vocab), 256, 2, len(vocab)\n",
    "ctx = d2l.try_gpu()\n",
    "num_epochs, num_steps, batch_size, lr, clipping_theta = 500, 35, 32, 5, 1\n",
    "prefixes = ['traveller', 'time traveller']\n",
    "print('Using', ctx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The actual invocation logic is identical to before and we re-use train_and_predict_rnn_gluon. The only\n",
    "difference is that we now instantiate two layers with LSTMs. This rather more complex architecture and\n",
    "the large number of epochs slow down training considerably."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JlfIQfiah9OR"
   },
   "outputs": [],
   "source": [
    "def init_lstm_state(batch_size, num_hiddens, device):\n",
    "    return  (torch.zeros(size=(batch_size, num_hiddens), device=device),\n",
    "             torch.zeros(size=(batch_size, num_hiddens), device=device))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1312
    },
    "colab_type": "code",
    "id": "BfgM_NPCiCr7",
    "outputId": "a60f4bff-46bd-4b35-d50f-02f8886f1cc8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 125, perplexity 1.837382, time 46.93 sec\n",
      "epoch 250, perplexity 1.044730, time 47.20 sec\n",
      " - traveller smiled. 'are you sure we can move freely in space\n",
      " - time traveller smiled. 'are you sure we can move freely in space\n",
      "epoch 375, perplexity 1.034328, time 47.27 sec\n",
      "epoch 500, perplexity 1.029750, time 59.48 sec\n",
      " - traveller smiled. 'are you sure we can move freely in space\n",
      " - time traveller smiled. 'are you sure we can move freely in space\n"
     ]
    }
   ],
   "source": [
    "lstm_layer = nn.LSTM(input_size=num_inputs, hidden_size=num_hiddens)\n",
    "model = RNNModel(lstm_layer, num_hiddens, len(vocab))\n",
    "model.to(ctx)\n",
    "train_and_predict_rnn_nn(model, num_hiddens, init_lstm_state, corpus_indices, vocab,\n",
    "                         ctx, num_epochs, num_steps, lr,\n",
    "                         clipping_theta, batch_size, prefixes, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Heading "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "• In deep recurrent neural networks, hidden state information is passed to the next time step of the\n",
    "current layer and the next layer of the current time step.\n",
    "\n",
    "• There exist many different flavors of deep RNNs, such as LSTMs, GRUs or regular RNNs. Conveniently\n",
    "these models are all available as parts of the rnn module in Gluon.\n",
    "\n",
    "• Initialization of the models requires care. Overall, deep RNNs require considerable amount of work\n",
    "(learning rate, clipping, etc) to ensure proper convergence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Heading "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Try to implement a two-layer RNN from scratch using the single layer implementation we discussed in\n",
    "Section 10.5.\n",
    "2. Replace the LSTM by a GRU and compare the accuracy.\n",
    "3. Increase the training data to include multiple books. How low can you go on the perplexity scale?\n",
    "4. Would you want to combine sources of different authors when modeling text? Why is this a good idea?\n",
    "What could go wrong?"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "lstm.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}