Ch10_Recurrent_Neural_Networks/Backpropagation_Through_Time.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backpropagation Through Time\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "So far we repeatedly alluded to things like *exploding gradients*,\n",
    "*vanishing gradients*, *truncating backprop*, and the need to\n",
    "*detach the computational graph*. For instance, in the previous\n",
    "section we invoked `s.detach()` on the sequence. None of this was really fully\n",
    "explained, in the interest of being able to build a model quickly and\n",
    "to see how it works. In this section we will delve a bit more deeply\n",
    "into the details of backpropagation for sequence models and why (and\n",
    "how) the math works. For a more detailed discussion, e.g. about\n",
    "randomization and backprop also see the paper by\n",
    "[Tallec and Ollivier, 2017](https://arxiv.org/abs/1705.08209).\n",
    "\n",
    "We encountered some of the effects of gradient explosion when we first\n",
    "implemented recurrent neural networks . In\n",
    "particular, if you solved the problems in the problem set, you would\n",
    "have seen that gradient clipping is vital to ensure proper\n",
    "convergence. To provide a better understanding of this issue, this\n",
    "section will review how gradients are computed for sequences. Note\n",
    "that there is nothing conceptually new in how it works. After all, we are still merely applying the chain rule to compute gradients. Nonetheless it is\n",
    "worth while reviewing backpropagation  for\n",
    "another time.\n",
    "\n",
    "Forward propagation in a recurrent neural network is relatively\n",
    "straightforward. Back-propagation through time is actually a specific\n",
    "application of back propagation in recurrent neural networks. It\n",
    "requires us to expand the recurrent neural network one time step at a time to\n",
    "obtain the dependencies between model variables and parameters. Then,\n",
    "based on the chain rule, we apply back propagation to compute and\n",
    "store gradients. Since sequences can be rather long this means that the dependency can be rather lengthy. E.g. for a sequence of 1000 characters the first symbol could potentially have significant influence on the symbol at position 1000. This is not really computationally feasible (it takes too long and requires too much memory) and it requires over 1000 matrix-vector products before we would arrive at that very elusive gradient. This is a process fraught with computational and statistical uncertainty. In the following we will address what happens and how to address this in practice.\n",
    "\n",
    "## A Simplified Recurrent Network\n",
    "\n",
    "We start with a simplified model of how an RNN works. This model ignores details about the specifics of the hidden state and how it is being updated. These details are immaterial to the analysis and would only serve to clutter the notation and make it look more intimidating.\n",
    "\n",
    "$$h_t = f(x_t, h_{t-1}, w) \\text{ and } o_t = g(h_t, w)$$\n",
    "\n",
    "Here $h_t$ denotes the hidden state, $x_t$ the input and $o_t$ the output. We have a chain of values $\\{\\ldots (h_{t-1}, x_{t-1}, o_{t-1}), (h_{t}, x_{t}, o_t), \\ldots\\}$ that depend on each other via recursive computation. The forward pass is fairly straightforward. All we need is to loop through the $(x_t, h_t, o_t)$ triples one step at a time. This is then evaluated by an objective function measuring the discrepancy between outputs $o_t$ and some desired target $y_t$\n",
    "\n",
    "$$L(x,y, w) = \\sum_{t=1}^T l(y_t, o_t).$$\n",
    "\n",
    "For backpropagation matters are a bit more tricky. Let's compute the gradients with regard to the parameters $w$ of the objective function $L$. We get that\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\partial_{w} L & = \\sum_{t=1}^T \\partial_w l(y_t, o_t) \\\\\n",
    "\t& = \\sum_{t=1}^T \\partial_{o_t} l(y_t, o_t) \\left[\\partial_w g(h_t, w) + \\partial_{h_t} g(h_t,w) \\partial_w h_t\\right]\n",
    "\\end{aligned}$$\n",
    "\n",
    "The first part of the derivative is easy to compute (this is after all the instantaneous loss gradient at time $t$). The second part is where things get tricky, since we need to compute the effect of the parameters on $h_t$. For each term we have the recursion:\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\t\\partial_w h_t & = \\partial_w f(x_t, h_{t-1}, w) + \\partial_h f(x_t, h_{t-1}, w) \\partial_w h_{t-1} \\\\\n",
    "\t& = \\sum_{i=t}^1 \\left[\\prod_{j=t}^i \\partial_h f(x_j, h_{j-1}, w) \\right] \\partial_w f(x_{i}, h_{i-1}, w)\n",
    "\\end{aligned}$$\n",
    "\n",
    "This chain can get *very* long whenever $t$ is large. While we can use the chain rule to compute $\\partial_w h_t$ recursively, this might not be ideal. Let's discuss a number of strategies for dealing with this problem:\n",
    "\n",
    "**Compute the full sum.** This is very slow and gradients can blow up,\n",
    "since subtle changes in the initial conditions can potentially affect\n",
    "the outcome a lot. That is, we could see things similar to the\n",
    "butterfly effect where minimal changes in the initial conditions lead to disproportionate changes in the outcome. This is actually quite undesirable in terms of the model that we want to estimate. After all, we are looking for robust estimators that generalize well. Hence this strategy is almost never used in practice.\n",
    "\n",
    "**Truncate the sum after $\\tau$ steps.** This is what we've been\n",
    "discussing so far. This leads to an *approximation* of the true\n",
    "gradient, simply by terminating the sum above at $\\partial_w\n",
    "h_{t-\\tau}$. The approximation error is thus given by $\\partial_h\n",
    "f(x_t, h_{t-1}, w) \\partial_w h_{t-1}$ (multiplied by a product of\n",
    "gradients involving $\\partial_h f$). In practice this works quite\n",
    "well. It is what is commonly referred to as truncated BPTT\n",
    "(backpropgation through time). One of the consequences of this is that\n",
    "the model focuses primarily on short-term influence rather than\n",
    "long-term consequences. This is actually *desirable*, since it biases\n",
    "the estimate towards simpler and more stable models.\n",
    "\n",
    "**Randomized Truncation.** Lastly we can replace $\\partial_w h_t$ by a\n",
    "random variable which is correct in expectation but which truncates\n",
    "the sequence. This is achieved by using a sequence of $\\xi_t$ where\n",
    "$\\mathbf{E}[\\xi_t] = 1$ and $\\Pr(\\xi_t = 0) = 1-\\pi$ and furthermore\n",
    "$\\Pr(\\xi_t = \\pi^{-1}) = \\pi$. We use this to replace the gradient:\n",
    "\n",
    "$$z_t  = \\partial_w f(x_t, h_{t-1}, w) + \\xi_t \\partial_h f(x_t, h_{t-1}, w) \\partial_w h_{t-1}$$\n",
    "\n",
    "It follows from the definition of $\\xi_t$ that $\\mathbf{E}[z_t] = \\partial_w h_t$. Whenever $\\xi_t = 0$ the expansion terminates at that point. This leads to a weighted sum of sequences of varying lengths where long sequences are rare but appropriately overweighted. [Tallec and Ollivier, 2017](https://arxiv.org/abs/1705.08209) proposed this in their paper. Unfortunately, while appealing in theory, the model does not work much better than simple truncation, most likely due to a number of factors. Firstly, the effect of an observation after a number of backpropagation steps into the past is quite sufficient to capture dependencies in practice. Secondly, the increased variance counteracts the fact that the gradient is more accurate. Thirdly, we actually *want* models that have only a short range of interaction. Hence BPTT has a slight regularizing effect which can be desirable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"91pt\" version=\"1.1\" viewBox=\"0 0 513 91\" width=\"513pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"M 0.703125 2.46875 L 0.703125 -9.875 L 7.703125 -9.875 L 7.703125 2.46875 Z M 1.484375 1.703125 L 6.921875 1.703125 L 6.921875 -9.078125 L 1.484375 -9.078125 Z M 1.484375 1.703125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M 1.640625 -5.34375 C 1.628906 -5 1.585938 -4.78125 1.515625 -4.6875 C 1.453125 -4.59375 1.316406 -4.546875 1.109375 -4.546875 C 0.953125 -4.546875 0.832031 -4.585938 0.75 -4.671875 C 0.664062 -4.765625 0.625 -4.890625 0.625 -5.046875 C 0.625 -5.046875 0.625 -5.082031 0.625 -5.15625 C 0.632812 -5.226562 0.640625 -5.300781 0.640625 -5.375 L 0.71875 -7.625 C 0.71875 -7.789062 0.757812 -7.910156 0.84375 -7.984375 C 0.925781 -8.066406 1.054688 -8.109375 1.234375 -8.109375 L 7.171875 -8.109375 C 7.347656 -8.109375 7.472656 -8.066406 7.546875 -7.984375 C 7.628906 -7.910156 7.675781 -7.789062 7.6875 -7.625 L 7.765625 -5.375 C 7.765625 -5.289062 7.765625 -5.210938 7.765625 -5.140625 C 7.765625 -5.066406 7.765625 -5.03125 7.765625 -5.03125 C 7.765625 -4.875 7.722656 -4.753906 7.640625 -4.671875 C 7.554688 -4.585938 7.425781 -4.546875 7.25 -4.546875 C 7.070312 -4.546875 6.945312 -4.59375 6.875 -4.6875 C 6.8125 -4.78125 6.773438 -5 6.765625 -5.34375 L 6.6875 -7.125 L 4.734375 -7.125 L 4.734375 -0.96875 L 6.046875 -0.96875 C 6.242188 -0.96875 6.382812 -0.925781 6.46875 -0.84375 C 6.550781 -0.769531 6.59375 -0.65625 6.59375 -0.5 C 6.59375 -0.332031 6.550781 -0.207031 6.46875 -0.125 C 6.382812 -0.0390625 6.269531 0 6.125 0 L 2.28125 0 C 2.125 0 2.003906 -0.0390625 1.921875 -0.125 C 1.847656 -0.207031 1.8125 -0.332031 1.8125 -0.5 C 1.8125 -0.664062 1.851562 -0.785156 1.9375 -0.859375 C 2.03125 -0.929688 2.179688 -0.96875 2.390625 -0.96875 L 3.671875 -0.96875 L 3.671875 -7.125 L 1.703125 -7.125 Z M 1.640625 -5.34375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-2\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-3\" overflow=\"visible\">\n",
       "<path d=\"M 1.625 -0.96875 L 1.625 -7.96875 L 1.5 -7.96875 C 1.1875 -7.96875 0.972656 -8.003906 0.859375 -8.078125 C 0.742188 -8.148438 0.6875 -8.273438 0.6875 -8.453125 C 0.6875 -8.617188 0.734375 -8.742188 0.828125 -8.828125 C 0.921875 -8.910156 1.054688 -8.953125 1.234375 -8.953125 L 2.265625 -8.953125 C 2.390625 -8.953125 2.484375 -8.925781 2.546875 -8.875 C 2.617188 -8.820312 2.65625 -8.753906 2.65625 -8.671875 L 2.65625 -5.578125 C 2.988281 -5.878906 3.316406 -6.101562 3.640625 -6.25 C 3.972656 -6.40625 4.3125 -6.484375 4.65625 -6.484375 C 5.34375 -6.484375 5.882812 -6.269531 6.28125 -5.84375 C 6.675781 -5.414062 6.875 -4.820312 6.875 -4.0625 L 6.875 -0.96875 L 7.203125 -0.96875 C 7.429688 -0.96875 7.59375 -0.925781 7.6875 -0.84375 C 7.789062 -0.769531 7.84375 -0.65625 7.84375 -0.5 C 7.84375 -0.332031 7.796875 -0.207031 7.703125 -0.125 C 7.609375 -0.0390625 7.472656 0 7.296875 0 L 5.46875 0 C 5.257812 0 5.113281 -0.0351562 5.03125 -0.109375 C 4.957031 -0.179688 4.921875 -0.3125 4.921875 -0.5 C 4.921875 -0.675781 4.96875 -0.796875 5.0625 -0.859375 C 5.15625 -0.929688 5.332031 -0.96875 5.59375 -0.96875 L 5.84375 -0.96875 L 5.84375 -3.8125 C 5.84375 -4.34375 5.71875 -4.75 5.46875 -5.03125 C 5.226562 -5.3125 4.878906 -5.453125 4.421875 -5.453125 C 3.953125 -5.453125 3.539062 -5.289062 3.1875 -4.96875 C 2.832031 -4.65625 2.65625 -4.269531 2.65625 -3.8125 L 2.65625 -0.96875 L 2.90625 -0.96875 C 3.175781 -0.96875 3.359375 -0.929688 3.453125 -0.859375 C 3.546875 -0.796875 3.59375 -0.675781 3.59375 -0.5 C 3.59375 -0.3125 3.554688 -0.179688 3.484375 -0.109375 C 3.410156 -0.0351562 3.265625 0 3.046875 0 L 1.234375 0 C 1.054688 0 0.921875 -0.0390625 0.828125 -0.125 C 0.742188 -0.207031 0.703125 -0.332031 0.703125 -0.5 C 0.703125 -0.675781 0.753906 -0.796875 0.859375 -0.859375 C 0.960938 -0.929688 1.171875 -0.96875 1.484375 -0.96875 Z M 1.625 -0.96875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-4\" overflow=\"visible\">\n",
       "<path d=\"M 1.890625 -2.859375 C 1.921875 -2.203125 2.140625 -1.691406 2.546875 -1.328125 C 2.960938 -0.972656 3.519531 -0.796875 4.21875 -0.796875 C 4.757812 -0.796875 5.34375 -0.9375 5.96875 -1.21875 C 6.59375 -1.507812 6.972656 -1.65625 7.109375 -1.65625 C 7.242188 -1.65625 7.359375 -1.609375 7.453125 -1.515625 C 7.546875 -1.429688 7.59375 -1.328125 7.59375 -1.203125 C 7.59375 -1.066406 7.539062 -0.941406 7.4375 -0.828125 C 7.34375 -0.722656 7.191406 -0.617188 6.984375 -0.515625 C 6.523438 -0.296875 6.050781 -0.125 5.5625 0 C 5.082031 0.113281 4.601562 0.171875 4.125 0.171875 C 3.09375 0.171875 2.265625 -0.125 1.640625 -0.71875 C 1.015625 -1.320312 0.703125 -2.117188 0.703125 -3.109375 C 0.703125 -4.117188 1.03125 -4.9375 1.6875 -5.5625 C 2.351562 -6.1875 3.210938 -6.5 4.265625 -6.5 C 5.234375 -6.5 6.03125 -6.207031 6.65625 -5.625 C 7.289062 -5.039062 7.609375 -4.332031 7.609375 -3.5 C 7.609375 -3.25 7.554688 -3.078125 7.453125 -2.984375 C 7.347656 -2.898438 7.054688 -2.859375 6.578125 -2.859375 Z M 1.921875 -3.75 L 6.375 -3.75 C 6.300781 -4.289062 6.070312 -4.726562 5.6875 -5.0625 C 5.300781 -5.394531 4.828125 -5.5625 4.265625 -5.5625 C 3.660156 -5.5625 3.148438 -5.398438 2.734375 -5.078125 C 2.328125 -4.753906 2.054688 -4.3125 1.921875 -3.75 Z M 1.921875 -3.75 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-5\" overflow=\"visible\">\n",
       "<path d=\"M 3.5625 -8.28125 C 3.5625 -8.78125 3.601562 -9.09375 3.6875 -9.21875 C 3.78125 -9.34375 3.972656 -9.40625 4.265625 -9.40625 C 4.554688 -9.40625 4.742188 -9.347656 4.828125 -9.234375 C 4.921875 -9.128906 4.96875 -8.878906 4.96875 -8.484375 C 4.96875 -8.085938 4.925781 -7.84375 4.84375 -7.75 C 4.757812 -7.65625 4.566406 -7.609375 4.265625 -7.609375 C 3.972656 -7.609375 3.78125 -7.648438 3.6875 -7.734375 C 3.601562 -7.828125 3.5625 -8.007812 3.5625 -8.28125 Z M 4.828125 -0.96875 L 7.03125 -0.96875 C 7.207031 -0.96875 7.335938 -0.925781 7.421875 -0.84375 C 7.503906 -0.769531 7.546875 -0.65625 7.546875 -0.5 C 7.546875 -0.3125 7.503906 -0.179688 7.421875 -0.109375 C 7.347656 -0.0351562 7.203125 0 6.984375 0 L 1.6875 0 C 1.5 0 1.359375 -0.0390625 1.265625 -0.125 C 1.179688 -0.207031 1.140625 -0.332031 1.140625 -0.5 C 1.140625 -0.65625 1.1875 -0.769531 1.28125 -0.84375 C 1.375 -0.925781 1.515625 -0.96875 1.703125 -0.96875 L 3.796875 -0.96875 L 3.796875 -5.34375 L 2.46875 -5.34375 C 2.28125 -5.34375 2.132812 -5.382812 2.03125 -5.46875 C 1.9375 -5.550781 1.890625 -5.675781 1.890625 -5.84375 C 1.890625 -6 1.929688 -6.113281 2.015625 -6.1875 C 2.109375 -6.269531 2.25 -6.3125 2.4375 -6.3125 L 4.4375 -6.3125 C 4.5625 -6.3125 4.65625 -6.285156 4.71875 -6.234375 C 4.789062 -6.191406 4.828125 -6.125 4.828125 -6.03125 Z M 4.828125 -0.96875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-6\" overflow=\"visible\">\n",
       "<path d=\"M 6.734375 -3.8125 C 6.734375 -4.445312 6.675781 -4.878906 6.5625 -5.109375 C 6.445312 -5.335938 6.25 -5.453125 5.96875 -5.453125 C 5.625 -5.453125 5.351562 -5.304688 5.15625 -5.015625 C 4.957031 -4.722656 4.859375 -4.320312 4.859375 -3.8125 L 4.859375 -0.96875 L 4.984375 -0.96875 C 5.265625 -0.96875 5.453125 -0.929688 5.546875 -0.859375 C 5.640625 -0.796875 5.6875 -0.675781 5.6875 -0.5 C 5.6875 -0.332031 5.644531 -0.207031 5.5625 -0.125 C 5.488281 -0.0390625 5.367188 0 5.203125 0 L 4.34375 0 C 4.144531 0 4.007812 -0.046875 3.9375 -0.140625 C 3.863281 -0.234375 3.828125 -0.425781 3.828125 -0.71875 L 3.828125 -4.34375 C 3.828125 -4.6875 3.75 -4.957031 3.59375 -5.15625 C 3.445312 -5.351562 3.238281 -5.453125 2.96875 -5.453125 C 2.757812 -5.453125 2.566406 -5.390625 2.390625 -5.265625 C 2.222656 -5.148438 2.082031 -4.972656 1.96875 -4.734375 L 1.96875 -0.96875 L 2.21875 -0.96875 C 2.488281 -0.96875 2.671875 -0.929688 2.765625 -0.859375 C 2.859375 -0.796875 2.90625 -0.675781 2.90625 -0.5 C 2.90625 -0.3125 2.867188 -0.179688 2.796875 -0.109375 C 2.722656 -0.0351562 2.578125 0 2.359375 0 L 0.53125 0 C 0.351562 0 0.222656 -0.0351562 0.140625 -0.109375 C 0.0546875 -0.191406 0.015625 -0.320312 0.015625 -0.5 C 0.015625 -0.675781 0.0664062 -0.796875 0.171875 -0.859375 C 0.273438 -0.929688 0.484375 -0.96875 0.796875 -0.96875 L 0.9375 -0.96875 L 0.9375 -5.34375 L 0.8125 -5.34375 C 0.488281 -5.34375 0.269531 -5.378906 0.15625 -5.453125 C 0.0390625 -5.523438 -0.015625 -5.65625 -0.015625 -5.84375 C -0.015625 -6 0.03125 -6.113281 0.125 -6.1875 C 0.21875 -6.269531 0.351562 -6.3125 0.53125 -6.3125 L 1.578125 -6.3125 C 1.734375 -6.3125 1.832031 -6.285156 1.875 -6.234375 C 1.925781 -6.179688 1.957031 -6.070312 1.96875 -5.90625 C 2.15625 -6.101562 2.351562 -6.25 2.5625 -6.34375 C 2.78125 -6.445312 3.019531 -6.5 3.28125 -6.5 C 3.601562 -6.5 3.867188 -6.425781 4.078125 -6.28125 C 4.285156 -6.144531 4.441406 -5.9375 4.546875 -5.65625 C 4.734375 -5.9375 4.953125 -6.144531 5.203125 -6.28125 C 5.453125 -6.425781 5.738281 -6.5 6.0625 -6.5 C 6.65625 -6.5 7.085938 -6.304688 7.359375 -5.921875 C 7.628906 -5.535156 7.765625 -4.914062 7.765625 -4.0625 L 7.765625 -0.96875 L 7.875 -0.96875 C 8.15625 -0.96875 8.34375 -0.929688 8.4375 -0.859375 C 8.53125 -0.796875 8.578125 -0.675781 8.578125 -0.5 C 8.578125 -0.332031 8.535156 -0.207031 8.453125 -0.125 C 8.367188 -0.0390625 8.25 0 8.09375 0 L 7.265625 0 C 7.085938 0 6.953125 -0.0507812 6.859375 -0.15625 C 6.773438 -0.269531 6.734375 -0.445312 6.734375 -0.6875 Z M 6.734375 -3.8125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-7\" overflow=\"visible\">\n",
       "<path d=\"M 4.21875 -3.359375 L 5.6875 -7.640625 C 5.75 -7.804688 5.832031 -7.925781 5.9375 -8 C 6.050781 -8.070312 6.210938 -8.109375 6.421875 -8.109375 L 7.546875 -8.109375 C 7.691406 -8.109375 7.800781 -8.066406 7.875 -7.984375 C 7.945312 -7.910156 7.984375 -7.796875 7.984375 -7.640625 C 7.984375 -7.472656 7.9375 -7.351562 7.84375 -7.28125 C 7.757812 -7.21875 7.585938 -7.1875 7.328125 -7.1875 L 7.21875 -7.1875 L 7.421875 -0.96875 L 7.578125 -0.96875 C 7.867188 -0.96875 8.0625 -0.929688 8.15625 -0.859375 C 8.257812 -0.796875 8.3125 -0.675781 8.3125 -0.5 C 8.3125 -0.332031 8.269531 -0.207031 8.1875 -0.125 C 8.113281 -0.0390625 8 0 7.84375 0 L 5.921875 0 C 5.660156 0 5.476562 -0.0351562 5.375 -0.109375 C 5.28125 -0.179688 5.234375 -0.3125 5.234375 -0.5 C 5.234375 -0.675781 5.289062 -0.796875 5.40625 -0.859375 C 5.519531 -0.929688 5.738281 -0.96875 6.0625 -0.96875 L 6.359375 -0.96875 L 6.21875 -6.84375 L 4.859375 -2.90625 C 4.796875 -2.71875 4.710938 -2.585938 4.609375 -2.515625 C 4.503906 -2.441406 4.351562 -2.40625 4.15625 -2.40625 C 3.96875 -2.40625 3.820312 -2.441406 3.71875 -2.515625 C 3.613281 -2.585938 3.53125 -2.71875 3.46875 -2.90625 L 2.09375 -6.84375 L 1.96875 -0.96875 L 2.4375 -0.96875 C 2.707031 -0.96875 2.894531 -0.929688 3 -0.859375 C 3.113281 -0.785156 3.171875 -0.664062 3.171875 -0.5 C 3.171875 -0.3125 3.117188 -0.179688 3.015625 -0.109375 C 2.921875 -0.0351562 2.738281 0 2.46875 0 L 0.5625 0 C 0.40625 0 0.285156 -0.0390625 0.203125 -0.125 C 0.117188 -0.207031 0.078125 -0.332031 0.078125 -0.5 C 0.078125 -0.675781 0.128906 -0.796875 0.234375 -0.859375 C 0.335938 -0.929688 0.535156 -0.96875 0.828125 -0.96875 L 0.984375 -0.96875 L 1.171875 -7.1875 L 1.0625 -7.1875 C 0.8125 -7.1875 0.640625 -7.21875 0.546875 -7.28125 C 0.453125 -7.351562 0.40625 -7.472656 0.40625 -7.640625 C 0.40625 -7.796875 0.441406 -7.910156 0.515625 -7.984375 C 0.597656 -8.066406 0.710938 -8.109375 0.859375 -8.109375 L 1.96875 -8.109375 C 2.1875 -8.109375 2.347656 -8.070312 2.453125 -8 C 2.566406 -7.925781 2.65625 -7.804688 2.71875 -7.640625 Z M 4.21875 -3.359375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-8\" overflow=\"visible\">\n",
       "<path d=\"M 5.71875 -3.015625 C 5.46875 -3.097656 5.222656 -3.160156 4.984375 -3.203125 C 4.753906 -3.242188 4.519531 -3.265625 4.28125 -3.265625 C 3.582031 -3.265625 3.03125 -3.144531 2.625 -2.90625 C 2.21875 -2.664062 2.015625 -2.347656 2.015625 -1.953125 C 2.015625 -1.640625 2.140625 -1.378906 2.390625 -1.171875 C 2.648438 -0.972656 2.972656 -0.875 3.359375 -0.875 C 3.941406 -0.875 4.476562 -1.023438 4.96875 -1.328125 C 5.46875 -1.640625 5.71875 -1.957031 5.71875 -2.28125 Z M 5.765625 -0.875 C 5.390625 -0.53125 4.984375 -0.269531 4.546875 -0.09375 C 4.109375 0.0820312 3.65625 0.171875 3.1875 0.171875 C 2.507812 0.171875 1.960938 -0.015625 1.546875 -0.390625 C 1.128906 -0.773438 0.921875 -1.269531 0.921875 -1.875 C 0.921875 -2.59375 1.195312 -3.148438 1.75 -3.546875 C 2.300781 -3.941406 3.082031 -4.140625 4.09375 -4.140625 C 4.351562 -4.140625 4.617188 -4.125 4.890625 -4.09375 C 5.160156 -4.0625 5.4375 -4.007812 5.71875 -3.9375 L 5.71875 -3.984375 C 5.71875 -4.503906 5.59375 -4.890625 5.34375 -5.140625 C 5.09375 -5.390625 4.703125 -5.515625 4.171875 -5.515625 C 3.816406 -5.515625 3.421875 -5.414062 2.984375 -5.21875 C 2.546875 -5.03125 2.253906 -4.9375 2.109375 -4.9375 C 1.984375 -4.9375 1.875 -4.988281 1.78125 -5.09375 C 1.695312 -5.207031 1.65625 -5.347656 1.65625 -5.515625 C 1.65625 -5.785156 1.90625 -6.015625 2.40625 -6.203125 C 2.914062 -6.390625 3.5625 -6.484375 4.34375 -6.484375 C 5.144531 -6.484375 5.75 -6.273438 6.15625 -5.859375 C 6.570312 -5.453125 6.78125 -4.851562 6.78125 -4.0625 L 6.78125 -0.96875 L 7.109375 -0.96875 C 7.429688 -0.96875 7.644531 -0.929688 7.75 -0.859375 C 7.863281 -0.796875 7.921875 -0.675781 7.921875 -0.5 C 7.921875 -0.332031 7.875 -0.207031 7.78125 -0.125 C 7.695312 -0.0390625 7.5625 0 7.375 0 L 6.296875 0 C 6.171875 0 6.066406 -0.0351562 5.984375 -0.109375 C 5.898438 -0.191406 5.847656 -0.304688 5.828125 -0.453125 Z M 5.765625 -0.875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-9\" overflow=\"visible\">\n",
       "<path d=\"M 6.25 -5.90625 L 6.25 -5.9375 C 6.25 -6.082031 6.289062 -6.191406 6.375 -6.265625 C 6.46875 -6.335938 6.601562 -6.375 6.78125 -6.375 C 6.96875 -6.375 7.097656 -6.316406 7.171875 -6.203125 C 7.242188 -6.097656 7.28125 -5.875 7.28125 -5.53125 L 7.28125 -4.265625 C 7.28125 -4.085938 7.234375 -3.957031 7.140625 -3.875 C 7.054688 -3.789062 6.929688 -3.75 6.765625 -3.75 C 6.640625 -3.75 6.53125 -3.785156 6.4375 -3.859375 C 6.351562 -3.929688 6.25 -4.078125 6.125 -4.296875 C 5.90625 -4.671875 5.640625 -4.953125 5.328125 -5.140625 C 5.015625 -5.328125 4.65625 -5.421875 4.25 -5.421875 C 3.53125 -5.421875 2.945312 -5.207031 2.5 -4.78125 C 2.0625 -4.363281 1.84375 -3.828125 1.84375 -3.171875 C 1.84375 -2.484375 2.054688 -1.925781 2.484375 -1.5 C 2.910156 -1.070312 3.457031 -0.859375 4.125 -0.859375 C 4.4375 -0.859375 4.726562 -0.898438 5 -0.984375 C 5.28125 -1.066406 5.539062 -1.191406 5.78125 -1.359375 C 5.863281 -1.410156 5.976562 -1.492188 6.125 -1.609375 C 6.425781 -1.847656 6.660156 -1.96875 6.828125 -1.96875 C 6.972656 -1.96875 7.085938 -1.921875 7.171875 -1.828125 C 7.265625 -1.742188 7.3125 -1.632812 7.3125 -1.5 C 7.3125 -1.132812 6.96875 -0.765625 6.28125 -0.390625 C 5.601562 -0.015625 4.878906 0.171875 4.109375 0.171875 C 3.117188 0.171875 2.296875 -0.144531 1.640625 -0.78125 C 0.984375 -1.414062 0.65625 -2.207031 0.65625 -3.15625 C 0.65625 -4.101562 0.984375 -4.894531 1.640625 -5.53125 C 2.296875 -6.164062 3.117188 -6.484375 4.109375 -6.484375 C 4.453125 -6.484375 4.796875 -6.4375 5.140625 -6.34375 C 5.492188 -6.25 5.863281 -6.101562 6.25 -5.90625 Z M 6.25 -5.90625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-10\" overflow=\"visible\">\n",
       "<path d=\"M 1.65625 -0.96875 L 1.65625 -5.34375 L 1.53125 -5.34375 C 1.21875 -5.34375 1.003906 -5.378906 0.890625 -5.453125 C 0.773438 -5.523438 0.71875 -5.65625 0.71875 -5.84375 C 0.71875 -6.007812 0.765625 -6.128906 0.859375 -6.203125 C 0.953125 -6.273438 1.101562 -6.3125 1.3125 -6.3125 L 2.296875 -6.3125 C 2.421875 -6.3125 2.515625 -6.285156 2.578125 -6.234375 C 2.648438 -6.191406 2.6875 -6.125 2.6875 -6.03125 L 2.6875 -5.578125 C 3.019531 -5.878906 3.347656 -6.101562 3.671875 -6.25 C 3.992188 -6.40625 4.332031 -6.484375 4.6875 -6.484375 C 5.375 -6.484375 5.914062 -6.269531 6.3125 -5.84375 C 6.707031 -5.414062 6.90625 -4.820312 6.90625 -4.0625 L 6.90625 -0.96875 L 7.21875 -0.96875 C 7.445312 -0.96875 7.613281 -0.925781 7.71875 -0.84375 C 7.820312 -0.769531 7.875 -0.65625 7.875 -0.5 C 7.875 -0.3125 7.816406 -0.179688 7.703125 -0.109375 C 7.597656 -0.0351562 7.410156 0 7.140625 0 L 5.625 0 C 5.332031 0 5.144531 -0.03125 5.0625 -0.09375 C 4.976562 -0.164062 4.9375 -0.300781 4.9375 -0.5 C 4.9375 -0.675781 4.984375 -0.796875 5.078125 -0.859375 C 5.179688 -0.929688 5.363281 -0.96875 5.625 -0.96875 L 5.859375 -0.96875 L 5.859375 -3.8125 C 5.859375 -4.34375 5.738281 -4.75 5.5 -5.03125 C 5.257812 -5.3125 4.910156 -5.453125 4.453125 -5.453125 C 3.984375 -5.453125 3.570312 -5.289062 3.21875 -4.96875 C 2.863281 -4.65625 2.6875 -4.269531 2.6875 -3.8125 L 2.6875 -0.96875 L 2.921875 -0.96875 C 3.203125 -0.96875 3.390625 -0.929688 3.484375 -0.859375 C 3.578125 -0.796875 3.625 -0.675781 3.625 -0.5 C 3.625 -0.300781 3.582031 -0.164062 3.5 -0.09375 C 3.414062 -0.03125 3.21875 0 2.90625 0 L 1.28125 0 C 1.09375 0 0.953125 -0.0351562 0.859375 -0.109375 C 0.765625 -0.191406 0.71875 -0.320312 0.71875 -0.5 C 0.71875 -0.675781 0.769531 -0.796875 0.875 -0.859375 C 0.988281 -0.929688 1.203125 -0.96875 1.515625 -0.96875 Z M 1.65625 -0.96875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-11\" overflow=\"visible\">\n",
       "<path d=\"M 4.5625 -0.953125 C 5.207031 -0.953125 5.726562 -1.15625 6.125 -1.5625 C 6.53125 -1.96875 6.734375 -2.492188 6.734375 -3.140625 C 6.734375 -3.773438 6.53125 -4.296875 6.125 -4.703125 C 5.71875 -5.109375 5.195312 -5.3125 4.5625 -5.3125 C 3.925781 -5.3125 3.40625 -5.109375 3 -4.703125 C 2.601562 -4.296875 2.40625 -3.773438 2.40625 -3.140625 C 2.40625 -2.503906 2.601562 -1.976562 3 -1.5625 C 3.40625 -1.15625 3.925781 -0.953125 4.5625 -0.953125 Z M 2.4375 -5.328125 C 2.6875 -5.648438 3.003906 -5.894531 3.390625 -6.0625 C 3.773438 -6.226562 4.207031 -6.3125 4.6875 -6.3125 C 5.601562 -6.3125 6.363281 -6.003906 6.96875 -5.390625 C 7.582031 -4.773438 7.890625 -4.007812 7.890625 -3.09375 C 7.890625 -2.195312 7.578125 -1.4375 6.953125 -0.8125 C 6.328125 -0.1875 5.5625 0.125 4.65625 0.125 C 4.175781 0.125 3.757812 0.0390625 3.40625 -0.125 C 3.050781 -0.300781 2.757812 -0.554688 2.53125 -0.890625 L 2.53125 -0.34375 C 2.53125 -0.226562 2.507812 -0.140625 2.46875 -0.078125 C 2.425781 -0.0234375 2.351562 0 2.25 0 L 1 0 C 0.820312 0 0.691406 -0.0390625 0.609375 -0.125 C 0.523438 -0.207031 0.484375 -0.332031 0.484375 -0.5 C 0.484375 -0.675781 0.535156 -0.796875 0.640625 -0.859375 C 0.742188 -0.929688 0.953125 -0.96875 1.265625 -0.96875 L 1.40625 -0.96875 L 1.40625 -7.96875 L 1.265625 -7.96875 C 0.960938 -7.96875 0.753906 -8.003906 0.640625 -8.078125 C 0.523438 -8.148438 0.46875 -8.273438 0.46875 -8.453125 C 0.46875 -8.617188 0.507812 -8.742188 0.59375 -8.828125 C 0.6875 -8.910156 0.820312 -8.953125 1 -8.953125 L 2.046875 -8.953125 C 2.179688 -8.953125 2.28125 -8.925781 2.34375 -8.875 C 2.40625 -8.832031 2.4375 -8.765625 2.4375 -8.671875 Z M 2.4375 -5.328125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-12\" overflow=\"visible\">\n",
       "<path d=\"M 3.6875 1.75 L 3.984375 1.75 C 4.316406 1.75 4.539062 1.785156 4.65625 1.859375 C 4.769531 1.929688 4.828125 2.0625 4.828125 2.25 C 4.828125 2.40625 4.78125 2.523438 4.6875 2.609375 C 4.59375 2.691406 4.457031 2.734375 4.28125 2.734375 L 1.140625 2.734375 C 0.921875 2.734375 0.769531 2.695312 0.6875 2.625 C 0.601562 2.550781 0.5625 2.425781 0.5625 2.25 C 0.5625 2.082031 0.601562 1.957031 0.6875 1.875 C 0.78125 1.789062 0.914062 1.75 1.09375 1.75 L 2.578125 1.75 L 3.6875 -0.4375 L 1.171875 -5.34375 C 0.867188 -5.34375 0.660156 -5.378906 0.546875 -5.453125 C 0.441406 -5.523438 0.390625 -5.65625 0.390625 -5.84375 C 0.390625 -6 0.429688 -6.113281 0.515625 -6.1875 C 0.609375 -6.269531 0.75 -6.3125 0.9375 -6.3125 L 2.84375 -6.3125 C 3.101562 -6.3125 3.273438 -6.273438 3.359375 -6.203125 C 3.441406 -6.140625 3.484375 -6.019531 3.484375 -5.84375 C 3.484375 -5.65625 3.425781 -5.523438 3.3125 -5.453125 C 3.195312 -5.378906 2.988281 -5.34375 2.6875 -5.34375 L 2.4375 -5.34375 L 4.28125 -1.578125 L 6.171875 -5.34375 L 5.90625 -5.34375 C 5.59375 -5.34375 5.375 -5.378906 5.25 -5.453125 C 5.132812 -5.523438 5.078125 -5.65625 5.078125 -5.84375 C 5.078125 -6.019531 5.117188 -6.140625 5.203125 -6.203125 C 5.296875 -6.273438 5.472656 -6.3125 5.734375 -6.3125 L 7.421875 -6.3125 C 7.609375 -6.3125 7.742188 -6.269531 7.828125 -6.1875 C 7.921875 -6.113281 7.96875 -6 7.96875 -5.84375 C 7.96875 -5.664062 7.914062 -5.539062 7.8125 -5.46875 C 7.707031 -5.394531 7.53125 -5.351562 7.28125 -5.34375 Z M 3.6875 1.75 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-13\" overflow=\"visible\">\n",
       "<path d=\"M 2.5625 -0.96875 L 3.015625 -0.96875 C 3.285156 -0.96875 3.472656 -0.929688 3.578125 -0.859375 C 3.691406 -0.785156 3.75 -0.664062 3.75 -0.5 C 3.75 -0.3125 3.703125 -0.179688 3.609375 -0.109375 C 3.515625 -0.0351562 3.304688 0 2.984375 0 L 1.171875 0 C 1.003906 0 0.878906 -0.0390625 0.796875 -0.125 C 0.710938 -0.207031 0.671875 -0.332031 0.671875 -0.5 C 0.671875 -0.675781 0.71875 -0.796875 0.8125 -0.859375 C 0.914062 -0.929688 1.101562 -0.96875 1.375 -0.96875 L 1.5 -0.96875 L 1.5 -7.125 L 1.390625 -7.125 C 1.117188 -7.125 0.929688 -7.160156 0.828125 -7.234375 C 0.722656 -7.304688 0.671875 -7.429688 0.671875 -7.609375 C 0.671875 -7.785156 0.710938 -7.910156 0.796875 -7.984375 C 0.878906 -8.066406 1.007812 -8.109375 1.1875 -8.109375 L 2.90625 -8.109375 C 3.269531 -8.109375 3.5 -8.070312 3.59375 -8 C 3.695312 -7.925781 3.75 -7.796875 3.75 -7.609375 C 3.75 -7.429688 3.6875 -7.304688 3.5625 -7.234375 C 3.445312 -7.160156 3.21875 -7.125 2.875 -7.125 L 2.5625 -7.125 L 2.5625 -4.75 L 5.84375 -4.75 L 5.84375 -7.125 L 5.375 -7.125 C 5.113281 -7.125 4.925781 -7.160156 4.8125 -7.234375 C 4.707031 -7.316406 4.65625 -7.441406 4.65625 -7.609375 C 4.65625 -7.796875 4.703125 -7.925781 4.796875 -8 C 4.898438 -8.070312 5.15625 -8.109375 5.5625 -8.109375 L 7.21875 -8.109375 C 7.382812 -8.109375 7.507812 -8.066406 7.59375 -7.984375 C 7.675781 -7.898438 7.71875 -7.773438 7.71875 -7.609375 C 7.71875 -7.429688 7.671875 -7.304688 7.578125 -7.234375 C 7.484375 -7.160156 7.3125 -7.125 7.0625 -7.125 L 6.90625 -7.125 L 6.90625 -0.96875 L 7.015625 -0.96875 C 7.285156 -0.96875 7.46875 -0.929688 7.5625 -0.859375 C 7.664062 -0.785156 7.71875 -0.664062 7.71875 -0.5 C 7.71875 -0.3125 7.671875 -0.179688 7.578125 -0.109375 C 7.484375 -0.0351562 7.265625 0 6.921875 0 L 5.5 0 C 5.132812 0 4.898438 -0.0351562 4.796875 -0.109375 C 4.703125 -0.179688 4.65625 -0.3125 4.65625 -0.5 C 4.65625 -0.675781 4.710938 -0.796875 4.828125 -0.859375 C 4.941406 -0.929688 5.164062 -0.96875 5.5 -0.96875 L 5.84375 -0.96875 L 5.84375 -3.8125 L 2.5625 -3.8125 Z M 2.5625 -0.96875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-14\" overflow=\"visible\">\n",
       "<path d=\"M 2.859375 -1.125 C 2.859375 -1.476562 2.984375 -1.769531 3.234375 -2 C 3.484375 -2.226562 3.804688 -2.34375 4.203125 -2.34375 C 4.609375 -2.34375 4.929688 -2.226562 5.171875 -2 C 5.421875 -1.78125 5.546875 -1.488281 5.546875 -1.125 C 5.546875 -0.757812 5.421875 -0.460938 5.171875 -0.234375 C 4.921875 -0.015625 4.597656 0.09375 4.203125 0.09375 C 3.796875 0.09375 3.46875 -0.015625 3.21875 -0.234375 C 2.976562 -0.460938 2.859375 -0.757812 2.859375 -1.125 Z M 2.859375 -1.125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-15\" overflow=\"visible\">\n",
       "<path d=\"M 6.375 -1.171875 L 6.375 -2.796875 L 4.671875 -2.796875 C 4.535156 -2.796875 4.429688 -2.832031 4.359375 -2.90625 C 4.296875 -2.976562 4.265625 -3.09375 4.265625 -3.25 C 4.265625 -3.414062 4.296875 -3.53125 4.359375 -3.59375 C 4.421875 -3.664062 4.535156 -3.703125 4.703125 -3.703125 L 7.65625 -3.703125 C 7.769531 -3.703125 7.851562 -3.660156 7.90625 -3.578125 C 7.96875 -3.503906 8 -3.394531 8 -3.25 C 8 -3.082031 7.960938 -2.960938 7.890625 -2.890625 C 7.816406 -2.828125 7.6875 -2.796875 7.5 -2.796875 L 7.40625 -2.796875 L 7.40625 -0.75 C 7.40625 -0.625 7.390625 -0.535156 7.359375 -0.484375 C 7.335938 -0.441406 7.285156 -0.394531 7.203125 -0.34375 C 6.941406 -0.1875 6.5625 -0.0546875 6.0625 0.046875 C 5.5625 0.160156 5.054688 0.21875 4.546875 0.21875 C 3.328125 0.21875 2.34375 -0.171875 1.59375 -0.953125 C 0.84375 -1.734375 0.46875 -2.75 0.46875 -4 C 0.46875 -5.257812 0.832031 -6.296875 1.5625 -7.109375 C 2.300781 -7.921875 3.238281 -8.328125 4.375 -8.328125 C 4.6875 -8.328125 5 -8.289062 5.3125 -8.21875 C 5.625 -8.15625 5.953125 -8.050781 6.296875 -7.90625 C 6.335938 -8.03125 6.394531 -8.117188 6.46875 -8.171875 C 6.550781 -8.222656 6.65625 -8.25 6.78125 -8.25 C 7 -8.25 7.144531 -8.179688 7.21875 -8.046875 C 7.289062 -7.921875 7.328125 -7.628906 7.328125 -7.171875 L 7.328125 -6.234375 C 7.328125 -6.015625 7.285156 -5.859375 7.203125 -5.765625 C 7.128906 -5.679688 6.992188 -5.640625 6.796875 -5.640625 C 6.566406 -5.640625 6.398438 -5.785156 6.296875 -6.078125 C 6.253906 -6.160156 6.222656 -6.226562 6.203125 -6.28125 C 6.035156 -6.632812 5.789062 -6.910156 5.46875 -7.109375 C 5.15625 -7.304688 4.796875 -7.40625 4.390625 -7.40625 C 3.578125 -7.40625 2.925781 -7.109375 2.4375 -6.515625 C 1.945312 -5.929688 1.703125 -5.132812 1.703125 -4.125 C 1.703125 -3.09375 1.972656 -2.269531 2.515625 -1.65625 C 3.054688 -1.050781 3.769531 -0.75 4.65625 -0.75 C 4.914062 -0.75 5.179688 -0.78125 5.453125 -0.84375 C 5.734375 -0.914062 6.039062 -1.023438 6.375 -1.171875 Z M 6.375 -1.171875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-16\" overflow=\"visible\">\n",
       "<path d=\"M 2.4375 -1.75 L 3.609375 -5.84375 C 3.648438 -5.988281 3.695312 -6.078125 3.75 -6.109375 C 3.800781 -6.148438 3.894531 -6.171875 4.03125 -6.171875 L 4.34375 -6.171875 C 4.476562 -6.171875 4.570312 -6.148438 4.625 -6.109375 C 4.675781 -6.066406 4.722656 -5.984375 4.765625 -5.859375 L 6.109375 -1.75 L 6.78125 -7.1875 L 5.703125 -7.1875 C 5.492188 -7.1875 5.335938 -7.222656 5.234375 -7.296875 C 5.140625 -7.367188 5.09375 -7.484375 5.09375 -7.640625 C 5.09375 -7.816406 5.140625 -7.9375 5.234375 -8 C 5.328125 -8.070312 5.5 -8.109375 5.75 -8.109375 L 7.96875 -8.109375 C 8.0625 -8.109375 8.144531 -8.0625 8.21875 -7.96875 C 8.300781 -7.882812 8.34375 -7.773438 8.34375 -7.640625 C 8.34375 -7.484375 8.300781 -7.367188 8.21875 -7.296875 C 8.132812 -7.222656 8.003906 -7.1875 7.828125 -7.1875 L 7.75 -7.1875 L 6.78125 -0.34375 C 6.757812 -0.207031 6.726562 -0.113281 6.6875 -0.0625 C 6.65625 -0.0195312 6.597656 0 6.515625 0 L 6.015625 0 C 5.910156 0 5.832031 -0.0195312 5.78125 -0.0625 C 5.726562 -0.101562 5.679688 -0.179688 5.640625 -0.296875 L 4.15625 -5 L 2.84375 -0.28125 C 2.8125 -0.164062 2.769531 -0.0859375 2.71875 -0.046875 C 2.664062 -0.015625 2.582031 0 2.46875 0 L 1.96875 0 C 1.875 0 1.804688 -0.0195312 1.765625 -0.0625 C 1.722656 -0.101562 1.691406 -0.203125 1.671875 -0.359375 L 0.625 -7.1875 L 0.546875 -7.1875 C 0.367188 -7.1875 0.234375 -7.222656 0.140625 -7.296875 C 0.0546875 -7.378906 0.015625 -7.492188 0.015625 -7.640625 C 0.015625 -7.773438 0.0507812 -7.882812 0.125 -7.96875 C 0.195312 -8.0625 0.285156 -8.109375 0.390625 -8.109375 L 2.609375 -8.109375 C 2.859375 -8.109375 3.03125 -8.070312 3.125 -8 C 3.21875 -7.9375 3.265625 -7.816406 3.265625 -7.640625 C 3.265625 -7.484375 3.210938 -7.367188 3.109375 -7.296875 C 3.015625 -7.222656 2.863281 -7.1875 2.65625 -7.1875 L 1.703125 -7.1875 Z M 2.4375 -1.75 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-17\" overflow=\"visible\">\n",
       "<path d=\"M 3.734375 -7.96875 L 2.578125 -7.96875 C 2.140625 -7.96875 1.863281 -8 1.75 -8.0625 C 1.632812 -8.132812 1.578125 -8.265625 1.578125 -8.453125 C 1.578125 -8.617188 1.625 -8.742188 1.71875 -8.828125 C 1.8125 -8.910156 1.945312 -8.953125 2.125 -8.953125 L 4.375 -8.953125 C 4.5 -8.953125 4.59375 -8.925781 4.65625 -8.875 C 4.726562 -8.820312 4.765625 -8.753906 4.765625 -8.671875 L 4.765625 -0.96875 L 6.921875 -0.96875 C 7.085938 -0.96875 7.210938 -0.925781 7.296875 -0.84375 C 7.378906 -0.769531 7.421875 -0.65625 7.421875 -0.5 C 7.421875 -0.3125 7.378906 -0.179688 7.296875 -0.109375 C 7.222656 -0.0351562 7.078125 0 6.859375 0 L 1.8125 0 C 1.632812 0 1.5 -0.0390625 1.40625 -0.125 C 1.3125 -0.207031 1.265625 -0.332031 1.265625 -0.5 C 1.265625 -0.65625 1.3125 -0.769531 1.40625 -0.84375 C 1.507812 -0.925781 1.65625 -0.96875 1.84375 -0.96875 L 3.734375 -0.96875 Z M 3.734375 -7.96875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-18\" overflow=\"visible\">\n",
       "<path d=\"M 1.21875 -0.5 L 1.21875 -1.703125 C 1.21875 -1.910156 1.253906 -2.054688 1.328125 -2.140625 C 1.398438 -2.222656 1.523438 -2.265625 1.703125 -2.265625 C 1.878906 -2.265625 2.023438 -2.140625 2.140625 -1.890625 C 2.210938 -1.734375 2.28125 -1.613281 2.34375 -1.53125 C 2.53125 -1.289062 2.8125 -1.09375 3.1875 -0.9375 C 3.5625 -0.789062 3.972656 -0.71875 4.421875 -0.71875 C 4.898438 -0.71875 5.285156 -0.816406 5.578125 -1.015625 C 5.867188 -1.210938 6.015625 -1.472656 6.015625 -1.796875 C 6.015625 -2.117188 5.894531 -2.347656 5.65625 -2.484375 C 5.425781 -2.617188 5.03125 -2.6875 4.46875 -2.6875 L 4.09375 -2.6875 C 3.15625 -2.6875 2.441406 -2.847656 1.953125 -3.171875 C 1.472656 -3.492188 1.234375 -3.960938 1.234375 -4.578125 C 1.234375 -5.179688 1.46875 -5.648438 1.9375 -5.984375 C 2.414062 -6.316406 3.085938 -6.484375 3.953125 -6.484375 C 4.304688 -6.484375 4.664062 -6.429688 5.03125 -6.328125 C 5.394531 -6.234375 5.582031 -6.1875 5.59375 -6.1875 C 5.644531 -6.1875 5.738281 -6.226562 5.875 -6.3125 C 6.019531 -6.40625 6.160156 -6.453125 6.296875 -6.453125 C 6.453125 -6.453125 6.566406 -6.394531 6.640625 -6.28125 C 6.722656 -6.164062 6.765625 -5.992188 6.765625 -5.765625 L 6.765625 -4.96875 C 6.765625 -4.726562 6.726562 -4.550781 6.65625 -4.4375 C 6.582031 -4.332031 6.460938 -4.28125 6.296875 -4.28125 C 6.160156 -4.28125 5.992188 -4.398438 5.796875 -4.640625 C 5.648438 -4.796875 5.535156 -4.914062 5.453125 -5 C 5.210938 -5.21875 4.953125 -5.375 4.671875 -5.46875 C 4.398438 -5.570312 4.082031 -5.625 3.71875 -5.625 C 3.28125 -5.625 2.929688 -5.53125 2.671875 -5.34375 C 2.410156 -5.15625 2.28125 -4.914062 2.28125 -4.625 C 2.28125 -4.332031 2.414062 -4.125 2.6875 -4 C 2.957031 -3.882812 3.535156 -3.816406 4.421875 -3.796875 C 5.367188 -3.773438 6.0625 -3.609375 6.5 -3.296875 C 6.9375 -2.992188 7.15625 -2.519531 7.15625 -1.875 C 7.15625 -1.257812 6.910156 -0.765625 6.421875 -0.390625 C 5.941406 -0.0234375 5.296875 0.15625 4.484375 0.15625 C 4.222656 0.15625 3.929688 0.128906 3.609375 0.078125 C 3.285156 0.0351562 2.882812 -0.0390625 2.40625 -0.15625 C 2.25 -0.0507812 2.113281 0.0234375 2 0.078125 C 1.882812 0.128906 1.785156 0.15625 1.703125 0.15625 C 1.546875 0.15625 1.425781 0.101562 1.34375 0 C 1.257812 -0.113281 1.21875 -0.28125 1.21875 -0.5 Z M 1.21875 -0.5 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<path d=\"M -107.4375 189.394531 L 392.664062 188.40625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(100%,100%,100%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 396.664062 188.398438 L 392.660156 186.90625 L 392.664062 189.90625 Z M 396.664062 188.398438 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(100%,100%,100%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M -107.4375 189.394531 L -65.894531 189.3125 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M -61.894531 189.304688 L -65.898438 187.8125 L -65.890625 190.8125 Z M -61.894531 189.304688 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"8.7222\" xlink:href=\"#glyph0-1\" y=\"19\"/>\n",
       "  <use x=\"17.1236\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"25.525\" xlink:href=\"#glyph0-3\" y=\"19\"/>\n",
       "  <use x=\"33.9264\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"42.3278\" xlink:href=\"#glyph0-4\" y=\"19\"/>\n",
       "  <use x=\"50.7292\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"59.1306\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"67.532\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"75.9334\" xlink:href=\"#glyph0-1\" y=\"19\"/>\n",
       "  <use x=\"84.3348\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"92.7362\" xlink:href=\"#glyph0-5\" y=\"19\"/>\n",
       "  <use x=\"101.1376\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"109.539\" xlink:href=\"#glyph0-6\" y=\"19\"/>\n",
       "  <use x=\"117.9404\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"126.3418\" xlink:href=\"#glyph0-4\" y=\"19\"/>\n",
       "  <use x=\"134.7432\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"143.1446\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"151.546\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"159.9474\" xlink:href=\"#glyph0-7\" y=\"19\"/>\n",
       "  <use x=\"168.3488\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"176.7502\" xlink:href=\"#glyph0-8\" y=\"19\"/>\n",
       "  <use x=\"185.1516\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"193.553\" xlink:href=\"#glyph0-9\" y=\"19\"/>\n",
       "  <use x=\"201.9544\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"210.3558\" xlink:href=\"#glyph0-3\" y=\"19\"/>\n",
       "  <use x=\"218.7572\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"227.1586\" xlink:href=\"#glyph0-5\" y=\"19\"/>\n",
       "  <use x=\"235.56\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"243.9614\" xlink:href=\"#glyph0-10\" y=\"19\"/>\n",
       "  <use x=\"252.3628\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"260.7642\" xlink:href=\"#glyph0-4\" y=\"19\"/>\n",
       "  <use x=\"269.1656\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"277.567\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"285.9684\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"294.3698\" xlink:href=\"#glyph0-11\" y=\"19\"/>\n",
       "  <use x=\"302.7712\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"311.1726\" xlink:href=\"#glyph0-12\" y=\"19\"/>\n",
       "  <use x=\"319.574\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"327.9754\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"336.3768\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"344.7782\" xlink:href=\"#glyph0-13\" y=\"19\"/>\n",
       "  <use x=\"353.1796\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"361.581\" xlink:href=\"#glyph0-14\" y=\"19\"/>\n",
       "  <use x=\"369.9824\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"378.3838\" xlink:href=\"#glyph0-15\" y=\"19\"/>\n",
       "  <use x=\"386.7852\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"395.1866\" xlink:href=\"#glyph0-14\" y=\"19\"/>\n",
       "  <use x=\"403.588\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"411.9894\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"420.3908\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"428.7922\" xlink:href=\"#glyph0-16\" y=\"19\"/>\n",
       "  <use x=\"437.1936\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"445.595\" xlink:href=\"#glyph0-4\" y=\"19\"/>\n",
       "  <use x=\"453.9964\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"462.3978\" xlink:href=\"#glyph0-17\" y=\"19\"/>\n",
       "  <use x=\"470.7992\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"479.2006\" xlink:href=\"#glyph0-17\" y=\"19\"/>\n",
       "  <use x=\"487.602\" xlink:href=\"#glyph0-2\" y=\"19\"/>\n",
       "  <use x=\"496.0034\" xlink:href=\"#glyph0-18\" y=\"19\"/>\n",
       "</g>\n",
       "<path d=\"M -107.4375 231.394531 L 392.664062 230.40625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 396.664062 230.398438 L 392.660156 228.90625 L 392.664062 231.90625 Z M 396.664062 230.398438 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M -107.4375 209.785156 L -43.332031 209.988281 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M -39.332031 210 L -43.328125 208.488281 L -43.335938 211.488281 Z M -39.332031 210 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M -37.4375 209.785156 L 26.667969 209.988281 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 30.667969 210 L 26.671875 208.488281 L 26.664062 211.488281 Z M 30.667969 210 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 32.5625 209.785156 L 96.667969 209.988281 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 100.667969 210 L 96.671875 208.488281 L 96.664062 211.488281 Z M 100.667969 210 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 102.5625 209.785156 L 166.667969 209.988281 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 170.667969 210 L 166.671875 208.488281 L 166.664062 211.488281 Z M 170.667969 210 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 172.5625 209.785156 L 236.667969 209.988281 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 240.667969 210 L 236.671875 208.488281 L 236.664062 211.488281 Z M 240.667969 210 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 242.5625 209.785156 L 306.667969 209.988281 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 310.667969 210 L 306.671875 208.488281 L 306.664062 211.488281 Z M 310.667969 210 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 312.5625 209.785156 L 376.667969 209.988281 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 380.667969 210 L 376.671875 208.488281 L 376.664062 211.488281 Z M 380.667969 210 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M -58.992188 189.296875 L 42.101562 189.015625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 46.101562 189.003906 L 42.097656 187.515625 L 42.105469 190.515625 Z M 46.101562 189.003906 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 48 189 L 77.101562 189.015625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 81.101562 189.015625 L 77.101562 187.515625 L 77.101562 190.515625 Z M 81.101562 189.015625 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 82 189.019531 L 110.601562 189.332031 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 114.601562 189.375 L 110.617188 187.832031 L 110.585938 190.832031 Z M 114.601562 189.375 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 116.503906 189.394531 L 229.09375 188.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 233.09375 188.730469 L 229.085938 187.25 L 229.101562 190.25 Z M 233.09375 188.730469 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 236 189.394531 L 329.105469 188.570312 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 333.105469 188.535156 L 329.09375 187.074219 L 329.117188 190.070312 Z M 333.105469 188.535156 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 330.0625 188.394531 L 392.664062 188.394531 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "<path d=\"M 396.664062 188.394531 L 392.664062 186.894531 L 392.664062 189.894531 Z M 396.664062 188.394531 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,111,-144)\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "SVG('../img/truncated-bptt.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Fig.10.7.1: From top to bottom: randomized BPTT, regularly truncated BPTT and full BPTT\n",
    "\n",
    "The picture above illustrates the three cases when analyzing the first few words of *The Time Machine*: randomized truncation partitions the text into segments of varying length. Regular truncated BPTT breaks it into sequences of the same length, and full BPTT leads to a computationally infeasible expression.\n",
    "\n",
    "## The Computational Graph\n",
    "\n",
    "In order to visualize the dependencies between model variables and parameters during computation in a recurrent neural network, we can draw a computational graph for the model, as shown below. For example, the computation of the hidden states of time step 3 $\\mathbf{h}_3$ depends on the model parameters $\\mathbf{W}_{hx}$ and $\\mathbf{W}_{hh}$, the hidden state of the last time step $\\mathbf{h}_2$, and the input of the current time step $\\mathbf{x}_3$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"132pt\" version=\"1.1\" viewBox=\"0 0 303 132\" width=\"303pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M 0.375 -3.9375 L 0.375 -3.703125 C 0.546875 -3.71875 0.609375 -3.71875 0.6875 -3.71875 C 0.96875 -3.71875 1.109375 -3.53125 1.203125 -3.09375 L 1.453125 -1.734375 L 1.0625 -1.09375 C 0.875 -0.78125 0.796875 -0.734375 0.640625 -0.734375 C 0.453125 -0.734375 0.34375 -0.921875 0.078125 -0.921875 C -0.203125 -0.921875 -0.40625 -0.703125 -0.40625 -0.40625 C -0.40625 -0.09375 -0.1875 0.109375 0.15625 0.109375 C 0.609375 0.109375 0.8125 -0.0625 1.34375 -0.921875 L 1.5625 -1.28125 C 1.75 -0.25 1.984375 0.109375 2.4375 0.109375 C 2.859375 0.109375 3.234375 -0.1875 3.78125 -0.984375 L 3.59375 -1.109375 C 3.25 -0.65625 3.15625 -0.578125 3 -0.578125 C 2.828125 -0.578125 2.734375 -0.703125 2.65625 -1.046875 L 2.375 -2.46875 C 2.65625 -2.984375 3 -3.3125 3.203125 -3.3125 C 3.40625 -3.3125 3.5 -3.125 3.71875 -3.125 C 3.984375 -3.125 4.21875 -3.34375 4.21875 -3.609375 C 4.21875 -3.890625 3.984375 -4.15625 3.703125 -4.15625 C 3.28125 -4.15625 2.984375 -3.9375 2.28125 -2.921875 C 2.140625 -3.6875 2.03125 -3.90625 1.859375 -4.15625 Z M 0.375 -3.9375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-2\" overflow=\"visible\">\n",
       "<path d=\"M 4.28125 -1.28125 C 3.890625 -0.71875 3.71875 -0.53125 3.609375 -0.53125 C 3.53125 -0.53125 3.453125 -0.609375 3.453125 -0.671875 C 3.453125 -0.703125 3.453125 -0.71875 3.484375 -0.78125 L 4.015625 -2.421875 C 4.1875 -2.984375 4.28125 -3.3125 4.28125 -3.484375 C 4.28125 -3.90625 4.03125 -4.15625 3.59375 -4.15625 C 2.984375 -4.15625 2.5625 -3.8125 1.671875 -2.546875 L 2.640625 -6.296875 C 1.953125 -6.15625 1.5 -6.078125 0.765625 -6.015625 L 0.765625 -5.78125 L 0.96875 -5.78125 C 1.21875 -5.78125 1.34375 -5.6875 1.34375 -5.484375 C 1.34375 -5.40625 1.3125 -5.25 1.25 -5.0625 L -0.109375 0 L 0.96875 0 C 1.4375 -1.578125 1.609375 -2.015625 2.046875 -2.65625 C 2.4375 -3.234375 2.75 -3.515625 3 -3.515625 C 3.109375 -3.515625 3.171875 -3.46875 3.171875 -3.328125 C 3.171875 -3.265625 3.15625 -3.1875 3.109375 -3.046875 L 2.9375 -2.515625 C 2.484375 -1.1875 2.390625 -0.859375 2.390625 -0.5 C 2.390625 -0.140625 2.625 0.078125 2.984375 0.078125 C 3.53125 0.078125 3.890625 -0.21875 4.484375 -1.140625 Z M 4.28125 -1.28125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-3\" overflow=\"visible\">\n",
       "<path d=\"M 2.515625 -4.15625 C 1.90625 -4.15625 1.328125 -3.921875 0.875 -3.46875 C 0.328125 -2.9375 -0.03125 -2.109375 -0.03125 -1.21875 C -0.03125 -0.328125 0.609375 0.109375 1.453125 0.109375 C 2.046875 0.109375 2.578125 -0.109375 3.03125 -0.5625 C 3.578125 -1.109375 3.96875 -1.953125 3.96875 -2.796875 C 3.96875 -3.625 3.421875 -4.15625 2.5625 -4.15625 Z M 2.890625 -3.34375 C 2.890625 -2.5625 2.5625 -1.3125 2.125 -0.578125 C 1.9375 -0.28125 1.71875 -0.140625 1.484375 -0.140625 C 1.21875 -0.140625 1.046875 -0.328125 1.046875 -0.71875 C 1.046875 -1.640625 1.390625 -3.015625 1.9375 -3.625 C 2.109375 -3.8125 2.28125 -3.890625 2.46875 -3.890625 C 2.734375 -3.890625 2.890625 -3.6875 2.890625 -3.34375 Z M 2.890625 -3.34375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-4\" overflow=\"visible\">\n",
       "<path d=\"M 8.453125 -6.015625 L 6.78125 -6.015625 L 6.78125 -5.796875 C 7.28125 -5.765625 7.34375 -5.71875 7.34375 -5.390625 C 7.34375 -5.28125 7.3125 -5.21875 7.203125 -4.984375 L 5.8125 -2.03125 L 5.515625 -4.65625 C 5.484375 -5 5.453125 -5.203125 5.453125 -5.3125 C 5.453125 -5.65625 5.53125 -5.75 6.015625 -5.796875 L 6.015625 -6.015625 L 3.609375 -6.015625 L 3.609375 -5.8125 C 4.15625 -5.734375 4.234375 -5.578125 4.234375 -4.859375 L 2.890625 -2.03125 L 2.53125 -5.171875 C 2.53125 -5.21875 2.53125 -5.296875 2.53125 -5.34375 C 2.53125 -5.625 2.65625 -5.734375 3.078125 -5.796875 L 3.078125 -6.015625 L 0.578125 -6.015625 L 0.578125 -5.8125 C 1.15625 -5.703125 1.1875 -5.625 1.28125 -4.984375 L 1.984375 0.15625 L 2.234375 0.15625 L 4.34375 -4.234375 L 4.875 0.15625 L 5.125 0.15625 L 7.671875 -5.0625 C 7.96875 -5.671875 8.078125 -5.75 8.453125 -5.8125 Z M 8.453125 -6.015625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-5\" overflow=\"visible\">\n",
       "<path d=\"M -0.421875 1.640625 C 0.640625 1.1875 1.296875 0.4375 1.296875 -0.359375 C 1.296875 -0.84375 1.03125 -1.203125 0.546875 -1.203125 C 0.1875 -1.203125 -0.109375 -0.984375 -0.109375 -0.546875 C -0.109375 -0.3125 -0.03125 -0.125 0.25 0.03125 C 0.46875 0.15625 0.515625 0.25 0.515625 0.375 C 0.515625 0.6875 0.171875 1.03125 -0.546875 1.40625 Z M -0.421875 1.640625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-6\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-1\" overflow=\"visible\">\n",
       "<path d=\"M 2.8125 -4.734375 C 2.265625 -4.625 2.234375 -4.640625 1.625 -4.5 L 1.46875 -4.46875 L 1.46875 -4.359375 C 1.59375 -4.375 1.75 -4.375 1.875 -4.375 L 1.890625 -4.375 C 2.046875 -4.375 2.140625 -4.296875 2.140625 -4.15625 C 2.140625 -4.109375 2.125 -4.03125 2.09375 -3.90625 L 1.140625 -0.484375 C 1.0625 -0.21875 0.90625 -0.140625 0.34375 -0.109375 L 0.34375 0 L 2.296875 0 L 2.296875 -0.109375 L 2.15625 -0.109375 C 1.84375 -0.109375 1.6875 -0.171875 1.6875 -0.328125 C 1.6875 -0.40625 1.765625 -0.671875 1.8125 -0.84375 L 2.828125 -4.484375 C 2.84375 -4.578125 2.859375 -4.640625 2.859375 -4.6875 C 2.859375 -4.71875 2.84375 -4.734375 2.828125 -4.734375 Z M 2.8125 -4.734375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.703125 -3.546875 C 0.953125 -4.015625 1.3125 -4.21875 1.734375 -4.21875 C 2.28125 -4.21875 2.5625 -3.859375 2.5625 -3.375 C 2.5625 -3.21875 2.546875 -3.09375 2.5 -3 C 2.328125 -2.625 2.015625 -2.171875 1.546875 -1.6875 L 0.078125 -0.125 L 0.078125 0 L 2.5625 0 L 2.90625 -0.9375 L 2.796875 -0.984375 C 2.625 -0.640625 2.484375 -0.53125 2.078125 -0.53125 L 0.8125 -0.53125 L 0.8125 -0.5625 L 2.125 -1.90625 C 2.953125 -2.765625 3.15625 -3.078125 3.15625 -3.546875 C 3.15625 -4.21875 2.703125 -4.734375 2.03125 -4.734375 C 1.703125 -4.734375 1.40625 -4.640625 1.140625 -4.4375 C 0.890625 -4.25 0.734375 -4.015625 0.5625 -3.59375 Z M 0.703125 -3.546875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-3\" overflow=\"visible\">\n",
       "<path d=\"M 1.25 -3.96875 C 1.5 -4.296875 1.734375 -4.4375 2.03125 -4.4375 C 2.40625 -4.4375 2.640625 -4.1875 2.640625 -3.796875 C 2.640625 -3.5 2.5 -3.265625 2.21875 -3.0625 C 1.90625 -2.84375 1.609375 -2.75 0.984375 -2.640625 L 0.984375 -2.53125 L 1.15625 -2.53125 C 1.828125 -2.53125 2.28125 -2.171875 2.28125 -1.421875 C 2.28125 -1.09375 2.1875 -0.75 2.015625 -0.53125 C 1.828125 -0.34375 1.59375 -0.21875 1.359375 -0.21875 C 1.21875 -0.21875 1.046875 -0.28125 0.859375 -0.4375 C 0.671875 -0.609375 0.546875 -0.65625 0.40625 -0.65625 C 0.234375 -0.65625 0.109375 -0.53125 0.109375 -0.34375 C 0.109375 -0.125 0.390625 0.046875 0.890625 0.046875 C 1.390625 0.046875 1.84375 -0.078125 2.1875 -0.390625 C 2.578125 -0.734375 2.828125 -1.21875 2.828125 -1.78125 C 2.828125 -2.28125 2.640625 -2.609375 2.171875 -2.78125 L 2.171875 -2.8125 C 2.96875 -3.09375 3.25 -3.359375 3.25 -3.875 C 3.25 -4.375 2.875 -4.734375 2.328125 -4.734375 C 1.8125 -4.734375 1.359375 -4.515625 1.140625 -4 Z M 1.25 -3.96875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-4\" overflow=\"visible\">\n",
       "<path d=\"M 3.484375 -2.96875 C 3.484375 -4.109375 3.078125 -4.734375 2.359375 -4.734375 C 1.265625 -4.734375 0.21875 -3.15625 0.21875 -1.515625 C 0.21875 -0.53125 0.609375 0.046875 1.328125 0.046875 C 1.625 0.046875 1.921875 -0.046875 2.21875 -0.25 C 2.9375 -0.765625 3.484375 -1.953125 3.484375 -2.96875 Z M 2.921875 -3.625 C 2.921875 -2.71875 2.65625 -1.5625 2.25 -0.84375 C 1.984375 -0.390625 1.671875 -0.140625 1.34375 -0.140625 C 1.015625 -0.140625 0.78125 -0.5 0.78125 -1.09375 C 0.78125 -1.84375 1.140625 -3.21875 1.5 -3.875 C 1.75 -4.328125 2.015625 -4.53125 2.34375 -4.53125 C 2.765625 -4.53125 2.921875 -4.1875 2.921875 -3.625 Z M 2.921875 -3.625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-5\" overflow=\"visible\">\n",
       "<path d=\"M 3.28125 -2.109375 C 3.28125 -2.71875 2.859375 -3.09375 2.28125 -3.09375 C 2 -3.09375 1.71875 -3.03125 1.453125 -2.875 C 0.71875 -2.484375 0.1875 -1.6875 0.1875 -0.890625 C 0.1875 -0.28125 0.5625 0.078125 1.21875 0.078125 C 1.6875 0.078125 2.125 -0.09375 2.53125 -0.5 C 3 -0.953125 3.28125 -1.578125 3.28125 -2.109375 Z M 2.6875 -2.28125 C 2.6875 -1.75 2.484375 -1.09375 2.171875 -0.65625 C 1.890625 -0.265625 1.625 -0.0625 1.328125 -0.0625 C 1 -0.0625 0.78125 -0.28125 0.78125 -0.703125 C 0.78125 -1.265625 0.96875 -1.8125 1.296875 -2.296875 C 1.578125 -2.71875 1.890625 -2.9375 2.1875 -2.9375 C 2.484375 -2.9375 2.6875 -2.703125 2.6875 -2.28125 Z M 2.6875 -2.28125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-6\" overflow=\"visible\">\n",
       "<path d=\"M 3.25 -0.828125 C 2.875 -0.34375 2.796875 -0.265625 2.6875 -0.265625 C 2.625 -0.265625 2.5625 -0.328125 2.5625 -0.390625 C 2.5625 -0.46875 2.6875 -0.90625 2.796875 -1.265625 C 3.015625 -1.921875 3.140625 -2.4375 3.140625 -2.625 C 3.140625 -2.875 3 -3.09375 2.6875 -3.09375 C 2.25 -3.09375 1.703125 -2.625 1.078125 -1.609375 L 1.921875 -4.734375 L 1.875 -4.78125 C 1.46875 -4.6875 1.1875 -4.640625 0.765625 -4.59375 L 0.765625 -4.484375 L 0.953125 -4.484375 C 1.078125 -4.484375 1.25 -4.4375 1.25 -4.3125 C 1.25 -4.21875 1.21875 -4.09375 1.15625 -3.875 L 0.140625 0 L 0.65625 0 C 0.9375 -1.078125 1.046875 -1.328125 1.328125 -1.75 C 1.6875 -2.296875 2.140625 -2.734375 2.390625 -2.734375 C 2.5625 -2.734375 2.59375 -2.65625 2.59375 -2.5625 C 2.59375 -2.53125 2.5625 -2.4375 2.546875 -2.328125 L 2.15625 -0.875 C 2.0625 -0.53125 2.015625 -0.3125 2.015625 -0.234375 C 2.015625 -0.046875 2.140625 0.0625 2.328125 0.0625 C 2.6875 0.0625 2.921875 -0.125 3.34375 -0.734375 Z M 3.25 -0.828125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-7\" overflow=\"visible\">\n",
       "<path d=\"M 1.703125 -2.484375 C 1.609375 -2.96875 1.515625 -3.09375 1.34375 -3.09375 C 1.1875 -3.09375 0.953125 -3.03125 0.53125 -2.890625 L 0.453125 -2.859375 L 0.46875 -2.75 L 0.59375 -2.78125 C 0.734375 -2.8125 0.8125 -2.828125 0.875 -2.828125 C 1.046875 -2.828125 1.09375 -2.765625 1.1875 -2.359375 L 1.390625 -1.484375 L 0.8125 -0.671875 C 0.671875 -0.46875 0.53125 -0.328125 0.453125 -0.328125 C 0.40625 -0.328125 0.34375 -0.34375 0.265625 -0.390625 C 0.1875 -0.4375 0.109375 -0.46875 0.046875 -0.46875 C -0.09375 -0.46875 -0.1875 -0.359375 -0.1875 -0.21875 C -0.1875 -0.03125 -0.0625 0.078125 0.15625 0.078125 C 0.375 0.078125 0.515625 0.015625 0.828125 -0.40625 L 1.4375 -1.234375 L 1.640625 -0.40625 C 1.734375 -0.046875 1.828125 0.078125 2.0625 0.078125 C 2.328125 0.078125 2.5 -0.09375 2.90625 -0.71875 L 2.8125 -0.78125 C 2.75 -0.71875 2.71875 -0.671875 2.65625 -0.59375 C 2.5 -0.375 2.421875 -0.3125 2.328125 -0.3125 C 2.234375 -0.3125 2.171875 -0.40625 2.125 -0.59375 L 1.890625 -1.53125 C 1.859375 -1.703125 1.84375 -1.796875 1.84375 -1.84375 C 2.15625 -2.390625 2.40625 -2.6875 2.53125 -2.6875 C 2.6875 -2.6875 2.765625 -2.578125 2.890625 -2.578125 C 3.03125 -2.578125 3.125 -2.6875 3.125 -2.828125 C 3.125 -2.984375 3.015625 -3.09375 2.84375 -3.09375 C 2.53125 -3.09375 2.28125 -2.828125 1.78125 -2.09375 Z M 1.703125 -2.484375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-8\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-1\" overflow=\"visible\">\n",
       "<path d=\"M 2.1875 -1.671875 C 2.03125 -2.46875 1.5625 -3.96875 1.421875 -3.96875 L 1.390625 -3.96875 C 0.90625 -3.890625 0.609375 -3.84375 0.421875 -3.8125 C 0.328125 -3.78125 0.234375 -3.765625 0.140625 -3.75 L 0.140625 -3.59375 C 0.234375 -3.625 0.3125 -3.640625 0.40625 -3.640625 C 0.78125 -3.640625 0.9375 -3.5 1.109375 -3.0625 C 1.34375 -2.453125 1.84375 -0.4375 1.84375 -0.078125 C 1.84375 0.03125 1.8125 0.140625 1.75 0.25 C 1.6875 0.359375 1.28125 0.890625 1.109375 1.0625 C 0.90625 1.28125 0.796875 1.359375 0.6875 1.359375 C 0.453125 1.359375 0.453125 1.109375 0.140625 1.109375 C -0.0625 1.109375 -0.21875 1.265625 -0.21875 1.46875 C -0.21875 1.703125 -0.03125 1.859375 0.25 1.859375 C 0.75 1.859375 1.71875 0.8125 2.65625 -0.734375 C 3.40625 -1.96875 3.828125 -2.96875 3.828125 -3.46875 C 3.828125 -3.75 3.640625 -3.96875 3.359375 -3.96875 C 3.15625 -3.96875 3 -3.828125 3 -3.640625 C 3 -3.484375 3.0625 -3.40625 3.234375 -3.28125 C 3.40625 -3.171875 3.46875 -3.125 3.46875 -3 C 3.46875 -2.59375 3.140625 -1.953125 2.375 -0.6875 Z M 2.1875 -1.671875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-2\" overflow=\"visible\">\n",
       "<path d=\"M 5.03125 -1.625 L 4.84375 -1.671875 C 4.59375 -1.15625 4.40625 -0.859375 4.109375 -0.671875 C 3.671875 -0.375 3.390625 -0.328125 2.5 -0.328125 C 1.75 -0.328125 1.546875 -0.40625 1.546875 -0.6875 C 1.546875 -0.8125 1.578125 -0.96875 1.640625 -1.171875 L 2.71875 -5.046875 C 2.859375 -5.546875 3.09375 -5.703125 3.625 -5.734375 L 3.625 -5.875 L 1.171875 -5.875 L 1.171875 -5.734375 C 1.734375 -5.703125 1.859375 -5.609375 1.859375 -5.34375 C 1.859375 -5.21875 1.84375 -5.03125 1.765625 -4.78125 L 0.671875 -0.84375 C 0.5 -0.28125 0.4375 -0.203125 -0.078125 -0.140625 L -0.078125 0 L 4.515625 0 Z M 5.03125 -1.625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<path d=\"M 8.074219 246.5 L 30.574219 246.5 L 30.574219 269 L 8.074219 269 Z M 8.074219 246.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"8.32237\" xlink:href=\"#glyph0-1\" y=\"84.25\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"12.82237\" xlink:href=\"#glyph1-1\" y=\"89.25\"/>\n",
       "</g>\n",
       "<path d=\"M 262.214844 213.796875 C 266.609375 218.1875 266.609375 225.3125 262.214844 229.703125 C 257.820312 234.097656 250.699219 234.097656 246.304688 229.703125 C 241.914062 225.3125 241.914062 218.1875 246.304688 213.796875 C 250.699219 209.402344 257.820312 209.402344 262.214844 213.796875 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 105.925781 246.5 L 128.425781 246.5 L 128.425781 269 L 105.925781 269 Z M 105.925781 246.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"105.9256\" xlink:href=\"#glyph0-2\" y=\"84.75\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"110.9296\" xlink:href=\"#glyph1-1\" y=\"88.75\"/>\n",
       "</g>\n",
       "<path d=\"M 128.425781 257.75 L 132.441406 257.75 M 150.042969 257.75 L 150.957031 257.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 154.957031 257.75 L 150.957031 257.75 M 150.957031 256.25 L 154.957031 257.75 L 150.957031 259.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 128.425781 221.75 L 136.945312 221.75 M 147.945312 221.75 L 151.054688 221.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 155.054688 221.75 L 151.054688 221.75 M 151.054688 220.25 L 155.054688 221.75 L 151.054688 223.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 8.171875 210.5 L 30.671875 210.5 L 30.671875 233 L 8.171875 233 Z M 8.171875 210.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"8.42237\" xlink:href=\"#glyph0-1\" y=\"48.25\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"12.92237\" xlink:href=\"#glyph1-2\" y=\"53.25\"/>\n",
       "</g>\n",
       "<path d=\"M 8.171875 174.5 L 30.671875 174.5 L 30.671875 197 L 8.171875 197 Z M 8.171875 174.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"8.42237\" xlink:href=\"#glyph0-1\" y=\"12.25\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"12.92237\" xlink:href=\"#glyph1-3\" y=\"17.25\"/>\n",
       "</g>\n",
       "<path d=\"M 105.925781 210.5 L 128.425781 210.5 L 128.425781 233 L 105.925781 233 Z M 105.925781 210.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"105.9256\" xlink:href=\"#glyph0-2\" y=\"48.75\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"110.9296\" xlink:href=\"#glyph1-2\" y=\"52.75\"/>\n",
       "</g>\n",
       "<path d=\"M 106.027344 174.5 L 128.527344 174.5 L 128.527344 197 L 106.027344 197 Z M 106.027344 174.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"106.0256\" xlink:href=\"#glyph0-2\" y=\"12.75\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"111.0296\" xlink:href=\"#glyph1-3\" y=\"16.75\"/>\n",
       "</g>\n",
       "<path d=\"M 199.78125 246.5 L 222.28125 246.5 L 222.28125 269 L 199.78125 269 Z M 199.78125 246.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"200.0329\" xlink:href=\"#glyph0-3\" y=\"84.75\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"204.5329\" xlink:href=\"#glyph1-1\" y=\"88.75\"/>\n",
       "</g>\n",
       "<path d=\"M 199.984375 210.5 L 222.484375 210.5 L 222.484375 233 L 199.984375 233 Z M 199.984375 210.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"200.2329\" xlink:href=\"#glyph0-3\" y=\"48.75\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"204.7329\" xlink:href=\"#glyph1-2\" y=\"52.75\"/>\n",
       "</g>\n",
       "<path d=\"M 199.882812 174.5 L 222.382812 174.5 L 222.382812 197 L 199.882812 197 Z M 199.882812 174.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"200.1329\" xlink:href=\"#glyph0-3\" y=\"12.75\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"204.6329\" xlink:href=\"#glyph1-3\" y=\"16.75\"/>\n",
       "</g>\n",
       "<path d=\"M 286.039062 246.5 L 308.539062 246.5 L 308.539062 269 L 286.039062 269 Z M 286.039062 246.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"286.5401\" xlink:href=\"#glyph2-1\" y=\"84.25\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"290.5361\" xlink:href=\"#glyph1-1\" y=\"89.25\"/>\n",
       "</g>\n",
       "<path d=\"M 286.039062 210.5 L 308.539062 210.5 L 308.539062 233 L 286.039062 233 Z M 286.039062 210.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"286.5401\" xlink:href=\"#glyph2-1\" y=\"48.25\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"290.5361\" xlink:href=\"#glyph1-2\" y=\"53.25\"/>\n",
       "</g>\n",
       "<path d=\"M 285.738281 174.5 L 308.238281 174.5 L 308.238281 197 L 285.738281 197 Z M 285.738281 174.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"286.2402\" xlink:href=\"#glyph2-1\" y=\"12.25\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"290.2362\" xlink:href=\"#glyph1-3\" y=\"17.25\"/>\n",
       "</g>\n",
       "<path d=\"M 105.925781 282.5 L 128.425781 282.5 L 128.425781 305 L 105.925781 305 Z M 105.925781 282.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"105.9256\" xlink:href=\"#glyph0-2\" y=\"120.75\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"110.9296\" xlink:href=\"#glyph1-4\" y=\"124.75\"/>\n",
       "</g>\n",
       "<path d=\"M 154.207031 282.5 L 182.203125 282.5 L 182.203125 305 L 154.207031 305 Z M 154.207031 282.5 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"153.7048\" xlink:href=\"#glyph0-4\" y=\"120.75\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"161.7058\" xlink:href=\"#glyph1-5\" y=\"124.75\"/>\n",
       "  <use x=\"165.2058\" xlink:href=\"#glyph1-6\" y=\"124.75\"/>\n",
       "</g>\n",
       "<path d=\"M 47.25 282.5 L 93.25 282.5 L 93.25 305 L 47.25 305 Z M 47.25 282.5 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"45.32\" xlink:href=\"#glyph0-4\" y=\"120.75\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"53.321\" xlink:href=\"#glyph1-6\" y=\"124.75\"/>\n",
       "  <use x=\"56.821\" xlink:href=\"#glyph1-7\" y=\"124.75\"/>\n",
       "  <use x=\"59.929\" xlink:href=\"#glyph1-8\" y=\"124.75\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"61.679\" xlink:href=\"#glyph0-5\" y=\"120.75\"/>\n",
       "  <use x=\"63.929\" xlink:href=\"#glyph0-6\" y=\"120.75\"/>\n",
       "  <use x=\"66.179\" xlink:href=\"#glyph0-4\" y=\"120.75\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"74.18\" xlink:href=\"#glyph1-6\" y=\"124.75\"/>\n",
       "  <use x=\"77.68\" xlink:href=\"#glyph1-6\" y=\"124.75\"/>\n",
       "</g>\n",
       "<path d=\"M 30.574219 257.75 L 34.742188 257.75 M 52.859375 257.75 L 53.101562 257.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 57.101562 257.75 L 53.101562 257.75 M 53.101562 256.25 L 57.101562 257.75 L 53.101562 259.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 30.671875 221.75 L 39.246094 221.75 M 50.246094 221.75 L 53.199219 221.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 57.199219 221.75 L 53.199219 221.75 M 53.199219 220.25 L 57.199219 221.75 L 53.199219 223.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 30.671875 185.75 L 53.199219 185.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 57.199219 185.75 L 53.199219 185.75 M 53.199219 184.25 L 57.199219 185.75 L 53.199219 187.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 128.527344 185.75 L 151.054688 185.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 155.054688 185.75 L 151.054688 185.75 M 151.054688 184.25 L 155.054688 185.75 L 151.054688 187.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 286.039062 248.335938 L 267.414062 232.753906 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 264.347656 230.1875 L 267.414062 232.753906 M 266.453125 233.90625 L 264.347656 230.1875 L 268.378906 231.605469 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 286.039062 221.75 L 271.410156 221.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 267.410156 221.75 L 271.410156 221.75 M 271.410156 223.25 L 267.410156 221.75 L 271.410156 220.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 285.738281 195.230469 L 267.375 210.699219 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 264.316406 213.277344 L 267.375 210.699219 M 268.34375 211.847656 L 264.316406 213.277344 L 266.410156 209.550781 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 222.28125 248.382812 L 241.082031 232.726562 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 244.15625 230.164062 L 241.082031 232.726562 M 240.121094 231.574219 L 244.15625 230.164062 L 242.042969 233.878906 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 222.484375 221.75 L 237.109375 221.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 241.109375 221.75 L 237.109375 221.75 M 237.109375 220.25 L 241.109375 221.75 L 237.109375 223.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 222.382812 195.140625 L 241.09375 210.757812 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 244.164062 213.324219 L 241.09375 210.757812 M 242.054688 209.609375 L 244.164062 213.324219 L 240.132812 211.910156 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 105.925781 285.121094 L 83.859375 268.1875 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 80.683594 265.753906 L 83.859375 268.1875 M 82.945312 269.378906 L 80.683594 265.753906 L 84.769531 267 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 81.5 257.75 L 100.027344 257.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 104.027344 257.75 L 100.027344 257.75 M 100.027344 256.25 L 104.027344 257.75 L 100.027344 259.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 105.925781 213.101562 L 83.949219 196.203125 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 80.777344 193.765625 L 83.949219 196.203125 M 83.035156 197.394531 L 80.777344 193.765625 L 84.863281 195.015625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 55.410156 282.5 C 47.25 274.511719 38.769531 262.382812 38.785156 246.5 C 38.800781 228.292969 49.972656 210.246094 59.019531 198.613281 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 61.539062 195.507812 L 59.019531 198.613281 M 57.855469 197.667969 L 61.539062 195.507812 L 60.183594 199.558594 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 81.601562 185.75 L 100.128906 185.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 104.128906 185.75 L 100.128906 185.75 M 100.128906 184.25 L 104.128906 185.75 L 100.128906 187.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 70.25 282.5 L 70.25 274.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 70.25 270.898438 L 70.25 274.898438 M 68.75 274.898438 L 70.25 270.898438 L 71.75 274.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 154.207031 283.417969 C 145.699219 275.453125 136.460938 263.003906 136.460938 246.5 C 136.460938 228.277344 147.722656 210.210938 156.832031 198.578125 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 159.363281 195.476562 L 156.832031 198.578125 M 155.671875 197.628906 L 159.363281 195.476562 L 157.996094 199.523438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 168.175781 282.5 L 168.152344 274.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 168.140625 270.898438 L 168.152344 274.898438 M 166.652344 274.902344 L 168.140625 270.898438 L 169.652344 274.894531 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 78.203125 249.796875 C 82.597656 254.1875 82.597656 261.3125 78.203125 265.703125 C 73.8125 270.097656 66.6875 270.097656 62.296875 265.703125 C 57.902344 261.3125 57.902344 254.1875 62.296875 249.796875 C 66.6875 245.402344 73.8125 245.402344 78.203125 249.796875 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 81.601562 221.75 L 100.027344 221.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 104.027344 221.75 L 100.027344 221.75 M 100.027344 220.25 L 104.027344 221.75 L 100.027344 223.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 78.304688 177.796875 C 82.699219 182.1875 82.699219 189.3125 78.304688 193.703125 C 73.910156 198.097656 66.789062 198.097656 62.394531 193.703125 C 58 189.3125 58 182.1875 62.394531 177.796875 C 66.789062 173.402344 73.910156 173.402344 78.304688 177.796875 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 78.304688 213.796875 C 82.699219 218.1875 82.699219 225.3125 78.304688 229.703125 C 73.910156 234.097656 66.789062 234.097656 62.394531 229.703125 C 58 225.3125 58 218.1875 62.394531 213.796875 C 66.789062 209.402344 73.910156 209.402344 78.304688 213.796875 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 105.925781 249.101562 L 83.949219 232.203125 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 80.777344 229.765625 L 83.949219 232.203125 M 83.035156 233.394531 L 80.777344 229.765625 L 84.863281 231.015625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 176.058594 249.796875 C 180.453125 254.1875 180.453125 261.3125 176.058594 265.703125 C 171.667969 270.097656 164.542969 270.097656 160.148438 265.703125 C 155.757812 261.3125 155.757812 254.1875 160.148438 249.796875 C 164.542969 245.402344 171.667969 245.402344 176.058594 249.796875 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 179.355469 257.75 L 193.882812 257.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 197.882812 257.75 L 193.882812 257.75 M 193.882812 256.25 L 197.882812 257.75 L 193.882812 259.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 176.160156 213.796875 C 180.554688 218.1875 180.554688 225.3125 176.160156 229.703125 C 171.765625 234.097656 164.644531 234.097656 160.25 229.703125 C 155.855469 225.3125 155.855469 218.1875 160.25 213.796875 C 164.644531 209.402344 171.765625 209.402344 176.160156 213.796875 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 179.457031 221.75 L 194.082031 221.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 198.082031 221.75 L 194.082031 221.75 M 194.082031 220.25 L 198.082031 221.75 L 194.082031 223.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 176.160156 177.796875 C 180.554688 182.1875 180.554688 189.3125 176.160156 193.703125 C 171.765625 198.097656 164.644531 198.097656 160.25 193.703125 C 155.855469 189.3125 155.855469 182.1875 160.25 177.796875 C 164.644531 173.402344 171.765625 173.402344 176.160156 177.796875 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 179.457031 185.75 L 193.984375 185.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 197.984375 185.75 L 193.984375 185.75 M 193.984375 184.25 L 197.984375 185.75 L 193.984375 187.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 242.808594 174.5 L 265.308594 174.5 L 265.308594 197 L 242.808594 197 Z M 242.808594 174.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"244.5585\" xlink:href=\"#glyph2-2\" y=\"13.25\"/>\n",
       "</g>\n",
       "<path d=\"M 254.199219 210.5 L 254.15625 202.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 254.132812 198.898438 L 254.15625 202.898438 M 252.65625 202.90625 L 254.132812 198.898438 L 255.65625 202.890625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 59.023438 282.5 C 53.175781 275.398438 47.234375 265.625 47.25 255.257812 C 47.261719 246.050781 51.972656 238.324219 57.136719 232.621094 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 59.972656 229.804688 L 57.136719 232.621094 M 56.078125 231.558594 L 59.972656 229.804688 L 58.191406 233.6875 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 156.15625 282.5 C 150.300781 275.734375 144.507812 266.492188 144.507812 256.410156 C 144.507812 246.933594 149.621094 238.78125 155.101562 232.769531 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "<path d=\"M 157.933594 229.941406 L 155.101562 232.765625 M 154.042969 231.703125 L 157.933594 229.941406 L 156.160156 233.828125 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-7,-174)\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SVG('../img/rnn-bptt.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Fig.10.7.2: Computational dependencies for a recurrent neural network model with three time steps. Boxes represent variables (not shaded) or parameters (shaded) and circles represent operators. \n",
    "\n",
    "## BPTT in Detail\n",
    "\n",
    "Now that we discussed the general principle let's discuss BPTT in detail, distinguishing between different sets of weight matrices ($\\mathbf{W}_{hx}, \\mathbf{W}_{hh}$ and $\\mathbf{W}_{oh}$) in a simple linear latent variable model:\n",
    "\n",
    "$$\\mathbf{h}_t = \\mathbf{W}_{hx} \\mathbf{x}_t + \\mathbf{W}_{hh} \\mathbf{h}_{t-1} \\text{ and }\n",
    "\\mathbf{o}_t = \\mathbf{W}_{oh} \\mathbf{h}_t$$\n",
    "\n",
    "Following the discussion in `chapter_backprop` we compute gradients $\\partial L/\\partial \\mathbf{W}_{hx}$, $\\partial L/\\partial \\mathbf{W}_{hh}$, and $\\partial L/\\partial \\mathbf{W}_{oh}$ for\n",
    "$L(\\mathbf{x}, \\mathbf{y}, \\mathbf{W}) = \\sum_{t=1}^T l(\\mathbf{o}_t, y_t)$.\n",
    "Taking the derivatives with respect to $W_{oh}$ is fairly straightforward and we obtain\n",
    "\n",
    "$$\\partial_{\\mathbf{W}_{oh}} L = \\sum_{t=1}^T \\mathrm{prod}\n",
    "\\left(\\partial_{\\mathbf{o}_t} l(\\mathbf{o}_t, y_t), \\mathbf{h}_t\\right)$$\n",
    "\n",
    "The dependency on $\\mathbf{W}_{hx}$ and $\\mathbf{W}_{hh}$ is a bit more tricky since it involves a chain of derivatives. We begin with\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\partial_{\\mathbf{W}_{hh}} L & = \\sum_{t=1}^T \\mathrm{prod}\n",
    "\\left(\\partial_{\\mathbf{o}_t} l(\\mathbf{o}_t, y_t), \\mathbf{W}_{oh}, \\partial_{\\mathbf{W}_{hh}} \\mathbf{h}_t\\right) \\\\\n",
    "\\partial_{\\mathbf{W}_{hx}} L & = \\sum_{t=1}^T \\mathrm{prod}\n",
    "\\left(\\partial_{\\mathbf{o}_t} l(\\mathbf{o}_t, y_t), \\mathbf{W}_{oh}, \\partial_{\\mathbf{W}_{hx}} \\mathbf{h}_t\\right)\n",
    "\\end{aligned}$$\n",
    "\n",
    "After all, hidden states depend on each other and on past inputs. The key quantity is how past hidden states affect future hidden states.\n",
    "\n",
    "$$\\partial_{\\mathbf{h}_t} \\mathbf{h}_{t+1} = \\mathbf{W}_{hh}^\\top\n",
    "\\text{ and thus }\n",
    "\\partial_{\\mathbf{h}_t} \\mathbf{h}_T = \\left(\\mathbf{W}_{hh}^\\top\\right)^{T-t}$$\n",
    "\n",
    "Chaining terms together yields\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\partial_{\\mathbf{W}_{hh}} \\mathbf{h}_t & = \\sum_{j=1}^t \\left(\\mathbf{W}_{hh}^\\top\\right)^{t-j} \\mathbf{h}_j \\\\\n",
    "\\partial_{\\mathbf{W}_{hx}} \\mathbf{h}_t & = \\sum_{j=1}^t \\left(\\mathbf{W}_{hh}^\\top\\right)^{t-j} \\mathbf{x}_j.\n",
    "\\end{aligned}$$\n",
    "\n",
    "A number of things follow from this potentially very intimidating expression. Firstly, it pays to store intermediate results, i.e. powers of $\\mathbf{W}_{hh}$ as we work our way through the terms of the loss function $L$. Secondly, this simple *linear* example already exhibits some key problems of long sequence models: it involves potentially very large powers $\\mathbf{W}_{hh}^j$. In it, eigenvalues smaller than $1$ vanish for large $j$ and eigenvalues larger than $1$ diverge. This is numerically unstable and gives undue importance to potentially irrelvant past detail. One way to address this is to truncate the sum at a computationally convenient size. Later on in this chapter we will see how more sophisticated sequence models such as LSTMs can alleviate this further. In code, this truncation is effected by *detaching* the gradient after a given number of steps.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Back-propagation through time is merely an application of backprop to sequence models with a hidden state.\n",
    "* Truncation is needed for computational convencient and numerical stability.\n",
    "* High powers of matrices can lead top divergent and vanishing eigenvalues. This manifests itself in the form of exploding or vanishing gradients.\n",
    "* For efficient computation intermediate values are cached.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Assume that we have a symmetric matrix $\\mathbf{M} \\in \\mathbb{R}^{n \\times n}$ with eigenvalues $\\lambda_i$. Without loss of generality assume that they are ordered in ascending order $\\lambda_i \\leq \\lambda_{i+1}$. Show that $\\mathbf{M}^k$ has eigenvalues $\\lambda_i^k$.\n",
    "1. Prove that for a random vector $\\mathbf{x} \\in \\mathbb{R}^n$ with high probability $\\mathbf{M}^k \\mathbf{x}$ will by very much aligned with the largest eigenvector $\\mathbf{v}_n$ of $\\mathbf{M}$. Formalize this statement.\n",
    "1. What does the above result mean for gradients in a recurrent neural network?\n",
    "1. Besides gradient clipping, can you think of any other methods to cope with gradient explosion in recurrent neural networks?\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}