07-optimization_2moons.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AZM-JDrE4tN1"
   },
   "source": [
    "# torch.optim: optimization package in PyTorch\n",
    "\n",
    "In the following notebook, we shall take a closer look at the `torch.optim` package.\n",
    "\n",
    "The reference documentation for the package is [here](https://pytorch.org/docs/stable/optim.html)\n",
    "\n",
    "The package typically provides gradient-descent optimization algorithms.\n",
    "\n",
    "Currently, the following algorithms are available:\n",
    "\n",
    "- Stochastic Gradient Descent (SGD)\n",
    "  - with heavy-ball and nesterov momentum variants (optional)\n",
    "  - with L2 weight decay (optional)\n",
    "- Adadelta\n",
    "- Adagrad\n",
    "- Adam\n",
    "- SparseAdam\n",
    "- Adamax\n",
    "- Averaged SGD\n",
    "- L-BFGS\n",
    "- RMSProp\n",
    "- RProp (resilient backprop)\n",
    "\n",
    "The optim package is available via `import torch.optim as optim`\n",
    "\n",
    "It makes no assumption on the function it is optimizing, and treats functions as a blackbox `f(x)`.\n",
    "\n",
    "The package is designed to work well with neural networks defined using  `torch.nn.Module`, but does not rely on `torch.nn`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "HkWHjSQv5JlI"
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import numpy as np\n",
    "\n",
    "# imports\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch import optim\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GbqLHgUWRHze"
   },
   "source": [
    "## A walkthrough of the API\n",
    "\n",
    "The optim package has optimizers placed in two buckets:\n",
    "\n",
    "(1) The optimizer evaluates f(x) only once\n",
    "    - examples: SGD, RMSProp, RProp, Adagrad, Adam, etc.\n",
    "(2) The optimizer evaluates f(x) multiple times, with different `x`\n",
    "    - example: L-BFGS, evolutionary algorithms\n",
    "\n",
    "\n",
    "**Case (1): Evaluate-once optimizers**\n",
    "\n",
    "In the case of (1), when f(x) is evaluated only once, we provide a simple interface that is roughly as follows:\n",
    "\n",
    "```\n",
    "y = f(x) # f is the function we are optimizing\n",
    "x        # params to optimize. a Tensor or list of Tensors with requires_grad=True\n",
    "\n",
    "# construct an optimizer object\n",
    "optimizer = optim.SGD(x, lr=0.1, momentum=0.9)\n",
    "\n",
    "# one gradient-descent step\n",
    "optimizer.zero_grad()\n",
    "y = f(x)\n",
    "y.backward()\n",
    "optimizer.step()\n",
    "```\n",
    "\n",
    "Here's an example of how these optimizers will be used with `nn.Module` based neural networks:\n",
    "\n",
    "```\n",
    "model = torchvision.models.resnet18()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9)\n",
    "\n",
    "for inputs, labels in dataset:\n",
    "    optimizer.zero_grad()\n",
    "    outputs = model(inputs)\n",
    "    loss = loss_fn(outputs, labels)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "```\n",
    "\n",
    "**Case (2): Optimizer does multiple evaluations of f(x)**\n",
    "\n",
    "In this case the API is a little more involved. One has to define the function `f(x)` as a python closure, and give that to `optimizer.step`.\n",
    "\n",
    "For example:\n",
    "\n",
    "```\n",
    "for input, target in dataset:\n",
    "    def closure():\n",
    "        optimizer.zero_grad()\n",
    "        output = model(input)\n",
    "        loss = loss_fn(output, target)\n",
    "        loss.backward()\n",
    "        return loss\n",
    "    optimizer.step(closure)\n",
    "```\n",
    "\n",
    "## Classifying two-moons dataset using Logistic Regression\n",
    "\n",
    "Let us work through a small example of classifying the noisy two-moons dataset generated via scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_moons(n_samples=100, shuffle=True, noise=None):\n",
    "    n_samples_out = int(n_samples / 2)\n",
    "    n_samples_in = n_samples - n_samples_out\n",
    "\n",
    "    outer_circ_x = np.cos(np.linspace(0, np.pi, n_samples_out))\n",
    "    outer_circ_y = np.sin(np.linspace(0, np.pi, n_samples_out))\n",
    "    inner_circ_x = 1 - np.cos(np.linspace(0, np.pi, n_samples_in))\n",
    "    inner_circ_y = 1 - np.sin(np.linspace(0, np.pi, n_samples_in)) - .5\n",
    "    \n",
    "    X = np.vstack((np.append(outer_circ_x, inner_circ_x),\n",
    "                   np.append(outer_circ_y, inner_circ_y))).T\n",
    "    y = np.hstack([np.zeros(n_samples_out, dtype=np.intp),\n",
    "                   np.ones(n_samples_in, dtype=np.intp)])\n",
    "\n",
    "    if shuffle:\n",
    "        perm = np.random.permutation(n_samples)\n",
    "        X = X[perm]\n",
    "        y = y[perm]\n",
    "        \n",
    "    if noise is not None:\n",
    "        X += np.random.randn(*X.shape) * noise\n",
    "\n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "Bc7xMONi6JJc"
   },
   "outputs": [],
   "source": [
    "X, y = make_moons(100, shuffle=True, noise=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x1188b8490>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd4VNXWwOHfmj6pJAQQqSooogIKYu+9gZ+9d7Fg7+XasRfELjYs1y6Wiyh2rCggIgiKFJHeAqnTz/r+OENIMpMQYDKTst/n4YE5dSUks+bssraoKoZhGIaxliPTARiGYRhNi0kMhmEYRg0mMRiGYRg1mMRgGIZh1GASg2EYhlGDSQyGYRhGDSYxGIZhGDWYxGAYhmHUYBKDYRiGUYMr0wFsjKKiIu3evXumwzAMw2hWJk+evFJV263vuJQkBhF5ETgSWK6q2yfZfypwPSBAGXCRqk6N7/snvi0GRFV1wPru1717dyZNmpSK0A3DMFoNEZnfkONS1ZQ0Cji0nv3zgH1UdQfgLmBkrf37qWq/hiQFwzAMo3Gl5IlBVb8Vke717P+x2ssJQOdU3NcwDMNIvUx0Pp8LfFLttQKfichkERmSgXgMwzCMatLa+Swi+2Enhj2rbd5TVReJSHvgcxH5U1W/TXLuEGAIQNeuXdMSr2EYRmuUticGEekDPA8MVtVVa7er6qL438uB94GByc5X1ZGqOkBVB7Rrt95OdcMwDGMjpSUxiEhXYDRwuqrOqrY9W0Ry1/4bOBiYno6YDMMwjORSNVz1DWBfoEhEFgK3AW4AVX0GuBVoCzwlIrBuWGoH4P34Nhfwuqp+moqYjNattLiMV+94h+/em4DH5+Hw8w7guKuPwuVullN3DCOtpDku7TlgwAA18xiMuoQCIc7f4WpWLFxFNBwFwOv30G//7Rn2vxszHJ1hZI6ITG7ItABTEsNocb5560dWL1tTlRQAQoEwv309ndm/zctgZIbRPJjEYLQ407+bSbAilGSPMGvS3LTHYxjNjUkMRovTqWdHPD53wnaHU+jQrSgDERlG82ISg9HiHHL2fjhdzhrbHE4H+UV59Ns/oZSXYRi1mMRgNDuqyrTvZvLFa98yf8aChP0FHdrw4Je30aXX5ri9LtweF9vt0YtHxt+J0+lMckXDMKozY/eMZmX1sjVcs//trFhgz5GMxSx2PrQf/3nzyhpDUbfZuQcvzhhB8dLVuDwu8gpzMxWyYTQ75onBaFbuP+NxFv29lEB5kEB5kHAgzKRPf+O94WOSHl+4WYFJCoaxgUxiMJqNipIKpo6fQSwaq7E9FAgz5pnPMhSVYbQ8pimpmYrFYkz85Dcmf/E7BR3yOfiMfSjq1HaTrmlZFisWrCIrz09uQU6KIk2dcCiKPUk+USgQTm8whtGCmcTQDEXCEa4/eBizf51LoDyI2+vi9btHc/voaxlwcN+NuuZP/5vE8AuepbKkEsuy6H9QX65/5VJy2mSnOPqNV9A+n826t2fBX4trbHe6new+OGntRcMwNoJpSmqGPn3hK2ZNmkOgPAhAJBQlVBninlMeTWhmaYjZv83j7pOHs3rpGkKBMJFQlEmfTeW2ox9Ideib7NpRl+DP8eH22vMUfNleCjdrw5l3nrjec+fPXMgtg+/n6IIzOaPHUD56ehzNsSSMYTQ288TQDH3x2reEKhNn9sYiMWZNnsu2u/TcoOu9+/D/CAcjNbZFw1H+mjibhX8voXPPjoDd1DT2uS8YPeJjKtZUsvNhO3LmHSfSrnNbKkoqGP/OBNYsL2GHvbZl+z17IXW1+2yCbXfpyYszH+Xj575g4V+L2WGv3hx0xt74c/z1nrdk3jIu2/UmAuVBVJWKkkpGXvsqS+Yu44IHz0h5nIbRnJnE0Ay5PMn/21QVdx376rN4zjLUSvzk7PK4WLlwVVViePLyFxn30jdVSenzV8Yz4X+Tuf7VSxh24nCsmEU4GMHjc7PD3r2568PrEyaapUJRp7acefv6nxCqe+v+DwkFwjWeEEKVIT568lNOuemYJtmnYhiZYpqSmqEjhhyEL9ubsD2nIJut+nXf4Ov13bc3bm9iQomEImzZpxsAxUtX88nzX9V4UrFiFpXlAe464REqSwMEK0JYMYtgRYjfx8/gkxe+onxNBfNnLCAUSFa7KH1mTpiVtJnN7XWzcNaSDERkGE2XSQzN0L4n7s5ex+2Kx+/B43fjz/WTU5DNnR9cv1HNN8dcfgRZuX6crnU/Dr5sL4MvOZS8tvYcgDlT5yetPxQJRpIWrAtVhhh1yxuc0PF8Lt3tJo5tdy6v3/Nextr0u/TaPOn3JhKK0K7Lpo3mMoyWxjQlNUMOh4PrXrqE468exLRvZ5JflMuuR/XH6098imiIgg5tePrXB3nljreZ+Olv5BXmcNxVR3HQGftUHdO+a1GNMtZVsTgdICRtiiotLkctJRKy+y9ev+d9ijq15eAz992oODfFidcdzYQxkwlVrhvW6vG5GXj4ThRtXpj2eAyjKUvJQj0i8iJwJLBcVROqlIn9UW0EcDhQCZylqr/G950J/Cd+6DBVfXl99zML9WTGFXv+h78mzamRILxZHvw5PtYsL23QNTpvszkvzRzRWCHWa+K433jsoudYubgYcQgHnroXQx87Z6MTqmE0Nw1dqCdViWFvoBx4pY7EcDhwKXZi2AUYoaq7iEghMAkYACgwGeivqqvru59JDJlRtrqcB89+komf/obDIeS3y+Oq5y4iK8/PDYfchVpKqDKMx+8mHIgkbTbKbpPFB8Xrzf1VQoEQ374zgX9nLmSLHbqy5zG74PF5NvprUFXK11TgzfLi8SY2jRlGS9bQxJCSpiRV/VZEutdzyGDspKHABBFpIyIdsdeJ/lxViwFE5HPgUOCNVMRlpFZuQQ53fnA9FSUVBMqDtN28sKrd/vX5zzD+7R9ZvayE7ffqxb2nPkbxksT83nuXrRt8v+ULVnLprjcRKAsQKA/iz/Hxwk2v8/iEeyjcrGCjvgYRMSOQDGM90tX53AmoXh95YXxbXduNFCpbXc6SecuIxTZ88lsy2fnZFHVqW6MzN6dNNkcMOYjTbjmOfvtuzyWPnYM3a90ne4dD8OX4OO/+0xp8nxEXjWTN8pKqiXyB8iCrFq/m6StH1XmOqjL+7R+5ZNcbOaPHJTx+6QsUL633ATRjZk+ZxxV7/odD3CcyuM0ZPHfdq0TCkfWfaBiNrNl0PovIEGAIQNeuXTMcTfNQUVLBA2c9ycRPp+BwOvFle7n8qfPZ69hdG/3eex27K/nt8vjv3e+xePZSeg3swWm3Hk+3bTs36HzLspg0bipWzKqxPRaN8eNHdTcjvnz7W7z3yJiqkVJjR37Od+/+xHPTHiG/KG/jv6AUWzJvGVftc2tV0qssDfDhk5+ydP4KbnnrqgxHZ7R26UoMi4Au1V53jm9bhN2cVH37N8kuoKojgZFg9zE0RpAtzR3HPcz072cSCUUBu2zG/Wc+Trsubek1cMNmR2+MPnv3ps/evTfqXBGpc+itw5F8e9nqct5+8CMi1WZxRyMxytdU8v6IjznrrpOrtgfKAyydt5yizm0z0rT03vAxhEM1nw5CgTAT/jeJ5QtW0r6LWYLUyJx0NSV9BJwhtl2BElVdAowDDhaRAhEpAA6ObzM2gqoyY8IsRt36Ji/e9DrTv/8znhTWCQcivP3QRxmKsOFEhN0HD0iYOe3yuNj7uN2SnjN36vykHcqRUIRfv5wG2N+jl/7zBsd3OI8r9rqFEzcfwkPnPkU0kjgUtzH9PXkusUgdE+5qFQk0jHRLyRODiLyB/cm/SEQWArcBbgBVfQYYiz0iaTb2cNWz4/uKReQuYGL8Uneu7Yg2Noyq8sBZT/D96J8JVYZxuBxJ33hUtdm88Vz65PnM+e0fVi8vIRKM4Pa6adelLRc+cmbS49tuXpB0roWI0KF7ewDGPPsZo0d8bJfpDtj7v3nzB7Lz/Fw0/OxG+1pq67HTlvw1aU7C/1E4FKHzNpunLQ7DSCYlw1XTzQxXXScWixEsDzJ1/EzuPfXRpLOQa3O6nTz2491s3X+rNES4aWLRGL98MoWFfy2m23Zd6H9wn3rXbb58j5uZNWkO0WpvuN4sDw9/fQfb7NyD07a8mGX/rEg4z5vl4cM1rzRKbadklsxbxgV9r6nqYwDw+j3scmR/08dgNJq0zmNIt5aSGCpKK4mEIrRpl7/B56oqb97/AW/d/wGhyhDikIRmo/p0374rz/3+8Abft6krXVXGsJOGM/37P3G6nbg9Li5/6nz2OWF3AI7KO51gtTfjtZwuB6NXjSIrt/4qrak0e8o8nrj0BWZMmIU/x8eRQw7irGEn4faY+RVG40jrPAZjw6xetoYHznyC3775A4COW7Tn2lGXbFC57Lcf/JD/Dnsvafnthlg4azGlxWUtbj3kvLa5PPD5raxetoay1RV06rFZjaeAXgN78NtX0xPOa9e5CH+OL52h0mPHLXj0+2FpvadhNIQpopdmqsq1B9zBlK+nEw1HiYajLPhrMdcfdCcrF61q0DUsy+LN+z5oWFKoq6beRpbobi4KOrSha69OCU1DFzx4Br5sb42RTd4sD5c8cW6jrB9hGM2RSQxp9scPf7L835UJnY7RSJSPn/uiQdcIByNUlgXq3O/2uPBlefFmeTng1L3x+GuWkHC6nOy4/w7rXdymJeqx4xY88ct97HPC7nTquRkDD9+JB764jV0O3ynToRlGk9FyPzI2UUuTdHyCvTxnQ9cF8Po9tGmfn7TkRNdtO3HoOQeQnednz2N3wZ/jo6y4nKnfTK+aG1DUuS3Xjhq6SV9Hc9Zt287c9PoVmQ7DMJoskxjSrGf/LYnVms0L4M3yssOe2zboGiLC+Q+cxqMXPFujjLTX72HoY+ey0wE71Dj+7jE3Mvf3+cyeMo8O3dvRZ+/eptnEMIw6mcSQZt227czOh/Zj0qe/2WPpsYeP5hZmc+Dpezf4Ogeeujf+bB+jbnmTpfNX0LVXJ86//zT67ZdQ3BaALft0q1qNzTAMoz5muGoGRCNR3n1kDGOe/YxQIMzuRw2gXdciPhv1DZVlAXY+tB9nDzvZlEUwDCOlzDyGZuTBc55k/Ns/VjULOZwOcgtyuO7lS2jXpS3dt+timn4MSovL+PSFr5j58990374LR15wMG07blz5caN1MomhmVi+YCVnbXNZjcJva7k8LlxuJ0Wd2jJszA106tExAxEaTcGy+SsYuvMNBCuChAJh3F43bo+LR769k636ds90eEYz0dDEYIarZtic3/6pcyWxaDhKsCLEor+XcN2Bd2JZiZ3WRvqpKj9/PJn7Tn+Mh899imnfzWz0ez57zcuUFZdV9UtFQvaQ5eFDnm30exutj+l8zrDNurerUdcnGVWlbHU507//c6PLWBupoarce9oIfvpoEsGKECLw9Vs/cszlh3PO3ac02n0njpuKZSU+3f/961zCwfAmLXdqGLWZJ4YM22IHe7SQaz2zkEWEkpVlaYrKWCscDPPzx5P5bvTPlK+p4PdvZ1QlBQBVCFWGeG/4GJbMXdZocXj9yd/4HU5H2gr/Ga2HeWJoAu7++EYeOucpfvlkCmpZ9jyHWh8OI6Eo2+3e8PWSjU03dfwf3Hb0A6gqqL3ozw77bJu8FIkIEz/9jUEXH9IosRx+3gG89+jHhAPr5q24PS72PHZXkxiMlDOJIYXCwTBTvrJrIPXbf3uy87IadF5uQQ53vH8dgfIAZcXl3HTEPSyZu7zqTcCX7eX/Lj+cws3MCJR0CVQEueWo+2qUxQb47ctpOJxOYtGazX9OpyOlRfgsy+K94WMY/ejHlK2uYNtde7J1/y2ZNXkuTpcTtSy69e7C5U+dl7J7GsZaJjGkyNRv4p8u4x/1Y5EYV4y8gANPbfikNX+OH3+On8cn3MuYZz7j23cnkJOfxeBLDmPXI/s3VuhGEj+PmQxJhgirgkhiW7+qsvvg9Q72aLCnrxzFJy98VfV08ttX0/Hn+LjtvWuoWFPJ5j02Y+v+W5phzEajSNUKbocCIwAn8Lyq3ldr/3Bgv/jLLKC9qraJ74sB0+L7/lXVQamIKZ0qywLcMijx0+WjQ55l2116bvAwU3+2j+OvHsTxVze7b0WLESgPoklGgVkxi34H7MCMH//C5XKCgFrK7aOvJTs/OyX3Li0uY+xzXxCuNYQ5HAzz/eifuWrkhSm5j2HUZZMTg4g4gSeBg4CFwEQR+UhVZ6w9RlWvrHb8pcCO1S4RUNV+mxpHJv30UfI5FdFojC9eHc+Zd5yU5oiMTdX/oD7EoomJwZft5aTrBtNrYA9+/WIaTreT/gf1wev3puzeC2ctwe11JySGWNTir19mp+w+hlGXVIxKGgjMVtW5qhoG3gQG13P8ycAbKbhvk1FZFsBKUhgvFolRXlJ3eWyj6WrftR0n3XA03ixvVXONL9tL/4P7stOBfcjOz2avY3dl90E7pzQpgD2EORxKnPAoDqFr784pvZdhJJOKpqROwIJqrxcCuyQ7UES6AVsAX1Xb7BORSUAUuE9VP0hBTGk14OC+PJNkBrkv28vug1LX7myk1xm3ncBOB+zAuJe+JhgIs9+Je7DrUf0bvV2/cLMCdh+8Mz99NKnGKCSPz83JN/xfo97bMCD9nc8nAe+qavUhHd1UdZGIbAl8JSLTVHVO7RNFZAgwBKBr167pibaBOm7ZgWOvPJL3HxtLqDKEKviyfex8aL86q50aTc/Cv5cw5ctp5LTJZtej+uPP9rH9ntuyfQPLoafSdaMuYeQ1r/DJi18RDUfZfKsOXPbU+aZCrpEWm1wrSUR2A25X1UPir28EUNV7kxw7BRiqqj/Wca1RwBhVfbe+ezbVWklTx//BuJe+JhyKsP9Je7LrUf1xOMwcwqZOVXn6ylF8PPJzEMHpciAi3PvJzfTebZuMxhaLxYiEoviyUttcZbROaSuiJyIuYBZwALAImAicoqp/1DquF/ApsIXGbyoiBUClqoZEpAj4CRhcveM6maaaGIzm6eexvzLsxEeqZjOvldc2l7eXPGcmkBktRtqK6KlqFLgEGAfMBN5W1T9E5E4RqT7e8iTgTa2ZibYFJonIVOBr7D6GepOCYaTaJy98mZAUwC5iOP2HPzMQkWFkVkr6GFR1LDC21rZba72+Pcl5PwI71N5uGOkUCUWT7xA7ORhGa2NmPhut3gGn7Mnv4/9IeGqwYhbb79krQ1HVzbIsPhv1DWOe/ZxwKMwBp+zF4EsOq9EPMWvyHCaMmYwvy8u+J+5O+67tMhix0dyYhXqMVi8Wi3HLoPuZ9t1MguVBXB4nTqeT61+5lL2O3TXT4SW497QR/PjhxKpE5vF76Na7M4/9eDdOl5PHL3mez14eTyQYxuFy4nA6uOq5CznglL0yHLmRaQ3tYzBPDEar53Q6Gfa/G5j8+e/8MvZX8trmcuDpe9Nxiw6ZDi3BP38s4If3f6lasAcgHAiz8K/F/PjhRPLa5vL5K+OraixZ8aawR85/hl0O34mcNqkp22G0bCYxGAbgcDjY+ZB+7HxI067OMv37PyHJ/LpAeZApX04jFrOSdqQ7XQ4mjfuNfU/cIw1RGs2dGWRvGM1I4WZtcDgTh8+6vW6KOrfF4XDUMTNbEDOnxmgg85NiGM3Izof1w+v3JFQEd7ocHHLWvhx42l54kqz2ZsVi7Hxo034aMpoOkxgMoxlxe9w8/M0ddN56c7xZHnxZXvKL8rjzw+sp6tSW7ffclsFDD8Hj9+D2uPBmefD4Pdz42uVk5fozHb7RTJhRSdglEb56/Xvef3wsFSWVDDysHweduS9bbN8VZ5LHdsPItH9mLOD2/3uQZfNXIA6hQ7d23PzGFfTotwUA//65iJ8//hVftpe9jt2FNu3yMxyx0RSkrSRGJqQ6MTx95UuMfb7W7Fexl9wc+tg5Zpif0aSEAiFO6XYRZavKqP7rm52fxX//eSplCwYZLU/aSmI0dysXFzPm2c8TR3IolBWXM3zIs0wd/0fykw0jib9/ncvbD37IJy98SUVJRcqv/8P7vxAJRqj9mS4WifH1m0nrUxrGBmn1w1X/+mU2Lo8rYbWstUKVId6873367rNdmiMzmhvLsrj/jMf54YOJRCNR3B4XT185invG3pTS0t0rFxUn/XkNVoZYsXBVyu5jtF6t/omh7eYFWFb9zWlL561IUzRGc/bduxP48cOJhCpDxCIxghUhAuVBbvu/B4lFY+u/QANtM7AHbm/iZzp/jo/eu/ZM2X2M1qvVJ4Ztdu5B+y5tEUfyVbmcLgc77J3+hVqM5ueTF79KOrksEo4y8+e/U3afPnv3ZusBW+GtNizV43PTpVcnBpghqUYKtPrEICLc/9kt9BrYA4ez5rfD4RB82T5OvtEsp2isn2UlrvsN9kRlXc9T6YYQEe799D+cduvxdN66I5v32IyTb/w/Hv7mDjOKzkgJMyqpmuULVvLduz/xxWvfsXrZGvrs05sz7ziRTj06pvxeLZlaxWjZExD6AsQHWaciWach0rLftL547VtGXDQy4akhpyCbd5Y+j8vd6rv0jAwzw1UbkaqybP4KvFleCtqb8eHVqVWOrjwCrJXA2g5SP/gOwNHmkUyG1uhisRh3Hvcwv37xO8HKEB6vG4fDwR0fXMdOB/bJdHiGkd7qqiJyKDACcALPq+p9tfafBTyIvfQnwBOq+nx835nAf+Lbh6nqy6mIqbFM+WoaD571JKWryrAsi20G9uTmN66gaPPCTIfWJGjgA7DWsC4pAAQg+Dka/Qdxdc9QZI3P6XRy++hrmfbdTKZ8NY28wlz2PWkP8+HBaHZSseazE3vN54OAhdhrPp9cfYnOeGIYoKqX1Dq3EJgEDAAUmAz0V9XV9d0zU08MS+Yu4/w+V1eVNAa7c3rzHh154Y/hdRQva12s1ZdCaFziDslG8u5C/EemPyjDMID0TnAbCMxW1bmqGgbeBAY38NxDgM9VtTieDD4HDk1BTI3io6fHEYvUXOoxFrVYuXAVM36alaGomhhXd8CdZIeC0/TVGEZzkIrE0AlYUO31wvi22o4Vkd9F5F0R6bKB5zYJS+YsIxpJHI8uIqw0E4sAkKyTSPyxEnB0BPdOmQjJMIwNlK7hqv8DuqtqH+yngg3uRxCRISIySUQmrViRmQln/fbfHm+1dXXXikaibL3zVhmIqIlKGH0k4D3ANLUZRjORisSwCOhS7XVn1nUyA6Cqq1R1bcP880D/hp5b7RojVXWAqg5o1y4zC5sfcta+5LfLxeVZ12fvzfKyz4m7N8llIDNBy0eChmtttaDyFdRKfd0gwzBSLxWJYSLQU0S2EBEPcBLwUfUDRKR64/IgYGb83+OAg0WkQEQKgIPj25okf46fpybez+Chh9ChWzu6b9eFCx46g2teuDjToTUd4V+AaOJ2cUHsn3RHYxjGRtjk4aqqGhWRS7Df0J3Ai6r6h4jcCUxS1Y+Ay0RkEPY7RjFwVvzcYhG5Czu5ANypqsWbGlNjyi/K48KHz+LCh8/KdChNk7MzxOYkbtcwONqnPx7DaCJUFSK/QvRPcHYFz+5NdtKnmeBmpJSGJ6HF5wDBals94N0LR8HTmQrLMDJKrUp09dkQ/QvUsvvhHEVI4RuIsyhtcZj1GIyMEM8AyL8HpADwYyeFA5D8hzIdmmE0OlULTehjAy1/DCIzQCuBIGgFxBahJTelP8gGMMVbjJRz+I9EfYdBbAk48hFHbqZDyrjipasZPeJjfh8/g817dOS4q46sWobTaP5UQ2jpfRB4F4igrq2RvDsQz472AYH3gdqVd6MQ/h7VMHb3bNNhEoPRKESc4Opc9Vo1gla+A8HRgAPxnwD+/2uybayptGz+Ci4acB3B8iCRUJS/fpnN96MncMvbV7PL4emb2/HnL3/z1gMfsnjOUvrs3ZsTrh1Mu85t03b/lkzXXAWhb6l684/+iRafBUUfIK4tgLrW49B69mWOaUpqoEWzlzD9hz+pLAsk7Fu+YCXL/103t0JVmf7Dn4x59nOmfDWtznLMrYWqoqvPg7L7IPI7RH5DS+9C11xKc+zj2lCjbn2TijWVREL2aC3LUkKVYYYPeSZtPxvfv/8z1+x/Oz+8/zNzp85nzDOfMaTv1SyZtywt92/JNLa4ZlKoEkYrXrT/6TuAxM/hAu4+iPgbP8gNZJ4Y1mPNihJuHfwAc6f+g9PtIhaJcsYdJ3LCNYOYN/1fhp04nKXzloEIHboVcc2LF/PMVS8zb9q/qCoOh4P2XYt4+Js7yC/Ky/SXkxnhHyEylZod0gEI/WAnCk/fTEWWFr9+/jtWLDEBlBWXs2rx6pR8ao9FY0z89DcWz1nKVn2702ef3lUTCi3L4rGLnyNUua7tOxqJUVlSyahb3+LGVy/b5Pu3atF/QTygtRNDzB6BBEjOdWhoAmhJvJ/BD+JG8u9Je7gNYRLDetxx7EPMmjTHXpoxYP9ivXrH23Tcsj3DhzxLWXF51bEL/lzMNfvdjipEw+vG8i/6ewkjLn6OW9++Ou3xNwUanhj/ZagtApGJLT4x5BbmULx0TcJ2tZSsvE3/tLhycTFX7nkLJatKiYZjON1Oum3biQe/vA1/jp+Vi4qpLE180rUs5bcvp23y/Vs915ZJkgKAG9x2uXVxtoV24yAwFo1MA9eWiH8w4miaHxZNU1I9lv+7Yl1SqCZYEWLULW8SCSdO5IqEojWSAtifzn78cCKxWNNrS0wHcbQFfEn2eMDR8suVH3PFkfiya5ZScXtd7HJEf7Lzsjb5+o+c+zQrFq4kUBYkEooQLA8y9/d/GXXrWwBk52fV2WSV365pvjE1J+JsD74jSPgZFw+Sfc66l+ID3/5I7hU4sk9vskkBTGKoV8nKshrlL6orXVVOuDLZp4TkNKYpXd6xWfEfAZLkR00c4D0k/fGk2WHn7s8RQw7C7XWTnZ+Fx+9hu917cc2LF23ytcPBML9+OY1YtOYbfyQU4YvXvgUgOy+L3Y4agNtb82fZl+XlhGsbWgjZqI/k3w05F4AUAl578lrhW4jTrgmq0dlYK49Gl++BLt8da9WJaHRB/RfNINOUVI9uvTsn7Rx1eVz03bc3P4+dQrA8WGOf0+XAsmp7FazlAAAgAElEQVQmARFhuz23abVLO4qjEAqeR9dcBhoAFCQPKXgScWRnOrxGJyJc+PCZnHzj//HP9AW069KWzbfaLCXXtizFHtmSZF+1ZHH1Cxcz7KThTP16Om6vm0gowjFXHckBp+6VkjhaOxEXkjMUcoYm7FOrHF11MmgpVf9Xkalo8UnQ7usmN1QVTGKolziEfU/YnXGjvqnqPHR73eQW5jD0sXMoXvIIsybNIRTve/D4PXTbtjOrlthtusGKEN4sLx6fm6tGXpjJLyXjxDMA2n0f74wTcPVqddVW84vy6Lvvdim9pi/LS6+BPZnx06waH2Jcbid7HrtL1eusXD/3fHwTyxesZOWiYrpt24ns/JaflJuE4MfxwpLVE7hl97uFvgJf01uCxiSGOoRDEa7e51b++WNBVVIQEfof3IdrXriY/KI87hv3H0aPGMu4l75GVTn4rH059oojsCzlq/9+x9+/zqVb784cdMa+5LQxv4QiDnD3znQYLc41L17M5XvcTDgYIVgRwp/jI79dHufde2rCse27FNG+S/pKMBigsYVAYuc/GoLYwrTH0xCmVlIdxjz7Oc9c/XKNZTwBfNle3ln2Ar4k6zIYRqZUlFby9Rs/sPDvxfTccUv2Om5XPN5kK+kZ6abBz9CS6xJH5kkW0uYZxLtr2mJpaK0k88RQh/Hv/JiQFAAcTgczfprFTgfskIGoWhcNT0TLnwdrqd2Zl31uWguONSfZeVkcecFBmQ7DSMa7Pzg7QfQfILJ2I7i2Bs8u9ZyYOSYx1CE7P/kwQlXFn5Ns6KWRSlble1B6B1WT4qKz0cD7UPSRPTzQMJoJERcUvoWWPwHBMYDDLgeTc2GT7Wczw1XrcNSFByeMPQfIaZPNNmYZz0alGoayu6k5UzoCWoqWP5v8nNgSrJLbsFYchlV8jj3LtJUJBULMn7GAstXl6z/YSCtx5ODIuwFH++9xtP8WR+6VTbIUxlrmiaEO/Q/qy3FXD+Kt+z/AHZ/L4M3ycM/Ym3E4TD5tVNF5JB+CGYXwdwlbNbYIXTk43oYbhdgce12IvNtwZB3b2NFmnKry5n3v8/o9oxGHEA3H2OeE3bhy5IWmn8HYKClJDCJyKDACewW351X1vlr7rwLOw17BbQVwjqrOj++LAWvn5f+rqoNSEVMqnHn7CRx14UFM+3YmuYU59N13O5yull8NNOMcBaBJlgcFcCT2MWj5k3Z9+xpVKoNQdg/qH4RIy35z/OK1b3n97tEEq/WJffvuBLx+D1c8c0EGIzOaq01ODGLXTX4SOAhYCEwUkY9UdUa1w6YAA1S1UkQuAh4ATozvC6hqv02NozEsmbeM/w57j9/H/0FRp7ZEIzEGHrZjpsNq8cTZHvUMiK8fHam2w49kn594QugnkpcujkFsgV3LpgV76/4PaiQFgHAgzOevjOfiR8/G42t6E6iMpi0VbSIDgdmqOlftpYveBGrMs1fVr1WrxmpNADrTxC2Zt4yLdrqOz18Zz5K5y5n23UzuPP5hxjz7WaZDaxWkzXBw7wR4QXIAH2Rfhvj2SzzY2S75RTRqP320cKuXJRboA1CFiiTF84zmRcO/YK06HmtZP6wVh6HBcY1+z1Qkhk5A9aIfC+Pb6nIu8Em11z4RmSQiE0Tk6BTEkxL/HfYegfJgjXLJocoQz13/GpFwpJ4zjVQQRxscbV9F2n2KFLyEtP8JR865yY/NHgIJHXke8O6DtILE0Hu3bUg2uCW3IJv8IrN6XlOjGkGD47BKH0Qr30KtugcLaPgXtPg8u2y9Vtr9Z2uuxap8t1FjTGsvqoicBgwAHqy2uVt8wsUpwKMiknTIj4gMiSeQSStWrEh2SEpN/fqPpDX01VKWzF3e6Pc3bOLshHj61ltTSXwHQs7ldnKQbMBjz3vIvz99gWbQOfecgi/bh8OxLjt4szxcPOIcM1AiTVQDWBWvYq06HWv1ZXap+WTHWaXoykFoyfVQ+Rxaeg+6Yj80Oif58WUPUnN0Hvbr8odQbbxFnlLR+bwI6FLtdef4thpE5EDgZmAf1XXFy1V1UfzvuSLyDbAjkPBdUtWRwEiwZz6nIO56FXUuZOk/iQkgGonRxpQqbnIc2eegWSfbI5oc7ZC6mpdaoC2278qTE+/jtbveZeaEv+m4VQdOuekY+u6zHfOm/8snL3xJWXE5ux01gD2OHmgGUKSYahBddQJE52O/iQsaGo/mXo6jWtltAC0fAbF/Wdd3FgANomuuQ4reS7x4dHbym1ql9hOE5KTwK1knFYlhItBTRLbATggnYX/6ryIiOwLPAoeq6vJq2wuASlUNiUgRsAd2x3TaTflqGs9e/Qr/zFhAQYd8dh+0M7MmzyUcWLfqlcfnZpcjdiKvrXk831iqYQiOQYPf2G/gWSch7p4pubaIv9XWYuqyTSdufO3yGtvGjfqax4c+TyQcxYpZfD/6Zz568lPu++yWVlvptzFo5ehqSQHsodYBKBuO+o+rue5CcCw1BlSsPT76J2qVJq7R4OgIsSTJQbxJmk9TZ5OfM1U1ClwCjANmAm+r6h8icqeIrB16+iCQA7wjIr+JyEfx7dsCk0RkKvA1cF+t0UxpMf2HP7ll0H3MmfoPsUiMlQuL+WzUN/Q/uA9ZuX78uT7cXjcDD9+Ja0ddku7wWgzVALrqOLT0Dgh9CoHX0VXHYgXGZjq0FqeyLMDjQ58nFAhXNYkGK0L8NWkOX7/5Q4aja2FCn5PY3AOIGyJTam2s62lNSfZ2LLmXk7jIlR+yz8ceENo4UvKxQVXHAmNrbbu12r8PrOO8H4GMFx166T9v1FgPFyBYGWLqV3/wxqJnWbFgFQXt882TwibSyrfi9WLW/hLF7D+lN6O+A5tkXfrmavr3f+J0OxOKegYrQnzz1o8cdPo+mQmsJXIUAkLipEwLJL/mJv9gqHgFqD682AHuHRFHYrOQ+A5B88qh/CGwSkB8kH0ekt24ZfxbzfPkykWreO761/j5419xe90cft4BnHrLcXi8bubPSF76Nha1F0zvtm2TH13bPATHkvSTFQKR6eDZKd0RtVjeLA91FU5OxTrTxjqSdRoaTPbU4EadPak+YExyLkHDv0D0b3uNBvGCZCNt6m5Bd2Qdi/qPsSdxir9RnxTWahWJoaKkgosH3EDJytKqx+p3h49h1uQ53PvJf+jUczNKVpQmnCdOB3lFpqM5ZersKLNANn3tY2Od7ffohdfvIVBW85HBl+XliPOTPsAbG0k8O6K510PZPdToP9AAFB+Dth1dNapOxA+Fb0P4Z4j+Ac7O4N1vvU/LItJoHc3JtIqxbJ+9/A2VZYEaw0/DgTDTvpvJ3N/nc9adJ+HNqvkf483ycuyVR+DxulmxcBWPXjiS07cayuV73MyPHyYfimbUT7JOBWp/WhW7zIVrm0yE1GI5XU7uGXsTuYU5ZOX58efY/WTHXzuIfvttv97zS1aWMuqWN7h45+u5dfD9/P5t2rv+mhXJOobE/oMwxJaglW/WPFYE8e5ql5H3HdIkm1BbxRPDzAl/J19bweFg7u/zOfC0vbn5jSt55qpRLJm7jOw22Zx43dGccO0gVi5axQU7XkNlSYBYNMbSecu597QRnH7b8ZxwjVlIfYN494es06DyFZD4j57kIAUjm2z54eas505b8tbikfz6+e+Ur6mk737bUbR54XrPW7OihAv6XUtZcTmRUIS/J8/l1y9/5+JHz+bw88zTRlKR6fbPdELzXRBC46COyZlNVatIDN2264zH5yYcTJyx3HHLDgDsdtQAdjtqALFoDIfTUfVG9faDHxEotZPCWsGKEK/c/g5HXXQI/myzNkNDiQiSdy2afTqEf7XLVXgGpqXNtLVye9zsckT/DTrn3Yf/R9mqMiLhdYUMQ5VhnrnqZQ48bW9TeykZySN5vS7indPNS6toSjrs3AMSxm273E46btWB3rttXWO70+Ws8el1ylfTiEYS/8OdLgf/zkyYx2c0gDg3Q/yHI97dTFJogn4ZO6VGUlhLHMI/fyxIcoaBa2t7zkHCW6ofyTo9ExFtklaRGAo3K+CRb++kZ/8tcbocuNxOdj1yAA9+cdt6mzDad00+gzYajlK4WZvGCNcwMqqgjp/raDhKvhmMkZSIIAXPgbOLPZBCcgAv5AxFvHtkOrwN1iqakgC26tudpybeT6AiiMvtxO1pWI3+E64dxNRvpteY5+D2uNhhr21p17ltY4VrGBlz3FVH8sePf9Xol3O6nfTsvxUdurWeUiMbSlxdoOgziE4Daw24+yKO/PWf2AS1iieG6vzZvgYnBYC++2zHpU+eR3abrKqRHTsd1Idb3r6qEaM0jMzZ+dAdOeuuE/H6PWTlZeH1e9i6/1bcPvqaTIfW5IkI4u6DePdutkkBQLSuWTBN2IABA3TSpElpvWc0EmXR7KXkF+XSpl3z/Q83jIYKlAeYM3U+BR3y6dSjY6bDMVJARCbHq1nXq9U0JW0ql9tlZkAbzVYoEOK7935m8eylbNGnG7sd1X+9hfT8OX6236NXmiI0mhKTGIwWQWMrIDoHXF0QZ33rRLU+y+av4LLdbiJQHiRQHsSf66NwswIe++lu8gpN/a8NpVYFhL4BDYF3T8TZPtMhpZxJDEazphpDS2+BwEd23RkNo97dkTYjEDFzTAAeOf8Z1qxYVw4mUBZkWXA5z13/Glc/d9EGX2/2lHksmr2ULXboStderSsJa+gHdM1QqormlcbQ3CtwZDevCWzrY/oYjGbNKh8J5U9Qs4CZF/yDceQPy1RYTUYkHOHInNOwoomrfWXnZ/HB6pcbfK2KkgpuPOwe5v4+H6fLQSwSY8cDduCWd67G4234gI7mSq1KdMXu9gI5NfiQtm8izWAtkIb2MbS6UUlGC1P5ColVLUMQ+BB7qZDWbd60f5OUadg4jw19gdlT5hKqDFFZGiAUCPPrl9N47a53UnODpi78LZBs3lMYDYxOdzSNyiQGo1lRtewF0gNj0OhC0LoWUo+SuFJW61G8dDUX7nQtV+97W9L3MpfHxb4nNnziVSwa49t3fyISqplsw4EwY5/7clPDbR40RPJa5hZosnLyzVdK+hhE5FBgBHZ5wedV9b5a+73AK0B/YBVwoqr+E993I3AudqGRy1R1XCpiMloejS1Gi08Da3X8U3DUrreU8GgPOLvbJY5bqduPfYh/pv9LrFYTkjgEX7aX9l2KOO++Uxt8vVg0VqM6cXXJClS2SJ49SFoPSbIQ36FpD6cxbfITg9jFbp4EDgN6AyeLSO3GtnOB1araAxgO3B8/tzf2GtHbAYcCT4kpnmPUQVcPhdhie8ESKoCQnSTwsu4zjtNezCT/TjQ8Fav4bKzle2CtOhUN/Zyx2NNp2fwVzJkyLyEpALTvWsTNr1/Bs1MfIqdNdoOv6fF52Kpf94TtDocw4OB+mxJusyHOIsi9FnupzbVvnX7w7hdPGi1HKpqSBgKzVXWuqoaBN4Ha9agHA2t7ud4FDhC7SNFg4E1VDanqPGB2/HqGUYPGFkF0NlD7zS4Eru6QdQq4+4L//5C279nnFJ8O4R/AWgGRiejq89Hg1+kOPe3K11TYy3om4c3ysssR/XE6N/zz11UjL8Sf68fttZOwx+8hpyCbCx4+Y5PibU4c2Wcgbd+BrLPBfzJS8BSS/0iLKxufiqakTkD1kosLgV3qOkZVoyJSArSNb59Q69zWNf7NaBgNgDiTd6RqBEfef2psslYeQ2KndBAtuxvx7ddYUTYJXbfthMOR+JnP7XWx+6D1DkipU48dt+DFGcP539Of8c/0BfTapQdHDDmo1a2FLu5tEPf1mQ6jUTWbeQwiMgQYAtC1a9cMR2OknXMLeyH0hP4Ep70AUG3Rv5JfJ7YA1QgiLXd4pdvj5rKnz+eR854mHIygluLxe8gvyuX4qwdt0rWLOrXl7GEnpyhSo6lKRWJYBHSp9rpzfFuyYxaKiAvIx+6Ebsi5AKjqSGAk2PMYUhC30YyIOCH/QXT1xUCYdY8OFlT+F/XuhXh3W3eCoy1YS5NcyYFaQcTZchMDwP4n7UmnHh15/7GxLP93BQMP3ZEjLzx4g/oVjNTQyCy0/HGITANXNyRnKOJp2i3mmzzBLf5GPws4APtNfSJwiqr+Ue2YocAOqnqhiJwEHKOqJ4jIdsDr2P0KmwNfAj1VtY6lkGxmglvrZVW+C6W3Yg9HrUZykPYTqtbPtSpeh7JhicfhBN9hONo8ko5wjVZOIzPQVSdjN2uufa/1Qf5DOPwHpz2etE1wU3sW0SXAOGAm8Laq/iEid4rI2ufWF4C2IjIbuAq4IX7uH8DbwAzgU2Do+pKC0cqFvyfxzX7tvl+q/ilZJ9sLpiSIQXAcqq1kiKWRUVp2PxCgZudYEMqG0ZSrTqSkj0FVxwJja227tdq/g8DxdZx7N3B3KuIwWoF6PzesSxgiUv+EXw3atZUMozFFpiXfbq0CLYuvFd30mJnPRrMi/qOAOp4EPLUGw3l2J+mPuHPzJvsLaTRfSZ8AHHWt8mjPt2mqTGIwmhfvgeDdu9ovlQfwQd4DCTOdJfea+Nq7nvgWJ+BH8oa1uHHnRuZo+BeslUeiy7bBWjYAq/xxqlrEsy8EaicAH2Sd2KRHxjWb4aqGASDigDYjIDIJDY0HyQXfIUhkClbpPYhra/AdjjiyEFdXKBqLVr4K4cng2hLJPhtxbZXpL8NoITQyHS0+j6o5M1oK5c+j1hok7xbEfwxqLYeKZwAHaBT8g5Dc6zIZ9nqZsttGs6axZeiq4+1fSK0EssCRhbR9xyzYY2w0jfyFlj0Mkd/A2R7JvhjxH55wnLX6Ygh9SeLMSy/S/ifEkWNfT4MQWwKOdlXbMsGU3TZaBS292y55UTXxrRKsYrTktozGZTRPGluBteYmdNUgCH8Dugais9CSG7EqRiWeEJ1F0un44rLreq19KT7EtUVGk8KGMInBaN5CX5NY8dKC8PeoJq8GahjJaGwJuvJICL5H4pt9AMpHYJeDq8a1DUnrmmvUHuTQTJnEYDRvdRbjtdDSu5v0WHGjadHyx+wmyToHOqvdHFSN5AzFrrZanQ+yTm42TwfJmMRgNG++w4A6RncE3oXw+LSGYzQvqlGsirewVh0HgQ9Jut5C1cHRhOGn4u6NFL4Iru2xh6AWQs5QJPeGRo27sZlRSXWwLIsZP/5FRUkl2+3Ry9SYaaIk90Y0/AvEFiTZG0Ar30G8+6Y7LKMZUFV0zWUQ+gF7dnJ9vOA/KulTgHj6I0Uta2lPkxiSmD9jATccOoyKkkpEhGg4yrn3ncoxlx2R6dCMWsSRh+beDmuGklhmG6jdJmwYa0V+b2BScNjrfNQq7d6SmcRQi2VZ3HDIMFYtLq6xvOuLN73BNgN6sN3u22QuOCMp8e6MiiOxaVj8iL/2mlGGEReZRN3rgnsAAfd20OZxHM52aQws80wfQy0zfvyLitLKhDW/w4Ew/3vms8wEZdRLxAd592N3Asb7GyQL3APjfRCGkYSjLYgnyQ4P+E9Cij7G0fbNVpcUwDwxJFjbfFSbqlK6qiwDERkN4fAfgrq3QwMfgLUG8e0Lnt3tmdKGkYz3YOCuxO3iQnKHIo6CtIfUVJjEUMt2e/QiGk4s6+zL9rL3sbtmICKjocTVGcm9JNNhGM2EOLKg8FV78SddDYi9rkebEa06KYBpSkqQ0yab8+8/DW+Wt+rJwZvlpUuvTux/yp4Zjs4wjFQSd2+k3ddI4btI4ZtIu28RT/9Mh5Vx5okhiaMvPZytB2zF/575jJKVZex97K7sf+peeLxNtxqiYRgbR0TA3TPTYTQpm5QYRKQQeAvoDvwDnKCqq2sd0w94GsjDnj1yt6q+Fd83CtgHKIkffpaq/rYpMaVK7922ofduZgSSYTQm1ai9HnLlf0ErwN3Hrkrq3j7TobVqm9qUdAPwpar2xF6vOdl0v0rgDFXdDjgUeFRE2lTbf62q9ov/aRJJwTCM9NCSm6HipXgpihhEpqCrTsIK/7Hec5sCjc7GKhuBVfYQWtdqbc3QpiaGwcDL8X+/DBxd+wBVnaWqf8f/vRhYDrS+8V9GWqlG7KJommTSm9EkaGwFBD8mcWJiGIqPQ0MTMhFWg1kVo9CVx9hrLVQ8j646Fav03kyHlRKbmhg6qOraqlJLgQ71HSwiA7FnjsyptvluEfldRIaLmEV4jU1nVbyMLt8FXXEIumwgVuk9qCaONDMyLPZPPetux9A1QxOrmTYRGlsCZQ9jJ7UYYNn/rnyjRTw5rDcxiMgXIjI9yZ8aU0rVLmNZZylLEekIvAqcrevqId8I9AJ2BgqB6+s5f4iITBKRSStWrFj/V2a0SlblR1D2CGg59i9tECrfRMuGZzo0ozZnN9BQPQcohH9JWzgbJPQ1ScttE0aD49IdTcqtt/NZVQ+sa5+ILBORjqq6JP7Gv7yO4/KAj4GbVbXq+bDa00ZIRF4CrqknjpHASLBXcFtf3EYrVfEUibVvglD5Gpp7JSLpG4insVVoxbMQ+gYchfayor5D0nb/dNPIn/YyqrEl4N0b8R9Xb+lpcbZHfYdCcAz2J+4k19QQhKeCOMC1XROasOgkeWIQ6qz224xs6m/JR8CZwH3xvz+sfYCIeID3gVdU9d1a+9YmFcHun5i+ifEYrZ2V9LMJELVHvUh+WsJQa7W9Cpi1BohA7B90zUw0ZxaOnEvTEkM6WYFPoeQ6IIy9UNIktPIVaPs+4qj7ey7596JaCaEvEndqBNbcgEq8GVCyoM1TiKdvo3wNG8R3EJQOS7LDhfibf7HNTU2/9wEHicjfwIHx14jIABF5Pn7MCcDewFki8lv8T7/4vv+KyDRgGlAEJPtOG0bDuXon3+5oA5KXtjC04jWwSqhZpC0A5SNRq6Su01Jzb6scq+IlrOIhWCV3odG5jXs/jUDpzdhNd2s/+QchthxNthxmNSJupM0T4D2UdQveeABv/FoldkLXCrBWoKvPRq2KxvlCNoA4CiH/Puw4/dixeyH3KsTVI7PBpYA0xxWuBgwYoJMmTcp0GEYTpOGpaPHp1Bzp4oP8e3D4j0xbHNaqE+yF5GuTXKTNk4i3ccqrqFWMrvw/sFZjfw+cgBspeALx7t0494zMRItPsd+8a3P2xNHu4/Vew7IiEPkVwt/ZCVyDUPEcULsPIgvJvw3x/19KYt9UahVD8HMgBt79EedmmQ6pXiIyWVUHrO+4ptJgZxgpIZ6+SNvXwbM3OIrA3Q8peCKtSQGIr/ebbC3gCDRitU4tfxaslaxLjDEgiJbc0HhrYEuOvbpZMvU0IwFodC7WqlNh+Q6w+hyILUKyTsR+0krWMR2JJ72mQRyFSNaJSNYpTT4pbAhTEsNoccS9PVL4/PoPbMwYss5Gg19R88nFBa6tEddWjXfj0BckXWPAqoDYv+DqnvJbiqsL6uoJ0ZnUWBpT/Ej2GXWep9YadNUJoGXYAxotCH6GRudAzg0gr4JW1jrLCR5TzLKxmScGw6hFw1OxVl+KtfIorJI70NjiDb6GePpC/j12s4hkAx5w74QUjkxNjLGlaOWbaOU7dnNG1Y3rWoI2Vs++TScFT4Gzu91BLDnYaxqcDt66R2Fp5ej4CnvVm7MjdgITF7h3xG6/X3sTP/gORNx19CMZKWOeGAyjGg1+ia65ErsZQyE6Bw1+BG1HI65uG3Qth/9I1HcIROeBIx9x1jv/s8GsilHxyVUCIlB6J5p/Pw7/4UjWmWjZnaDVh+w67RpEjdiEJc7NoGgsRKdBbGX8fkX1nxSbRfLlWBWJzYeCkRD4AA28DziRrOPBl+YmwVbKJAbDiFO10NLbqPlmZQ9z1bLhSMGjG3xNETe4t05djNHZ8aQQb39f+2G75HrUuyv4j7HXMg68F1+dzALn5kibESmLoS52ldI+DR/G79oe+ISkay67tra/d1nH2wnBSCuTGAxjLWtFfIhpwg4Ip7Zuj8YWoeUv2iNxXFsi2ech7m3Xf17gYyBJR684IPgFknUCkn8HmnMhRKaBo4P96T3JqoSZJv6j0YqnwAqyLsM5wbW1nWCMjDGJwTDWkhzqrOriKEzZbTQ6F111XLwcRASiM9Hg51DwNOLdYz1nR5LHqEr1hCHOjuDsmLKYG4M4ctDCN2DVCfEV1OJicyA6A9zbZS64Vs50PhvGWtaqOjpo/ZB9bspuo2UPxsf8rx09ZBdg09JbWd+8IvEdjD0BLOGq4N0vZTGmTfCLWv0hMdBydM1l6/1eGI3HJAbDAFTDaPHJoEmakrJORvzHpO5m4V9I+qk/tiQ+dLNu4u4DWSdjz7R1YE9g80HulfZTQnMTGE3SDujYCnt0kpERpinJMABCX8XHzNeeBOZDXN1T20YveXUkAAeIL8n2Wkfl3Yj6j4hX8XQh/qOabxmGer+vTa9fpLUwTwyGARBbVEcJ6CAaW5jae2WfTY3x+QB4wX8Uds3J9RN3Hxy51+LIvbL5JgUA3zGsq5FUjXMzcHZJeziGzSQGwwB76GRdb8rBL9DIjJTdSrJOg6zjAA9ILuAF755I3q0pu0dzIdmngbufPTEOJ5AFkoe0GdEkR1K1FqaInmEAqooWnwSRGSSt0SNZSNuPEFfX1N3TKoboXHB2ap79AymiqhCZCOFfwdkevIcgjsabpd2aNbSInuljMAzik7MKX0bLHobKlxMP0DBa8RKSf1vq7ukoBE/qhsECaOQviEwHZyfwDGzUhW3UWg2h8YADvPvUu+5CfUQEPAPtP0aTYBKDYcSJ+MC7PxoYnaRzOArRPzISV0OohtE1l0LoJ3uyGwKOdlD430YphWFVfgClt2A3/wDE0Pz7cLSARWoM08dgGDW5togXdkvYUfciQE2AVrxgJwWC9ugqrYDYArSkztVyN/5escVQ+h/sJrfK+J8QlNyAxupaQc9oTjYpMYhIoYh8LiJ/x/8uqOO4WLXV2z6qtn0LEflZRGaLyFvS0CEZhvvSgRsAAAw9SURBVNFIxLkZePcnYaSMeJDsczISU4NUvkXifICYvcSmVZ7SW2npQ9hLeCYRHJfSexmZsalPDDcAX6pqT+DL+OtkAqraL/5nULXt9wPDVbUHsBpI3fRSw9hI0uZByDotPgtawL0jUvjflHY8p16yobZr1fEmvhHUKoHQJ3Xsjab0XkbmbGpiGAys7al7GTi6oSeKPRZtf+DdjTnfMBqLiAdH3nU4OkxBOvyJo+1bSFOv2+M9mKRdhs6udid3qoS+o97yqd59U3cvI2M2NTF0UNUl8X8vBeoqOO8TkUkiMkFE1r75twXWqFatCbgQ6LSJ8RhGSjWXsfSSezk42rNu4pwXJBtp80CKb+S0/yTjbOTV6apRDWGVP4e14kislYOwKl5FNcnKdcZGWe+oJBH5Aki2mOnN1V+oqopIXZMiuqnqIhHZEvhKRKYByeob1xfHEGAIQNeuTfmR3jDSTxyF/H979x7bVnnGcfz7cxMnDaFN2oSSQVWoxCgwROkCAjom1nXTVmkURoFO01Y0ELchhjYmyiZNU8dE2SSGYBuj4rJOQlwGbG25qOPSFpAgI7BCoFzKRQyqtM3oFTUXN372xzlp7cSOnTrxsePnI1k+8Xnj/Hzs+LHfc8770vQk1r0GEu1QdSyaeOHoH5EUPxusP8OKmlE9lHc4Zv3Yjh9A4h0O7FfZ+3us7wVouKtsinkpy1kYzGx+tnWStklqMbNOSS1AxkMSzGxLeP2hpPXAqcCjQIOkqvBbw9HAlmFyrABWQHCCW67czo2UWT8kt4Imo1h91HFGTLE6dNjFwMVj+DfqoeE2bNd1BGMZ9QfXdZeg+Jwx+7tpep+HxODZ33qgty2YpCh+SnFyjGOFdiWtBpaEy0uAVYMbSGqUVBMuNwFzgU0WnHK9Dlg03O87VwzJfaux7WdiXQuw7WeQ3PVTzDLMLOZQ7Tx0xHo06UZ0+PWoaQ2xST8r2t+3xKsEh8gOth8SrxYtx3hW6Aluy4GHJV0KfAxcBCCpFbjSzC4DTgDukpQkKETLzWxg4JkbgAcl3QT8B7inwDzOjZj1toXH5ad8Au15GrMEarwjslylwKwfetZg+x4FDNV9F2oXBl1XdYsjyaTYERg1DDkSS/FwP4srlI+V5CpecsePoO/FDGviqHkDmjAVAEu8CX1tEJsKNd9EsbriBi0yM8N2XRt03RyYl3ki1JyJGu6MrC/fkjuxrnnhZEcpNBkd8UJwBrvLKN+xkvzMZ+eyDautakh2YdZPcue12Gffx/beiu3+NdZ1dlAoxrPE64OKAsFy30vBDu6IKNaIGu+DWAvBUVi1MOGY4FwTLwqjwsdKci7eCt2fEOxITWFJqJoB3avCweIG3iATYGA7r4bmDeP3KJi+NjKesGY9wbr4aUWPNEDx2dC8Hvo/AiYE52uM1+chAv6NwVU8HXZVOB9A6r/DRKj/MdJErPth0j81h2wP7H+nSCkjEGvIMkdFDcQyjn5TVJJQ1UxUNcOLwijzwuAqnqqmo6mPQe2CYOdl1UmoYTmx+svDFpmO24fgcM3BU4GWNku8h3U/nl83WO23yTi9pmLBtnLjlnclOQfBp86GWzOvrD0fEu8yZJA61ULVrDHPNhrMerGdV0Ffe3jmchKrOh413pv1nA3FJkHjPdiua4LuIwDiqPEOVALfGNzY8cLgXA6qW4T1rg12xto+guEmJqCG21G24SFKjO29DfpeAXph4EDExCZsz7Jhh81Q/MvQ/GIw+Q9A9ZfK5jG7Q+eFwbkcpDg03gd9L2F9L6FYM0z8zugOTjfWuh9h6AisfdDzBGbLh53pTZrgZxNXGC8MzuVBikHNXFQzN+ooh8ayDcvdH158d6M7yF8NzlWCmrlk/HevPgVpmGG0XUXywuBcBdDhvwBN4uDMdHFQPZq0LMpYrkR5V5JzFUBV06F5LbbvYUh0QNXxqG7x6A/L7cYFLwzOVQjFGlH9FVHHcGXAu5Kcc86l8cLgnHMujRcG55xzabwwOOecS1NQYZA0RdLTkjaH10MGUJH0NUkbUy49ks4L1/1V0kcp62YXksc551zhCv3GsBR41syOA54Nf05jZuvMbLaZzQbmEUzW+q+UJj8fWG9mGwvM45xzrkCFFoaFwMpweSVwXo72i4CnzCzTTN7OuYiZ9ZLcs5zktlaSW08mufMKbP9/o47liqzQwjDNzDrD5a3AtBztFwMPDLrtt5LekPQHSTUF5nHOFcB2XgX77g8mIaIXejdgn12AJXdEHc0VUc7CIOkZSW9muCxMbWdmxsEBfTPdTwtwMrA25eYbgVnAacAU4IZhfv9ySe2S2ru6unLFds6NkCXeC+ZrSBuFNQnWE5wx7SpGzjOfzWx+tnWStklqMbPO8I1/+zB3dRHwDzNLpNz3wLeNXkn3AdcPk2MFsAKgtbU1awFyzh2i/ZuDSXyG/Hf1BsNouIpRaFfSamBJuLwEWDVM2+8xqBspLCYomLD1PCCP+Qadc2Oi6liwTFOVxstmpjo3OgotDMuBb0jaDMwPf0ZSq6S7BxpJOgaYDmwY9Pv3S+oAOoAm4KYC8zjnDpGqT4TqWUB80Io4qlscSSYXDQW7BspLa2urtbe3Rx3DuXHHkp9je38D3U8A+6F6Npq0DFV/MepobhRIetXMWnO189FVnXMHKFaPJt+CTboZSCL5W0Ql8mfdOTdEMAe0j5hTqfyZd845l8YLg3POuTReGJxzzqXxwuCccy6NFwbnnHNpvDA455xLU5YnuEnqAj4eo7tvAv43Rvc91so1e7nmhvLNXq65oXyzl0LuGWbWnKtRWRaGsSSpPZ8zA0tRuWYv19xQvtnLNTeUb/Zyyu1dSc4559J4YXDOOZfGC8NQK6IOUIByzV6uuaF8s5drbijf7GWT2/cxOOecS+PfGJxzzqWp+MIg6UJJb0lKSsp6xICkb0l6V9L7kpYWM2M2kqZIelrS5vC6MUu7fkkbw8vqYudMyTHsNpRUI+mhcH1bOMFT5PLIfYmkrpRtfFkUOQeTdK+k7ZIyzoyowO3h43pD0pxiZ8wmj+znSNqdss1/VeyMmUiaLmmdpE3h+8pPMrQp2e1+gJlV9AU4ATgeWA+0ZmkzAfgAmEkwvdXrwIklkP13wNJweSlwS5Z2n5dA1pzbELga+Eu4vBh4qExyXwL8MeqsGbJ/FZgDvJll/QLgKUDAGUBb1JlHkP0c4PGoc2bI1QLMCZcPB97L8Hop2e0+cKn4bwxm9raZvZuj2enA+2b2oZn1AQ8CC8c+XU4LgZXh8kqCebNLVT7bMPXxPAJ8PZwPPEql+tznZGbPAzuGabIQ+JsFXgYaBuZhj1oe2UuSmXWa2Wvh8l7gbeCoQc1KdrsPqPjCkKejgE9Sfv6UoU92FKaZWWe4vBWYlqVdraR2SS9Liqp45LMND7Qxs/3AbmBqUdJll+9zf0HYLfCIpOnFiVawUn1d5+tMSa9LekrSSVGHGSzsCj0VaBu0quS3e0XM4CbpGeDIDKt+aWarip1nJIbLnvqDmZmkbIeYzTCzLZJmAs9J6jCzD0Y7awVbAzxgZr2SriD41jMv4kzj3WsEr+vPJS0A/gkcF3GmAyTVA48C15nZnqjzjFRFFAYzm1/gXWwBUj8FHh3eNuaGyy5pm6QWM+sMv4puz3IfW8LrDyWtJ/gUU+zCkM82HGjzqYLJhicDnxUnXlY5c5tZasa7Cfb9lIPIXteFSn2zNbMnJf1ZUpOZRT0WEZKqCYrC/Wb2WIYmJb/dvSspP68Ax0k6VlKcYMdoZEf3pFgNLAmXlwBDvv1IapRUEy43AXOBTUVLeFA+2zD18SwCnrNwb12EcuYe1D98LkG/cjlYDfwwPErmDGB3StdkSZN05MD+J0mnE7yXRf0hgjDTPcDbZnZrlmalv92j3vsd9QU4n6CPrxfYBqwNb/8C8GRKuwUERxh8QNAFVQrZpwLPApuBZ4Ap4e2twN3h8llAB8HRNB3ApRHmHbINgWXAueFyLfB34H3g38DMqLdxnrlvBt4Kt/E6YFbUmcNcDwCdQCJ8jV8KXAlcGa4X8KfwcXWQ5ai8Es1+Tco2fxk4K+rMYa6vAAa8AWwMLwvKZbsPXPzMZ+ecc2m8K8k551waLwzOOefSeGFwzjmXxguDc865NF4YnHPOpfHC4JxzLo0XBuecc2m8MDjnnEvzf/w/aGi2PpkFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X[:, 0], X[:, 1], c=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WtiJwD-vRnyC"
   },
   "source": [
    "The goal is to build a classifier that classifies the white points from the black ones.\n",
    "\n",
    "Let us create weights and biases for a model `y = sigmoid(W * X + b)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "fESxbldq9GOy"
   },
   "outputs": [],
   "source": [
    "weight = torch.randn(2, requires_grad=True)\n",
    "bias = torch.randn(1, requires_grad=True)\n",
    "\n",
    "X = torch.tensor(X, dtype=torch.float32) # convert from numpy to PyTorch format\n",
    "y = torch.tensor(y, dtype=torch.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DTOEsYEVSlyr"
   },
   "source": [
    "Now, let's create an Stochastic Gradient Descent optimizer that will optimize the weights and biases of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "E5eqxh9wAKgL"
   },
   "outputs": [],
   "source": [
    "optimizer = optim.SGD([weight, bias], lr=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iakcZKoNS26B"
   },
   "source": [
    "We shall use the Binary Cross Entropy loss, which implicitly will do the Sigmoid operation to convert the inputs to logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "pyMVxlCrBICE"
   },
   "outputs": [],
   "source": [
    "loss_fn = F.binary_cross_entropy_with_logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iowMShh3TCpn"
   },
   "source": [
    "Now, let's train!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 379
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 5595,
     "status": "ok",
     "timestamp": 1529905103810,
     "user": {
      "displayName": "Soumith Chintala",
      "photoUrl": "//lh5.googleusercontent.com/-ASzCTwFtgW0/AAAAAAAAAAI/AAAAAAAASL4/OWJyxUEEJ64/s50-c-k-no/photo.jpg",
      "userId": "106447253626219410322"
     },
     "user_tz": 240
    },
    "id": "DXNRgUW9B7Ct",
    "outputId": "8c5f7c60-81a0-4876-85a1-7bb80e93f879"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x118acc250>]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl4FeXd//H3NwskrGEJCIRVUEFlkbBvLtWq4I6KIiIqKFql1dbWLvbp82ufPtparYoLBRUEFURBBbcKKPsSVkGQPYQ9gux78v39cQ59qGWJkMkk53xe13UuMnNmzvnONZAPc8/c923ujoiIxK+EsAsQEZFwKQhEROKcgkBEJM4pCERE4pyCQEQkzikIRETinIJARCTOKQhEROKcgkBEJM4lhV1AQVStWtXr1asXdhkiIiXK3Llzv3X39FNtVyKCoF69emRlZYVdhohIiWJm2QXZTk1DIiJxLvAgMLNEM5tvZuOiy0PMbKGZLTKz0WZWLugaRETkxIriimAAsPSY5Z+5ezN3bwqsA35SBDWIiMgJBBoEZpYBdAUGH13n7rui7xmQCmgcbBGREAV9RfAs8BiQf+xKM3sN2AycBzwfcA0iInISgQWBmXUDtrr73O+/5+59gJpEmoxuPcH+/cwsy8yycnNzgypTRCTuBXlF0AG41szWAm8Dl5rZ8KNvuntedP1Nx9vZ3Qe5e6a7Z6ann/IxWBEROU2BBYG7P+7uGe5eD+gBTAR6mVlD+Nc9gmuBZUHVMHHZFkZl5QT18SIiMaGoO5QZMNTMKkR/Xgj0D+KL3J0RM9fxxfJcalRMoVMjXVWIiBxPkXQoc/cv3L2bu+e7ewd3v9DdL3D3nkefIipsZsazPZrTqFo5Hhg+j+VbdgfxNSIiJV5M9ywun5LMkLtakVIqkT6vzSF398GwSxIRKXZiOggAaqWlMqR3Jtv2HuTeYVnsP5QXdkkiIsVKzAcBQNOMNJ7r0YJF63cw4O355OWrD5uIyFFxEQQAV5x/Fk90a8JnX2/hT+OXnnoHEZE4USKGoS4sfTrUJ2f7fl6dtoaMSqnc3bF+2CWJiIQuroIA4DddG7Nhxz7+3/ivqZmWwpUX1Ai7JBGRUMVN09BRiQnGs7e2oHntNAa8vYCstdvDLklEJFRxFwQAqaUSGdK7FTXTUrl3WBYrt+4JuyQRkdDEZRAAVC5biqF9WpOUYPR+dTZbdx0IuyQRkVDEbRAA1KlShtfuas13+w7R+7U57DpwOOySRESKXFwHAcCFGRV5+Y6WrNiym37Dsjh4RB3ORCS+xH0QAHQ+J52/3tyMmau388jIhepwJiJxJe4eHz2R61vUInf3Qf700VKqlCvFH649n8hI2SIisU1BcIy+nRvw7Z6DvDJ5NZXLluKnPzon7JJERAKnIPieX111Htv2HuLZz1dQpWwperWrF3ZJIiKBUhB8j5nxvzdeyHd7D/HEB0uoWKYU1zarGXZZIiKB0c3i40hKTGBgz4toVa8yj4xcwKRlW8MuSUQkMAqCE0hJTmRw70zOPas8/UfMZY6GohCRGKUgOIkKKckMvbs1NSumcvfrc1iycWfYJYmIFDoFwSlULVeaN+5tQ/nSSdw5ZLbGJRKRmKMgKIBaaakMv7cNZnDH4FnkbN8XdkkiIoUm8CAws0Qzm29m46LLI8zsGzNbbGavmlly0DUUhgbp5XjjnjbsO3SEnoNnsUWD1IlIjCiKK4IBwLFzQ44AzgMuBFKBe4ughkLRuEYFht7dmm17DtJz8Cy27TkYdkkiImcs0CAwswygKzD46Dp3/8ijgNlARpA1FLYWdSox5K5W5Gzfxx1DZrNzn0YsFZGSLegrgmeBx4D8778RbRLqBXxyvB3NrJ+ZZZlZVm5ubrBV/kBtG1Rh0J2ZrNq6hztfm81uDV8tIiVYYEFgZt2Are4+9wSbvAhMdvcpx3vT3Qe5e6a7Z6anpwdV5mnrck46A3texJINO+nz2hz2HjwSdkkiIqclyCuCDsC1ZrYWeBu41MyGA5jZ74F04JEAvz9wlzepznO3tWB+zg7uGTqH/Yc0l4GIlDyBBYG7P+7uGe5eD+gBTHT3O8zsXuDHwG3u/h9NRiXN1RfW4G+3NGP2mu30HZbFgcMKAxEpWcLoR/AyUB2YYWYLzOyJEGooVNc1r8VT3ZsxbdW33PfGXIWBiJQoRTL6qLt/AXwR/TkmRzzt3jKD/HznsXcX0X/4XF7u1ZLSSYlhlyUickrqWVyIbmlVmz/feCGTvsml//B5mv9YREoEBUEhu611Hf7nhguZuGwrDygMRKQEUBAE4PY2dfjTDRcwYdlWXRmISLGnIAhIzzZ1/3VlcL9uIItIMaYgCNDtber8655BP4WBiBRTCoKA3da6Dk/d1JQpK3LV6UxEiiUFQRG4pVVtnr65GTNWbeOu12ZrOAoRKVYUBEXkxosyeObW5mRlf8edr85mlwaqE5FiQkFQhK5rXosXbmvBwpwd3DF4Fjv2HQq7JBERBUFRu+rCGrzSqyXLNu2mx6CZfKvJbUQkZAqCEFzWuDpD7spk7ba93PrKDDbv1LSXIhIeBUFIOjVKZ9jdbdiy6yA3vzKdddv2hV2SiMQpBUGIWtevzIh727D7wBFufmU6K7fuDrskEYlDCoKQNaudxtv92pKXD7e8MpPFG3aGXZKIxBkFQTFw3lkVeOf+dqQmJ3LboJnMWr0t7JJEJI4oCIqJ+lXLMrp/O6pVKM2dr85m0rKtYZckInFCQVCM1KiYyqj72nFO9fL0HZbF+ws2hF2SiMQBBUExU6Vcad7s24bMepUY8PYCXp+2JuySRCTGKQiKofIpybzepzVXNKnOf334NX/77BvcPeyyRCRGKQiKqZTkRF7seRG3ZGbw3MSV/GbsYvLyFQYiUvgCDwIzSzSz+WY2Lrr8EzNbaWZuZlWD/v6SLCkxgSdvakr/i8/mzVnreHDEPM1pICKFriiuCAYAS49Zngb8CMgugu8u8cyMX155Hr/r1oRPlmym96uz2blfI5eKSOEJNAjMLAPoCgw+us7d57v72iC/Nxbd07E+f+/RnHnrvtP4RCJSqIK+IngWeAzI/6E7mlk/M8sys6zc3NzCr6wEuq55LV67qzU52/dx44vTWLFFQ1KIyJkLLAjMrBuw1d3nns7+7j7I3TPdPTM9Pb2Qqyu5Ojaqysj72nEoz+n+8gxmr9kedkkiUsIFeUXQAbjWzNYCbwOXmtnwAL8vblxQqyJjHmhPlXKluGPILMYv2hR2SSJSggUWBO7+uLtnuHs9oAcw0d3vCOr74k3tymV49/72NK1VkQffnMfgKavDLklESqgi70dgZg+b2XogA1hkZoNPtY8cX6WypRh+bxuuuuAs/jh+Kf/1wRL1NRCRH8xKQo/VzMxMz8rKCruMYis/3/mfj5YyeOoaftS4Os/d1pwypZLCLktEQmZmc90981TbqWdxDEhIMH7brQl/uPZ8Ji7bQo9BM9m6W4+XikjBKAhiSO/29XilVyYrtuzhhoHT+WazHi8VkVNTEMSYy5tUZ9R97Ticl0/3l6Yzebn6YIjIySkIYtCFGRUZ+2AHalVKpc/rcxg+U6N5iMiJKQhiVM20VEb3b0/nRlX57djF/PeHX+uJIhE5LgVBDCtXOonBvVtxd4f6vDptDX2HZbHn4JGwyxKRYkZBEOMSE4wnrmnCn264gC+X53LTi9PJ2b4v7LJEpBhREMSJnm3qMrRPazbt3M91A6cxZ63GKBKRCAVBHOnYqCpjH+xAWmoyt/9jJqOycsIuSUSKAQVBnGmQXo4xD3SgTf0qPDZ6Ef9v3NccyfvBo4SLSAxREMShimWSeb1PK/p0qMeQqWvo8/ocdu7TrGci8UpBEKeSEhP4/TXn8+RNFzJz9TauGzhVE92IxCkFQZy7tVUd3urblj0H87jhxen88+stYZckIkVMQSBk1qvMhw91oEF6WfoOy+K5CSvIV+czkbihIBAAalRMZdR97bixRS3+9s/l3D98LrsP6L6BSDxQEMi/pCQn8vQtzfj9NU2YsGwr1w+cxqrcPWGXJSIBUxDIvzEz+nSoz/B72vDdvsNc98I0Pl2yOeyyRCRACgI5rnZnV+HDhzpydnpZ7ntjLn/5dJkGrROJUQoCOaFaaamMvK8dt2bWZuCkVdz12my27z0UdlkiUsgUBHJSKcmJPNm9KX++8UJmrd7ONc9PZWHOjrDLEpFCFHgQmFmimc03s3HR5fpmNsvMVprZSDMrFXQNcuZua12H0f3bAXDzyzN4c9Y63NVUJBILiuKKYACw9JjlJ4Fn3L0h8B1wTxHUIIWgaUYaHz7UkTYNKvPrMV/x6DsL2X8oL+yyROQMBRoEZpYBdAUGR5cNuBQYHd1kKHB9kDVI4apcthSv92nNgMsaMWb+Bq4fOI3VesRUpEQL+orgWeAx4OjwllWAHe5+dJqs9UCt4+1oZv3MLMvMsnJzNQF7cZKYYPzs8nN4vU9rtu4+wDXPT2Xcoo1hlyUipymwIDCzbsBWd597Ovu7+yB3z3T3zPT09EKuTgpDl3PSGf9wJ849qzw/eXM+T7y/mINH1FQkUtIEeUXQAbjWzNYCbxNpEvo7kGZmSdFtMoANAdYgAasZfcS0b6f6DJuRTfeXZrBum6bCFClJAgsCd3/c3TPcvR7QA5jo7j2BSUD36Ga9gfeDqkGKRnJiAr/p2oRXerUke9teuj43hY+/2hR2WSJSQGH0I/gl8IiZrSRyz2BICDVIAH58/lmMf7gTDaqVo/+IeTzx/mIOHFZTkUhxZyXhWfDMzEzPysoKuwwpoENH8nnqk2UMnrqG82tW4PnbWtAgvVzYZYnEHTOb6+6Zp9pOPYul0JVKSuC33ZowpHcmG3bs55rnpzJm/vqwyxKRE1AQSGAua1ydjwd04vyaFfnZyIU8MmoBew8eOfWOIlKkChQEZjbAzCpYxBAzm2dmVwRdnJR8NSqm8mbfNjx8WSPGzt9At+ensnjDzrDLEpFjFPSK4G533wVcAVQCegH/G1hVElOSEhN45PJzeLNvW/YfyuOGF6cxeMpqTYcpUkwUNAgs+ufVwBvuvuSYdSIF0rZBFT4e0IlLzq3GH8cvpfdrs9m6+0DYZYnEvYIGwVwz+4xIEHxqZuX5v2EjRAqsUtlSvNKrJX+64QLmrN3OVc9OYcLSLWGXJRLXChoE9wC/Alq5+z4gGegTWFUS08yMnm3qMu6hjlSrkMI9Q7P43djFGslUJCQFDYJ2wDfuvsPM7gB+C+iOn5yRhtXKM/bB9vTtVJ83ZmZzzQu6kSwShoIGwUvAPjNrBjwKrAKGBVaVxI3SSYn8pmsTht/Tht0HDnPDi9N46YtVmh9ZpAgVNAiOeKQL8nXAC+4+ECgfXFkSbzo2qsonAzpzeZPqPPnJMm4bNJOc7Rq8TqQoFDQIdpvZ40QeGx1vZglE7hOIFJpKZUsx8PaLePrmZny9aRdX/X0K72TlaEpMkYAVNAhuBQ4S6U+wmcjw0X8JrCqJW2bGTS0z+HhAJ5rUrMAvRi/i/uFz2bbnYNilicSsAgVB9Jf/CKBidMKZA+6uewQSmNqVy/BW37b8+urzmLQslx8/O5nPlmwOuyyRmFTQISZuAWYDNwO3ALPMrPvJ9xI5M4kJRr/OZ/PhQx2pVj6Ffm/M5dFRC9m5/3DYpYnElAINQ21mC4HL3X1rdDkd+NzdmwVcH6BhqCUytPXzE1fw4herqFa+NE/e1JTO52gKU5GTKexhqBOOhkDUth+wr8gZK5WUwKNXnMu7/dtTplQid746m1+P+Yo9Gs1U5IwV9Jf5J2b2qZndZWZ3AeOBj4IrS+T4mtdOY/zDnejbqT5vzV7Hlc9OZvqqb8MuS6REK/AMZWZ2E5EJ6QGmuPuYwKr6HjUNyfFkrd3OL0YvYs23e+nVti6/uuo8ypZOCrsskWKjoE1DmqpSSrT9h/L462ff8Oq0NdRKS+XJm5rSoWHVsMsSKRYK5R6Bme02s13Hee02s12FV67I6UktlcjvujVh1H3tSE5MoOfgWTz+3lfsPqAni0QK6qRB4O7l3b3CcV7l3b3CyfY1sxQzm21mC81siZn9Ibr+0ugMZ4vNbKiZ6VpezlirepX5eEAn+nVuwMg567jimclMWrb11DuKSKBP/hwELo0+YtocuNLM2gNDgR7ufgGQDfQOsAaJIynJifz66sa827895VOS6PP6HB4ZuYDv9h4KuzSRYi2wIPCIPdHF5OgrDzjk7suj6/8J3BRUDRKfWtSpxIcPdeThSxvywcKNXP7Ml3y4cKPGLBI5gUD7AphZopktALYS+aU/G0gys6M3L7oDtU+wbz8zyzKzrNzc3CDLlBhUOimRR644lw8f6kjNtFQeems+fYdlsWnn/rBLEyl2iuSpITNLA8YADxEZvvopoDTwGdDN3ZufbH89NSRn4khePq9NW8vT//yGpIQEHrvyXO5oU5eEBE27LbGtsHsWnxF33wFMAq509xnu3sndWwOTgeUn31vkzCQlJtC3cwM+/WlnmtdO44n3l9D95eks37I77NJEioXAgsDM0qNXAphZKnA5sMzMqkXXlQZ+CbwcVA0ix6pbpSxv3NOap29uxppv99L1uSk8/dk3HDisuZIlvgV5RVADmGRmi4A5wD/dfRzwCzNbCiwCPnT3iQHWIPJvjs538PkjXbimaU2en7iSK5+dzLSVGqZC4pd6Fktcm7riW34z9iuyt+3jhha1+E3XxlQtVzrsskQKRbG6RyBSXHVsVJVPf9qZn1zSkHGLNnLZ01/y9ux15OcX//8giRQWBYHEvZTkRH7+43P56OFOnFu9PL967ytufmUGyzZrFBWJDwoCkahG1csz8r62PNW9Katz99D1uan8z0dL2as5DyTGKQhEjmFm3JJZm4mPXswtmRkMmryaH/3tSz7+apN6JkvMUhCIHEelsqX4841Nebd/eyqVKUX/EfPo/doc1ny7N+zSRAqdgkDkJFrWrcQHP+nA769pwvzs7/jxM5P566ffsP+Q+h5I7FAQiJxCUmICfTrUZ8KjXejatAYvTFrJj/72JZ8s3qzmIokJCgKRAqpWIYVnbm3OyH5tKZ+SxP3D53Lnq7NZlbvn1DuLFGMKApEfqE2DKox7qCP/dU0TFuTs4MpnJ/Pnj5ZqVjQpsRQEIqchKTGBuzrUZ9LPL+aGFrV4ZfJqLn36S96du16d0aTEURCInIGq5UrzVPdmjH2wAzXTUnn0nYXc9PJ0FuTsCLs0kQJTEIgUgua10xjTvz1/6d6UnO37uX7gNH7+zkK27joQdmkip6QgECkkCQnGzZm1mfTzLtzXpQHvL9jAJX/9ghe/WKmhrqVYUxCIFLLyKck8flVjPvtZF9qdXZWnPvmGy59R72QpvhQEIgGpX7Usg3tnMvyeNpRJTqL/iHncOmgmizfsDLs0kX+jIBAJWMdGVRn/cEf+eP0FrNy6h2temMrP31nIFt0/kGJCQSBSBJISE7ijbV2++MXF9OvUgA8WbOTiv3zBs58vZ98hjW4q4VIQiBShCinJPH51Yz5/pAuXnleNZz9fwSV//YJRWTnkqf+BhERBIBKCOlXKMLDnRYy+vx0101J5bPQiuj0/lSkrcsMuTeKQgkAkRJn1KvNe//a8cHsL9hw8TK8hs7nz1dks3aTZ0aToBBYEZpZiZrPNbKGZLTGzP0TXX2Zm88xsgZlNNbOGQdUgUhKYGd2a1uTzR7rw266NWZizg6ufm8LP31nIxh37wy5P4oAF9VyzmRlQ1t33mFkyMBUYAAwDrnP3pWb2ANDa3e862WdlZmZ6VlZWIHWKFDc79x1m4BcreX36Wgy4q0M9Hri4IRVTk8MuTUoYM5vr7pmn2i6wKwKPODo+b3L05dFXhej6isDGoGoQKYkqlknm11c3ZuKjXeh6YQ0GTV5N56cm8Y/Jq9VDWQIR2BUBgJklAnOBhsBAd/+lmXUCxgL7gV1AW3f/jwZRM+sH9AOoU6dOy+zs7MDqFCnOlmzcyZOffMPk5bnUrJjCzy4/hxsvyiAxwcIuTYq5gl4RBBoExxSTBowBHgL+G3jS3WeZ2S+Ac9393pPtr6YhEZi+8lue/GQZC9fvpFG1cvzix+dyeZPqRFphRf5T6E1Dx3L3HcAk4CqgmbvPir41EmhfFDWIlHTtG1Zl7IMdeLHnReS50++Nudz00nRmrNoWdmlSwgX51FB69EoAM0sFLgeWAhXN7JzoZkfXiUgBmBlXX1iDz37amSdvupCNOw5w2z9m0mvILL5arzGM5PQkBfjZNYCh0fsECcAodx9nZn2Bd80sH/gOuDvAGkRiUlJiAre2qsN1zWsxfGY2Ayet5JoXpnLVBWfxyOXn0Kh6+bBLlBKkSO4RnCndIxA5ud0HDjN4yhqGTF3DvkNHuL5FLX562TnUqVIm7NIkRMXqZvGZUhCIFMz2vYd4+ctVDJ2+lrx85+bM2jx0aUNqpqWGXZqEQEEgEse27DrAwEkreWv2Ogzj9jZ1eODis6lWISXs0qQIKQhEhPXf7eP5CSsZPW89yYlGr7Z1ub/L2VQpVzrs0qQIKAhE5F+yt+3l7xNWMHb+BlKSE7mzXT36dW5A5bKlwi5NAqQgEJH/sCp3D89NWMEHCzdSJjmR3u3rcW8nBUKsUhCIyAmt3Lqbv09YybhF/xcIfTs1oJICIaYoCETklJZv2c1zE1Yw/qtNlElOpFe7evTtVF/3EGKEgkBECmz5lt08PzFyhZCSlEivdnXp26kB6eUVCCWZgkBEfrCVW/fwwsTIPYRSSQnc3rou93VpQHU9dloiKQhE5LStzt3Di1+sYsz8DSQmGLdm1ua+Lg3IqKSeyiWJgkBEzti6bft46ctVjJ6bgzvc0KIW/S8+mwbp5cIuTQpAQSAihWbjjv0Mmryat2av43BePldfWIMHL2lI4xoVTr2zhEZBICKFLnf3QQZPXc3wGdnsPZTHZedV44FLGtKybqWwS5PjUBCISGB27DvE0OnZvDZ9DTv2HaZtg8o8cHFDOjWqqhnTihEFgYgEbu/BI7w1ex3/mLKaLbsOckGtCvTv0pArLzhLcyoXAwoCESkyB4/kMXb+Bl7+cjVrvt1LvSpl6Nf5bG68qBYpyYlhlxe3FAQiUuTy8p1Pl2zm5S9XsWj9TtLLl6ZPh3r0bFOXiqnJYZcXdxQEIhIad2fGqm289OUqpqz4lnKlk7i9TR36dKhHjYqaJKeoKAhEpFhYvGEngyavZvxXmzDg2uY16de5AeedpUdPg6YgEJFiJWf7PoZMXcPIOTnsP5xHl3PSua9zA9qdXUVPGgUk9CAwsxRgMlAaSAJGu/vvzWwKUD66WTVgtrtff7LPUhCIxI7v9h5i+Mxshs5Yy7d7DnF+zQr07dSArk1rkJyYEHZ5MaU4BIEBZd19j5klA1OBAe4+85ht3gXed/dhJ/ssBYFI7DlwOPKk0T+mrGZV7l5qVEzhrvb16NG6jm4sF5LQg+B7xZQhEgT93X1WdF0FIBuo6+67Tra/gkAkduXnO18s38o/Jq9hxuptlC2VyC2tatOnfX3qVNEgd2eioEGQFHARicBcoCEw8GgIRF0PTDhVCIhIbEtIMC49rzqXnledxRt28urUNbwxI5uh09dyRZOzuKdTfTLrVtJ9hAAV1RVBGjAGeMjdF0fXfQwMdvd3T7BPP6AfQJ06dVpmZ2cHXqeIFA+bdx5g2Iy1vDl7HTv2HaZpRkXu6Vifqy6oQakk3UcoqGLVNARgZk8A+9z9r2ZWFfgGqOXuB061r5qGROLT/kN5vDtvPa9OW8Pq3L1UK1+aO9vV5fY2dams+ZVPKfQgMLN04LC77zCzVOAz4El3H2dm9wPt3L13QT5LQSAS3/LznS9X5PLatLVMXp5LqaQErm9ekz4d6mso7JMoDvcIagBDo/cJEoBR7j4u+l4P4H8D/G4RiSEJCcYl51bjknOrsWLLbl6fvpb35m1gVNZ62tSvTJ8O9flR42ok6fHT06IOZSJSIu3Yd4iRc3IYNiObDTv2UystlV7t6nJrZm0qqdkIKAZNQ4VJQSAiJ3IkL5/Pl27l9elrmLl6O6WTEri+eS3ubF+X82tWDLu8UCkIRCTuLNu8i2EzshkzbwP7D+eRWbcSd7avx5XnnxWXTxspCEQkbu3cd5h35ubwxsxssrftI718aW5rVZvb29TlrIopYZdXZBQEIhL3jj5tNGz6Wr5YnkuCGVc0qU6vtnXjYrC74vDUkIhIqI592mjdtn2MmJXNyKwcPl68mbPTy9KzTV1uapkR92Mb6YpAROLKgcN5jF+0iTdmZrMgZwcpyQlc26wmd7StS9OMtLDLK1RqGhIROYXFG3YyYlY2Y+dvZP/hPJpmVKRnmzpc06wmZUqV/AYTBYGISAHtOnCYMfM2MGJWNsu37KF86SRuuKgWt7epU6JnUlMQiIj8QO5OVvZ3jJiZzUeLN3PoSD4t61bittZ16Na0BinJiWGX+IMoCEREzsB3ew/x7rz1vDlrHau/3UuFlCRuvCiDHq1rl5irBAWBiEghcHdmrdnOm7PW8cnizRzKy6dFnTRua1WHbs1qFOt7CQoCEZFCtn3vId6bt5635+SwcuseypVO4ppmNenRqjZNMyoWu34JCgIRkYAcvZfw9uwcxn+1kQOH8znvrPL0aFWb61vUIq1M8Rj0TkEgIlIEdh04zPsLNjJqTg5fbdhJqaQErjz/LG5tVZt2DaqQkBDeVYKCQESkiC3ZuJNRc3IYM38Duw4cIaNSKje3rE33zAxqpaUWeT0KAhGRkBw4nMenSzYzKiuHaSu3YQYdG1ale8sMfnz+WUX2GKqCQESkGMjZvo93563nnaz1bNixnwopSVzbvCY3twz+BrOCQESkGMnPd2as3sY70UHvDh7J55zq5ejeMoPrm9eiWoXCHx5bQSAiUkzt3H+Y8Ys2MXpuDvPW7SAxwejcqCrdW9bmssbVCq3pSEEgIlICrMrdw7tz1/PevA1s3nWACimRvgk3XpTBRXXSzqjpKPQgMLMUYDJQmsi8B6Pd/fcWOao/AjcDecBL7v7cyT5LQSAisS4v35m+6ltGz13Pp0s2c+BwPvVpb/6NAAAG00lEQVSrluXlO1py7lnlT+szi8PENAeBS919j5klA1PN7GOgMVAbOM/d882sWoA1iIiUCIkJRqdG6XRqlM7uA4f5ePFmxi/aRO3KwT92GlgQeORSY090MTn6cqA/cLu750e32xpUDSIiJVH5lGRuyazNLZm1i+T7EoL8cDNLNLMFwFbgn+4+CzgbuNXMsszsYzNrFGQNIiJycoEGgbvnuXtzIANobWYXELlncCDabvUP4NXj7Wtm/aJhkZWbmxtkmSIicS3QIDjK3XcAk4ArgfXAe9G3xgBNT7DPIHfPdPfM9PT0oihTRCQuBRYEZpZuZmnRn1OBy4FlwFjgkuhmXYDlQdUgIiKnFuRTQzWAoWaWSCRwRrn7ODObCowws58RuZl8b4A1iIjIKQT51NAioMVx1u8Augb1vSIi8sMUyT0CEREpvhQEIiJxrkSMNWRmuUD2ae5eFfi2EMspKeLxuOPxmCE+j1vHXDB13f2Uj12WiCA4E2aWVZCxNmJNPB53PB4zxOdx65gLl5qGRETinIJARCTOxUMQDAq7gJDE43HH4zFDfB63jrkQxfw9AhERObl4uCIQEZGTiOkgMLMrzewbM1tpZr8Ku54gmFltM5tkZl+b2RIzGxBdX9nM/mlmK6J/Vgq71sIWHeZ8vpmNiy7XN7NZ0fM90sxKhV1jYTOzNDMbbWbLzGypmbWL9XNtZj+L/t1ebGZvmVlKLJ5rM3vVzLaa2eJj1h333FrEc9HjX2RmF53Jd8dsEETHOBoIXAU0AW4zsybhVhWII8Cj7t4EaAs8GD3OXwET3L0RMCG6HGsGAEuPWX4SeMbdGwLfAfeEUlWw/g584u7nAc2IHH/MnmszqwU8DGS6+wVAItCD2DzXrxMZoflYJzq3VwGNoq9+wEtn8sUxGwRAa2Clu69290PA28B1IddU6Nx9k7vPi/68m8gvhlpEjnVodLOhwPXhVBgMM8sgMmbV4OiyAZcCo6ObxOIxVwQ6A0MA3P1QdOyumD7XRMZESzWzJKAMsIkYPNfuPhnY/r3VJzq31wHDPGImkGZmNU73u2M5CGoBOccsr4+ui1lmVo/IQH+zgOruvin61magekhlBeVZ4DEgP7pcBdjh7keiy7F4vusDucBr0SaxwWZWlhg+1+6+AfgrsI5IAOwE5hL75/qoE53bQv39FstBEFfMrBzwLvBTd9917HvR+aNj5vEwM+sGbHX3uWHXUsSSgIuAl9y9BbCX7zUDxeC5rkTkf7/1gZpAWf6z+SQuBHluYzkINgDHzvycEV0Xc8wsmUgIjHD3o7O/bTl6qRj9c2tY9QWgA3Ctma0l0uR3KZG287Ro8wHE5vleD6yPzv0NkaaRi4jtc/0jYI2757r7YSKzG3Yg9s/1USc6t4X6+y2Wg2AO0Cj6dEEpIjeYPgi5pkIXbRsfAix1978d89YHQO/oz72B94u6tqC4++PunuHu9Yic14nu3pPIdKjdo5vF1DEDuPtmIMfMzo2uugz4mhg+10SahNqaWZno3/WjxxzT5/oYJzq3HwB3Rp8eagvsPKYJ6Ydz95h9AVcTmQpzFfCbsOsJ6Bg7ErlcXAQsiL6uJtJmPgFYAXwOVA671oCO/2JgXPTnBsBsYCXwDlA67PoCON7mQFb0fI8FKsX6uQb+QGSa28XAG0DpWDzXwFtE7oMcJnL1d8+Jzi1gRJ6KXAV8ReSpqtP+bvUsFhGJc7HcNCQiIgWgIBARiXMKAhGROKcgEBGJcwoCEZE4pyAQCZiZXXx0hFSR4khBICIS5xQEIlFmdoeZzTazBWb2SnS+gz1m9kx0PPwJZpYe3ba5mc2MjgU/5phx4hua2edmttDM5pnZ2dGPL3fMPAIjor1kRYoFBYEIYGaNgVuBDu7eHMgDehIZ5CzL3c8HvgR+H91lGPBLd29KpGfn0fUjgIHu3gxoT6SnKERGhf0pkbkxGhAZL0ekWEg69SYiceEyoCUwJ/qf9VQiA3zlAyOj2wwH3ovOC5Dm7l9G1w8F3jGz8kAtdx8D4O4HAKKfN9vd10eXFwD1gKnBH5bIqSkIRCIMGOruj//bSrPffW+70x2T5eAxP+ehf3tSjKhpSCRiAtDdzKrBv+aKrUvk38jRUS5vB6a6+07gOzPrFF3fC/jSIzPErTez66OfUdrMyhTpUYicBv2vRARw96/N7LfAZ2aWQGQEyAeJTP7SOvreViL3ESAyJPDL0V/0q4E+0fW9gFfM7L+jn3FzER6GyGnR6KMiJ2Fme9y9XNh1iARJTUMiInFOVwQiInFOVwQiInFOQSAiEucUBCIicU5BICIS5xQEIiJxTkEgIhLn/j+wNMpEJvF5QwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_epochs = 100\n",
    "loss_epoch = [0] * n_epochs\n",
    "\n",
    "for epoch in range(n_epochs): # number of epochs through the dataset\n",
    "    for i in range(100): # size of dataset\n",
    "        optimizer.zero_grad()\n",
    "        y_predicted = torch.dot(X[i], weight) + bias\n",
    "        loss = loss_fn(y_predicted[0], y[i])\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        loss_epoch[epoch] += loss.item()\n",
    "\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.plot(loss_epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UxD65EV2XgFJ"
   },
   "source": [
    "An interesting exercise would be to see how different optimization algorithms fare in terms of rate of convergence.\n",
    "\n",
    "Let us construct a few, and plot their losses.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pL6hkbfHYJce"
   },
   "source": [
    "Now, let's construct a new model that is cloned 4 times, along with these optimizers which take each of the 4 cloned models:\n",
    "\n",
    "- Plain SGD\n",
    "- SGD with momentum\n",
    "- RMSProp\n",
    "- Adam\n",
    "\n",
    "We clone the model so that all optimizations can start from the same starting weights, for equal comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "iMsnXxT4YK9N"
   },
   "outputs": [],
   "source": [
    "weight = torch.randn(2, requires_grad=True)\n",
    "bias = torch.randn(1, requires_grad=True)\n",
    "\n",
    "params_sgd = [weight.data.clone().requires_grad_(), bias.data.clone().requires_grad_()]\n",
    "sgd = optim.SGD(params_sgd, lr=0.01)\n",
    "\n",
    "params_sgdmom = [weight.data.clone().requires_grad_(), bias.data.clone().requires_grad_()]\n",
    "sgdmom = optim.SGD(params_sgdmom, lr=0.01, momentum=0.5)\n",
    "\n",
    "params_rms = [weight.data.clone().requires_grad_(), bias.data.clone().requires_grad_()]\n",
    "rms = optim.RMSprop(params_rms, lr=0.01)\n",
    "\n",
    "params_adam = [weight.data.clone().requires_grad_(), bias.data.clone().requires_grad_()]\n",
    "adam = optim.Adam(params_adam, lr=0.001, betas=(0.9, 0.999))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4HFnHZMFZqU9"
   },
   "source": [
    "Next, let's make a function of the training loop, so that we can re-use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "-zzgL0cJUZVQ"
   },
   "outputs": [],
   "source": [
    "def train(optimizer, weight, bias):\n",
    "  loss_epoch = [0] * n_epochs\n",
    "\n",
    "  for epoch in range(n_epochs): # number of epochs through the dataset\n",
    "    for i in range(100): # size of dataset\n",
    "      optimizer.zero_grad()\n",
    "      y_predicted = torch.dot(X[i], weight) + bias\n",
    "      loss = loss_fn(y_predicted[0], y[i])\n",
    "      loss.backward()\n",
    "      optimizer.step()\n",
    "  \n",
    "      loss_epoch[epoch] += loss.item()\n",
    "  return loss_epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "KBIC42_QY3F5"
   },
   "outputs": [],
   "source": [
    "loss_sgd = train(sgd, params_sgd[0], params_sgd[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "EE9PYPKCaii2"
   },
   "outputs": [],
   "source": [
    "loss_sgdmom = train(sgdmom, params_sgdmom[0], params_sgdmom[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "Fh0o4DnlbBlm"
   },
   "outputs": [],
   "source": [
    "loss_rms = train(rms, params_rms[0], params_rms[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "V5mum3ofbCCK"
   },
   "outputs": [],
   "source": [
    "loss_adam = train(adam, params_adam[0], params_adam[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 378
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 767,
     "status": "ok",
     "timestamp": 1529905129675,
     "user": {
      "displayName": "Soumith Chintala",
      "photoUrl": "//lh5.googleusercontent.com/-ASzCTwFtgW0/AAAAAAAAAAI/AAAAAAAASL4/OWJyxUEEJ64/s50-c-k-no/photo.jpg",
      "userId": "106447253626219410322"
     },
     "user_tz": 240
    },
    "id": "9mupQH6uameb",
    "outputId": "35dd96e7-c1a1-417c-86af-ea25806d0297"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x118bb8950>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd8VGXe///XNb2kF3oNIi0gTaQIossiNsS1Iiquurp776pfXFncvS34u2+9bbuWXevaXdfexYaFIosFEBUBRSCQUNPLTCbTrt8fZyYkIYEkZDJJ5vP0cR6nzCmfAcw75zrnXEdprRFCCCEaMsW7ACGEEB2TBIQQQohGSUAIIYRolASEEEKIRklACCGEaJQEhBBCiEZJQAghhGiUBIQQQohGSUAIIYRolCXeBRyJrKwsPWDAgHiXIYQQncratWuLtNbZh1uvUwfEgAEDWLNmTbzLEEKITkUptaM560kTkxBCiEZJQAghhGiUBIQQQohGdeprEEIkqkAgQEFBAT6fL96liA7M4XDQp08frFZrq7aXgBCiEyooKCA5OZkBAwaglIp3OaID0lpTXFxMQUEBAwcObNU+pIlJiE7I5/ORmZkp4SCapJQiMzPziM4yJSCE6KQkHMThHOm/kYQMiJqtW9n3f/9H2O+PdylCCNFhJWRABAoKKHnmWTz/+U+8SxEi4QwYMICioqJ4lyGaISEDwj1pEqaUFCrffz/epQghRIeVkHcxKZuN5BkzqPzoI8I1NZjs9niXJESn4vF4OO+88ygoKCAUCnHTTTeRnJzMddddh9vtZsqUKWzbto13332X4uJi5s6dy65du5g0aRJa63iXL5opIQMCIOWUWZS//jqeVatIPumkeJcjRKvd+s4PbNxd0ab7HN4rhVvOGNHk5x988AG9evViyZIlAJSXl5Obm8uKFSsYOHAgc+fOPVDfrbdy/PHHc/PNN7NkyRKeeOKJNq1VxE5CNjEBuCdOxJSaSsX7H8S7FCE6nZEjR7J06VIWLVrEypUr2b59Ozk5ObX329cNiBUrVnDRRRcBcNppp5Genh6XmkXLJewZhLJaSf7lDCrfe5+wz4fJ4Yh3SUK0yqF+04+Vo48+mnXr1vHee+9x44038otf/KLdaxCxl7BnEAAps04h7PXi+fzzeJciRKeye/duXC4XF110EQsXLmTVqlVs27aNvLw8AF566aXadadNm8a///1vAN5//31KS0vjUbJohYQ9gwBwHzcBc1oaFe9/QPKMGfEuR4hO4/vvv2fhwoWYTCasVisPP/wwe/bsYdasWbjdbo499tjadW+55Rbmzp3LiBEjmDx5Mv369Ytj5aIlEjogjGamX1K+ZIk0MwnRAieffDInn3xyvWVVVVVs3rwZrTW///3vGT9+PACZmZl89NFH8ShTHKGEbmIC424m7fVStWx5vEsRolP75z//yejRoxkxYgTl5eVcddVV8S5JHKGEPoMAcE2YgDkzk4r33ydl1smH30AI0agFCxawYMGCeJch2lDCn0Eoi4WUWbOoWraMUJUn3uUIIUSHkfABAZBy2qnomhqqPvs03qUIIUSHIQEBOEePxtKzJxVL3ot3KUII0WFIQADKZCLllFOoWrWKUFlZvMsRQogOQQIiIuW0UyEQoGLp0niXIkSX19ouv6+44go2btwIwO233167PC8vj9zc3Darr73dd999eL3eeJdxEAmICMfw4dj696fiPWlmEqKjevzxxxk+fDhQPyA6OwmIDk4pRcppp+L98iuChYXxLkeIDs3j8XDaaadxzDHHkJubW9u1xnvvvcfQoUMZN24c11xzDaeffjoAxcXFzJw5kxEjRnDFFVc02uX3K6+8wnXXXQfA/fffT05ODgDbtm1jypQpAEyfPp01a9Zwww03UF1dzejRo5k3bx4AoVCI3/zmN4wYMYKZM2dSXV190DEuvfRSfve73zFx4kRycnJYtmwZl112GcOGDePSSy+tXe+FF15g5MiR5ObmsmjRotrlSUlJLFy4kBEjRjBjxgy++uorpk+fTk5ODm+//XZtHQsXLuTYY49l1KhRPProowAsW7aM6dOnc8455zB06FDmzZuH1poHHniA3bt3c+KJJ3LiiSfWHifq1Vdfra2tufW3lYR/DqKulNNOo+ihh6l4/wMyLrk43uUI0Tzv3wB7v2/bffYYCafc0eTHjXX37fP5uOqqq1rd5ffUqVO56667AFi5ciWZmZns2rWLlStXMm3atHrr3nHHHfzjH/9g/fr1gNHEtGXLFl544QX++c9/ct555/Haa6/V9iJbV2lpKatXr+btt99m9uzZrFq1iscff5xjjz2W9evX061bNxYtWsTatWtJT09n5syZvPnmm8yZMwePx8NJJ53E3XffzVlnncWNN97I0qVL2bhxI/Pnz2f27Nk88cQTpKam8vXXX1NTU8OUKVOYOXMmAN988w0//PADvXr1YsqUKaxatYprrrmGv/3tb3z22WdkZWUd9q/mcPWPHj36sPtoLjmDqMM+aBD2YcMoX/JuvEsRokNr2N13amoqmzdvPqIuv3v06EFVVRWVlZXk5+dz4YUXsmLFClauXMnUqVMPW9PAgQNrfziOGzeutuPAhs444wyUUowcOZLu3bszcuRITCYTI0aMIC8vj6+//prp06eTnZ2NxWJh3rx5rFixAgCbzcasWbNq/wxOOOEErFYrI0eOrD3eRx99xLPPPsvo0aM57rjjKC4uZsuWLQBMmDCBPn36YDKZGD16dJM1Hsrh6m9LcgbRQOrpp7P/7rvx79iBrX//eJcjxOEd4jf9WGmsu+/Zs2cf8X4nT57MU089xZAhQ5g6dSpPPvkkq1ev5q9//etht7XXeTOk2WxutImp7nomk6neNiaTiWAwiNVqbfIYVqsVpdRB20e3BdBa8/e///2gvqqWLVt2UI3RbRqKHgPA5/O1qP62JGcQDaScdiooRfm7chYhRFMadve9bt06hgwZcsRdfk+dOpV77rmHadOmMWbMGD777DPsdjupqakHrWu1WgkEAm3+3SZMmMDy5cspKioiFArxwgsvcMIJJzR7+5NPPpmHH364traffvoJj+fQvTQkJydTWVlZO9+9e3c2bdpEOBzmjTfeaN0XaQMxCwil1JNKqf1KqQ11lt2tlNqslPpOKfWGUiqtzmd/Vkr9rJT6USkVt06RrD164Dr2WCreeVfenStEE77//nsmTJjA6NGjufXWW7nxxhtxOp089NBDzJo1i3HjxpGcnFz7g/2WW25hxYoVjBgxgtdff73JLr+nTp1Kfn4+06ZNw2w207dvX44//vhG173yyisZNWpU7UXqttKzZ0/uuOMOTjzxRI455hjGjRvHmWee2eztr7jiCoYPH87YsWPJzc3lqquuOuxv9ldeeSWzZs2qvUh9xx13cPrppzN58mR69ux5RN/nSKhY/RBUSk0DqoBntda5kWUzgU+11kGl1J0AWutFSqnhwAvABKAX8DFwtNY6dKhjjB8/Xq9Zs6bNay995RX23nQzA155BefIzntvtei6Nm3axLBhw+JdxkGqqqpISkqq7fJ78ODB0oFfnDX2b0UptVZrPf5w28bsDEJrvQIoabDsI611NEq/APpEps8EXtRa12ittwM/Y4RFTKz4qZBT7l9Jfknj9x2nzJyJslqpkGYmIVpEuvzuWuJ5DeIy4P3IdG8gv85nBZFlMVETDLNpTwWlXn+jn5tTU3GfMI2K995Dhw55EiOEqGPBggWsX7+ejRs38vzzz+NyueJdkjgCcQkIpdR/A0Hg+VZse6VSao1Sak1hKx9oS3EYN29VVDfdLph6+hkECwvxfvllq44hhBCdXbsHhFLqUuB0YJ4+cAFkF9C3zmp9IssOorV+TGs9Xms9Pjs7u1U1pDiN29gqfE3fAZF04nRMycmUv/V2q44hhBCdXbsGhFJqFvAnYLbWuu4FgLeBC5RSdqXUQGAw8FWs6qgNiOqmA8Jkt5My62Qqli4l3AH7SBFCiFiL5W2uLwCrgSFKqQKl1OXAP4BkYKlSar1S6hEArfUPwMvARuAD4PeHu4PpSNQ2MR3iDAIgdfZstNdL5ccfx6oUIYTosGL2JLXWem4jiw/ugOXA+rcBt8WqnrrcNgsmdehrEADOceOw9u5N+Vtvk9oGT4kKIURnkpBPUptMimSH9bBnEMpkImX2GXhWryawb387VSdE56O1JhwOx/QYIbmjsN0lZEAApDgth7wGEZU6ezaEw/JMhBAN5OXlMWTIEC655BJyc3Mxm82H7Qr7hx9+qH0Ce9SoUWzZsoW8vLza7q+HDRvGOeecU/tuhAEDBrBo0SLGjh3LK6+8wvr165k4cSKjRo3irLPOqu2yY/r06Vx77bWMHj2a3NxcvvoqZpcwE0rCdtaX4rBS6Tt8x1b2gQNxHDOK8rffJvPyy9qhMiFa5s6v7mRzyeY23efQjKEsmrDosOtt2bKFZ555hokTJ6KUOmxX2I888gjXXnst8+bNw+/3EwqF2LdvHz/++CNPPPEEU6ZM4bLLLuOhhx7i+uuvByAzM5N169YBMGrUKP7+979zwgkncPPNN3Prrbdy3333AeD1elm/fj0rVqzgsssuY8OGDU3WLZoncc8gmtHEFJU6ezY1P/6Ib3Pb/k8oRGfXv39/Jk6cCDSvK+xJkyZx++23c+edd7Jjxw6cTicAffv2rX0p0EUXXcTnn39ee4zzzz8fMN45UVZWVttx3vz582u74YYD3YtPmzaNiooKyuT98kcscc8gnBbyipp3+2rKqaey7447KX/jTRx/viHGlQnRMs35TT9W3G537XRzusK+8MILOe6441iyZAmnnnoqjz76KDk5OfW6t4b63V3XPcahHGofonXkDKIZLOnpJE+fTvk776Bj0L2wEIli27Zt5OTkcM0113DmmWfy3XffAbBz505Wr14NwL///e9Ge3BNTU0lPT2dlStXAvDcc8/V64Y72r34559/TmpqaqNdhIuWSdyAcFqbdZE6KvVXZxEqKaGqzimtEKJlXn75ZXJzcxk9ejQbNmzgkksuAWDIkCE8+OCDDBs2jNLSUn73u981uv0zzzzDwoULGTVqFOvXr+fmm2+u/czhcDBmzBh++9vfNvpKU9FyMevuuz0cSXff93+8hXs//omfbzsFi/nwOamDQbZMPxHn6GPo+49/tOqYQrSVjtrdd2vk5eVx+umnH9FF5enTp3PPPfcwfvxhe7BOOB2yu++OLsVpXH5pzp1MAMpiIXX2bKqWLSdYXBzL0oQQokNI3IBwHL7DvobSzpoDwSDl77wTq7KESDgDBgw44ltSly1bJmcPMZC4AVHbYV/zX/JtHzwYx8iRlL/+hryOVAjR5SVuQDSzw76G0n51FjU//YRv48ZYlCWEEB1G4gZEM7r8bnS7U09F2WyUv/Z6LMoSQogOI2EDIrmVZxDm1FSSZ86k/N13Cft8sShNCCE6hIQNiNZcg4hKO+ccwhUVVC5d2tZlCdFlPP300/zhD3+IdxniCCRmQISCJIWqsKhQi88gAFwTjsXarx9lr7wag+KEEKJjSMyA2PgmprsHMMJe2OJrEGC8JyLt7LPxfvUV/kgnZEIkmjlz5jBu3DhGjBjBY489BsBTTz3F0UcfzYQJE1i1alXtuu+88w7HHXccY8aMYcaMGezbtw+AxYsXM3/+fKZOnUr//v15/fXX+dOf/sTIkSOZNWsWAenaJq4Ss7M+m9H5V5Y9REUzH5RrKHXOHAofeICy116j2x//2JbVCdEie2+/nZpNbdvTsH3YUHr85S+HXOfJJ58kIyOD6upqjj32WE477TRuueUW1q5dS2pqKieeeCJjxowB4Pjjj+eLL75AKcXjjz/OXXfdxV//+lcAtm7dymeffcbGjRuZNGkSr732GnfddRdnnXUWS5YsYc6cOW363UTzJXRAZFqDlLSiiQnA2r0bSSecQNkbb5J9zTUoq7UtKxSiw3vggQd44403AMjPz+e5555j+vTpZGdnA0Y33T/99BMABQUFnH/++ezZswe/38/AgQNr93PKKafUdgseCoXqdRmeJ2focZWYAWE1AiLDFiCvFRepo9LOOYeqTz+lavlykmfMaKvqhGiRw/2mHwvLli3j448/ZvXq1bhcLqZPn87QoUPZ2MTzQVdffTXXXXcds2fPZtmyZSxevLj2s7rdgjfsMjzaTbiIj8S8BmFzAZBuCbTqInVU0rSpWLp1o/Sll9uqMiE6hfLyctLT03G5XGzevJkvvviC6upqli9fTnFxMYFAgFdeeaXe+r179waMHllF55CgAWGcQaRZAq26SB2lLBbSzjkbz+ef4y/Y1VbVCdHhzZo1i2AwyLBhw7jhhhuYOHEiPXv2ZPHixUyaNIkpU6bU60F08eLFnHvuuYwbN46srKw4Vi5aIjG7+/YUw905vN/3OhbunMiGW09udQ2B3bv5ecYvyfzNb+i24P+1ej9CtERX6u5bxJZ0991SkSamZFMNVTVBgqFwq3dl7dXLuFj92mtov7+tKhRCiLhLzICwOECZSDIZP9Crao7sQlj6BecTKiqi8tNP26I6IYToEBIzIJQCqxuXqgFa191GXe7jj8faqxelL77UFtUJ0SyduXlYtI8j/TeSmAEBYHPh0kZne0dyJxOAMptJO+88vF98Qc327W1RnRCH5HA4KC4ulpAQTdJaU1xcjMPhaPU+EvM5CACbGweRgDiCO5mi0s7+FYX/+AdlL71M9xsWHfH+hDiUPn36UFBQQGFhYbxLER2Yw+GgT58+rd4+cQPC6sYWbpszCABLdjbJv5xB2euvk33tNZicziPepxBNsVqt9Z5GFiIWYtbEpJR6Uim1Xym1oc6yDKXUUqXUlsg4PbJcKaUeUEr9rJT6Tik1NlZ11bK5sIWrgSO/BhGVMW8e4YoKyt99t032J4QQ8RTLaxBPA7MaLLsB+ERrPRj4JDIPcAowODJcCTwcw7oMVheWUCQg2uAMAsA5bhz2IUMoff7f0jYshOj0YhYQWusVQEmDxWcC0efsnwHm1Fn+rDZ8AaQppXrGqjYAbG7MQS9Ktc01CAClFOnzLqRm82aq161rk30KIUS8tPddTN211nsi03uB7pHp3kB+nfUKIssOopS6Uim1Rim15ogu0NncKL+HJLul1V1+Nyb19NMxpaRQ+vzzbbZPIYSIh7jd5qqNNpgWt8NorR/TWo/XWo+PdivcKlYXBLykOKxt1sQEYHK5SPvVr6j4aCmBffvbbL9CCNHe2jsg9kWbjiLj6E/QXUDfOuv1iSyLHZsb/F5SnNY2u0gdlT73AgiFKHtJHpwTQnRe7R0QbwPzI9PzgbfqLL8kcjfTRKC8TlNUbNjcEPCQaje16RkEgK1/f9zTplL68suEpX8mIUQnFcvbXF8AVgNDlFIFSqnLgTuAXyqltgAzIvMA7wHbgJ+BfwL/Fau6almNDvsyHeE2u0hdV8YllxAqKqJiyXttvm8hhGgPMXtQTms9t4mPftHIuhr4faxqaVT0vdS2EOt9re/NtSnuyZOxDz6KkmeeIXXOmbVvyRJCiM4igftiir521B+TMwilFOmXXELN5s14v/yqzfcvhBCxlrgBYY2+djRIZU2QULjtH2xLPeMMzOnplDz7bJvvWwghYi1xA8KWBECa1Th7qGrDZyGiTA4HaRecT9Vnn+HfsaPN9y+EELGUwAFhnEGkmo27jNr6Tqao9LlzwWKh5NnnYrJ/IYSIlcQNCOuB144ClMfgOgSAtVs3Uk89lbLXXydYWhqTYwghRCwkbkBELlInR147GquAAMi47DJ0dTVlL74Ys2MIIURbS/iASLUYwVBUVROzQzmGHI172lRK/vU8YZ8vZscRQoi2lLgB0aCJqbAydgEBkHnZ5YSKiyl/863DryyEEB1A4gZE5AzCoX3YLKaYB4TruAk4cnMpeeopdCgU02MJIURbSNyAMFvBbEMFvGQn2WMeEEopMq+4HP+OHVR+8klMjyWEEG0hcQMCjGYmv4esZDuFMbwGEZX8y19i7duX4ieekDfOCSE6vMQOiEiX3+1xBgGgzGYyL78M37ff4f3yy5gfTwghjoQERMBDdrI9pncx1ZV61llYsrMpeuTRdjmeEEK0VmIHRKSJKTvJRrHHTzDU9r26NmSy28m47DK8X3yB95tvYn48IYRorcQOiGgTU7IdraHE2z4v90k//zzMaWkUy1mEEKIDk4CINDFB7J+FiDK5XGTMv4Sq5cvxbdrULscUQoiWSuyAiDYxtXNAAKTPm4cpKYmiRx9rt2MKIURLJGRA5JXn8cwPz1ButYHfS1aSERBFVe33/mhzSgrp8+ZR+eGH1GzZ0m7HFUKI5krIgNhStoV71tzDXrMZAp7agGjPMwiAjEvnY3I6KXzwoXY9rhBCNEdCBoTbYnSz4TVZwO/BbbfgtpnbPSAs6emkX3wxlR98gO/Hn9r12EIIcTgJGRCuSEd9HrMZQn4IBdvtaeqGMn99KSa3m6IHH2z3YwshxKEkZEC4rdEzCLOxIOAhO8lOUTufQQCY09LImH8JlR99hG/z5nY/vhBCNCUhA6L2DMKkjAWRO5nicQYBkDF/PqbkZDmLEEJ0KM0KCKXUtUqpFGV4Qim1Tik1M9bFxUrtNQgVDQjjYbn2vgYRZU5NJWP+fCqXfkz19xviUoMQQjTU3DOIy7TWFcBMIB24GLgjZlXFWLSJyaMiPapG7mQqrw5QE4zPuxoyLp2POS2NwnvvjcvxhRCioeYGRORXbU4FntNa/1BnWadjNVuxmqx4iPS9VOdhueJ2fBaiLnNSEplXXYXnP//B88UXcalBCCHqam5ArFVKfYQREB8qpZKB2PdsF0NuqxuPjpwtRLr8hvZ/FqKu9AvnYunRg/1/u1feFyGEiLvmBsTlwA3AsVprL2AFfh2zqtqB2+rGq4PGTBz6Y2qMyW4n++o/4PvuOyo//jhudQghBDQ/ICYBP2qty5RSFwE3AuWtPahSaoFS6gel1Aal1AtKKYdSaqBS6kul1M9KqZeUUrbW7r85XFYXnnDAmPF7yYoGRJzuZIpKPfNMbDk5FN53PzoYjGstQojE1tyAeBjwKqWOAf4IbAWebc0BlVK9gWuA8VrrXMAMXADcCdyrtT4KKMU4a4kZt8WNJxS53uCvIivJyKN4PAtRl7JYyP5/1+LfupWy11+Pay1CiMTW3IAIaqNR/EzgH1rrB4HkIziuBXAqpSyAC9gDnAS8Gvn8GWDOEez/sNxWN96Qz5gJeLFbzKQ6rXE/gwDj3dXOceMofODvhKo88S5HCJGgmhsQlUqpP2Pc3rpEKWXCuA7RYlrrXcA9wE6MYCgH1gJlWkcvClAA9G7N/pvLZXXhiQaE3wsQ12ch6lJK0X3RnwgVFVH8xOPxLkcIkaCaGxDnAzUYz0PsBfoAd7fmgEqpdIwzkYFAL8ANzGrB9lcqpdYopdYUFha2pgQgchdTwBN5J0QVAFlJtg4READOUaNIOe00Sp56msDevfEuRwiRgJoVEJFQeB5IVUqdDvi01q26BgHMALZrrQu11gHgdWAKkBZpcgIjgHY1UctjWuvxWuvx2dnZrSwh0sQU8BoBEYieQTgo6gBNTFHZCxZAOEzhvffFuxQhRAJqblcb5wFfAecC5wFfKqXOaeUxdwITlVIupZQCfgFsBD4DovucD7zVyv03i8viwhP0oG2uA01MSR2jiSnK1qc3GZdcTPlbb0kXHEKIdtfcJqb/xngGYr7W+hJgAnBTaw6otf4S42L0OuD7SA2PAYuA65RSPwOZwBOt2X9zua1uwjqMz+Y+0MSUbMPjD+Gp6Ti3l2b+9reYs7LY97//iw536mcThRCdTHMDwqS13l9nvrgF2x5Ea32L1nqo1jpXa32x1rpGa71Naz1Ba32U1vpcrXVMf5Wv7Y/J6jzQxFT76tGOcxZhTkqi23XXUf3tt5S/9Xa8yxFCJJDm/pD/QCn1oVLqUqXUpcAS4L3YlRV7te+EsNnr3cUEHSsgAFLnnInzmGPY/9e/EqqsjHc5QogE0dyL1AsxmoFGRYbHtNaLYllYrNW+E8LiAL/xrEG3ZAcAe8s7VkAok4nuN95IqLiYInl/tRCinVgOv4pBa/0a8FoMa2lXtU1MFisEjIDol2mERl5xx3s4zTkyl7RzzqbkX/8i7ZyzsR91VLxLEkJ0cYc8g1BKVSqlKhoZKpVSFe1VZCzUvjTIYq1tYkqyW8hOtpNX1PECAozbXk1uN3sX3yq9vQohYu6QAaG1TtZapzQyJGutU9qryFioPYMwW2qbmAAGZrrZ3kEDwpKRQbfr/4h3zRrKX38j3uUIIbq4hHwnNdS5BqFMRhNT5DfygVnuDtnEFJV29tk4x41j/113ESwpiXc5QoguLGEDovYMwmQCHYagcWF6QJaboio/lb5APMtrkjKZ6Ln4FkIeD/vvalVvJ0II0SwJGxBOixMAryny5tRIM9PArMiF6iJvXOpqDvvgwWRefjnlb74prycVQsRMwgaExWTBYXbgIXKxN3In04As48xiW1FVvEprlqzf/RZr/37suelmwt6OG2ZCiM4rYQMCIl1+R1+tHbmTaUCmERAd+QwCwORw0Ou22wjk57NfOvMTQsRAQgeE2+rGWxsQxhmEw2qmV6qjQ1+ojnKNH0/6vHmU/utfeNeujXc5QoguRgJCh4wZX1nt8gFZHfdW14a6XbcAa69e7PnvGwn7fPEuRwjRhSR0QLgsdZqYKg+8lGdAB7/VtS6T203P//0f/Hl5FN57b7zLEUJ0IQkdEG6rG0/0LaeVe2qX52S5KfMGKPX441RZy7gnTSL9wgspeeZZPKtXx7scIUQXkfAB4Q36wJFWLyCiF6q3d5KzCIBuC6/HNnAgu//8F0Ll5fEuRwjRBSR8QHgCHkjpBRV1AiIreidT5wkIk9NJr7vuIlhUxN7/+d94lyOE6AISOiBcVpcREMk9651B9MtwYVKdKyDA6PE1679+R8W771L+7pJ4lyOE6OQSOiCMJiYv4aQe9QLCZjHRJ93Ftk4WEABZV16Jc8wY9t5yC/4dO+JdjhCiE0vsgIh0+V2dnA1V+yB04F3Unek0+TXFAAAgAElEQVROprqUxULve+4Gs5ld1/2RsL9zXGgXQnQ8CR0QtT26ujKMDvs8hbWfDcx0kVfk7ZTvXbD27k2v22/D98MPFP71r/EuRwjRSSV0QNT26OpMMxZU7q79bECWm6qaIEVVnfM38OQZM0i/6CJKnnmWyk8+iXc5QohOSAIC8DqSjAV17mQaGLmTqbM8Ud2Ybn9aiGP4cHbf8Ge5HiGEaDEJCMBjjwREvYfljGVbCzt2r66HYrLZ6P3AAyiTiYKrr5FeX4UQLZLQAVF7DcJsAWWuFxB90p2kOq2s31nW1Oadgq1Pb3rdcw81W7aw5+ZbOuU1FSFEfCR0QETvYvKEqiG5R70mJpNJMbZfGut2lsarvDaTNPV4sq+5mop336X0uefiXY4QopNI7ICIXoMIeCMPy+2u9/nYfuls2V9FeXXHfP1oS2RedRVJv/gF++68i6rPV8W7HCFEJyABAZGnqXvU69EVYGz/dAC+6QJnEcpkotedd2IfNIhdCxZQs21bvEsSQnRwCR0QDosDhWq0PyaAY/qmYVKwrpNfh4gyJ7np+/BDKKuV/N/9jlBZ1/heQojYiEtAKKXSlFKvKqU2K6U2KaUmKaUylFJLlVJbIuP0WNdhUiacFueB/phqymvfLAeQZLcwpEcK63Z0/jOIKGvv3vT5x98J7t5DwTXXypPWQogmxesM4n7gA631UOAYYBNwA/CJ1now8ElkPuai/TGR3NNY0KCZaVz/NNbnlxEKd527f1xjx9Lz9tvwfvUVe/78F3Q4HO+ShBAdULsHhFIqFZgGPAGgtfZrrcuAM4FnIqs9A8xpj3oOdPkdCYiKgy9UV9UE2bK/sj3KaTepZ5xB9nXXUbFkCYV/+1u8yxFCdEDxOIMYCBQCTymlvlFKPa6UcgPdtdbRiwB7ge7tUcyBLr97GQsq61+HGNvPaOla24WamaIyf3MFaXMvoPjxJyh57l/xLkcI0cHEIyAswFjgYa31GMBDg+YkbTzN1WibjlLqSqXUGqXUmsLCwsZWaRG31R25zbWHsaBBQPTPdJHptrFuR9e7oKuUoseNNxq3v952G+Vvvx3vkoQQHUg8AqIAKNBafxmZfxUjMPYppXoCRMb7G9tYa/2Y1nq81np8dnb2ERfjtkSamBwpYEs66E4mpRRj+qV3iVtdG6PMZnr/7a+4jjuO3X/+C5UffxzvkoQQHUS7B4TWei+Qr5QaEln0C2Aj8DYwP7JsPvBWe9RT28QEjT4sBzC2fxrbijyUeLrmHT8mu50+Dz6II3cEuxZch+c//4l3SUKIDiBedzFdDTyvlPoOGA3cDtwB/FIptQWYEZmPudq7mKDRh+UAxvXrOg/MNcWc5Kbfo49iGziQ/P/6PZ4vv4p3SUKIOItLQGit10eaiUZpredorUu11sVa619orQdrrWdorUvao5baaxDQ6MNyAKP6pGEzm/j856L2KCluzGlp9HvqSax9epN/1VUSEkIkuIR+khqMJiZfyEcwHIw0Me2BBs8FOG1mph2dxQcb9hLuQs9DNMaSmUn/Z56RkBBCSEBEe3T1Br3GGUQ4ANUHn7ycktuTPeU+vi3oenczNdQwJKpWrIh3SUKIOJCAqNeja+RW14qDL1TPGNYdq1nx/oaDr1F0RdGQsA0cSP7v/0DFBx/EuyQhRDuTgKjXo2vkYbmKXQetl+qyMuWoLN7fsCdhXrpjhMTTOEeOZNd1f6TstdfiXZIQoh0lfEDUvlUu4IHsIYCCPd81uu6puT3JL6nmh90V7VhhfJlTUuj3xOO4J09mz3/fSNEjjyZMQAqR6BI+IDKdmQAUeguNh+W6DYOCrxtd95fDu2M2Kd77/uA7nboyk9NJ34ceJGX2GRTedx97b1mMDgbjXZYQIsYSPiD6JvcFIL8y31jQZ7wREI38lpzutjF5UCbvfZ84zUxRymaj1513knnllZS9/DIFv/8DoSrP4TcUQnRaCR8QKbYU0u3p7KzcaSzocyz4yqB4a6Prn5Lbk7xiL5v3dq3eXZtDKUW36xbQ45abqVq5kh0XXoi/4ODrNUKIriHhAwKgb0rf+gEBTTYzzRzRHZOCt9YffKdTokifO5e+jz1GYM8e8s47D+/atfEuSQgRAxIQGM1M+RWRJqasIWBPaTIgspLszBzegxe+2omnJnHb4ZOOn8KAl17CnJzMjkt/TemLLyZcs5sQXZ0EBNAvuR97PHvwh/xgMkHvsVDQ9BPEV52QQ3l1gJe+zm/HKjsee85ABrz8Eu5JE9m7+Fb23PBnwtXV8S5LCNFGJCAwziA0moKqAmNBnwmw74d676eua0y/dCYMyOCJz7cTCCX26zrNqan0feQRsv7wB8rffpu8uRfiz8uLd1lCiDYgAQH0S+kHQEFlNCCOBR2G3d80uc2V03LYVVadcLe8NkaZTGT/4ff0ffQRgnv2sP1XZ8vLh4ToAiQgMJqYAHZWRC9UjzfGTVyHADhpaDcGZbt5dPk2aXuPSJo2jYFvvoF9+DB2/2kRuxfdILfCCtGJSUAAafY0kq3JB+5kcmVAxiAoWNPkNiaT4sppOWzcU8Gqn4vbqdKOz9qzJ/2ffpqs//ovyt95h+1z5uBd0/SfoxCi45KAwLi/v09ynwMBAUYzUxMPzEXNGdOb7il27v7oR0JdvBvwllAWC9nXXE3/554Fpdhx8SXsu/tuwv6u+UY+IboqCYiIfin9DtzqCkYzU9U+KG/6TiW7xcyfTxnGt/llvPDVzibXS1SucePIefMN0s47j5InnmT7nLPwrmv6uo4QomORgIjol9yP3VW7jRcHwYEH5nZ+ccjtzhzdi8mDMrnrg80UVtbEuMrOx+R20/PWxfT95z8J+6rZMW8ee2+7nbBHrk0I0dFJQET0Te5LUAfZ44ncldRjpNH993cvH3I7pRT/35m5VAdC/N/7m9qh0s4paerx5Lz9DukXXkjpc8+x9bTTqfjwI7nAL0QHJgEREb3VtbaZyWSG0RfC1k+g/ND9DR3VLYmrpg3i9XW7+GKbXLBuijnJTY+bbqT/C//GnJbGrmuvJf+qq/Dv2BHv0oQQjZCAiKi91bXuheoxFxnPQ6z/92G3/8NJR9E3w8kfX/6WUo9cjD0U15gxDHz1Fbr/5c9Ur13H1tPPYN9ddxOqTLwOEIXoyCQgIrKcWTgtzvoBkTEQBkyF9f+C8KGfmHZYzfx97lgKK2u45sVv5K6mw1AWCxmXXMKgD94ndfYZlDz1FFtnnWL06RQIxLs8IQQSELWit7rWu5MJYOwlUJoHOz4/7D5G903jf+aMYOWWIu7+8MfYFNrFWLKz6XXbbQx45RVsAwawd/GtbDtjtlyfEKIDkICoo19yvwMvDooadgbYU2Hdc83ax/nH9uPC4/rxyPKtvPtd4nYJ3lLO3BH0/9dz9HnoQTCb2XXtteSddz5VK1dKUAgRJxIQdUQDIqzrNCdZnTDyHNj0NlSXNWs/t5wxnHH907nupW/5bPP+GFXb9SilSD7pJHLeepOet/0voeJi8n9zJTvmXkjVqlUSFEK0MwmIOvqm9MUf9rPf2+CH+tiLIeiDb5p3FmG3mHly/rEc3SOJq55bKyHRQspiIe3ssxn0wfv0WHwLgT17yL/8CvLOO5/Kjz9GH+Z6kBCibUhA1BG9k2lHRYPbLnuOhkEnwfK7oHJfs/aV6rLy/OUTJSSOgLLZSL/gAgYt/YgeixcTKi2l4A9Xs232bMpee0267hAixiQg6jg6/WgUinX719X/QCk49R7jLGLpTc3eX92QuOLZNTy9ars0k7SCyWYj/YLzGfTB+/S6606U2cKe/76Rn3/xC4oeeYRgSUm8SxSiS5KAqCPdkU5uVi6rdq06+MPMQTD5GvjuJcg7/B1NUakuKy9eOYmThnZj8TsbWfTad9QEQ21YdeJQFgups2cz8M036PvE4ziOHkLhfffz8wnT2b3oBqq//z7eJQrRpcQtIJRSZqXUN0qpdyPzA5VSXyqlflZKvaSUssWjrim9p/B90feU15Qf/OHUP0JqP1hyPYSaf69+kt3CoxeN45qTjuLlNQWc9+gXbCusasOqE4tSiqQpU+j3xOPkLHmXtHPPoWLpUvLOPY/tvzqb0hdfkvdQCNEG4nkGcS1Qt/OiO4F7tdZHAaXA5fEoakqvKYR1mNW7Vx/8oc0Fp9wJhZtgxT0t2q/JpLhu5hAeuWgseUUeTrl/JU98vp2wPFB3ROyDBtHj5psZvHwZ3W+6ER0MsnfxYrZMm8buP/8F79dfS7OeEK0Ul4BQSvUBTgMej8wr4CTg1cgqzwBz4lHbyKyRpNhSWLlrZeMrDD0VRl0Ay++A715p8f5n5fZk6YJpHH9UFv/z7kbOf2w1G3Y1crYiWsScnEzGvHkMfOtNBrz4AimnnkLlhx+y4+JL2HryLAr//g95V7YQLaTi8duVUupV4P+AZOB64FLgi8jZA0qpvsD7WuvcRra9ErgSoF+/fuN2xKCjt4XLF/L13q/59LxPMalGMjRYA/86G/K/hIvfgAHHt/gYWmteW7eL29/bRKnXz7nj+nD9zCF0S3G0wTcQAGGvl4qPPqL8zbfwfvklaI1j5EhSTj2VlFknY+3ZM94lChEXSqm1Wuvxh1uv3c8glFKnA/u11mtbs73W+jGt9Xit9fjs7Ow2rs5wfO/jKfYV81PpT42vYLHD+c9B+kB48ULYv7nFx1BKcc64Pnx2/XR+MzWHN77ZxQl3L+O2JRvZX+E7wm8gAEwuF2lz5tD/6ac4atlndFu4EB0Ksv/OO/n5xJPIu2AuxU8+hT+/6ZdCCZHI2v0MQin1f8DFQBBwACnAG8DJQA+tdVApNQlYrLU++VD7Gj9+vF4Tg/cdF3oLOemVk7h27LVcMfKKplcs2wmPz4CQH87/V6vOJKLyijzc/8kW3lq/C4vZxPnj+3LplAEMyk5q9T5F4/x5eVR88CEVH3xAzWYj3O1Dh5J80kkknXgijhHDUSa5wU90Xc09g4hLE1PtwZWaDlyvtT5dKfUK8JrW+kWl1CPAd1rrhw61fawCAuDcd87FbXXz9KynD71iyTb49/lQsh3OuM/oIvwI5BV5eGT5Vl5bV0AgpJk6OIuLJ/bnxKHdsJrlh1Zb8+fnU7n0Yyo//pjq9eshHMbSrRvuaVNJOn4q7imTMScnx7tMIdpUZwyIHOBFIAP4BrhIa33Id3jGMiDuX3c/T214ipUXrCTZdpgfENVl8MqlsO0zmHAVzLgFbO4jOn5hZQ0vfrWT57/cyd4KH1lJNmYf05tfje3NiF4pGNf1RVsKlpZStXw5VcuW41m1inBlJZjNOI85BvfkybgnT8Y5Mhdltca7VCGOSKcIiCMVy4BYs3cNv/7w19w17S5OGXjK4TcIBeCjm+DLhyF9AJzxAOSccMR1BENhPt28nze+2cUnm/bjD4UZkOliVm5PTsntwcjeqZhMEhZtTQeDVH/7LVUrVuL5z3/wbdgAWmNyuXCOH4f7uONwTZiAY9gwlMUS73KFaBEJiCMUDAc5880zcVgcvHLGK43fzdSYvFXw9tVQshWOmQsn/gXS+rVJTWVeP+99v5f3N+xh9dZigmFNdrKd6Udnc+LQbkwZlEWqS367jYVQWRmeL77E+9WXeL74Ev+2bQAolwvX6NE4x43FNWYMjlHHYE46srNHIWJNAqINvLP1Hf7y+V+454R7OHnAIa+X1xeohuV3wuqHjFeWjrsUpl0PyT3arLYyr59PNu3nsx/3s+KnQip8QZSC3F6pTD4qk4kDMxnbP51UpwRGLAT276d67Vq8X6/Bu2YNNVu2gNZgMmEfPBjnqFE4jxmFY9Qo7IMGoczmeJcsRC0JiDYQCof41du/QqN5Y/YbmE0t/J+8fBesuDvSTbiC3F/BcVdB73FtWmcwFGbdzjL+s7WI//xczDf5pQRCGqVgSPdkxvZPZ3TfNMb0TWNQdpI0ScVAqLKS6m+/o3rdOqq//Zbq778nXFEBgHI6cQwfjjN3BPZhw3AMH449J0eapkTcSEC0kY/yPuKPy//I7cffzhmDzmjdTkq2wZePwjfPg78Seo0xmp9yzwZ3VtsWDHj9Qdbnl7Emr5Sv80pYv7OMypogAG6bmeG9UhjRK5XhvVIY1iOFwd2TcFjlN9y2pMNh/Hk78H3/HdUbfsC3YQO+TZvQPuMZF2WzYT/qKOxDh+IYOgT70UdjP/poLBkZca5cJAIJiDYS1mHOf/d8PAEPb815C6vpCJpsfBXw7QvGGcXe70GZjfdMDD0NhpwKyd3brvA6wmHNtiIP6/PL2LCrnA27yvlhdwXVAaNXWbNJMSDTxeBuyRzdPYlB3ZIYlJ1ETrYbl01+y20rOhTCv307vo0b8W3cRM1PP+Lb/COhOt2VmzMzjeAYNAjbUYOw5+Rgy8nBkp0td66JNiMB0YaW5S/j6k+vZsG4BVyWe1nb7HTfRqPr8B/egLJIdyG9xxmBkXMi9DkWLLHr0DYU1uwo9rB5byWb9lTw075KtuyrIq/YQ93+A3ulOhiQ5TaGTBf9Mtz0y3DRL9NFkl3C40hprQkVFVGzZQu+n36iZssW/D9vpWbrVsJVB3r8NSUlYRsw4MDQvz+2/v2w9u2LOS1NwkO0iAREG9Jas2DZAj7d+Sl/m/43ZvSf0ZY7h/0bYfN7sOVD2LXWuLBtdRkh0W8S9JtohIcjpe2O24SaYIjtRR62FXrYVljFtkIP24s95BV5KPXW7+I8zWWlb7qLPulOeqUZQ+80Jz1THfRMc5Dltsv1jlbSWhPcvx//tm3UbN2Gf9s2/Hl5+PPyCOzZY/y7iTAlJ2Pr2xdr377Y+vbB2rs31j59sPbqhbVXL0xOZxy/ieiIJCDamC/o4/KPLufHkh95fObjjO42OjYHqi6DvJWwfQXsXA17NwAaUJB1NPQeCz2PgR4jjcGRGps6GlHuDZBf6mVHsZedJV4KSr0UlFZTUOplV1k1vkD9d0VbzYpuyQ56pDrokeKgW4qdbskOuqfYyU42prOT7aQ5rRIkLRD2+QgUFODfuRP/jp0E8vPxF+QT2JlPYNcudKB+kJvT07H26oWlZw+sPXpi7dkDS/ceWHt0x9KjB5bsbEx2e5y+jYgHCYgYKPGVcPF7F1PuL+fpk5/mqPSjYn9QXzkUfA271hlnF7vWgqfwwOepfSF7KHQbBtlDjBDJGgzO9NjXVofWmlJvgN1l1ewuq2ZvhY/dZT72VfjYW26M91fWUBW5WF6XxaTIcNvISrKTlWwny20jM8lGhttOpttGhttGRpKNDJeNdJeNFKdFmlSaoMNhgoWFBAoKCOzeTWDXbmO8dw/BPXsI7N5D2HPwy5TMaWlYunUzhuzsOkMWlsxMzFlZWLKyMCUlyZ99FyABESP5Fflc9P5FVAeruX789Zx79Lnt/z9M5T7Y+50x7N8M+zdB0Y9Gp4FRzgzjNakZOcaT3Wn9I+N+kNwTzPG5flBVE2R/hY/CyhoKq2rYX1FDUVV08FNcVUOxx09RVc1BZyRRZpMizWkl1WUl3WUj3WUl1WkjzWUl1WmtHac4I2OHlRSnhRSHVe7WAkJVVQT37iWwdx/BfXsJ7t9PYN8+gvsLCRYWEty/n2BREYQOfjWustkwZ2ZiycjAnJmBJT0Dc0YG5ox0LOnpxnRaOua0VMxpaZhTU+UZkA5IAiKG9nn2cdOqm1i9ZzVTe0/l5kk308Pddg/BtUooaFzsLtoCRT8ZT3KXbIPibVCxC6OZKkKZIaU3pPaBlF6RobfxIF9KL0jqbgzW+L6bwusPUlzlp8Tjp8Trp9RjTJd6/ZR5A5R5A7XT5dXGtNd/6Pd92ywmUhxGWCQ7LCTXji0k2etOW0hyWHDbjWm3LTK2m3HbLdgtpi79m7QOhwmVlREsLCJYWEiouIhgUTHB4mJCJSUES4oJFZcQLC0hVFJae/tuY0wpKZhTU+sNptQUzCmpmFNSMKUkR6aTMSWnYE5OwpScjDkpCWWLy5uHuzwJiBgL6zAvbH6Be9feS0iHOHXgqcwfMZ+j04+OSz2HFKyB8gIo3Q5l+cZ0eb7xIF9FAVTsrn/2EeVINYLC3Q2SssEdGVyZxvMbrkxjcGYYTVoxvOuqufzBMBU+IzwqfEZwVFQHqPAFjXFkutJnjKt8ASp9QSp9Qapqgo02gTXGYlK4bEZYuO0W3DYzLpsFl82My27BZTXjspuNeZsFp9WYdtrMkWkLTpsJh/XA506rGYfNhM3c+cIn7PUSKi0lWFZGqKSUUFnZgaG83BjKyghVlBMui8xXVjZ6llKXcjgwJSVhTjJCw+R2Y0pyY3YnRaYj4+jgcjUYO41plwvlcHS6P9dYkYBoJwWVBTy78Vne/PlNqoPVjM4ezYn9TmR63+kMTBnYOf5BhsNQXQqVu6FyL1Tugap9ULU/Mi4Ez37j2ofvEK9HtSUbQeFMNcaONHCmGUHjiI5TwZ5i3JFlT44MkWlz/LsFCYc1Vf4gnpogVb4glTXGtKfGCBFPTRCPP4SnJog3Mvb4jWlvTQhPZFuvP0S135hv6WvHlQKHxQgTh8WEw2bGYTHjsBqBYgwmHBYzdqsJe52xo8HYbjFht5iwWequFx3M2CxGINki61hMqt3+zWqtCXu8hCvKCVVWEa6sIFRRQbiyklBFJeGqSmN5VdWBaY8nMm9MhzweCDYv1FEK5YwEhtOJyeFAuZyYnC5j2umoP+1wYnI6UHYHymE3ljscxthmx+SwG/N2Y6zsdmPabu/wzWoSEO2svKacV396lQ/zPmRTySYAerh7MDJrJKOyRjE8czg5aTlkOjI7R2g0JegHbzF4iyLjEmNcXWoM3hLwlR2Y91UY88FmvCXP4gBbEtiTjLCxJxndptvcYI2Mba7ItMu4FdjmNsZWF1idRrOY1WXsy+oy5i1OI3zi8OeutcYfCkfCIoQvYASH1x+iOjJdHQjiC4Qj0yFqApHPAiFjeWSZLxDGFwjhCxrTNdFxIIQvGMYfbPyaTUsohREo0dCoEx5W84GxPTpvNmG1mLCalTEdWcdSZ94YVCSAjGmr2Vgn+pmxPDJtNoKqdh2TCatF1W5rjn5mUpgUEAgYweH1GmOP98C010vYa4x1dTVhbzXh6mrC1Q3mfdVobzXhmhq010u4uhrtb+SsurksFkw2mxEWdjvKZsNkt6Gsttp5ZbcZy22R5bY6g9VaZ95qzFutxnpWK8pmxT5wIPbBg1v59ywBETd7PXtZlr+MtfvW8n3R9+yq2lX7WbI1mX4p/eiV1Ise7h70cPUgy5lFljOLDEcGaY40Um2pWDvAb9NtKuCDmgojMGrKI+NKY1lNJdRUGcv9HmPaHx2i8x4IeMDvhdAhXxPSOGUyQsPiMILEYjeCw2KvMzjAbIusZwNzZLnZFlkeGZvtRuDUfmY9sE502mSNTFsj05b6y02WA5+ZzG0SXuGwEUY1gTC+YAh/sE6IRKZrgsbn/lC49nN/JFz8wbrLjelAnWV1P4+OgyFNIGSsHwgZgz8YJhA2lrfHjxeLSWGJhIwxNqaNIDkQOGZTg+kG82alMEe2N5sUFjS2cAhHyI81FMAeCmAL+bEGjHlLKIA1GMAS9GMN+jEHjGXmYABzwI85FMAcCGAK+DEFA5gCAUxBP6ZAABU0lqtgABUwBgIBVMAPwcj0YZrfkn99GX0WLWzVn5kERAdSVF3ET6U/sb18O9vLt7OzYid7vXvZU7UHX6jx36ydFifJ1mSSbEkk2ZJwW9y4rC5cFhdOixOHxYHD4sButtcONrMNq8mKzWzDZrJhNVuxmCxYlAWLyYLVZMyblRmzyYxFWTCbzJiUCYvJgkmZMCtz7VgpVX9M+zU/HFIoCAGvMfg9Ru+5gWojQAK+yGfVEKw25mvHPmN5qMa4LhOoNsbByGfBGuNaTNBnnCmFIkOwpnWh1BImSyQsLEaYRKdNFiNAaqejn0cGZa7zeWSsTAfmVfQzU/11o9PKVGe96Diybt3PaqdV/W3rfWaqN2hlIowipBUBrQiFIYQiEFIEUQS1IhCGkFaEwppAODoPQa0IhhVBDYGwMQTDEAgb+wloYz6oFYGQJqSNz4KR/QTD2givyH4C4XDt9sY6mmBY4Q9HgjWsCIbCBDXG/sMQ1jqynzChsCYUNuajy2P9o9Okw1jCISzhINZwCGs4GJk3ls2ZNpz/N3dKq/bd3ICQvhLaQfQMYXKvyfWWa62p8FdQXF1Msa+Y4upiymvKKaspo9xfTpW/iqpAFZX+SrxBL0W+IrwBL9XBanxBH76Qj7A+8maFllAoTMq4iFo7HQmOQ47rhEvD5SjqLa+dpn4Y1Q2nup81Z3nD74AJsEcGFOCMDIfYVmtAG+PoNIAOG0fVB+brrU+d9TUNljVcjzr7bbgsCDoI6Mi3jNYCBKM/rRrus05dzZ2PEU0H+OUCjL9uc2SIsNCMH4ZN/nJ0mOWqqT/Zpv88dDPWMVnHAa0LiOaSgIgjpRSp9lRS7ankkNPi7bXWBHWQmmANvpCPQCiAP+zHH/ITCAeMIRQgqIMEw8YQCodq50M6RCgcqjcO6zAhHTL6CIrMh3WYMMZYa127TEf/08YQxvgcQKNr19eRf+7R6eg20fXqftZwuuH3rbvNQdO68eX1d8Jh12ly2ya091l4S+tr2c6bEV711mkYZo0sqzOtUE0HYt3jNrH9gWmaMR/dZ8NjNJivHTU89qG24xDrNlLD4ZY3VkP9f6wHrZ+T1ffgWtqYBEQnppTCqqxYbVaSSIp3OUKILqaZ79EUQgiRaCQghBBCNEoCQgghRKMkIIQQQjRKAkIIIUSjJCCEEEI0SgJCCCFEoyQghBBCNKpT98WklCoEdrRy8yygqA3L6SwS8Xsn4neGxPzeifidoeXfu7/WOvtwK3XqgDgSSqk1zemsqqtJxO+diN8ZEvN7J+J3hth9b2liEkII0SgJCCGEEI1K5IB4LMQwzdEAAAWhSURBVN4FxEkifu9E/M6QmN87Eb8zxOh7J+w1CCGEEIeWyGcQQgghDiEhA0IpNUsp9aNS6mel1A3xricWlFJ9lVKfKaU2KqV+UEpdG1meoZRaqpTaEhmnx7vWWFBKmZVS3yil3o3MD1RKfRn5O39JKWWLd41tSSmVppR6VSm1WSm1SSk1KRH+rpVSCyL/vjcopV5QSjm64t+1UupJpdR+pdSGOssa/ftVhgci3/87pdTY1h434QJCKWUGHgROAYYDc5VSw+NbVUwEgT9qrYcDE4HfR77nDcAnWuvBwCeR+a7oWmBTnfk7gXu11kcBpcDlcakqdu4HPtBaDwWOwfjuXfrvWinVG7gGGK+1zsV4iegFdM2/66eBWQ2WNfX3ewowODJcCTzc2oMmXEAAE4CftdbbtNZ+4EXgzDjX1Oa01nu01usi05UYPzB6Y3zXZyKrPQPMiU+FsaOU6gOcBjwemVfAScCrkVW61PdWSqUC04AnALTWfq11Gf9/e3cTYlUdxnH8+yszfAmsKCmt1IqIoLQgJAskW4VUCytIS4R2bVxEYRRR0C6qRVFCLyhJ9KblMrKwXPiaRmC7ihzxDSrDojD9tfj/p27TmZkaZuYO5/4+m5nzv4dzz+GZO885z7nn+fdArCmzYk6RNAmYChyihbG2/Rnww4DhweJ7J7DexXZghqSLRvK+vZggZgEHOpb76lhrSZoDLAB2ADNtH6ovHQZmdmm3xtILwCPA6bp8PvCT7T/qcttiPhc4BrxRy2qvSppGy2Nt+yDwLPA9JTEcB/bQ7lh3Giy+o/Y/rhcTRE+RNB14H1ht++fO1+zOmeDbQdJS4KjtPd3el3E0CbgeeNn2AuAXBpSTWhrrcylny3OBi4Fp/LsM0xPGKr69mCAOApd0LM+uY60j6SxKcthge2MdPtJ/uVl/Hu3W/o2RRcAdkr6jlA9vpdTnZ9QyBLQv5n1An+0ddfk9SsJoe6xvA761fcz2SWAjJf5tjnWnweI7av/jejFB7AKurN90mEy5qbW5y/s06mrd/TXga9vPdby0GVhZf18JfDje+zaWbK+xPdv2HEpsP7G9HPgUWFZXa9Vx2z4MHJB0VR1aAuyn5bGmlJYWSppa/977j7u1sR5gsPhuBh6o32ZaCBzvKEX9Lz35oJyk2yl16jOB120/0+VdGnWSbgY+B77i71r8Y5T7EO8Al1I64d5je+DNr1aQtBh42PZSSfMoVxTnAXuBFbZ/7+b+jSZJ8yk35ScD3wCrKCeArY61pKeAeynf2tsLPEipt7cq1pLeAhZTurYeAZ4EPqAhvjVZvkgpt/0KrLK9e0Tv24sJIiIihteLJaaIiPgPkiAiIqJREkRERDRKgoiIiEZJEBER0SgJIqJLJC3u7zYbMRElQURERKMkiIhhSFohaaekfZLW1rkmTkh6vs5FsEXSBXXd+ZK21z78mzp69F8h6WNJX0r6QtLldfPTO+Zx2FAfcoqYEJIgIoYg6WrKk7qLbM8HTgHLKY3hdtu+BthKebIVYD3wqO1rKU+x949vAF6yfR1wE6X7KJQuu6spc5PMo/QSipgQJg2/SkRPWwLcAOyqJ/dTKE3RTgNv13XeBDbWeRlm2N5ax9cB70o6B5hlexOA7d8A6vZ22u6ry/uAOcC2sT+siOElQUQMTcA622v+MSg9MWC9kfas6ewRdIp8JmMCSYkpYmhbgGWSLoS/5gG+jPLZ6e8Yeh+wzfZx4EdJt9Tx+4GtdUa/Pkl31W2cLWnquB5FxAjkbCViCLb3S3oc+EjSGcBJ4CHKpDw31teOUu5TQGm7/EpNAP1dVaEki7WSnq7buHscDyNiRNLNNWIEJJ2wPb3b+xExllJiioiIRrmCiIiIRrmCiIiIRkkQERHRKAkiIiIaJUFERESjJIiIiGiUBBEREY3+BEe/psMVRXXUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "legend_sgd, = plt.plot(loss_sgd, label='sgd')\n",
    "legend_sgdmom, = plt.plot(loss_sgdmom, label='sgd with momentum')\n",
    "legend_rms, = plt.plot(loss_rms, label='rmsprop')\n",
    "legend_adam, = plt.plot(loss_adam, label='adam')\n",
    "plt.legend(handles=[legend_sgd, legend_sgdmom, legend_rms, legend_adam])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7KfGaiSkcUg2"
   },
   "source": [
    "## Exercise1: play with the hyperparameters of the optimizers and make training convergence faster\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "VrjtEv7Wc7bN"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Yg86x9loeDs_"
   },
   "source": [
    "## Exercise2: train on the MNIST dataset\n",
    "\n",
    "The MNIST dataset is available via the following code snippet. Build a small logistic regression model that trains on it using different optimization algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "ezE7x5U-c9RU"
   },
   "outputs": [],
   "source": [
    "from torchvision import datasets, transforms\n",
    "mnist_train = datasets.MNIST('../data', train=True, download=True,\n",
    "                       transform=transforms.Compose([\n",
    "                           transforms.ToTensor(),\n",
    "                           transforms.Normalize((0.1307,), (0.3081,))\n",
    "                       ]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "R8LbsqlFeqmx"
   },
   "outputs": [],
   "source": [
    "print(\"Size of MNIST training set: \", len(mnist_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "Uxn19ivve0vn"
   },
   "outputs": [],
   "source": [
    "# Example of getting a particular sample:\n",
    "data, label = mnist_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "tmdXYzzAfEGj"
   },
   "outputs": [],
   "source": [
    "print(data.shape)\n",
    "print(label.item()) # label is a scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "0rlPXIcpfG6_"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "optimization_tutorial.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}