Lesson 03 - TF_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to TensorFlow\n",
    "\n",
    "https://www.youtube.com/watch?v=q5iL3XYFv2M\n",
    "\n",
    "## Backpropagation on zero hidden layer classification case\n",
    "Suppose we are required to learn the function that maps $x$ (the inputs) to $y$ (the outputs). In this particular instance we restrict ourselves to the case that $y=\\sigma(Wx)$. The maths behind regression is shown below:\n",
    "\\begin{align}\n",
    "z_i =& Wx_i\\\\\n",
    "a_i =& \\sigma(z_i)\n",
    "\\end{align}\n",
    "where $\\sigma(z_i) = 1/(1+exp(-z_i))$ is the sigmoid function. $a_i$ is commonly known as the activation. **The loss function** is $$\\mathcal{L} = \\frac{1}{2N}\\sum_i (y_i-a_i)^2$$. \n",
    "\n",
    "**We need to adjust the $W$ to minimise the loss function**. We use the chain rule:\n",
    "\\begin{align}\n",
    "\\frac{d \\mathcal{L}}{dW} =& \\sum_i \\frac{d\\mathcal{L}}{da_i}\\frac{da_i}{dz_i}\\frac{dz_i}{dW}\\\\\n",
    "=& \\frac{1}{N}\\sum_i -(y_i-a_i)\\cdot\\frac{\\exp(-z)}{(1+\\exp(-z))^2}\\cdot x_i\\\\\n",
    "=& \\frac{1}{N}\\sum_i -(y_i-a_i)\\cdot a_i(1-a_i)\\cdot x_i\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "np.random.seed(42)\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1111b02b0>]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgUAAAFkCAYAAACw3EhvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XeYlPW5//H3LRqNeESxgEaOPYgmMS7xGI16khh7YuWo\nq8ZeUNQE24mxK9bYo8QutqxdL0wwKMYW63FXxfxEk9gbKIIYFZXy/f3xHRSQhZ1hd5+Z2ffruuaC\nfXaemQ9zLTv33M+3REoJSZKkBYoOIEmSqoNFgSRJAiwKJElSiUWBJEkCLAokSVKJRYEkSQIsCiRJ\nUolFgSRJAiwKJElSiUWBJEkCKigKImKjiBgeEW9HxPSI2KYN5/w4Ipoj4rOI+EdE7FlZXEmS1FEq\n6RR0B54FDgbmuXFCRKwE/Am4H1gbuBC4MiI2reC5JUlSB4n52RApIqYD26WUhs/lPmcBW6aUvjfT\nsSagR0ppq4qfXJIktavOGFPwQ2DUbMdGAut3wnNLkqQ2WrATnqM3MG62Y+OAxSNi4ZTS57OfEBFL\nAZsDrwGfdXhCSZLqxyLASsDIlNIH5ZzYGUVBJTYHbiw6hCRJNWw34I/lnNAZRcFYoNdsx3oBH82p\nS1DyGsANN9xAv379OjBafRk8eDDnn39+0TFqjq9bmcaMYfDuu3P+DTeA/z/L4s9a+XzNyjdmzBh2\n3313KL2XlqMzioLHgS1nO7ZZ6XhrPgPo168fDQ0NHZWr7vTo0cPXqwK+buXrATT06we+bmXxZ618\nHfGaTZ0K48fD++/DBx/k2/jxMGFCvk2cmP/88MNZb+uuCyNHtmuUjlb25feyi4KI6A6sBkTp0CoR\nsTYwIaX0ZkScASyfUpqxFsGlwKDSLISrgU2AAYAzDyRJ7SKl/Eb+zjuz3saNg7Fj85/jxuVCYMKE\nr5+/wAKw5JLQs+dXf/bqBX37whJL5Nvqq3f+v6uzVdIp+AHwAHmNggScWzp+LbAPeWBhnxl3Tim9\nFhFbA+cDhwFvAfumlGafkSBJ0hxNnQpvvAGvvZZvr7+eb2++mW9vvQWTJ896zlJLwXLL5Tf3FVaA\n/v1h2WVhmWW+ui29dL7fEkvkwqCrK7soSCk9xFymMqaU9p7DsYeB/uU+lySp65g2Lb/h/+MfX90e\nfxxWXTUXANOmfXXf5ZaDFVeE//xPWGed/KY/47bcctC7Nyy8cGH/lJpVrbMPVIHGxsaiI9QkX7fy\n+YpVxp+1bPr0/OY/ejQ8/zy88EK+vfQSfF4afr7IIrldv9JKjfz4x7kwWGUVWHll6NMnf1/tz6Kg\njvgLpzK+buXzFatMV/xZmzIlv+E3N0NLS749/zx8/HH+fs+esNZasP76sO++eUJL3775E39u53e9\n16xIFgWSpHbz1lu55f/EE/nW0gKffQYR+c2+oQG23x6+9z347ndzqz9i3o+rzmFRIEmqSEr5uv9D\nD8Ejj+Tb66/n7620Eqy3HgwYkKfyrb02/Md/FBpXbWBRIElqs3ffhfvug/vvz7e334Zu3fJgvx12\ngI02gg02yCP+VXssCiRJrZo2LV8GGDEC7rkHnnkmH//+92GXXWCTTWDDDe0C1AuLAknSLCZPhlGj\n4M474e6782p/Sy8Nm28ORx4Jm26a5/ir/lgUSJL4/HP4y1/gpptyIfDJJ3lg4H77wbbb5nEB3boV\nnVIdzaJAkrqo6dPh4Yfhuuvgjjtg0qQ8I+CYY/L4APe76nosCiSpi3n9dbj2Whg2DF59NS8KdOih\neYzAWmsVnU5FsiiQpC5g2rR8eWDo0DxgcNFFYaedcmGw0UauFaDMokCS6tjEiXDFFfCHP+SlhRsa\n8tc77wyLLVZ0OlUbiwJJqkOvvQYXXABXXpl3GNxlF7j55jxg0K6AWmNRIEl15O9/hyFD4NZb83bA\nhx8Ogwa5mJDaxqJAkurAc8/BqafC7bfnJYZ//3vYa688dkBqqwWKDiBJqtyLL8KOO+YVBp95Bq66\nKu9HcPDBFgQqn0WBJNWgt9+G/ffPUwibm+Gaa3KBsM8+sNBCRadTrfLygSTVkI8/hjPOgPPOg+7d\n858DB8LCCxedTPXAokCSakBK0NQERx0FEybAEUfA0UfD4osXnUz1xMsHklTlRo+GjTeG3XaD9deH\nMWPyDAMLArU3iwJJqlKTJ+d9CBoacndg1Ci47bY8u0DqCF4+kKQqdP/9cOCB8OabcNJJ+VLBN75R\ndCrVOzsFklRFPv44FwM/+xl861v50sFxx1kQqHPYKZCkKvHoo7DHHjB2bN6r4IADYAE/uqkT+eMm\nSQX74os8dmDjjaF377w64cCBFgTqfHYKJKlAr76aNyt65pk8o+Doo6Fbt6JTqauyKJCkgtx5J+y9\nN/TsmS8drLtu0YnU1dmckqRO9sUX8KtfwQ47wCabQEuLBYGqg50CSepE48bBgAHw5JN5J8NBgyCi\n6FRSZlEgSZ2kuRm22w6mTIEHH4QNNig6kTQrLx9IUif44x9hww1hueXg6actCFSdLAokqQOlBCef\nnPct2GknePhhWGGFolNJc+blA0nqIF98kVcnHDYMTjstr0Xg+AFVM4sCSeoAkybBjjvmzsCNN8Ku\nuxadSJo3iwJJamfvvgubb543M7r3Xvjxj4tOJLWNRYEktaPXXsubGU2enBckWnPNohNJbedAQ0lq\nJy++mGcYAPztbxYEqj0WBZLUDlpaYKONYMkl4ZFHYOWVi04klc+iQJLmU3NzXq54lVXgoYfyWgRS\nLXJMgSTNh5YW2HRTWGMNGDkSFl+86ERS5ewUSFKFnn02DypcfXX4y18sCFT7LAokqQLPPZcvGay6\nau4Q9OhRdCJp/lkUSFKZ/vlP2GwzWGmlvA7BEksUnUhqHxYFklSGt9/OYwh69swdgiWXLDqR1H4s\nCiSpjT74IHcIpk/PHYKlly46kdS+nH0gSW3w8cew9dbw3nt5YaI+fYpOJLU/iwJJmoepU/O2xy+8\nAA88AH37Fp1I6hgWBZI0FynBoYfCfffBPfdA//5FJ5I6jkWBJM3FuefCpZfCVVflNQmkeuZAQ0lq\nxW23wVFHwbHHwj77FJ1G6ngWBZI0B08+Cb/8Jey6K5x6atFppM5RUVEQEYMi4tWImBwRT0TEuvO4\n/24R8WxEfBIR70TEVRHRs7LIktSx3nkHtt8eGhrg6qshouhEUucouyiIiJ2Bc4ETgXWA54CRETHH\nGbsR8SPgWuAKYE1gAPBfwOUVZpakDvPZZ7kg6NYN7rgDFl646ERS56mkUzAYuCyldF1K6UVgIPAp\n0NoVtx8Cr6aULkkpvZ5Segy4jFwYSFLVSAkGDoTRo+HOO6FXr6ITSZ2rrKIgIhYC+gP3zziWUkrA\nKGD9Vk57HOgTEVuWHqMX8D/AnysJLEkd5cIL4dpr4cor4Qc/KDqN1PnK7RQsDXQDxs12fBzQe04n\nlDoDuwM3R8QXwLvAROCQMp9bkjrMAw/AEUfAkUfCbrsVnUYqRoevUxARawIXAicB9wLLAeeQLyHs\nN7dzBw8eTI/Z9iNtbGyksbGxQ7JK6preeQd22QV+/GM488yi00ht19TURFNT0yzHJk2aVPHjRe7+\nt/HO+fLBp8COKaXhMx0fBvRIKW0/h3OuAxZJKe0007EfAY8Ay6WUZu86EBENQHNzczMNDQ1l/HMk\ndbiWlrysX3NzHp5f46ZOhZ/+FF5+GZ55BpZdtuhE0vxpaWmhf156s39KqaWcc8u6fJBSmgI0A5vM\nOBYRUfr6sVZOWxSYOtux6UACnOgjqVDHHguPPQY332xBIFUy++A8YP+I2CMi1gAuJb/xDwOIiDMi\n4tqZ7n83sGNEDIyIlUtdgguBJ1NKY+cvviRVbvhwOPtsOOss2HDDotNIxSt7TEFK6ZbSmgSnAL2A\nZ4HNU0rvl+7SG+gz0/2vjYjFgEHksQQfkmcv/GY+s0tSxV5/HfbcE7bbDg4/vOg0UnWoaKBhSmko\nMLSV7+09h2OXAJdU8lyS1N6mTs3LF/foAddc44qF0gzukiipyzn11Ly3wcMPwxJLFJ1Gqh5uiCSp\nS3nkERgyBE48ETbYoOg0UnWxKJDUZUycmBcm2nBD+O1vi04jVR+LAkldQkqw//7w73/DDTfkDY8k\nzcoxBZK6hOuvh9tvh1tvhT595n1/qSuyUyCp7r3xBhx6KPzylzBgQNFppOplUSCprk2fDvvsA4sv\nDhddVHQaqbp5+UBSXRs6FO6/H+691+mH0rzYKZBUt/7xDzj6aDj4YNh006LTSNXPokBSXZo2LS9j\n/K1v5f0NJM2blw8k1aWLLsqrFj7yCHTvXnQaqTbYKZBUd15+OW+JfOih8KMfFZ1Gqh0WBZLqyoxF\ninr1gtNOKzqNVFu8fCCprlx5JTzwQJ5tsNhiRaeRaoudAkl14+234cgj87oEzjaQymdRIKkupJSn\nHi66KJxzTtFppNrk5QNJdeGuu2D4cLjtNlhyyaLTSLXJToGkmvfvf+eZBj//OeywQ9FppNplUSCp\n5p1wAkycCBdfDBFFp5Fql5cPJNW0lpa8UNGZZ8KKKxadRqptdgok1axp0+DAA2GtteDXvy46jVT7\n7BRIqllDh0JzMzz6KCy0UNFppNpnp0BSTRo3Do47Lq9euP76RaeR6oNFgaSa9JvfwIILwumnF51E\nqh9ePpBUcx5/HIYNg0svhaWWKjqNVD/sFEiqKdOmwSGHQEMD7Ldf0Wmk+mKnQFJNueKKPA3x8ceh\nW7ei00j1xU6BpJoxfjz89rew997wwx8WnUaqPxYFkmrG8cfD9Ol5oSJJ7c/LB5JqwujRcPnlcO65\nsOyyRaeR6pOdAklVL6W8YuHqq8OgQUWnkeqXnQJJVe+uu+CBB+DPf3blQqkj2SmQVNU+/xyOPBK2\n2AK22qroNFJ9s1MgqapdcAG8/jrcfXfRSaT6Z6dAUtUaOxZOOw0OPhjWXLPoNFL9syiQVLVOOCHv\nb3DSSUUnkboGLx9Iqkp//ztcdVWegtizZ9FppK7BToGkqnT00bDyyvnSgaTOYadAUtW57z645x64\n7Tb4xjeKTiN1HXYKJFWVadPyFMQNNoAddig6jdS12CmQVFWuvz4vafz44xBRdBqpa7FTIKlqfPop\nHHss7LSTuyBKRbAokFQ1LrwQ3n8fTj+96CRS12RRIKkqfPBB3hL5oINg1VWLTiN1TRYFkqrC6afn\n3RCPO67oJFLXZVEgqXCvvw4XXwxHHQXLLFN0GqnrsiiQVLgTToAll4TBg4tOInVtTkmUVKjRo/M0\nxEsugcUWKzqN1LXZKZBUqGOOyQML99uv6CSS7BRIKszDD8OIEXDzzbDQQkWnkWSnQFIhUoLf/hYa\nGmDAgKLTSIIKi4KIGBQRr0bE5Ih4IiLWncf9vxERp0XEaxHxWUS8EhF7VZRYUl0YMQIefTRPRVzA\njydSVSj78kFE7AycCxwAPAUMBkZGxLdTSuNbOe1WYBlgb+BlYDnsUkhd1vTpeTnjjTeGzTYrOo2k\nGSoZUzAYuCyldB1ARAwEtgb2Ac6e/c4RsQWwEbBKSunD0uE3KosrqR7ccgs891zuFLjpkVQ9yvq0\nHhELAf2B+2ccSyklYBSwfiun/QJ4GvjfiHgrIl6KiN9FxCIVZpZUw6ZMgeOPh5//PG+PLKl6lNsp\nWBroBoyb7fg4oG8r56xC7hR8BmxXeow/AD2Bfct8fkk1btgw+Ne/4Lbbik4iaXadMSVxAWA6sGtK\n6WOAiDgcuDUiDk4pfd7aiYMHD6ZHjx6zHGtsbKSxsbEj80rqIJ99BiefDI2NsPbaRaeRal9TUxNN\nTU2zHJs0aVLFj1duUTAemAb0mu14L2BsK+e8C7w9oyAoGQMEsAJ54OEcnX/++TQ0NJQZUVK1uuwy\nGDs2FwaS5t+cPii3tLTQv3//ih6vrDEFKaUpQDOwyYxjERGlrx9r5bRHgeUjYtGZjvUldw/eKiut\npJr1ySd5+uFee8HqqxedRtKcVDIt8Dxg/4jYIyLWAC4FFgWGAUTEGRFx7Uz3/yPwAXBNRPSLiI3J\nsxSumtulA0n15fe/h4kT8yBDSdWp7DEFKaVbImJp4BTyZYNngc1TSu+X7tIb6DPT/T+JiE2B3wP/\nRy4Qbgb81SB1EZMmwdlnwwEHwIorFp1GUmsqGmiYUhoKDG3le3vP4dg/gM0reS5Jte/882Hy5Lys\nsaTq5aqCkjrUBx/AeefBoEGw/PJFp5E0NxYFkjrU736XlzX+3/8tOomkebEokNRhxo3LAwx/9StY\nZpmi00iaF4sCSR3m7LNhwQXhyCOLTiKpLSwKJHWId9+FoUPh8MNhySWLTiOpLSwKJHWIM8+ERRaB\nX/+66CSS2sqiQFK7e+utvKTxEUfAbNuXSKpiFgWS2t0ZZ0D37nDYYUUnkVQOiwJJ7eqNN+CKK+Co\no2DxxYtOI6kcFgWS2tVpp+VLBoccUnQSSeWyKJDUbl57Da6+Go4+GhZbrOg0ksplUSCp3Zx2GvTs\nCQcfXHQSSZWwKJDULl59FYYNy12C7t2LTiOpEhYFktrFkCGw1FJw0EFFJ5FUqYq2Tpakmb3yClx7\nbd78aNFFi04jqVJ2CiTNtyFD8oZHAwcWnUTS/LBTIGm+/OtfcN11cM458M1vFp1G0vywUyBpvgwZ\nAssuCwceWHQSSfPLToGkiv3rX3DDDXDuuXYJpHpgp0BSxWZ0CQ44oOgkktqDnQJJFXnzzdwlOO88\nuwRSvbBTIKkiV16ZuwT77190EkntxU6BpIqMGAHHXGCXQKondgokVaRnT7sEUr2p7k7BmDFFJ5A0\nm7EPjqE3cPhWY1jkhaLTSPqa+XjvjJRSOyZpHxHRADQ3Aw1Fh5EkqYa0AP3zX/unlFrKObe6OwU3\n3AD9+hWdQlLJG2/AMduP4UZ29/+nVK3GjIHdd6/o1OouCvr1gwZ7BVK1OP5CGL80MB7/f0p1yIGG\nktrkn//MzYG99y46iaSOYlEgqU2GDIFevWD77YtOIqmjVPflA0lVYUaX4IILYOGFi04jqaPYKZA0\nTzO6BK5LINU3OwWS5mrmLsEiixSdRlJHslMgaa5OPRV697ZLIHUFdgokteqll+DGG+HCC+0SSF2B\nnQJJrTr1VFhuOdhvv6KTSOoMdgokzdGLL0JTE1x0kV0CqauwUyBpjk49FZZf3i6B1JXYKZD0NTO6\nBJdc4roEUldip0DS15xyCqywAuyzT9FJJHUmOwWSZjFmDNx0EwwdapdA6mrsFEiaxcknQ58+bnwk\ndUV2CiR96fnn4ZZb4PLL7RJIXZGdAklfOvlkWGkl2HPPopNIKoKdAkkAPPss3H47XHMNLLRQ0Wkk\nFcFOgSQATjwRVlsNdt+96CSSimKnQBJPPw3Dh8P118OC/laQuiw7BZI48UTo2xcaG4tOIqlIfiaQ\nurgnnoARI+CPf4Ru3YpOI6lIdgqkLu744+E734Gddy46iaSi2SmQurAHH4RRo+COO2ABPyJIXV5F\nvwYiYlBEvBoRkyPiiYhYt43n/SgipkRESyXPK6n9pJS7BA0NsN12RaeRVA3KLgoiYmfgXOBEYB3g\nOWBkRCw9j/N6ANcCoyrIKamd3Xsv/O1vMGQIRBSdRlI1qKRTMBi4LKV0XUrpRWAg8Ckwr/3ULgVu\nBJ6o4DkltaOU4LjjYIMNYIstik4jqVqUVRRExEJAf+D+GcdSSon86X/9uZy3N7AycHJlMSW1p+HD\n89oEdgkkzazcgYZLA92AcbMdHwf0ndMJEbE6cDqwYUppevgbSCrU9Olwwgnw05/CT35SdBpJ1aRD\nZx9ExALkSwYnppRennG4recPHjyYHj16zHKssbGRRldYkSp2880wejQ89ljRSSTNr6amJpqammY5\nNmnSpIofL3L3v413zpcPPgV2TCkNn+n4MKBHSmn72e7fA5gITOWrYmCB0t+nApullB6cw/M0AM3N\nzc00NDSU8++RNBdTpkC/frDmmvkSQkVaWqB/f2huzlMXJFWVlpYW+vfvD9A/pVTWbL+yOgUppSkR\n0QxsAgwHiHw9YBPgojmc8hHwndmODQJ+AuwIvFbO80uaP1dfDa+8ktclkKTZVXL54DxgWKk4eIo8\nG2FRYBhARJwBLJ9S2rM0CPGFmU+OiPeAz1JKY+YnuKTyTJ4Mp5yS9zf43veKTiOpGpVdFKSUbimt\nSXAK0At4Ftg8pfR+6S69gT7tF1FSe7jkEnjvPTjZOUCSWlHRQMOU0lBgaCvf23se556MUxOlTjVp\nEpxxBuy7L6y2WtFpJFUrVzuXuoDzzoNPP83LGktSaywKpDo3blwuCg45BL71raLTSKpmFgVSnRsy\nBLp1g2OOKTqJpGpnUSDVsZdfhksvzQVBz55Fp5FU7SwKpDp23HHQqxccdljRSSTVgg5d5lhScZqb\n4aab4Mor4ZvfLDqNpFpgp0CqU7/5TV7SeM89i04iqVbYKZDq0H33wahRcNddsKD/yyW1kZ0Cqc5M\nnw5HHw0bbADbbFN0Gkm1xM8QUp254QZ49ll49FGINm9ULkl2CqS68umncOyxMGBA7hRIUjksCqQ6\ncsEFeQXDM84oOomkWmRRINWJGcXAoEFueiSpMhYFUp04+eQ80+C444pOIqlWOdBQqgNjxsDll8NZ\nZ8FSSxWdRlKtslMg1YGjjoI+ffJOiJJUKTsFUo0bORL+/Ge45RZYeOGi00iqZXYKpBo2dSoMHgwb\nbZSnIUrS/LBTINWwSy+FF1/Mmx+5UJGk+WWnQKpREybAiSfCPvvAOusUnUZSPbAokGrUSSfBlClw\n2mlFJ5FUL7x8INWgMWNg6FA4/XTo1avoNJLqhZ0CqcakBIcdBiuuCL/6VdFpJNUTOwVSjbnjDhg1\nCu6+2ymIktqXnQKphnzySZ6C+POf55sktSeLAqmGnHEGvPde3g1RktqbRYFUI/75T/jd7+Doo2HV\nVYtOI6keWRRINSClPKhwueXgN78pOo2keuVAQ6kGDB8O99yTBxkuumjRaSTVKzsFUpX7+GM49FDY\nckvYbrui00iqZxYFUpU78UQYPx4uucT9DSR1LC8fSFXs2WfhwgthyBBYeeWi00iqd3YKpCo1bRoc\neCCssQYccUTRaSR1BXYKpCp12WXw1FPwt7/BQgsVnUZSV2CnQKpC774LxxwD++8PP/pR0WkkdRUW\nBVKVSQkOPhgWWQTOPLPoNJK6Ei8fSFXmttvgrrvg1luhZ8+i00jqSuwUSFXkgw/gkENghx1gwICi\n00jqaiwKpCoyeDB88QVcfHHRSSR1RV4+kKrEiBFw/fVwzTV5jwNJ6mx2CqQq8NFHMHAgbLYZ7Lln\n0WkkdVUWBVIVGDwYJk7MaxO4lLGkonj5QCrY3XfD1VfDlVfCSisVnUZSV2anQCrQ+PF5gaKtt4Z9\n9ik6jaSuzqJAKkhKcNBBMGUKXHGFlw0kFc/LB1JBmpryQkU33+xsA0nVwU6BVIA334RBg2CXXWCn\nnYpOI0mZRYHUyaZNg913h8UWg0suKTqNJH3FywdSJzvjjLwd8gMPuLeBpOpip0DqRI8/DiedBMce\nCxtvXHQaSZqVRYHUSSZNgl13hfXWgxNOKDqNJH1dRUVBRAyKiFcjYnJEPBER687lvttHxL0R8V5E\nTIqIxyJis8ojS7UnpbyM8cSJcOONsKAX7iRVobKLgojYGTgXOBFYB3gOGBkRS7dyysbAvcCWQAPw\nAHB3RKxdUWKpBl16Kdx0E1x+uasWSqpelXQKBgOXpZSuSym9CAwEPgXmuB5bSmlwSumclFJzSunl\nlNKxwD+BX1ScWqohTz8Nv/41HHKI0w8lVbeyioKIWAjoD9w/41hKKQGjgPXb+BgB/AcwoZznlmrR\nhAkwYAB8//twzjlFp5GkuSv3yubSQDdg3GzHxwF92/gYRwHdgVvKfG6ppkyfnrdB/ve/4aGHYOGF\ni04kSXPXqcOdImJX4Hhgm5TS+Hndf/DgwfTo0WOWY42NjTQ2NnZQQqn9nHUW/OlPMGIErLhi0Wkk\n1aOmpiaamppmOTZp0qSKH6/comA8MA3oNdvxXsDYuZ0YEbsAlwMDUkoPtOXJzj//fBoaGsqMKBVv\nxIi8FsFxx8GWWxadRlK9mtMH5ZaWFvr371/R45U1piClNAVoBjaZcaw0RmAT4LHWzouIRuAqYJeU\n0l8qSirViJdeyusR/PzncPLJRaeRpLar5PLBecCwiGgGniLPRlgUGAYQEWcAy6eU9ix9vWvpe4cB\n/xcRM7oMk1NKH81XeqnKTJoE224Lyy8PN9wAC7g8mKQaUnZRkFK6pbQmwSnkywbPApunlN4v3aU3\n0GemU/YnD068pHSb4VpamcYo1aJp02C33WDcOHjqKVh88aITSVJ5KhpomFIaCgxt5Xt7z/b1Typ5\nDqnWHHss3HNPHk+w+upFp5Gk8rnYqtQOrrgizzY491zYfPOi00hSZbziKc2nkSPhoIPg4INh8OCi\n00hS5SwKpPkwejT8z//AFlvAhRdCRNGJJKlyFgVShd55B7beGlZbLW925M6HkmqdRYFUgQkTvho7\n8Kc/wWKLFZtHktqDn22kMn3ySV6Y6N134ZFH8poEklQPLAqkMnzxBey4Izz/PPz1r9CvX9GJJKn9\nWBRIbTRtGuyxBzzwQF6LYN11i04kSe3LokBqg+nT4cAD4dZb822TTeZ9jiTVGosCaR5mFARXXw3X\nXgs77FB0IknqGM4+kOZi+vS8MNFVV8E118Avf1l0IknqOBYFUitSgkMOyUsYX3017Lln0YkkqWN5\n+UCag2nT8iWDq67Kt732KjqRJHU8iwJpNl98kS8T3H47XHedlwwkdR0WBdJMPv0UBgyA+++H226D\n7bYrOpEkdR6LAqnkww9h223h6afhz3+Gn/2s6ESS1LksCiTg9ddhq63y0sWjRsH66xedSJI6n7MP\n1OW1tMAPfwiTJ8Pjj1sQSOq6LArUpY0YARtvDH365IKgb9+iE0lScSwK1CWlBOecA7/4RR478MAD\n0KtX0akkqVgWBepyJk/O0wyPOirfbr8duncvOpUkFc+BhupS3ngDtt8exoyBm26CnXcuOpEkVQ+L\nAnUZI0diZ/2IAAALOElEQVTC7rvnrsBjj8H3v190IkmqLl4+UN2bOhWOOQa22AJ+8IO8DoEFgSR9\nnZ0C1bW33oLGxjyz4Mwz8xiCBSyFJWmOLApUt26+OW973L07PPQQ/OhHRSeSpOrmZybVnQkTYNdd\nYZdd8nTDZ5+1IJCktrBToLryl7/AvvvmjY1uvDFfOogoOpUk1QY7BaoL778Pu+0GW24J3/kOPP98\n7hZYEEhS29kpUE1LCa67Dg4/PH997bV5YSKLAUkqn50C1azRo+EnP4G99sodghdfhD32sCCQpEpZ\nFKjmTJgAhxwC66wDY8fmcQQ33ADLLFN0MkmqbV4+UM34/HO47DI4+WSYMgV+97tcHHzjG0Unk6T6\nYKdAVW/69DyToF8/GDw4713wj3/kcQQWBJLUfiwKVLVSguHDoX//vGfB976XZxVceSX07l10Okmq\nPxYFqjrTp8Mdd+QxA9tuCz16wKOPwl13wZprFp1OkuqXRYGqxhdf5OmFa68NO+4ISy0FDz6Ybxts\nUHQ6Sap/FgUq3Icfwllnwcorw557Qp8+8MgjcP/98N//XXQ6Seo6nH2gwoweDUOH5umEU6bkcQOH\nHw5rrVV0MknqmiwK1KkmT87jBf7whzxOYLnl4IgjYODA/HdJUnEsCtThUoInn4RrroGbboKPPsor\nEd52G2yzDSy0UNEJJUlgUaAO9OKLuQhoasrrCvTpA4cdlscNrLZa0ekkSbOzKFC7GjMG7rwTbrkF\nnnsOFl88LzZ08cXw059Ct25FJ5QktcaiQPNl2rR8aWD48LyOwEsvQffusNVWcOKJeaOiRRYpOqUk\nqS0sClS2d9+F++6De+6BkSNh4sS8psA22+T9CH72M/jmN4tOKUkql0WB5mnChK/WDbj/fnjhhXz8\nBz/IGxJtuSX81395aUCSap1FgWaRErz2GjzxRC4EHnkE/v73/L2VV4ZNNoHjj8/jA5ZdttCokqR2\nZlHQxb3/PrS0QHNzHhvwxBPw3nv5e337wkYbwVFH5T9XXrnYrJKkjmVR0EVMmwYvv5xXERw9Os8M\neOYZePPN/P3FF8+XAA44ANZbL9+WWabYzJKkzmVRUGc+/RT+9a+8LsCLL+br/y+8kGcFfPZZvk/v\n3vDd70JjY96WuKEBVlkFFnAnDEnq0iwKakxKub3/+uvw6qvwyiu5A5C7AE1MmND45X179sxbDa+3\nHuy9d/772ms7FmB2TU1NNDY2zvuO+lIT4CtWPn/Wyudr1rkqKgoiYhBwJNAbeA44NKX0f3O5/4+B\nc4G1gDeA01JK11by3PVs2rT8hv/OO/n27ru5vf/WW/n2xhv5NuMTP8CSS8Kqq+ZP+j17NnHeeY2s\nvjp8+9t5mmBEcf+eWuEvnfJZFFTGn7Xy+Zp1rrKLgojYmfwGfwDwFDAYGBkR304pjZ/D/VcC/gQM\nBXYFfgZcGRHvpJTuqzx69Zs6Nc/h/+ADGD/+qz/ffz+/+c/4c+zYfBs/HqZP/+r8BRaA5ZeHFVbI\nSwR/97uw4oqw0kr5zxVXzEXBDNtsk5cQliSpEpV0CgYDl6WUrgOIiIHA1sA+wNlzuP9BwCsppaNL\nX78UERuWHqemioJx4+Dhh/OGPh99BJMmfXX78MOvbhMm5GLgo4/m/Dg9e+ZBfMssA716weqr5z97\n9cpFwPLL5x0Dl13Wuf+SpM5TVlEQEQsB/YHTZxxLKaWIGAWs38ppPwRGzXZsJHB+Oc9dDf7f/4Od\ndsp/X2yxPGJ/8cXzp/UllsgD+NZYI3+95JL5zX/JJWHppXMrf8ZtQUdySJKqULlvT0sD3YBxsx0f\nB/Rt5Zzerdx/8YhYOKX0+RzOWQRgzJgxZcbrWN27w4MPwqKLlv8J/rPP4O23862jTJo0iZaWlo57\ngjrl61amMWOYBLRU2f/PWuDPWvl8zco303tn2TvPREqp7XeOWA54G1g/pfTkTMfPAjZOKX2tWxAR\nLwFXp5TOmunYluRxBovOqSiIiF2BG8v5h0iSpFnsllL6YzknlNspGA9MA3rNdrwXMLaVc8a2cv+P\nWukSQL68sBvwGvBZK/eRJElftwiwEvm9tCxlFQUppSkR0QxsAgwHiIgofX1RK6c9Dmw527HNSsdb\ne54PgLKqG0mS9KXHKjmpkjXszgP2j4g9ImIN4FJgUWAYQEScEREzr0FwKbBKRJwVEX0j4mBgQOlx\nJElSlSh7HHxK6ZaIWBo4hXwZ4Flg85TS+6W79Ab6zHT/1yJia/Jsg8OAt4B9U0qzz0iQJEkFKmug\noSRJql9ugSNJkgCLAkmSVFITRUFEbB0RT0TEpxExISLuKDpTLYiIb0TEsxExPSK+V3SeahYRK0bE\nlRHxSunn7J8RcVJpFU/NJCIGRcSrETG59P9y3aIzVauIOCYinoqIjyJiXETcGRHfLjpXLYmI35R+\nhzk4fR4iYvmIuD4ixpd+jz0XEQ3lPEbVFwURsSNwHXAV8F1gA5yu2FZnkwd2OnBk3tYAAtgfWJO8\nN8dA4LQiQ1WbmTZEOxFYh7xL6sjS4GN93UbA74H1yJvBLQTcGxHfLDRVjSgVnAeQf840FxGxBPAo\n8DmwOdAPOAKYWNbjVPNAw4joRl7A6PiU0rBi09SW0qqR5wA7Ai8A308pjS42VW2JiCOBgSml1YrO\nUi0i4gngyZTSr0pfB/AmcFFKaU4bomkmpeLpPfIKsH8rOk81i4jFgGbypnrHA8+klA4vNlX1iogz\nyasN//f8PE61dwoagOUBIqIlIt6JiBERsVbBuapaRPQCLgd2ByYXHKeWLQFMKDpEtZhpQ7T7ZxxL\n+VPF3DZE06yWIHfu/Lmat0uAu1NKfy06SI34BfB0RNxSulTVEhH7lfsg1V4UrEJu6Z5IXhdha3Ir\n5MFSq0Rzdg0wNKX0TNFBalVErAYcQl58S9ncNkTr3flxakupq3IB8LeU0gtF56lmEbEL8H3gmKKz\n1JBVyF2Vl8irBv8BuCgiflnOgxRSFJRWPZw+l9u00mCcGfmGpJTuKr3J7U2utP+niOxFaetrFhGH\nAYsBMzagigJjF66Mn7WZz/kWcA9wc0rp6mKSqw4NJY9X2aXoINUsIlYgF0+7pZSmFJ2nhiwANKeU\njk8pPZdSugK4gjw2qs3KXtGwnZxD/jQ7N69QunQAfLkPZErpi4h4BfjPDspWrdrymr0K/ITcyv08\nfzD50tMRcWNKae8Oylet2vqzBuTRu8BfyZ/mDuzIYDWokg3RBETExcBWwEYppXeLzlPl+gPLAC3x\n1S+xbsDGEXEIsHCq5sFwxXmXmd4rS8YAO5TzIIUUBaUNjz6Y1/1Kmy99DvSltLlD6brmSsDrHRix\n6pTxmh0KHDvToeXJO2XtBDzVMemqV1tfN/iyQ/BX4P+AfToyVy2qcEO0Lq9UEGwL/HdK6Y2i89SA\nUeSZZjMbRn6DO9OCoFWPkt8rZ9aXMt8ri+oUtElK6d8RcSlwckS8Rf7HHU2+fHBroeGqVErprZm/\njohPyJcQXkkpvVNMqupX6hA8SO62HA0sO+NDSkpp9mvoXdl5wLBScfAUeermlxuiaVYRMRRoBLYB\nPikNAgaYlFJyW/g5SCl9Qp4x9aXS77EPUkqzfxLWV84HHo2IY4BbyNNg9yNPs26zqi4KSo4EppDX\nKvgm8CTw05TSpEJT1RYr63nblDxQZxXyFDvIxVQity5FmzZE06wGkn+GHpzt+N7k32lqG3+HzUNK\n6emI2B44kzyF81XgVymlm8p5nKpep0CSJHWeap+SKEmSOolFgSRJAiwKJElSiUWBJEkCLAokSVKJ\nRYEkSQIsCiRJUolFgSRJAiwKJElSiUWBJEkCLAokSVLJ/wfpKl3Mcu8JEQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10f14b0b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def sigmoid(z):\n",
    "    return 1/(1+np.exp(-z))\n",
    "\n",
    "z = np.arange(-5,5,0.1)\n",
    "\n",
    "plt.plot(z,sigmoid(z))\n",
    "plt.plot([0,0],[0,1],'r')\n",
    "plt.plot([-6,6],[0.5,0.5],'r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "N = 1000\n",
    "D = 5\n",
    "\n",
    "X = 5*np.random.randn(N,D)\n",
    "w = np.random.randn(D,1)\n",
    "y = X.dot(w)\n",
    "\n",
    "y[y<=0] = 0 \n",
    "y[y>0] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_X = X[1:100]\n",
    "test_X = X[100:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000, 5)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(99, 5)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(900, 5)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def dL_dw(X,e,a):\n",
    "    return -X.T.dot(e*a*(1-a))/len(X)\n",
    "\n",
    "def gradient_descent(gamma=5e-1, n_epochs=1000, batch_size=100, decay=0.9):\n",
    "    epoch_run = int(len(X)/batch_size)\n",
    "    \n",
    "    # get starting conditions\n",
    "    w = np.random.randn(D,1)\n",
    "    params = []\n",
    "    loss = np.zeros((n_epochs,1))\n",
    "    for i in range(n_epochs):\n",
    "        params.append(w)\n",
    "        \n",
    "        for j in range(epoch_run):\n",
    "            idx = np.random.choice(len(X),batch_size,replace=False)\n",
    "            a = sigmoid(X[idx].dot(w)) # Activation function\n",
    "            e = y[idx] - a # Really important that you use y_obs and not y (you do not have access to true y)\n",
    "            #update parameters\n",
    "            w = w - gamma*dL_dw(X[idx],e,a)\n",
    "        loss[i] = 0.5*e.T.dot(e)/len(e)    \n",
    "        gamma = gamma*decay #decay the learning parameter\n",
    "        \n",
    "    return params, loss\n",
    "        \n",
    "params, loss = gradient_descent()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f06656d3d30>]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhoAAAFkCAYAAABmeZIKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xu8HdP9//HXRy4IlZQgblXaCtG65KBNKZqU1F3V7eCr\npXwpihCUb/3cW5dKKl8NqW9L4nLUNaWoNtrGXeQc0pJES+NOmoSESOR2Pr8/1l6d2fvs+zn7ZG/e\nz8djHnvPmjUza2bPnvnMmjUz5u6IiIiI1MIqK7sAIiIi8smlQENERERqRoGGiIiI1IwCDREREakZ\nBRoiIiJSMwo0REREpGYUaIiIiEjNKNAQERGRmlGgISIiIjWjQENERERqpqpAw8xONrNZZrbYzJ42\nsx2L5B1kZndl8reb2al58qxiZpeY2b/MbJGZvWxmP6mmbCIiIlI/Kg40zOww4GrgAmB7YBrwsJn1\nLzBKH+AV4BzgnQJ5fgycAJwEbAmcDZxtZqdUWj4RERGpH1bpS9XM7GngGXc/LdNvwBvAGHe/ssS4\ns4DR7j4mJ/1+4F13Pz6VdhewyN2PrqiAIiIiUjcqqtEws15AE/BITPMQqUwChnSiHE8Cw8zsS5n5\nbAvsDDzYiWmKiIjIStazwvz9gR7A7Jz02cDATpTjcmAtYKaZrSAEQP/j7rfny2xm6wDDgVeBjzsx\nXxERkU+b1YDPAw+7+7xaz6zSQKMQAyq7BpPtMOAI4HBgOrAdcI2Zve3uN+fJPxy4tRPzExER+bQ7\nErit1jOpNNCYC6wA1s9JX4+OtRyVuBL4qbvfmel/0cw+D5wL5As0XgW45ZZb2GqrrToxW6nEiBEj\nGD169MouxqeK1nn30zrvflrn3WvGjBkcddRRkDmW1lpFgYa7LzOzVmAYcB/8pzHoMGBMsXFL6EPH\nGpF2Crch+Rhgq622YvDgwZ2YrVSib9++Wt/dTOu8+2mddz+t85WmW5oeVHPpZBQwPhNwTAFGEAKF\nmwDMbALwprufl+nvBQwiXF7pDWyUaey50N1fyUzzfuB/zOwN4EVgcGa6/1flcomIiEgdqDjQcPc7\nMs/MuJhwCeV5YLi7z8lk2RhYnhplQ+A5khqLkZluMjA0k3YKcAnwS8JlmLeB6zJpIiIi0qCqagzq\n7mOBsQWGDc3pf40St9G6+0fAGZlOREREPiH0rhMpW3Nz88ouwqeO1nn30zrvflrnn2wVPxm0HpjZ\nYKC1tbVVDYhEREQq0NbWRlNTE0CTu7fVen6q0RAREZGaUaAhIiIiNaNAQ0RERGpGgYaIiIjUjAIN\nERERqRkFGiIiIlIzCjRERESkZhRoiIiISM0o0BAREZGaUaAhIiIiNaNAQ0RERGpGgYaIiIjUjAIN\nERERqRkFGiIiIlIzCjRERESkZho60Bg/fmWXQERERIpp6EBjzJiVXQIREREppqEDDREREalvCjRE\nRESkZhRoiIiISM0o0BAREZGaqSrQMLOTzWyWmS02s6fNbMcieQeZ2V2Z/O1mdmqBfBua2c1mNtfM\nFpnZNDMbXE35REREpD5UHGiY2WHA1cAFwPbANOBhM+tfYJQ+wCvAOcA7BabZD3gCWAIMB7YCzgTe\nr7R8IiIiUj96VjHOCGCcu08AMLMTgX2AY4ErczO7+1RgaibvFQWm+WPgdXc/LpX2WhVlExERkTpS\nUY2GmfUCmoBHYpq7OzAJGNKJcuwHTDWzO8xstpm1mdlxJccSERGRulbppZP+QA9gdk76bGBAJ8qx\nOfBD4CVgT+B6YIyZHdWJaYqIiMhKVs2lk3wM8E6Mvwowxd3Pz/RPM7OtCcHHLZ0tnIiIiKwclQYa\nc4EVwPo56evRsZajEu8AM3LSZgAHFR9tBPvv3zcrpbm5mebm5k4URURE5JOhpaWFlpaWrLQFCxZ0\naxkqCjTcfZmZtQLDgPsAzMwy/Z1588gTwMCctIGUbBA6mvvu0x2wIiIi+eQ7+W5ra6OpqanbylDN\npZNRwPhMwDGFcBdKH+AmADObALzp7udl+nsBgwiXV3oDG5nZtsBCd38lM83RwBNmdi5wB/BV4Djg\n+CqXS0REROpAxYGGu9+ReWbGxYRLKM8Dw919TibLxsDy1CgbAs+RtOEYmekmA0Mz05xqZt8BLgfO\nB2YBp7n77RUvkYiIiNSNqhqDuvtYYGyBYUNz+l+jjLtb3P1B4MFqyiMiIiL1Se86ERERkZpp6EDD\nbGWXQERERIpp6ECjR4+VXQIREREppqEDjVUauvQiIiKffA19qFagISIiUt8a+lCtQENERKS+NfSh\nWoGGiIhIfWvoQ7Uag4qIiNS3hg40VKMhIiJS3xr6UK0aDRERkfrW0IGGajRERETqW0MfqhVoiIiI\n1LeGPlQr0BAREalvDX2oVqAhIiJS3xr6UK3GoCIiIvWtoQON1VZb2SUQERGRYho60PjsZ1d2CURE\nRKSYhg403Fd2CURERKQYBRoiIiJSMwo0REREpGYaOtAQERGR+tbQgYZqNEREROpbQwca7e0ruwQi\nIiJSTEMHGqrREBERqW9VBRpmdrKZzTKzxWb2tJntWCTvIDO7K5O/3cxOLTHtczP5RlVTNhEREakf\nFQcaZnYYcDVwAbA9MA142Mz6FxilD/AKcA7wTolp7wgcn5lmSarREBERqW/V1GiMAMa5+wR3nwmc\nCCwCjs2X2d2nuvs57n4HsLTQRM1sTeAW4DhgfjkFUaAhIiJS3yoKNMysF9AEPBLT3N2BScCQTpbl\nl8D97v7nckdQoCEiIlLfelaYvz/QA5idkz4bGFhtIczscGA7YIdKxlOgISIiUt8qDTQKMaCqw76Z\nbQz8AtjD3ZdVMu6rr45g//37ZqU1NzfT3NxcTVFEREQ+UVpaWmhpaclKW7BgQbeWwbyCaoHMpZNF\nwHfd/b5U+k1AX3f/TonxZwGj3X1MKu0A4B5gBSFggVBr4pm0VT2nkGY2GGj98pdb+fvfB5ddfhER\nkU+7trY2mpqaAJrcva3W86uojUamxqEVGBbTzMwy/U9WWYZJwFcIl062zXRTCQ1Dt80NMrLLU+Uc\nRUREpFtUc+lkFDDezFqBKYS7UPoANwGY2QTgTXc/L9PfCxhEqK3oDWxkZtsCC939FXf/CJienoGZ\nfQTMc/cZxQqiQENERKS+VRxouPsdmWdmXAysDzwPDHf3OZksGwPLU6NsCDxH0oZjZKabDAwtNJvy\nylJZ2UVERKR7VdUY1N3HAmMLDBua0/8alV+iKRSAiIiISAPRu05ERESkZhRoiIiISM0o0BAREZGa\nUaAhIiIiNaNAQ0RERGqmoQMNERERqW8NHWioRkNERKS+NXSg0d6+sksgIiIixTR0oKEaDRERkfrW\n0IGGiIiI1LeGDjRUoyEiIlLfFGiIiIhIzSjQEBERkZpRoCEiIiI109CBhoiIiNS3hg40VKMhIiJS\n3xRoiIiISM0o0BAREZGaUaAhIiIiNdPQgYaIiIjUt4YONFSjISIiUt8UaIiIiEjNKNAQERGRmqkq\n0DCzk81slpktNrOnzWzHInkHmdldmfztZnZqnjznmtkUM/vAzGab2b1mtkWpcijQEBERqW8VBxpm\ndhhwNXABsD0wDXjYzPoXGKUP8ApwDvBOgTzfAP4X+CrwLaAX8EczW73S8omIiEj96FnFOCOAce4+\nAcDMTgT2AY4FrszN7O5TgamZvFfkm6C7753uN7PvA/8GmoDHCxVENRoiIiL1raIaDTPrRTj4PxLT\n3N2BScCQLixXP8CB94plUqAhIiJS3yq9dNIf6AHMzkmfDQzoigKZmQG/AB539+nF8irQEBERqW/V\nXDrJxwg1EF1hLDAI2LlURgUaIiIi9a3SQGMusAJYPyd9PTrWclTMzK4F9ga+4e6FGo7+x8KFI9h/\n/75Zac3NzTQ3N3e2KCIiIg2vpaWFlpaWrLQFCxZ0axnMK6wWMLOngWfc/bRMvwGvA2Pc/aoS484C\nRrv7mDzDrgUOAHZz93+VmM5goHWttVpZsGBwReUXERH5NGtra6OpqQmgyd3baj2/ai6djALGm1kr\nMIVwF0of4CYAM5sAvOnu52X6exEuhRjQG9jIzLYFFrr7K5k8Y4FmYH/gIzOLNSYL3P3jQgXRpRMR\nEZH6VnGg4e53ZJ6ZcTHhEsrzwHB3n5PJsjGwPDXKhsBzJG04Rma6ycDQTNqJmeF/zZndMcCEwmWp\ntPQiIiLSnapqDOruYwmNNvMNG5rT/xol7m5x94Z+FLqIiIjk19AHeNVoiIiI1DcFGiIiIlIzCjRE\nRESkZhRoiIiISM00dKAhIiIi9a2hAw3VaIiIiNQ3BRoiIiJSMwo0REREpGYUaIiIiEjNNHSgISIi\nIvWtoQMN1WiIiIjUNwUaIiIiUjMKNERERKRmFGiIiIhIzTR0oCEiIiL1reEDDdVqiIiI1C8FGiIi\nIlIzCjRERESkZhRoiIiISM0o0BAREZGaUaAhIiIiNaNAQ0RERGpGgYaIiIjUjAINERERqZmqAg0z\nO9nMZpnZYjN72sx2LJJ3kJndlcnfbmandnaaaQo0RERE6lfFgYaZHQZcDVwAbA9MAx42s/4FRukD\nvAKcA7zTRdP8DwUaIiIi9auaGo0RwDh3n+DuM4ETgUXAsfkyu/tUdz/H3e8AlnbFNLOnX8USiIiI\nSLeoKNAws15AE/BITHN3ByYBQ6opQGenqUBDRESkflVao9Ef6AHMzkmfDQyosgydmqYCDRERkfrV\ns4umY0BXH/LLmOYIDjusL716JSnNzc00Nzd3cVFEREQaT0tLCy0tLVlpCxYs6NYyVBpozAVWAOvn\npK9HxxqJbpjmaFpaBtOvX5VzFhER+QTLd/Ld1tZGU1NTt5Whoksn7r4MaAWGxTQzs0z/k9UUoLPT\n1KUTERGR+lXNpZNRwHgzawWmEO4Y6QPcBGBmE4A33f28TH8vYBDhUkhvYCMz2xZY6O6vlDPNYhRo\niIiI1K+KAw13vyPzfIuLCZc7ngeGu/ucTJaNgeWpUTYEniNpbzEy000GhpY5zSLlqXQJREREpLtU\n1RjU3ccCYwsMG5rT/xplXKIpNs3i41U6hoiIiHSXhn/XSXv7yi6BiIiIFNLwgYZqNEREROqXAg0R\nERGpGQUaIiIiUjMKNERERKRmFGiIiIhIzSjQEBERkZpRoCEiIiI1o0BDREREakaBhoiIiNSMAg0R\nERGpGQUaIiIiUjMKNERERKRmFGiIiIhIzSjQEBERkZpRoCEiIiI1o0BDREREakaBhoiIiNSMAg0R\nERGpGQUaIiIiUjMNH2i0t6/sEoiIiEghDR9oqEZDRESkfinQEBERkZqpKtAws5PNbJaZLTazp81s\nxxL5DzGzGZn808xsr5zha5jZtWb2hpktMrMXzeyEcsqiQENERKR+VRxomNlhwNXABcD2wDTgYTPr\nXyD/EOA24AZgO2AiMNHMBqWyjQb2BI4AtgR+AVxrZvuWKo8CDRERkfpVTY3GCGCcu09w95nAicAi\n4NgC+U8DHnL3Ue7+krtfALQBp6TyDAHGu/tj7v66u99ACGB2KlUYBRoiIiL1q6JAw8x6AU3AIzHN\n3R2YRAgW8hmSGZ72cE7+J4H9zWzDzHy+CXwpk68oBRoiIiL1q2eF+fsDPYDZOemzgYEFxhlQIP+A\nVP+PgF8Bb5rZcmAFcLy7P1GqQAo0RERE6lelgUYhBlRyyM/NfyrwVWBf4HVgV2Csmb3t7n8uNiEF\nGiIiIvWr0kBjLqG2Yf2c9PXoWGsRvVssv5mtBlwGHODuf8gMf8HMtgdGAkUCjRGMGNGXfv2SlObm\nZpqbm8tYFBERkU+2lpYWWlpastIWLFjQrWWoKNBw92Vm1goMA+4DMDPL9I8pMNpTeYbvkUkH6JXp\ncusmVlCyDclorr56MDsWvblWRETk0ynfyXdbWxtNTU3dVoZqLp2MAsZnAo4phLtQ+gA3AZjZBOBN\ndz8vk/8aYLKZnQE8ADQTGpQeD+DuH5rZZOAqM/sYeA3YHTgaOL1UYXTpREREpH5VHGi4+x2ZZ2Zc\nTLgk8jww3N3nZLJsDCxP5X/KzJoJl0cuA/5JuEwyPTXZw4CfAbcAaxOCjXPd/Vely1PpEoiIiEh3\nqaoxqLuPBcYWGDY0T9rdwN1Fpvdv4AfVlaWasURERKQ76F0nIiIiUjMKNERERKRmFGiIiIhIzSjQ\nEBERkZpRoCEiIiI1o0BDREREakaBhoiIiNRMwwca7e0ruwQiIiJSSMMHGqrREBERqV8KNERERKRm\nFGiIiIhIzSjQEBERkZpRoCEiIiI1o0BDREREakaBhoiIiNSMAg0RERGpGQUaIiIiUjMKNERERKRm\nFGiIiIhIzSjQEBERkZpRoCEiIiI1o0BDREREakaBhoiIiNSMAg0RERGpmaoCDTM72cxmmdliM3va\nzHYskf8QM5uRyT/NzPbKk2crM/udmc03s4Vm9oyZbVyqLAo0RERE6lfFgYaZHQZcDVwAbA9MAx42\ns/4F8g8BbgNuALYDJgITzWxQKs8XgMeA6cCuwFeAS4CPS5VHgYaIiEj9qqZGYwQwzt0nuPtM4ERg\nEXBsgfynAQ+5+yh3f8ndLwDagFNSeS4FHnD3c939b+4+y91/7+5zSxVGgYaIiEj9qijQMLNeQBPw\nSExzdwcmAUMKjDYkMzzt4ZjfzAzYB/inmf3BzGZnLsccUE6Z2tsrWQIRERHpTpXWaPQHegCzc9Jn\nAwMKjDOgRP71gDWBc4AHgT2Ae4F7zOwbpQqkGg0REZH61bOLpmNAJYf8dP4Y7Ex09zGZ738zs68T\nLss8VngyI7jyyr7cemuS0tzcTHNzcwVFERER+WRqaWmhpaUlK23BggXdWoZKA425wApg/Zz09ehY\naxG9WyL/XGA5MCMnzwxg5+LFGc3IkYNRXCEiItJRvpPvtrY2mpqauq0MFV06cfdlQCswLKZl2lgM\nA54sMNpT6fwZe2TS4zSfBQbm5NkCeK10mcopuYiIiKwM1Vw6GQWMN7NWYArhLpQ+wE0AZjYBeNPd\nz8vkvwaYbGZnAA8AzYQGpcenpnkVcLuZPQb8BdgL2BfYrVRhFGiIiIjUr4oDDXe/I/PMjIsJl0Se\nB4a7+5xMlo0Jl0Ji/qfMrBm4LNP9EzjA3aen8kw0sxOB8wiByUvAQe7+VOnyVLoEIiIi0l2qagzq\n7mOBsQWGDc2Tdjdwd4lp3kSmVqSyslQ6hoiIiHQXvetEREREakaBhoiIiNSMAg0RERGpGQUaIiIi\nUjMKNERERKRmFGiIiIhIzSjQEBERkZpRoCEiIiI1o0BDREREaqahAw0zBRoiIiL1TIGGiIiI1IwC\nDREREakZBRoiIiJSMw0daAC0t6/sEoiIiEghDR1orLKKajRERETqWUMHGrp0IiIiUt8UaIiIiEjN\nNHSgAQo0RERE6llDBxqq0RAREalvCjRERESkZhRoiIiISM0o0BAREZGaaehAAxRoiIiI1LOqAg0z\nO9nMZpnZYjN72sx2LJH/EDObkck/zcz2KpJ3nJm1m9mppcuhQENERKSeVRxomNlhwNXABcD2wDTg\nYTPrXyD/EOA24AZgO2AiMNHMBuXJeyCwE/BWeWVRoCEiIlLPqqnRGAGMc/cJ7j4TOBFYBBxbIP9p\nwEPuPsrdX3L3C4A24JR0JjPbCBgDHAEsL6cgCjRERETqW0WBhpn1ApqAR2KauzswCRhSYLQhmeFp\nD6fzm5kBE4Ar3X1G+eVRoCEiIlLPKq3R6A/0AGbnpM8GBhQYZ0AZ+X8MLHX3ayssjwINERGROtaz\ni6ZjQCWH/P/kN7Mm4FRCe4/KZqoaDRERkbpWaaAxF1gBrJ+Tvh4day2id0vk3wVYF3gjXEEBQq3J\nKDM73d03L1SYhQtHcPPNfXnmmSStubmZ5ubm0ksiIiLyCdfS0kJLS0tW2oIFC7q1DOYVVgmY2dPA\nM+5+WqbfgNeBMe5+VZ78twOru/sBqbQngGnufpKZfRbYIGe0PxLabNzo7v/MM83BQOs667Ry2mmD\nOf/8ihZBRETkU6utrY2mpiaAJndvq/X8qrl0MgoYb2atwBTCXSh9gJsAzGwC8Ka7n5fJfw0w2czO\nAB4AmgkNSo8HcPf3gffTMzCzZcC7+YKM7Hy6dCIiIlLPKg403P2OzDMzLiZcEnkeGO7uczJZNiZ1\ne6q7P2VmzcBlme6fwAHuPr3YbMovT4ULICIiIt2mqsag7j4WGFtg2NA8aXcDd1cw/YLtMtJWWQXa\n28udqoiIiHS3hn7XiS6diIiI1DcFGiIiIlIzDR1ogAINERGRetbQgYZqNEREROpbQwcaagwqIiJS\n3xRoiIiISM00fKCxYsXKLoWIiIgUokBDREREaqahA40ePXTpREREpJ41dKBhphoNERGRetbQgUaP\nHgo0RERE6llDBxpmunQiIiJSzxo60FCNhoiISH1r6EBDz9EQERGpbw0faKhGQ0REpH4p0BAREZGa\naehAQ8/REBERqW8NHWjoORoiIiL1raEDDdVoiIiI1LeGDjRUoyEiIlLfGjrQ0HM0RERE6ltDBxp6\njoaIiEh9a/hAQzUaIiIi9UuBhoiIiNRMVYGGmZ1sZrPMbLGZPW1mO5bIf4iZzcjkn2Zme6WG9TSz\nK8zsb2a20MzeMrPxZrZBycLr0omIiEhdqzjQMLPDgKuBC4DtgWnAw2bWv0D+IcBtwA3AdsBEYKKZ\nDcpk6ZNJvygzve8AA4HflSy8ajRERETqWjU1GiOAce4+wd1nAicCi4BjC+Q/DXjI3Ue5+0vufgHQ\nBpwC4O4fuPtwd7/b3f/p7lMyw5rMbOOihVeNhoiISF2rKNAws15AE/BITHN3ByYBQwqMNiQzPO3h\nIvkB+gEOzC9WHtVoiIiI1LdKazT6Az2A2Tnps4EBBcYZUEl+M1sVuBy4zd0XFiuMAg0REZH61rOL\npmOEGohO5TeznsCdmWEnlZrIs8+O4OOP+7L//klac3Mzzc3NFRRFRETkk6mlpYWWlpastAULFnRr\nGSoNNOYCK4D1c9LXo2OtRfRuOflTQcYmwNBStRkAX//6aF5/fTD33VdGyUVERD5l8p18t7W10dTU\n1G1lqOjSibsvA1qBYTHNzCzT/2SB0Z5K58/YI5MepxGDjM2BYe7+fjnl6e5LJ1Onwpgx3Tc/ERGR\nRlfNpZNRwHgzawWmEO5C6QPcBGBmE4A33f28TP5rgMlmdgbwANBMaFB6fCZ/D+Buwi2u+wK9zCzW\ngLyXCW7y6u67TnbMPC3k1FO7b54iIiKNrOJAw93vyDwz42LCJZHngeHuPieTZWNgeSr/U2bWDFyW\n6f4JHODu01P59818fz7zGdtwfBN4tFBZ1BhURESkvlXVGNTdxwJjCwwbmiftbkKtRb78rxHuZKmY\nnqMhIiJS3/SukyqoFkVERKQ8CjSqsHhx989TRESkETV0oNGjR3WXTubPh2eeqX6+ixZVP66IiMin\nSUMHGmbV1Wh897vwta9VP9/O1GhMnAiPPVb9+CIiIo2koQONHj2qCzRmzgyf+WpD3n0X/vrX4uN3\npkbjO9+BXXetfvw5c+Cpp0rna1SLF8OFF8Kygjc1SyWGDoXJk7t3niedBCeeWNk4zz0HnvOs4EWL\nYN684uMtXAiXXqp2UyL1rKEDjWJ3nVx7bdh55dOrV/hcmOfZo3vsAd/8ZvH5dmUbjeefh+XLS+eL\nhg6Fr3+96+Zfb55+Gi66KKyXao0dC6+8AgsWwG67heCxGh9+CHfcUd24y5eH+Xe3+fPDgf7jj0MZ\n/vKX7n/uy3XXwbhx5edva4PBg+HWW7PThw6F/v2Lj3vNNXD++aVPDmph9myYMaPrp7tsGYwf3zHw\nkurMnAkPPND56ej3qF7DBxqFzmR+9CP46lfzD+uZuan3gw86Dps1K3z26AFPFnjWaVe10Zg/H7bf\nHi64oPxxXnyxa+Zdr+Zn3tc7Z07xfIW4w8knw777wh/+AI8+CrfdVt20Tj8dDjsM3i/xnNoLL4Q9\n98xOO/546Nev8DhLl8LBB8Nrr1VXtkJ+85twoP/970OwAclnPvPnVxeILVoE555bfNrleued8PnK\nK9nplbSjWrQIpk+Hjz7qfHnK9ZWvwKBBXT/da6+F738/+xLr7Nnwz392ftqvvQa//W3np9PVli8v\nfGJYqXnzwv4/1orusEPYH3TG4YfDuuuG/209u+YaePnllV2Kjho+0MhXoxEjz0LV77FGI1+g0SPz\nRI/2drj++vzjd1WNRgxYKgkeqomqZ86EX/2q+DTHjQvlWbYMrr46/DG7OoL/6U/hX/8qnice1P/9\n7+rmEX+bBQuSnULv3tVNa3bmbTxLlmSnz5gBb7yR9F90EfzpT9l5StWEvPQS3H03XHJJdWUrpG/f\n8HnVVfCZz4Tv6WDg6aezD+ibbQYbbFD5fH79a7j88hDQdFZngpXVVgufS5bA1ltDd75PsdpguJT4\nH4j7h3PPhQEDYIstOj/toUPDQbOU/fcPbdmq8eCD8MILlY1z6aWhVmvu3Oz09vbK90M//3kI1p54\nIvTH4LNYzfE773Tc5zz7LNx/f/j+29+GACY3T2srfPnL5W/Df/xj4dqVjz8uPZ1eveC007LT3ngj\nlMs9nBztt1/h8f/yl3Ay0t0aPtCYMyfsPNPSgcDzz4d3lKQ31nw1GvfeGxqXpn/opUth2rSwY04f\nbMoNNBYuDD/6owWebRp3JGZJmnv+AChXqWvSc+fC6NFhpzV0KJxwQuHq7BdeCNfUL744HDxGjgx/\nhmLLOW9e2AmUsyNoaoIf/xj+53/g6KOL5+1sjUZcd8uXJ7/ZqqtWN63o9NOzL+UMGgSf+1zxceJO\nrdDvFLfB9OW7G25I2g89+WRYt/PmwS9/2TGgfuyxsN1Mn569zcYD75QpSVp6+JAhMHBgCCjffz9Z\n32YdaxQg1HbkC/riAaFHBY/aW7o0/CbXXRcORuPGwYYbJr91oQNBsbPI+NvGg0l6uXO9+mpY9nyX\nO+bPh003DQFgV/v3v8P/qtIDpnv47S6/PP/wFSvyX/4tJgbIpcpy//1wzz2lp/fEE/Czn2Wn7bNP\nqO2pRPx8ix1BAAAgAElEQVRNci839u8fLmfn88ADIRjIFbfJuD3F4DvWnOWz4Yawfs6rP3faKQRc\nr76apG2ySfa6O/PMcKJYqFYwt4Zt+PDCtSubbx4C/2KWL+/4vq3PfS6cLMT9dfy//vzn4RiQNnQo\n/OAHxedRCw0faEDYeaalD9Tbbx/eUZKunchXo3HLLeEzvVNbsgS22w6++MXsM/EPPyyvfLfdFs74\nCp31xY0wBhqvvBKWqW/f0gfaxYuTP6V79sZ/442hmu+MM2DttZM/WKEGevFg+N572WfquQHPsmVh\nx7JsWdgBXHZZWD8HHVS8rG1tcMUV2csczZoVDqax9iAe+O69N/sPXq64TlasSAKNVcrcyh95JP9v\n+9vfJu12ijVSTS9bXKf5LrPNmpUs58KFoZp8333hv/87BGLTpsHOO4daiYED4ZRTOrZfuOmm8Ln1\n1uHyTpRvfrk1MitWhJ3N2mtnp+dr57DBBh13wJCs5/ffL78tyjbbhHmedFI4GJ17btg242+fezab\nO69cjz6a1CQVO4hEl14K//hHOAvN9eyz8PrryX6gUhMnFm6kfdppYVnTB6Pp07NrMufOTS4dxP2B\ne/HLdltuWd6lm8mTk7LF7bcrLnkB7LILnHdeqEEo5u67wzqCsFxxm3QP+9xYA5i7z3n//fC/XLEi\nWR9xX7fvviEYyBX373FZ43b++uull2f58o5BWO7BP72PmDYtfF54YfjvtrYm40+aBGuu2bG2M23S\npKTB8zvvJNvINddkn4BC8Uc5tLcn6y4Gn2edFY4B9aChA4302dRbb4Vo7fXX89cIPPlk2Cj+8Ifk\nD53OFzfOtPjHgLBzjMo9AMYNaMIEOO64jhtw7tnI2Wcn32++ufhOZsqU0AbgL3+BtdaCI44I6c88\nA8ceW7ps558Pv/td2LBjOZYvz14PN9+cPc5pp4UdSzzr/n//D/7+9+z1VErun2XzzUPQMmBA6I8H\n4KeeSv7gLS2hy2fp0tAmIx6s0oFGPPAfd1zYKececNOWL4dvfSsc7POZPz8sa7HLMOkz/xhoxDLc\nfXfYYba3h2U+5piQ/sADoeFfrE5da62wLUOoBYrb0NSp2fNK1zbdd18I+u69N38bhQUL4NBDs9uD\n5LtO/+SThQ/20RlnhINkDEh//OOwHeZra5K7vb/0UnYgFLfvWJNSKLh+440QjJ18cnYN0W67hWWH\ncCCH8DuZhW07bdKk8PtB9sHkww/D/GMV+eLFoeZt9uzi20vaihXhbrLYSHv8+GR7hGSbjNs2hADx\ny19O+ocODZcObrgh+W0//jh7nLRly8K1+DfeSNbp+efDlVcmea66Cn7yE9h9944NyPPVhCxfHsqQ\nW0McTZuWfdKUPin70Y/CvrfQvvHgg8M6evvtUHO62mrhN9h99+wax/SBMb2N9+wZTpTWXjssVz5v\nvRXWWdyHxYAgNiiO7e8g/A/z1dj26hUC+2LS/5H4+4wfH/aXO+wQggRIasfiOsut9VixItTWHHlk\ndrp70m4vXctX6gQ3Hs+WLoW//a143m7n7g3XAYMBP/PMVk/O50M3Zoz7lCneIT12a6+dfB83zv/j\n0EMLj5PudtrJ/Xvf8w7a292vvtp93rwk7dxzs8c98MDku7v7ww+H79/5Tujfa6+O87v11ux5xPSB\nA8PnlVcmaQ88ULr8CxeGdZROGzIkfB59tPtpp2UP++ijZP79+4e08eM7TvfBB7PXx4oV7kuXhu+5\neS+7LKS//XZ2+rXXuh9wQHbavHnJ9/nz3V97zf3EE93vuitMY9q0MGzYsOx1vsYa7j/+cfa0Wls7\n/m5RnM8OOyRp++yTPf7//m92fxT7n3wy/Ebp9dva6j50aNJ/zDHFf59993W/7rqO6YcfnszvoYfc\nV189//h77FF42kccUd42fsstHZfN3f3DD8P37bYLXe548+dn93/4YTKde+4pPd8hQ0Le995zX7Ik\nSf/+95PvL73UsWz5umOOSfJ9/HHH4SeemP1/it2aaybfDz208PaSnv8HHyTfly5Nvm+2Wcj3rW+F\n/rvvdr/5Zve//jV7vV5zTeHl+P3v829zb76ZpE2fnl2ea691f/zxjtNK5/nXvzouz8svh2H77Zc9\nTnt79nqJ3nqr+G9wxx3u55zjPmtWktajh/uOO4bvcX653aJFYfp/+1vhaaf/I0cd5f7+++H7wIHu\nP/1p+B737zvtFPrPPz8p+9lnZy9Letpmxbeve+8N+7dCeXr0cF++3P2ii0L/sGHuP/lJx98irr+e\nPd3/9Kdk2FtvhTKA+7//nZQxvR4feqhj2Z99Nn953MP/6dhjk7TW1lYHHBjs3g3H7O6YSZcXOhNo\nHHNMx0DjBz9wnzSp8EaSu7OJdt21dP5nn3U/8kj3r389HPDefDMcSGbOTA6+p5ySTPOUUwpP6/77\nkx1MDDR2371jvm23DcN++cv807niiuT78ceXXobcQCLdDRrUMe3GG8NOa9myJG3EiMLTOO+8EMys\nu27YYaXHS3fpP02x7umnk+9rrZU9zN39scfC9969s/90vXq5n3BCdv5vfzscqB5/PPzx0mJ5tt8+\n9E+c6L7qqtnjn3FGdv9HH4UdSjotvUMG9//6r/KWM3bbbJM/fejQUK7Fi4uP369f4WFbb11+Ofba\ny/1nP0v68x2Uc7s//CG7/7e/DUHXeuuVN88NN0x+w0Lr7Uc/cv/HP7J/63zdcceFg/oZZ7g//3z+\nPDNnFp/Gmmu6//zn+Q/K6fkfd1zy/Xvf6zid9dcvPI9//at4Gc47r+M6dnefOjVJe+CB0usDwoE3\nfj/ySPc5c8J/NR7Y44nK4Ycn+dzd587Nnk4M2Aut19zuS1/K7t922/AZA4J868Q9CTBXW61jntx9\nwYUXJt8PPjh8brNNWLa43R96aNiO3ZOTpuXL82/b775bfJli0FJo+DXXuJ98cuHh77zj/sQT+Yel\nA4YZM8J88v0fVqzI3r/mBjPpfL/5TXba1KkKNEoXOhNo7Lxzx0Djy18OZw7l/AG22Sb8iO3t7uus\nUzr/kiXhjHnTTQvnOfJI95NOCmc56TOxYl2PHu633eb+1a/mH/6PfxQeN0bm4L755sn3rbYqb97l\ndieemHzv27dwvt69k5oaCBF5/J7eObS0lDffX/yi8LC33goBW+z/6KPs4bvsknyPO5YBA5K0H/7Q\n/ZVXwmfcccfArpyyrbFGshMtdoAvpzvppFCO3PRjjklqVsaOdX/mma79Xcvt8v3mhxySHURcckn1\n0+/ZM5zFxYBvzTU7Bm2x+8IX3M88s/j00kHmpZeGz3hmG7tS/88NNgif8Qw3Wr68/G2ks13cbtPd\nhRdm13Rcd13+WptSXXNz2F8OGBBqBEaNCunpGo329hBY5I77pz+5//GP2eup0u4LX8if/j//k3z/\n/OfLm1ZTU/70Sy5JprHjjmEfOXJkUiM4f352jVTsTj89uz93+99kk1CLW6g8RxwR/h/VrJff/S75\nPmZMEjjldueck13jW6ibN69jELL77go0Shc6E2jcfHPrf6Lj2Jm5Dx9e3g+69truF1/s/tnPhv50\nFXe+zj0c+PJF2LndLbe4f/e7lW9k6W6LLUrn2X///OnXX194nF//OtT8QOEz6HxdusbloINCND17\ndvYZUG6XrgXYYIMkwLj44vBnT581V9rts0/2TjH3klC622yzjmk77JCcjV51Vfhcbz33BQsqL0s5\ntUn5uni2e+ONoQYgpp96avi89dbOb0e16ubOza7+3nff/Pl+8pPkP1aoiweDeJa76aYda5QKdeuu\n2zEtbt/gvuee4WCaDg7SQXmhLlZfQ6j9mDGj/MtPte5iDWz6UnBnuvvuy18bs/rqhU8sdtghfC5c\n2HFYodqKSrpvfzv/f7rY5ceDDsru33TTULtZKH9ra/ZlptwAI3Zf/nJ2/2c+U3iahQLkznY9eoTP\n9GXLeMxKn6TGy+qxe/TRfCcxCjRKFzoTaLS2tmZVJaevdeY7O3zvveTPka/LvQaf27mXfya+997Z\n/fnOTEp1gwdXPk7sil0TnzMnOXs566zypxmDgs9+1jvIzbvrrtlBAISD6uTJ4fuBB4aD//Tp4Wy2\n3DLcdFOoVUq3TQH3b34z2Rnk2zkdeWTH/g02SM5s0rUfle4o1lyz+HX22OXbOcWzujvvzL6+P316\n+Hz55fxtNtJd7rqopuvXL1xWy3dwHzcunMGl0044Ifzus2eXnva994bgNt8llHjAjNftq+lWW839\nz3/OTkv/nnH7SG+nuQeU3OXOvey2srt8l2EPPrhjzeXEiYWnsckmoR1ZbvpGG5Uf1OV2n/lM/v9/\nvKRZqvva1/KnH3hgqE3J1x4stn2IXTyJ6N07aeuw6qrZbeLKqbGGsK2nT75i8JV7EtqnT8dx44lh\n+kSl2IncFVd0bEcWu3w1pC+9FGo00+2XYpeuBXJ3v/zypP+gg8Ll+exxujfQaOi7TiC0Xn7oodDi\n9kc/StLjXRhpffuG29geeSRpGZxWzr3f+W71y+fBB7P7x49Pvt9zT7jttpT4TIS0+LTTXr2y7wqJ\n94pH661XeLprrx1un9xuO/jhD0uXI+rfP9ztku85BPGukWjTTTveetbeDhtvHL5PnBiWYautwh0B\naWefDUcdlb8MBxwQHvy1887Z6VtumTwkaLvtOo6X+/C1z38+3E62YkVoKf7448mwSp9NsOaa5T2r\nI99dRH36hM/PfCasj+uuC8u+1VZhd/CFL4TW9sXufT/uOPjSlwrfLQAwbFi45z8+0Or007OHb7QR\n/OIX+R/StMYayV0Z0brrhs+11io8z6h//3AnVL7nycRnNaQftHXyyeHz4IOz7x4pNK+PPw63H6dv\nI3z88ew7hFZfPXucoUOz+w89NLv/ssvyz6sSBx4Y1nvPnvlvwyzXY4/BXnt1TN9vv+y7WyDMb++9\n809nyy2TdQvhTq777gt39CxZEraBffYJwz7/+fzTuPba8Eye+PsXsv76cMghHdOHD8/uX2ed7P54\nx9yiReHuofh03S23TPLkPsMmrts110z2C0uWZN+xs9FGxcsb9e2bfTdi3JekbyM+9tj8t5FPnhzu\n3FprrfCAxPHjs58blPtqi802K1yuefPCNrj11qH/rLPCA9t22ils17kP3dp99+z+s84Kd7scf3z4\nj917b8FF7h4ru3aiszUaue65J1RBL1oUIrcNN8yO9KJ0O4J0xJgvuhw92v2RR8J4L7yQP8811yTX\n5L7xjSS9d+9k3KeeCmcVc+fmPwvNvYyQ7yxmxozwGe+OiA2OchucvfZa/nLmrgf3/O05/u//ksse\nsVru0Uc7jhulGxudd15Yxptuyp7mOutkN7zaYoswbu5dQr/5TWjAVKzsuQ3U0uvuzTfD5Zzddgtn\nHr16hXFiY9ejj04i/v33z47+852B5KuaT3ebbeZ+++3ZaVOnhhqp2D4glh2y76yJ7S+eeKLwunXP\nbvQ7cmRo05Hv9xw5MlyqGD06uzyxwXE8Q3viCfc33kjO7LfbLgz/1a9Cf/qM/9e/DsPWWCNJu+ii\nkJbbkC42vEtX5ca7IvI1unMPVe/pRrUff5zdWDemz5+fXeuTO505c7LT0pc5YqO62J87nXQbpAMO\nyM6bO818NTPpy14QGhVPmxaWZcWK7LLEyzbDhoXPyy4rvn3NmhX2Ibnp+dLck9qcdCPMgQPD79De\nHpb1T39K1u+yZe7PPRe+v/JKyH/YYfnLEhuexv1LbL+y886h/4orwn9o4cL86zC3Rjh9t9/LLyft\nJc4+O4z/4Yfhsma60ewdd2RPIzZCjrVsEGpJ0w1mS7XHibWqf/97dk1IrCGaMKH4es/9H0Zxv3jg\ngWGbS7c5+fOfk+1myJDQ8H2bbZL/qnsoD4T/ZVpujVH68lVa4Vp6XTopXegigUYUd2rNzfl/gNiy\n9+tfT4anq7zS1e9puQ0OY/fQQ0kL8dtvTxrGxR14roULw4blnmzE6er3F17oWEV27rlJo68LLgjj\nLl4cgqvca865O9J4qSTfH+KDD7IPtjFP/EOdfHLYsZQS13VsLLdihfsqqyTTjJdcnnkm3EXz8suh\nPzd4u+eekB77Y2O0vn2z5xeHDxsWAqPYv2RJ+P3b28OOKt5m+cUvhuGzZyd38ZxyStJgNfc6/pw5\nYQf86qv5f/PY7btvWObrrw87xYkT85fTPcwr3XAvtsV4/PHi6zY2PPuv/8o/3XzSbXwOPjikxQPe\n3/8e+u+8M/TvuGMy3vLlye2CEJbLPbsx76WXdizH9dcnjXdvuSVp05O+Re/GG0PD53xlL7Q8uelP\nPtnxN8jNC9l3LEVnnZWsi/nzk5ODkSPDTj99y3vH6ubkf5fbNsw9bG/xgJYrfbfXbruFtHh5N70+\nIATK77yTbBtLliR3KKQvR/7jH2F50icZ7kmwfNdd4bNPn+R2zHK89172Op44MWljM3NmsqwQLj26\nh/3Nxx93nFa6oeioUUlAGU8ADz00/M/efz8Z58UXO94V5p5MJ14mGzEinEQsXBjWxbJlId8//hH+\ns+5JG5abby7+Hz7yyGQ7/eCD5JLeihVhf5QOkl98MfmePh7kExvMxrvG0gHLE08ktzofcUTh32Pq\n1ORumSheWk3PG0J7rrRlywpdelWgUbrQZQQa7mGDiDUbuRtCbMT27W+HM5jjjks2pnj//OTJYaPO\ndfDBHVtDP/54+KNdcUVyYPvrX8NOo5TnngvTeOqpjmXNtzN9662OO44//CG7Eax72JDvvz9cB2xv\nD3l+85v8ZXj77ewzd/fkLGLEiNLL4B6We+rU7LR0o9QvfCH/eLm3+P3lLyF93LhwL32sxRk+PHu8\n2NLcPanFmD+/cPnigXf58uQPf955YdgHH4Sde5zv6NHJeO3tScvv3NvEbr89eyeZT77tLx6oZs8O\nNSzxFsNC4nMT0juSUoHGvHnJXTm33RbS4oF11qzQf999oX+XXTqO/9FH4WD4wQehP143/v3vQ4Ab\n9euXHIDHjg15brwxCWLiASDt/PNDrWJaoeW5//7kTDo9z3z/jXRbjQ8/DC34n3++wArypBFvLH/a\nkiVhmxg/PrnTbOzYMCwexHPnv3RpdmAVpW9TjYFGbAeSPkOOt/jG+cfaoEcfDcPPPDM5aUhvM6+9\nlpwMfPvbYXjcr+y3X+HlL2bevNC+xj2pzUoHE9dfH07YinnnnaS2L55YvfVW8jsdckj55bn00uRk\n49FHOx5885k0KTQMLtVm5NhjS08LQrCdvkslphf6H8Z9V7xtPh3wLFyYBAwHHVR6/mkffRTWxc9/\nntQ0x5OYXO3toeb0gQfCvuq//9v9oosUaJQudJmBRvTLX4bq5LR4xnbNNdnpL76YVPuVIx4wiu3M\nyhEfjFVOoFHM2WeHA3C1IFSDRuPHJweaarS3hx3WrbeGatl8cu9ZTz/oyT3scI85JlT1py1c2DGt\nmBdfTB4WFhvLXnxx+eNHb7wRaj8+97nyzhTz/X4ffZT/gFRIe3uoJYtnau5h53HVVeWNG8VLMDE4\nipcQ99679HQeeSScsRfz61+H6cVakEpA8jyUUt55J+ykX3gh+yFecTrl/l9iTUOp9Xjttf6fAMo9\n+6Fb/fqVnk8MetKBRgwwYo3aFlsUDpRj3nPPDf35zvijuXND/lhTGC9DdEYMtKsVazmjeACuJNDo\njHhA/8pXkt/tu99N/g/pZyoVcuedybaWG2hsskn+ceJD7uLD2+Ll5OuuC/3xt99rr84tX6W6+4Fd\nPbu3RcjKkX58eNSvX3gkbG6jyUpf+xzfkBk/qxUbBUJ4VG+14jtFqrV0afa7QUq9BK0Us9D4NF/j\n3Ci97DfdFBp1peVr/AShkeIaa5RflkGDkt83Pso6t5FgOTbeuLx3a0TnnNPxcdZ9+mQvdylm4XHS\naeW+OTX9zoSRI8PbO2Mju9iItZztd+jQjo0oc8Xplfv47rS3387/KoB8Bgzo2AA5GjUq+yV4xcRt\nvVSj1tgob5ttwmdsxHjUUeFNtqWsuWZoqAnJ7zF0aDhcQXhfyBFHdGzUHe2yS9iPjRwZ+os9Cn+d\ndZJp/+pXhRtWV+LWWwu/lLEcuS/fi//xzr7wsFxbbgl33glf+1p4MVpTE9x1V3iH1TnnhP5SDj44\n+f7gg8mjyGfOTB5znisuZ9ynx/evxJcq9u8fGteeeWbly9RIPhWBRiHl3kFSTHx3R2cDjWjRouyd\n7Zw54c+wySZdM/1Siu3oW1paaK7Be7jjAXfMGPje97p88nl19e9WTKG3b5ajq9f5Kqtkt9qPB6xy\n7h4pR9zh5gaL5ajmdfX5jBhRft4YkMS7uSD/Ot9993BgiQFGDAgWLy5+0E/r0ycc1PK9T6fUXS69\ne4e3+FbCLNx10BV69+74Er7O2GgjuOiicJCF2u1bIrMkUHjzzWR733zzEChssUVl00vfBTRwYPG8\nixYlAVV8d0kMNMw63hH3SVTV7a1mdrKZzTKzxWb2tJntWCL/IWY2I5N/mpl1uFnLzC42s7fNbJGZ\n/cnMvlhN2bpbVx+wVl892Qgh7Lh32qnrdsKd0VLozWad1KNH+AOWeplRVzrwwPDGxe4KbKpVq3Ue\nxR1fVwUae+4Jt99e/+s1im++Td/aXmidp2/FXH/9EDhUcnu4WQg6N9+8ioJ+wpiFlzLGk71ab+dp\nG22Uvb8eOLDjm1K70uqrJzVnX/pS+Kw0sGl0FQcaZnYYcDVwAbA9MA142MzyVh6Z2RDgNuAGYDtg\nIjDRzAal8pwDnAKcAOwEfJSZZpnnCivP9deHywvdVQX4SdWjR23/7Ll69w5vSMz3rJJPk/i68K46\nWzULB+/cqvJ6dcIJ4RLDKhXuCXv3Dm/KjZdDRMoxbFh403Hu23Q/6aqp0RgBjHP3Ce4+EzgRWAQU\nejn5acBD7j7K3V9y9wuANkJgkc5zibvf7+4vAEcDGwIHVlG+brXttuHBLN15kBTpKt/8ZqjZyX2A\nl4jURu4Dxz4NKgo0zKwX0AQ8EtPc3YFJwJACow3JDE97OOY3s82BATnT/AB4psg0RaQL9OwZanYq\naZgqIlKJShuD9gd6ADkPvmU2UKhJzIAC+WOb8fUJt9kUy5NrNYAZ+Z6FLTWzYMEC2traVnYxPlW0\nzruf1nn30zrvXqljZ7dcPO6qu06MECx0Zf5ieT4PcFRX3LclFWkq5z4w6VJa591P67z7aZ2vFJ8H\nnqz1TCoNNOYCKwi1EGnr0bFGInq3RP53CUHF+jnTWA94rsA0HwaOBF4FPi6j3CIiIhKsRggyHu6O\nmVUUaLj7MjNrBYYB9wGYmWX6xxQY7ak8w/fIpOPus8zs3Uyev2WmuRbwVSDvnePuPo9wJ4uIiIhU\nruY1GVE1l05GAeMzAccUwl0ofYCbAMxsAvCmu5+XyX8NMNnMzgAeAJoJDUrTj5L5BfATM3uZUEtx\nCfAmkHpJtIiIiDSaigMNd78j88yMiwmXO54Hhrv7nEyWjYHlqfxPmVkzcFmm+ydwgLtPT+W50sz6\nAOOAfsBjwF7uvrS6xRIREZF6YO6VtOEUERERKV9VjyAXERERKYcCDREREamZhgw0Kn2pm+RnZuea\n2RQz+8DMZpvZvWa2RU6eVc3sl2Y218w+NLO7zGy9nDybmNkDZvaRmb1rZleaWUNuW90t8xu0m9mo\nVJrWeRczsw3N7ObMOl2Uebnj4Jw8RV/saGafNbNbzWyBmb1vZv9nZmt075I0BjNbxcwuMbN/Zdbn\ny2b2kzz5tM6rZGbfMLP7zOytzD5k/zx5Or1+zWwbM3s0c7x9zczOqrSsDbdjqvSlblLUN4D/JdxK\n/C2gF/BHM1s9lecXwD7Ad4FdCe+guTsOzBzcHiQ0LP4a8D3g+4TGwlJEJkA+nrANp2mddyEz6wc8\nASwBhgNbAWcC76fylPNix9sy4w4j/D67EhqwS0c/JqzLk4AtgbOBs83sP++40jrvtDUIN2OcTJ6H\nW3bF+jWzzxCetTELGAycBVxoZsdVVFJ3b6gOeBq4JtVvhFthz17ZZWv0jvCI+XZgl0z/WoSd83dS\neQZm8uyU6d8LWAb0T+U5gbAT77myl6leO2BN4CVgKPAXYJTWec3W9eXA5BJ53gZGpPrXAhYDh2b6\nt8r8Btun8gwn3GE3YGUvY711wP3ADTlpdwETtM5rsr7bgf1z0jq9foEfEh7U2TOV52fA9ErK11A1\nGlW+1E3K148QGb+X6W8inDWn1/dLwOsk6/trwN/dfW5qOg8DfYGta13gBvZL4H53/3NO+g5onXe1\n/YCpZnZH5hJhW/qMzMw2o/SLHb8GvO/u6acVTyL8X75a6wVoQE8Cw8zsSwBmti2wM6EmTuu8xrpw\n/X4NeNTdl6fyPAwMNLO+5ZanoQINir/UrdAL2KQMZmaEKvvHPXnGyQBgaWYDTUuv70IvzQP9JnmZ\n2eHAdsC5eQavj9Z5V9uccGb2ErAncD0wxsziy5IGUPrFjgOAf6cHuvsKQlCudd7R5cBvgZlmthRo\nBX7h7rdnhmud11ZXrd8u2dd01UvVVrZKX+omHY0FBgG7lJG33PWt3ySHmW1MCOj2cPdllYyK1nm1\nVgGmuPv5mf5pZrY1Ifi4pch4nX3546fZYcARwOHAdEJgfY2Zve3uNxcZT+u8trpi/Vrms+zfoNFq\nNKp5qZuUYGbXAnsDu7v726lB7wK9Lbx7Ji33pXi5v0fs12/SUROwLtBqZsvMbBmwG3Ba5sxvNrCq\n1nmXegeYkZM2A/hc5nv6xY5pues8986fHsBn0TrP50rgZ+5+p7u/6O63AqNJavG0zmurs+v33VSe\nfNOACn6Dhgo0MmeA8aVuQNZL3brtBTGfJJkg4wDgm+7+es7gVkLDoPT63oKwg47r+yngKzl3/ewJ\nLCCcyUi2ScBXCGd422a6qYQz6/h9GVrnXekJQoPatIHAaxBe7EjYoabXeXyxY3qd9zOz7VPTGEbY\nmVMulLMAAAHISURBVD9Tm2I3tD50PONtJ3PM0TqvrS5Yv1NSeXbNBCDRnsBL7r6gkgI1VAccSmg5\nezThtqlxwDxg3ZVdtkbrCJdL3ifc5rp+qlstJ88sYHfC2fgTwGOp4asQbs98CNiG0Gp5NnDJyl6+\nRulI3XWidV6T9bsD4U6ec4EvEKr0PwQOT+U5O7Mf2Y8QCE4kvJepdyrPg4RAcEdCw8aXgJtX9vLV\nYwfcSGjAvDewKfAdQnuAn2qdd9k6XoNwcrIdIYg7PdO/SVetX8KdKm8D4wmX1g8DFgI/qKisK3tl\nVbmCTyK85XUxIeLaYWWXqRG7zMa5Ik93dCrPqoRnbczN7JzvBNbLmc4mwO8zG+Bs4ApglZW9fI3S\nAX/OCTS0zrt+He8N/A1YBLwIHJsnz4WZneoiQsv6L+YM70eoeVpACNBvAPqs7GWrxy5zEBxFCJg/\nyhzgLiLn9mut806t490K7MN/05XrlxCkTM5M43VgZKVl1UvVREREpGYaqo2GiIiINBYFGiIiIlIz\nCjRERESkZhRoiIiISM0o0BAREZGaUaAhIiIiNaNAQ0RERGpGgYaIiIjUjAINERERqRkFGiIiIlIz\nCjRERESkZv4/50A3/lAQrHwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f06657154e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.31226165, -0.42375968],\n",
       "       [-0.34929182, -0.45341411],\n",
       "       [-1.3870532 , -1.79564317],\n",
       "       [-0.24588416, -0.33009019],\n",
       "       [ 0.53334152,  0.73282908]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.hstack([params[-1],w])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.00271073]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx = np.random.choice(len(X),20,replace=False)\n",
    "a = sigmoid(X[idx].dot(w)) # Activation function\n",
    "e = y[idx] - a\n",
    "0.5*e.T.dot(e)/len(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Classification error\n",
    "X_test = 5*np.random.randn(100,D)\n",
    "y_test = X_test.dot(w)\n",
    "y_test[y_test<=0] = 0 \n",
    "y_test[y_test>0] = 1\n",
    "\n",
    "y_inferred = sigmoid(X_test.dot(params[-1])) # Get a probability measure given X\n",
    "# y_inferred[y_inferred>0.5] = 1\n",
    "# y_inferred[y_inferred<=0.5] = 0\n",
    "\n",
    "# np.sum(y_test==y_inferred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross Entropy Error\n",
    "\n",
    "We shall focus on doing the same as above with a different loss function, the Cross Entropy Loss function.\n",
    "$$\\mathcal{L} = -\\frac{1}{N}\\sum_i y_i\\log(a_i)+(1-y_i)\\log(1-a_i)$$.\n",
    "\n",
    "The following remains the same:\n",
    "\\begin{align}\n",
    "z_i =& Wx_i\\\\\n",
    "a_i =& \\sigma(z_i)\n",
    "\\end{align}\n",
    "and the derivative,\n",
    "\\begin{align}\n",
    "\\frac{d \\mathcal{L}}{dW} =& \\sum_i \\frac{d\\mathcal{L}}{da_i}\\frac{da_i}{dz_i}\\frac{dz_i}{dW}\\\\\n",
    "=& \\frac{1}{N}\\sum_i -\\left(\\frac{y_i}{a_i}-\\frac{1-y_i}{1-a_i}\\right)\\cdot\\frac{\\exp(-z)}{(1+\\exp(-z))^2}\\cdot x_i\\\\\n",
    "=& \\frac{1}{N}\\sum_i -\\left(\\frac{y_i-a_i}{a_i(1-a_i)}\\right)\\cdot a_i(1-a_i)\\cdot x_i\\\\\n",
    "=& \\frac{1}{N}\\sum_i -(y_i-a_i)\\cdot x_i\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# from IPython.core.debugger import Tracer;\n",
    "\n",
    "def dL_dw(X,e,a):\n",
    "    return -X.T.dot(e)/len(X)\n",
    "\n",
    "def gradient_descent(gamma=1e-2, n_epochs=1000, batch_size=100, decay=0.9):\n",
    "    epoch_run = int(len(X)/batch_size)\n",
    "    \n",
    "    # get starting conditions\n",
    "    # NOTE: I initialised w such that its variance is 1/D\n",
    "    w = np.random.randn(D,1)*(1/np.sqrt(D))\n",
    "    params = []\n",
    "    loss = np.zeros((n_epochs,1))\n",
    "    for i in range(n_epochs):\n",
    "        params.append(w)\n",
    "        \n",
    "        for j in range(epoch_run):\n",
    "            idx = np.random.choice(len(X),batch_size,replace=False)\n",
    "            a = sigmoid(X[idx].dot(w)) # Activation function\n",
    "            e = y[idx] - a # Really important that you use y_obs and not y (you do not have access to true y)  \n",
    "            w = w - gamma*dL_dw(X[idx],e,a)\n",
    "#         \n",
    "        loss[i] = -np.mean(y[idx]*np.log(a+1e-10)+(1-y[idx])*np.log(1-a+1e-10))\n",
    "        gamma = gamma*decay #decay the learning parameter\n",
    "        \n",
    "    return params, loss\n",
    "        \n",
    "params, loss = gradient_descent()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x111350f98>]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhIAAAFkCAYAAAB1rtL+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xe8HFX9//H3hxYkkNAT0EhRBBQN5AYBJQLSEYKIAheQ\njkJAICqI8lNERJp0KQEpoV0BFb6A0g2IFJF7BRECSAm9mARuCOm5n98fZ49TttzdyS0b8no+HvPY\n3ZkzM2fPzs585pwzM+buAgAAKGKx/s4AAABYeBFIAACAwggkAABAYQQSAACgMAIJAABQGIEEAAAo\njEACAAAURiABAAAKI5AAAACFEUgAAIDCGg4kzGyUmd1qZm+YWZeZja5jnqXM7BQzm2Rms8zsJTM7\noFCOAQBA01iiwDwDJT0h6XJJf6xznpskrSLpQEkvSlpN1IYAALDQaziQcPc7Jd0pSWZm3aU3sx0k\njZK0tru/Xxr9aqPrBQAAzacvagV2kfS4pB+Z2etm9pyZnWlmS/fBugEAQC8q0rTRqLUVaiRmSfq6\npJUlXSxpRUkHV5rBzFaStL2kSaX5AABAfZaWtKaku9x9Sm+vrC8CicUkdUna292nS5KZfV/STWY2\nxt1nV5hne0nX9UHeAAD4qNpH0vW9vZK+CCTekvRGDCJKJkoySZ9Q6HyZN0mSrr32Wq2//vq9nkEE\nY8eO1TnnnNPf2VikUOZ9jzLve5R535o4caL23XdfqXQs7W19EUg8JOmbZraMu88ojVtXoZbi9Srz\nzJKk9ddfXyNGjOiDLEKSBg8eTHn3Mcq871HmfY8y7zd90jWgyH0kBprZcDPbsDRq7dLnYaXpp5rZ\n+NQs10uaIulKM1vfzL4i6QxJl1dp1gAAAAuJIldtjJT0T0ntklzSWZI6JJ1Umj5U0rCY2N0/lLSt\npOUl/UPSNZL+T9LRhXMNAACaQpH7SDygGgGIux9YYdzzCh0oAQDARwh3l8T/tLa29ncWFjmUed+j\nzPseZf7RZu7e33koY2YjJLW3t7fTQQcAgAZ0dHSopaVFklrcvaO310eNBAAAKIxAAgAAFEYgAQAA\nCiOQAAAAhRFIAACAwggkAABAYQQSAACgMAIJAABQGIEEAAAojEACAAAURiABAAAKI5AAAACFEUgA\nAIDCCCQAAEBhBBIAAKAwAgkAAFAYgQQAACiMQAIAABRGIAEAAAojkAAAAIURSAAAgMIIJAAAQGEE\nEgAAoDACCQAAUBiBBAAAKIxAAgAAFNZwIGFmo8zsVjN7w8y6zGx0A/N+2czmmllHo+sFAADNp0iN\nxEBJT0gaI8nrncnMBksaL+neAusEAABNaIlGZ3D3OyXdKUlmZg3Meomk6yR1Sdq10fUCAIDm0yd9\nJMzsQElrSTqpL9YHAAD6RsM1Eo0ys3Uk/UrS5u7e1VglBgAAaGa9GkiY2WIKzRknuvuLcXS9848d\nO1aDBw/OjGttbVVra2vPZRIAgIVUW1ub2traMuM6Ozv7NA/mXnd/yfKZzbokfd3db60yfbCk9yTN\nUxJALFZ6P0/Sdu5+f4X5Rkhqb29v14gRIwrnDwCARU1HR4daWlokqcXde/0qyd5u2pgmaYPcuCMk\nbSVpd0mTenn9AACgFzUcSJjZQEmfVlLDsLaZDZc01d1fM7NTJa3u7vt7qO54Jjf/u5JmufvEBcw7\nAADoZ0VqJEZKmqBwDwmXdFZp/HhJB0kaKmlYj+QOAAA0tSL3kXhANS4bdfcDu5n/JHEZKAAAHwk8\nawMAABRGIAEAAAojkAAAAIURSAAAgMIIJAAAQGEEEgAAoDACCQAAUBiBBAAAKIxAAgAAFEYgAQAA\nCiOQAAAAhRFIAACAwggkAABAYQQSAACgMAIJAABQGIEEAAAojEACAAAURiABAAAKI5AAAACFEUgA\nAIDCCCQAAEBhBBIAAKAwAgkAAFAYgQQAACiMQAIAABRGIAEAAAojkAAAAIU1HEiY2Sgzu9XM3jCz\nLjMb3U363czsbjN718w6zexhM9uueJYBAECzKFIjMVDSE5LGSPI60n9F0t2SdpQ0QtIESbeZ2fAC\n6wYAAE1kiUZncPc7Jd0pSWZmdaQfmxt1gpntKmkXSU82un4AANA8+ryPRCn4WE7S1L5eNwAA6Fn9\n0dnyWIXmkRv7Yd0AAKAHNdy0sSDMbG9JP5U02t0n9+W6AQBAz+uzQMLM9pJ0qaRvuvuEeuYZO3as\nBg8enBnX2tqq1tbWXsghAAALl7a2NrW1tWXGdXZ29mkezL2eCy+qzGzWJenr7n5rN+laJf1W0p7u\nfnsdyx0hqb29vV0jRowonD8AABY1HR0damlpkaQWd+/o7fU1XCNhZgMlfVpSvGJj7dKlnFPd/TUz\nO1XS6u6+fyn93pKuknSUpH+Y2ZDSfDPdfdqCfgEAANB/inS2HCnpn5LaFe4jcZakDkknlaYPlTQs\nlf5QSYtLulDSm6nh3GJZBgAAzaLIfSQeUI0AxN0PzH3eqkC+AADAQoBnbQAAgMIIJAAAQGEEEgAA\noDACCQAAUBiBBAAAKIxAAgAAFEYgAQAACiOQAAAAhRFIAACAwggkAABAYQQSAACgMAIJAABQGIEE\nAAAojEACAAAURiABAAAKI5AAAACFEUgAAIDCCCQAAEBhBBIAAKAwAgkAAFAYgQQAACiMQAIAABRG\nIAEAAAojkAAAAIURSAAAgMIIJAAAQGEEEgAAoDACCQAAUFjDgYSZjTKzW83sDTPrMrPRdcyzpZm1\nm9ksM3vezPYvll0AANBMitRIDJT0hKQxkry7xGa2pqTbJd0nabik8yT91sy2LbBuAADQRJZodAZ3\nv1PSnZJkZlbHLIdLesndjyt9fs7MNpc0VtI9ja4fAAA0j77oI7GppHtz4+6StFkfrBsAAPSivggk\nhkp6JzfuHUmDzGxAH6wfAAD0Eq7aAAAAhTXcR6KAtyUNyY0bImmau8+uNePYsWM1ePDgzLjW1la1\ntrb2bA4BAFgItbW1qa2tLTOus7OzT/Ng7t1eeFF9ZrMuSV9391trpDlN0o7uPjw17npJy7v7TlXm\nGSGpvb29XSNGjCicPwAAFjUdHR1qaWmRpBZ37+jt9RW5j8RAMxtuZhuWRq1d+jysNP1UMxufmuWS\nUprTzWxdMxsj6ZuSzl7g3AMAgH5VpI/ESEn/lNSucB+JsyR1SDqpNH2opGExsbtPkvQ1Sdso3H9i\nrKSD3T1/JQcAAFjIFLmPxAOqEYC4+4EVxv1VUkuj6wIAAM2NqzYAAEBhBBIAAKAwAgkAAFAYgQQA\nACiMQAIAABRGIAEAAAojkAAAAIURSAAAgMIIJAAAQGEEEgAAoDACCQAAUBiBBAAAKIxAAgAAFEYg\nAQAACiOQAAAAhRFIAACAwggkAABAYQQSAACgMAIJAABQGIEEAAAojEACAAAURiABAAAKI5AAAACF\nEUgAAIDCCCQAAEBhBBIAAKAwAgkAAFAYgQQAACisUCBhZkeY2ctmNtPMHjWzjbtJv4+ZPWFmH5rZ\nm2Z2uZmtWCzLAACgWTQcSJjZnpLOknSipI0kPSnpLjNbuUr6L0saL+kySZ+V9E1JX5R0acE8AwCA\nJlGkRmKspHHufrW7PyvpMEkzJB1UJf2mkl529wvd/RV3f1jSOIVgAgAALMQaCiTMbElJLZLui+Pc\n3SXdK2mzKrM9ImmYme1YWsYQSd+S9KciGQYAAM2j0RqJlSUtLumd3Ph3JA2tNEOpBmJfSTeY2RxJ\nb0l6T9KRDa4bAAA0mSV6ewVm9llJ50n6uaS7Ja0m6dcKzRuH1Jp37NixGjx4cGZca2urWltbeyWv\nAAAsTNra2tTW1pYZ19nZ2ad5sNAyUWfi0LQxQ9Lu7n5ravxVkga7+24V5rla0tLuvkdq3JclPShp\nNXfP127IzEZIam9vb9eIESMa+DoAACzaOjo61NLSIkkt7t7R2+trqGnD3edKape0dRxnZlb6/HCV\n2ZaRNC83rkuSS7JG1g8AAJpLkas2zpZ0qJntZ2brSbpEIVi4SpLM7FQzG59Kf5uk3c3sMDNbq1Qb\ncZ6kv7v72wuWfQAA0J8a7iPh7jeW7hnxC0lDJD0haXt3/28pyVBJw1Lpx5vZspKOUOgb8b7CVR/H\nL2DeAQBAPyvU2dLdL5J0UZVpB1YYd6GkC4usCwAANC+etQEAAAojkAAAAIURSAAAgMIIJAAAQGEE\nEgAAoDACCQAAUBiBBAAAKIxAAgAAFEYgAQAACmvqQKKBB5MCAIB+0NSBRFdXf+cAAADUQiABAAAK\nI5AAAACFEUgAAIDCmjqQoLMlAADNrakDCWokAABobgQSAACgMAIJAABQWFMHEvSRAACguTV1IEGN\nBAAAzY1AAgAAFEYgAQAACiOQAAAAhTV1IEFnSwAAmltTBxLUSAAA0NwIJAAAQGEEEgAAoLCmDiTo\nIwEAQHMrFEiY2RFm9rKZzTSzR81s427SL2Vmp5jZJDObZWYvmdkB3a2HGgkAAJrbEo3OYGZ7SjpL\n0nckPSZprKS7zOwz7j65ymw3SVpF0oGSXpS0muoIYggkAABobg0HEgqBwzh3v1qSzOwwSV+TdJCk\nM/KJzWwHSaMkre3u75dGv1rPiggkAABobg01bZjZkpJaJN0Xx7m7S7pX0mZVZttF0uOSfmRmr5vZ\nc2Z2ppkt3d36CCQAAGhujdZIrCxpcUnv5Ma/I2ndKvOsrVAjMUvS10vLuFjSipIOrrUyOlsCANDc\nijRtNGoxSV2S9nb36ZJkZt+XdJOZjXH32dVmPOussbrxxsGZca2trWptbe3N/AIAsFBoa2tTW1tb\nZlxnZ2ef5qHRQGKypPmShuTGD5H0dpV53pL0RgwiSiZKMkmfUOh8WdHRR5+jAw4Y0WAWAQBYNFQ6\nue7o6FBLS0uf5aGhPhLuPldSu6St4zgzs9Lnh6vM9pCk1c1smdS4dRVqKV6vtT76SAAA0NyK3Efi\nbEmHmtl+ZraepEskLSPpKkkys1PNbHwq/fWSpki60szWN7OvKFzdcXmtZg2JQAIAgGbXcB8Jd7/R\nzFaW9AuFJo0nJG3v7v8tJRkqaVgq/Ydmtq2kCyT9QyGouEHST7tfV6O5AwAAfalQZ0t3v0jSRVWm\nHVhh3POStm90PfPnN543AADQd3jWBgAAKKypAwn6SAAA0NyaOpCgRgIAgObW1IEENRIAADQ3AgkA\nAFBYUwcS8+b1dw4AAEAtTR1IzK55uyoAANDfmjqQmDu3v3MAAABqaepAghoJAACaW1MHEnPm9HcO\nAABALQQSAACgMAIJAABQGIEEAAAojEACAAAURiABAAAKI5AAAACFEUgAAIDCCCQAAEBhBBIAAKAw\nAgkAAFAYgQQAACiMQAIAABRGIAEAAAojkAAAAIURSAAAgMIIJAAAQGEEEgAAoDACCQAAUFihQMLM\njjCzl81sppk9amYb1znfl81srpl11JN+zhzJvUgOAQBAX2g4kDCzPSWdJelESRtJelLSXWa2cjfz\nDZY0XtK9jaxv7txGcwgAAPpKkRqJsZLGufvV7v6spMMkzZB0UDfzXSLpOkmPNrKyWbMK5BAAAPSJ\nhgIJM1tSUouk++I4d3eFWobNasx3oKS1JJ3UaAZnz250DgAA0FeWaDD9ypIWl/RObvw7ktatNIOZ\nrSPpV5I2d/cuM2tohdRIAADQvBoNJBpiZospNGec6O4vxtH1L2GsDjhgsAYOTMa0traqtbW1B3MJ\nAMDCqa2tTW1tbZlxnZ2dfZoH8wYuiyg1bcyQtLu735oaf5Wkwe6+Wy79YEnvSZqnJIBYrPR+nqTt\n3P3+CusZIaldatdTT43QBhs08pUAAFh0dXR0qKWlRZJa3L2uqyQXREN9JNx9rqR2SVvHcRbaKraW\n9HCFWaZJ2kDShpKGl4ZLJD1bev/37tZJHwkAAJpXkaaNsyVdZWbtkh5TuIpjGUlXSZKZnSppdXff\nv9QR85n0zGb2rqRZ7j6xnpXRRwIAgObVcCDh7jeW7hnxC0lDJD0haXt3/28pyVBJw3oqg9RIAADQ\nvAp1tnT3iyRdVGXagd3Me5IauAyUGgkAAJpXUz9rQ6JGAgCAZtb0gQQ1EgAANK+mDySokQAAoHk1\ndSCx2GLUSAAA0MyaOpBYailqJAAAaGZNH0hQIwEAQPNq+kCCGgkAAJpX0wcS1EgAANC8mj6QoEYC\nAIDm1fSBBDUSAAA0r6YPJKiRAACgeTV9IEGNBAAAzavpAwlqJAAAaF5NHUgMGCDNmNHfuQAAANU0\ndSCx3HLS++/3dy4AAEA1TR1IDBokvfdef+cCAABU09SBxHLLEUgAANDMmjqQGDxYmjpVcu/vnAAA\ngEqaOpBYbjlpzhxp5sz+zgkAAKikqQOJQYPC69Sp1dPccos0cWLf5AdYUO++K51+OrVsAD46mjqQ\nGDw4vE6ZUj3NbrtJG2zQN/kBFtQxx0jHHy+99VZ/5wQAekZTBxIrrRRe33knO/6886Sf/Sz53NXV\nd3kCFkS8wdq8ef2bDwDoKU0dSKy4Ynh9++3s+GOOkU4+ue/zAyyoxUr/OG79DuCjoqkDiaWXDv0k\n8oFEM/vwQ2nUKGnSpP7OCZpRDCS4YyvQfLq6pH//u79zsfBp6kBCkqZNk370o3D1Rt7cuZXnOfVU\n6be/7d18VfPYY9Lf/iZdckn/rL83vfKK9Prr/Z2LhduiEEhcfLE0dmx/5wJo3DnnSJ//vPTGGz27\n3JkzP9pN8E0fSAwdGl5feKF8WrWd8U9+Ih16aO/lafLk6r3u588Pr4sv3nvr7y9rrikNG9bfuVi4\nxe2iJwOJv/2tuTpvjhkjnXtuf+ei97mH/cyzz/Z3TtBTnngivH74Yc8ud5llpO9+t2eX2UyaPpB4\n+OHw+uqr5dO6u7/EyJEL3qntZz+Tfvzj5POcOdIqq4Raj0piILHEEgu23t72+uvS6NH921b/wAOS\nWe3Lez9qzMJrozuq116TPvig8rRRo6QvfWnB8tUMnnii/y+LdZcmTKgvH9OmhZrP732vfNpdd/Xc\nc4K+8x3pvvt6ZlkLg0a2gZdfTjrj33HHgu9LYi13kadOX3dd7aCyN2vJ335b2mef/ntadqFAwsyO\nMLOXzWymmT1qZhvXSLubmd1tZu+aWaeZPWxm29W7rmHDQnVwDCTSG1l3Z3Xt7dJ//1t9+mmnVa7p\nSDv55JAuv87bby9PO2lSuK+FlK2R6O+dYyVnny3ddpvU0dF92uuvlzo7ez4PP/xheH3ttfrS77GH\ntHHVLa2299+Xpk8vH//yy6Eqs6e/31tvhSr+vFpNG7/4hfSnP4X3l1+ePYP55CelLbaovr5G++S8\n+aY0blxj8xT10EPS/vvXTvPUU9JGG0lXXVU73RprSEccUf+6Y3v3hAlhO3MPNSY//3mSZvbs0Gwn\nSffcI331q9If/9j9smOgsFiFvegOO4RAfUFcd510773SZZdJu+++4Mu64Ybq0927308984w0fvyC\n5aOaX/9auuACaccdpdVXT04AX31V2nbbUAtcydprh1rr+fOlnXaSDjhgwfIRA4lGawzdpX33lbar\n+8gWfPhhqFFcUGecEfbT7e0hqKh04t2r3L2hQdKekmZJ2k/SepLGSZoqaeUq6c+R9ENJLZI+JekU\nSbMlDa+xjhGSvL293d3dV1ghbOazZrn/979xk3d/6qnkfVryt3B/9lmvaP78MP0LX6g8Pb8sd/e/\n/919hx2qz7fKKkn6X/4yjJszJ3y+4IJs2ra2sLyoq8v9rrvC66c+5X7CCZXzs/zy7uus477NNuHz\n7Nm18//hh+7XX18+/qijQr4eeaT2/LG8DzywclkviFheDz9cX/q4/unT3a+8MpTVzJnu774bpr/z\njvvkye6XXRbSdXVl5/34x7PL6+oKv6Pkfv/9PfKV/mennZK8pu2/fxj/29+WzxO/34orlpd1rbLv\n7neZOtX93//OjttmmzDPnDl1fZ2G5POz+urh89y51ee5666Qptp2X23ZtcRl3nNPeF1tNffXXy9f\nRvxN3N3Hjy//vz73nPutt5Yv/4knQtoddsiOj/95yf3FF+vLayXp/dhqqxVfTnpZtabvuGPtZay6\navn/ql4TJrjfcEPlac8+m/2uUihzd/fW1vD5ppuq51sK/33JfdNNq+fh0Ufd/+//audz9OiwnPvu\n6/YrZUyZEub7xCfKp3V1Jfl8//3stFGjipdp2ve+F5bz0EPuY8a4L798u0tySSO8wWN8kaFIjcRY\nSePc/Wp3f1bSYZJmSDqoSqAy1t1/7e7t7v6iu58g6T+Sdql3hfHMdZttsmcSlaK/fNXO+++Hn/Dq\nq7NnpPF9rcgz3yxywgnSnXeG95WqptO1H7GJ4557wuv112fTtrZKm2ySfH7gAWn77aVrrpFefFE6\n5RTpwQeT5aS/z3/+E85U/vpXacAA6cknQ7V3PJuNurrCGfzee2c7D82fnyy3uyr2d9/Nvqa/a7XO\nrpW0t4fvnO5wFM+AqlXZV3P88dKBB4bapHPPTar1hwwJw7HHhs/575bvQPXII9K//hXeL7105XVd\nc4302c823gQUv9Mbb2QfPFfPGU931bO/+U1yVp7fPioZNSrctG2LLUINjJSUzbRp3c9fVOwgHc/c\na1X1x2n5vkUXXxyaCaTGq/djx+CXXgqvb72V/b5vvhnKJZ5lz52blOeMGdLvfhfeb7ZZ5dqFmOd5\n86Tjjkv2Kekm18svrz+/06cn/4l87cAyy1SeZ7vtpFtvrTzNPex3qj348IMPpMMOS/Zbd9zRff6k\n7ptsPvxQOv/87H99q62kPfcsT/v669J665WPnzgxzB/X9fTTtdcZv8OLLyb77ZNPDjUxt90Waog2\n3VTaddfsfJ/9bGhu3Gyz8Dn+P2vtFy+7LNSaSKGMu7qSGsFVVilPn75QYPnlw+shh4T91IMPhs8z\nZ4YamUZq29Lidz7mGOmii6QVVii2nMIaiTokLSlprqTRufFXSbq5zmWYpFckjamRJlMjkY7cvvKV\n8uhVcm9vd58xI4lM43DiiSFCk9x/8IMkgnvttTBujTXcb7ut8tnSK69kI/l99kk+Dxnifscd5We9\ncTj++DDu7LOTfHd1uV9xRXImI7m/8EJId9tt4fNPflJ5OZXWsdVW4fUPf3D/+c/D+3/9K0l73XVJ\n2v/8J4ybNCm7jD/+sfx7p913X0i3887ZspDcDzooSXfWWe6/+U315ay1VpjnzTeTcSutFMZdeqn7\n738fyufMM0N0n9fZmax/s83C6513uh9xRHg/eXL5NvHKK+XldsUV4Qz0kUfcDz00Gf+XvyRpX3rJ\n/bHHsmcS//xn7XJyD+t76aVQxvG3+fjHkzLr6nIfOTJ8Pu208vkrbdfz52fzceON2d/hgw+Sz888\nkyzrjTdCmeSXe8wxYdwWW4TPV18d/iOvveY+b557R0c4w543L5u3555z/+IXy8+oKonrmjzZ/bOf\nTT7Hs8y8Bx8MNXjp7X3KFPeXXy7f5uIQz/ovu8z95JNDzZt7KK/Pfz6cwV59dUhz7rnJfH//e/I+\nnn3GYcqUsP2lx02a5L7kkuF9fh9x883ZtGecEca//XYybssts/uI558vr42aPDnUykmh7D/4IDnD\njcMGG5SX289+FqatvHLlcn366TD9Rz/KluO//+1++unuRx6ZbANx+jvvVF6We/J/vfDC7Pbx/vvu\n558fhr/+1b2lJdknR+n1pz3zTPZ7moXXM89033bbZPxee1XOU5x+ww3J+5/8JDst5jufhw8/zI6f\nPTt5/7vfVS+HWMM2fXrYF33602EfLLlvt115+vS+q9K2LIVa1fj+kEPqr6GYMcP94IPdd901u7y1\n1+7bGonGEkurSeqStElu/OmSHqlzGcdJmqwqTSGlNGWBRLqgF1us/IeQ3C+6KOysKk2TwsH8m990\n/3//L/mTpYef/Sz7I6WbTrq6koNDerjuuiR9evxRR4Vx8c++yirhIC+5r79+Nu0HHyR/5vTBTXIf\nPjxZfrrKNGws4fXyy8PBQHI/5ZTweerU8BrT7rxzqNqbMCG7jJ13DsFCPOjkxWBkl12SedJ/jGnT\nst+9mvwf6cEHy8vy+efD6/77Z+edOzebLh6cYxAhlX8vKTn4pw/E1Ybbby9fRvy9pNA8NGeO++67\nh6DsmmvCQW/+/NAs0tFRe/lz5oSDW/y87rrZ75jeiaWH994LB8pK09yzgfMqq2TLe9VVQ9NPep7v\nfS9Mj00b6WGppZL3W2wR0k2cGIKqU09NtjX38P+pVv0bl5EOBKSkGe3OO0Nzg3v5zvzgg5Pfa/HF\ns981nS7dxCmFAMs92Tbibya5//Snybh7703ep5uQYn6PPTY7rr09qdK/5x73J59MvueVV2bTfuc7\nIe8xWI+ByoMPJvPE/35sUurqch84MLucSy8t355iEPf662G+t97KTv/978t/h9NPD9O+//0kXVeX\n++c+F96vt171bfXXv06+6+TJIfj5xCeSNKuvnqznS1+qvJxf/jL8TunfLu/xxyvPG/dncdhhh/D/\nTAcU6f/1Gmsk7wcOzO4z0vlO5yF/UhWbwiT3H/4wCQz/9a/sgf3znw9p0ieEF18cXnfbrfw7po9d\nlbZlKZyApD8//XT5ctLOOius//bbK5ffsGEf4UBC0t6SPpC0VTfpygIJd/dPfrLyhhGHU08NbUSV\npknuG22UvH/00cpp0h57LBn/wQfZjTW/3vyBYKONwhDbX9PD4MHZzzff7H7AAeF9/k/5hS+Eg8m3\nv12+4cfh9NOTNrK4nP32ywYSkvvHPlb5AC65f/WryfeeNSsEQrfcktSoxDNYKZzRxPcjRyYBgBTO\nyru6khqCO+8s/+PE/in5If4pttoqzDNnTjjzee+96r9pHM47r3xcPNClz9obGWJwN2BAOMupFKT+\n5z/hddllay/rhRdCLUR6XLrmpVKNSiy/ast0Lz9YR/HzBRdkpx9+eJi+447df//0zjj2U5Lc9967\n8v/FPfu/OuaY7PLOOitsZ+l58weS0aPdl1uuPC8zZmQ/57/30UeHbSU9LvYDOvzwZNwf/1j9+z7x\nRPifpcehT8ylAAAbPUlEQVT9+c8h6MuXcVdX+XjJ/bjjQvAlJQema69Nyme11cK4iRPD53SwutZa\nYd927LHZg5oU/n+f+lSy/t/9Lju9paX8t/jyl8O09NnqpEnJCUi1IfYpGTgwLGfPPcPn/H5r551D\noFVrWaNGuf/iF8nnqVOztRkPPFB5voMOyn6O3z1ul+7lQWh6SPfpqvS/eeutcFKZHn/LLeVpY0AX\nTxjTJ6DxeCS5//jH4XX77ZP/8r33hv3Sn/+cXeb995evJ9ZIxeGuu8p/zygdQJ1xRuXvuNpqzd1H\nYrKk+ZKG5MYPkVTz/pNmtpekSyV9y90n1LOysWPHavTo0f8bZs4cLantf+1TeZ2dSRtw2tZbS9/4\nRrbtslrbsHvyPp3+mWeSnt15P/5x+Y2a/vnPMEyo8E3zfQsmTEh6qz/2WHZaV1e4udU114T7OFQy\nZUpyCVTM44wZ5W2ZM2dW71X8wgvJ973jjtDG+fWvJ71/022UL76YvH/8cekzn0k+r7FG6MW+8srh\n80UXlV+m29ZWOQ/PPBNeJ0wIvbeXWkraeef6HiPf3p68jw9xmzo1lPWbb3Y/fyUTJoQ2+223Db9L\npb4c8XKvSleEpD3/vLTsstlxTz0lfetboX019qXJ22GH6sucNau8r8WMGdn+HPm+LbEPwJJL1s6v\nlP2PpNvZ0/194v/lJz8J282mmybT8veS+MEPpL/8JTsu3/b93HOVyzn/G+Yf5Pf44+VXaJ1/fvm8\ntR4AOG1a+E3Sdtop5CntqqvCNp4fL4W+S3F7XWml0Icpvc7YFyc+sfjJJ8Pr6NHhSqrPfS78F/P9\nGh54IPu/yz9/KLbDv/Za+M0nTAhXy0jZ+dZcM+kzUs3994fX2E8gXjHR2Rn6cUW33y61tNRe1oMP\nZp+LtMUW4dL4Z54Jz0yq1hchv92mv8OUKeH/Vqtv1TXXVJ/mLp15pvT732fHx//gxz6WzX9c58sv\nh98nSl8Z8Y9/hNcZM5L92Pe/Lx19dNiGpHC1kCRtuWV5nvKXb3d0hCsxPv1p6e67k/FXXCGts07y\nOTwqok3S6Mwwb14f3xGu0chD0qOSzkt9NkmvSTq2xjytkj6UtHOd66hYIxHPcHbbLYm80tVz3/1u\nNvqNw7PPhjPKeDYgharASpHcLruE5oFjj82eDZ50UvUIV0rO/itFm+nItdKw+ebVp33uc7WnS6E6\nONYYxL4I++4bqnQHDao9bxzS1auHHZa8j/0R0sMJJ1ReRmzfLDqkz1bTwwsvNLacMWNCXi69NJwh\n1Eq7zDLVpw0bFvrCnHJKKMdK1Yi/+lV9edphh7AcKbSpSsmVHbHZoNFhySVDL/T0uNtvz9acxLPy\nOBxwQDij2WST7pd//PHZz0OGZP97Umjj/utfw/vYBFDv4J5tSlhzzfI08Uw0X/bps8cBA8JrPGM+\n4YSkX0N+2HjjMG3YsPJpcf5YC1fPEM/6Kw1PPx2aWmKT6SuvuC+xRJh23nlh3MknZ/s4HHlk+M6x\nqrzS0NVV/h9cd92kT9fmm4c+BoMGhX4G+aaTRoZ8WeSbHKoNm25aX7pq+5IvfrH8943DOuuE13RN\naK1hhx1CbWz8vNlmSc1tflhiiexxItZQXXhheY1MevjMZ5L33/lOeI19KeKQbmJrdPj618P2Uem3\nzO83zz/f/aGHmrhpo3SQ30PhKo305Z9TJK1Smn6qpPGp9HtLmqNwdceQ1DCo0UDi2mtDjn/yk+Sg\nFTthxqHSzqizs3o7c60hXQ367W9XP1AutVQ4cEuhenTppbPTuwskuvtTVRrOOit5v/322bZhKXTY\nOeqobGe3/FCparbSkP5TVyrjGGwUKeN6hiefTN7nv2el4bzzwkGvnp1erUBCCu2hsfo5XoqWHiqN\nqzQMGRLaXVdYITS1pKtqiwYSUrLTitvh4YdnO9lK2aaC3XZLApo4DB9e//qqNY0VGebNS4IpKTQL\n5NPEavVaw+67Zz+/8Ubl5rO47ay0UnmTT3rIN5vUGmbNqj7txRdD0+SYMWH/deKJ2X3DueeG6vWR\nI5N93B13hGnbbx/6cDz7bHk1/yGHZH93qfxgu8IKoZlkr7167veSQr+jceO6T3fBBdntLJ7g1DOs\nsEJ2HxO33/QJjpT09ag1XHRRKNdK++7hw7PbXxzSaeOJWD4IWHPNbD+jdKBS7TdJd/qtNpx0UvX9\ncrULDfLBXkeHe3t7kwcSpQP9GEmTJM2U9IikkalpV0r6S+rzBIXmkPxwRaOBROyQ8tOfJn+Q/J8s\nPWyzTfjjdnVlD7zpId1xMN+me+GFyfvNN6/eDv6FLyTtjpMmhXbzp54KZ8RS2MGnO3nFoaUl6eEs\nZTvvdDc89li40uOcc5JxlSLmWn9g9+rThg1LOrsdfnhog66W9uCDw2u1Wp5x47J/IrPkTxavYqg1\n/O1vyfvYkzv/J00Pd90VovR6dl7dBRJbbx3KKX22ne4EWGmYPj0cYOKVLunvuMkmYXm1dsaxM1e1\nId9ZN/07rLFGtg+DlA0mP/7xbH+HP/+5ep+V/GCWtP/3xDBiRHg95JBQw5DuOBrL7sYbq9dUSe53\n313e8TC2oefTxrPQNdcM0/NXeElJ34X01TH5Id3Zu9Z/6M03Q3+fPfdM+hVtu215H5BvfCPZx+U7\nFrsXP5MdOjTpkLzYYkltyIIM772X/c6vvpr0qUgPl1wSAsVTTw0nGu71nQRI2SA7bndS+RU19Qzx\n/imVpl1wQfiN8uPHji0fl+4jJoV97yGHNJaXeCJcazj7bPcNNwzvv/e92rUgcZg3L9tf5M03F5JA\notczVSWQcA/NDR9+mHRijD2T00Oszr7llmS+iy4qT/fzn2c3shdfzE7P9+AeMqTyD5m+NDJ/edzI\nkWHdXV2hyil9htXZmTTNrLhickmqFG6AEzeo9PLjEC/nnDs3/Fl3263+s+M4uGc7An3pS0k19YAB\nyQHtpJNCZ9L0mXPs3BnLMebZLJTbjTeGXtsTJyY9nmM14corh4PX+PHlHeQqVSunD8RxvekmmyOP\nTGoOrr02rC8GPumDZnqIO9h0VWGls5bW1pD3dHVnPeXqHi4Fy29H228fptW6umjcuMqd2L797dCZ\ncfbs8J3j9hGHeDYrhTPWGHCkm3fSB8GhQ5O8xnHbbRcODvl1b7NNqKpPdwqtViY77BBunFZpWr75\nY9CgsL58Ph56KOmoO39+9YAz3pCt0v8vjjvqqHBSEJsL0h0TH3wwnKBcdln28nD36rV56eas9HrS\nnZClcNCNJzzxQPSHP5QHoi+/nF1vep/m3lggkW5aXWWVpBPysstmO3ZWG2KzQfyvxkvn89v2c88l\n+6D895ZCeeZVuuotPRxwQNg+0mfeyy8fqvWl8pq2WsuRQqAa9z2V0t13X+UOm/Pnl181E08i77sv\nuUS2UufM/JDubJ/+f1arXbjsstCUe9hhYdvecsvs9JtvDuUeL+n9/vfLt/d58wgkQqZqBBJRrNrL\n/5hz5yZNBHffnaTPX6olhTOZ9A+Qv95Xqt7Wmm57jr3CzcJGWEs8a4/tprEd7ZVXsj3T3ZO77FWq\nnqx0ueYee5Sni1XRo0aV91iP4mVyo0dn8xAvG7vttiRtrHK89tqkN3hsv37oodrfPZ455u8wmb53\nxp/+VLm84zBpUjjDiX/Eyy4rv+eBexIQpX+nGExuvnnSvppu405fMRMDkHgZb/qupfmqx0svDfen\n+Mc/svejiAeiZ55JdkRHHhmmdXUlAVgc4k483sUvjo/BVbyULpo+PWwjO+8cymDmzOQMZvz4pOf/\n1luHGqFbb80GFUsvnSwrvU2kz4rjvQbifS/StRf77Ve5T9K0aeV9KQ49NJRh/qx01qzsd4rjZ8zI\njp8zp3K7dnT66eV3Tsyn+cc/wudadz9Mu/XW8vW99FKYdswxyd0u0+tJB8KzZmWvvthyy2x6KQRS\nlaSX+d575VdppId00Dl7dnLlzAorZAML9/L7DeSHp58O3zH+p8eOTfanSy1VOa/5q8k22CDcSyOv\nuzP4eLlpOuBwD4HWV75SfhnlkkuGfdpNN2XLYP78sC2k98UTJrh/7Wth+q67hsA43nU2f+WIe/Yq\nvHi138CB2ctAu7qS++xIyfe74opk3A03JE3b8SrA9dcPV69UKoNzzsmWWWw2Pv307KXH7uHEM31v\nk3T+CSTc6wok4plpPFDutFPYINyTm4Okb0+bvmGJFH7cuFHEg0Q99xuIG3n6h4ttgenr+Ku56qqQ\nNt6CN54R56tj3ZODaroWI96iu9INtCpF/O5ho4077Dg+fYOlWF0dq/Gl0FwzfXo4IKfFppg//Sns\n3ON9JKrdijxt+vQw79prZ8fHch8wIOwsPvax8rPt/EEhtm3ecUf19T32WHaH4B7OpDo7Q9733z/Z\nCcZ1t7WFs8d4+Vi81Xm858Arr4QzkrffDgfZgQOTmyHlxQ66cTv797/Lf7f0d4sdIONlq5deGg7g\nXV3lZ63VPPJI2EbefDOpPfjkJ5Pp+fuU5POR/vyrXyXVsWeemUyLl7Huu2/lQKKrK9uvRUpuRJYP\nzPOOP768ZiDKt5FXmj8tnyb2Zxg/vvZ8aflLqCt54IGwz4libVn83eOZ9Lnnhs+xZnPvvbOBZ9ov\nf5mkdw/b6+abZ5s/4yW8+W083nJ60KDsyZF7+B2q9Q/ZZJPk4BubK+INwm6+ufrtvtO1VLXKNn8D\nr2rlmg7o0/L75332yU6PfRaqiTVt1W6HX+0/EWsv11ij9nwzZ4YTlPS4Z59NOmK+8UbYbz3+eDYg\nv+eesF/5xjfKyzjOmw8iKnnssWQ/QSDhXlcgEauL778/HCjzO+j83RHzZxfpG7hMm1be9pc+o0o3\naXzwQfJjr7Za6EsQg5T8TYYqiTvYeMORM84IZ5FRemOOd+FL1zS8+271u67l29bT94aotPxo3rxw\ntvToo+Hz1Knhe1YSq2jrfT5GWtwRVLr7m5S0pbqHg0mtHU48s47BYy3dHXRef718e4lnFfFunfHg\nkH+2Sa3nR3R11Z7uHs72ttwyBCexb0M9d9Gs12c+E2rjolhDs9lm2bLbdddw1pcX+wqkA4l4kNln\nn3Bzt1hO+XKeNy/pyNnZGcble503YtKk7N0ojz66dvoHH0xufFVUPNNdb73QDFiP2Dk5LdZ+uocz\n/kaf5ZB22mmhD8LkyUkAky7PWPO3zDLh8xprhP1U2n77lf+3/va3ZHo8iTnxxO7zEwO0SveyyHv1\n1eSEotp2kG7Gyjv77OQqtgMOyE6bMSN759xGnXdeKNfolltCzVO6GbKSStPiuGnTkv1yvtm7nv9A\nPIGptj+uhkDC6wskYlQdD37dyd/Yp9of+Te/CQfJdPSbvqFR2pw5YYgReXcPAKtHej3xssdvfau+\nje7xx0MzS/q2zLWWX0SsCYid0hp1xx0hUMl76KHswXzatBBk5W+eE8VOl48/3v0699gjHIAaEQOJ\niy8On+NBvjfFtuD0rb172g9/GNZR7Uw4L96NMx1IxJu+tbaGoEcKgdDs2eW3WJ4xI7uuiROzt6ku\nQgoHwr4Q7wwaa6bq8c474UDcl9LlGZsnBwwIn2fNKm8qmj+//E656drHKVNCTVa8hX8tXV2hGeXm\nm+vPb0dHyN+4ceX7g/ggvfyD5qL44LV4RUZfOOKI0GG/klqBhHu45FgK21J38+XFpqhG9XUgsUT9\nd5xoLocfLq21VvbBV7XkH26UvulIWqWHpgwcWDltvKnPSitJl14qbb55fXmppa0tPHhKSm7qtPXW\n0k03dT9vS0sYTjyx+oOfll22+5sn1bL++tKf/ywNGlRs/mo3WMrfkGW55cJDbbbeOrnpTfrhNyHe\nrP7bpNV6fHI18cZByy0XXq+8MtwwqDfFB+3EB/v0hlVXDa9LLVVf+ljOZsm4WOZdXdKGGyZp0suP\nPvax8MCmKD6g6bbbytPWa/r06g9Z62lLLx1u/rTSSvXPs+qqyU2I+sprryU3aIplEx/kNGBAefrF\nFit//Hl6u1txxeo34Msz6/5Bc3kbbVT9QXjxf15tH/Pxj4ebalW7QV9v+M1vis970EHhhlX5/9wN\nN0if/GTteWs9dK2ZLLSBxOKLS1/7Wv3p83eTrPY0vUrSO9FqDj20/uXVstdeyfvBg8NGNHhwuKta\nvQfv+BTMSl58ccGe+HjKKeHgXu3uoj1txIhwd8xx47J3Y4xPFqwnkCjiW98Kd0qMTyxcaqnqZdpT\nzj1X2mWX4kFaPY46Khwk8oFbd9L/gfjfST/dsVE771x83t76zaspGvD0pU98Inkff6tKT6LM22OP\nECiddpr0+c/3Tt4aFQOJGMRXstZafZOXeg0dmv08cqT0xBPh/WGHhSFvjz26X+4yyzR2rOovC20g\n0aiNNw6vjzwiPfyw9IUvdD/Pv/6Vvb1tb54pVhPXedppPbO8VVddsB3jgAHh9tV96fDDw5AWD2K9\n9SdbYolw0O1LgwZJu+3Wu+sYMEA6+OD602+4YXgdNSoZF8940zURaC5//KM0fHj36YrU1vW2egKJ\nZvLee2F/kfbIIwsWaC9sFplAYvXVkx1f+nkAtaQj9CefTJoa0P9WXDG8LgzR+sJsnXXKA4bYLNhX\ntVJoXG8HpL0pNsUsvnj/5qNelU4w84HFR90i9nWLq6cGA33nssvCg9iq9XVB71l55dBPptLDh4AF\ndccdyYPDsHAgkMBCacUVpX326e9cLLr6unkLi4511sk+4RLNr9HHiAMAAPwPgQQAACiMQAIAABRG\nIAEAAAojkAAAAIURSAAAgMIIJAAAQGEEEgAAoDACCQAAUBiBBAAAKIxAAgAAFEYgAQAACiOQAAAA\nhRFIAACAwggkAABAYQQSAACgMAIJAABQGIEE/qetra2/s7DIocz7HmXe9yjzj7ZCgYSZHWFmL5vZ\nTDN71Mw27ib9lmbWbmazzOx5M9u/WHbRm/iz9z3KvO9R5n2PMv9oaziQMLM9JZ0l6URJG0l6UtJd\nZrZylfRrSrpd0n2Shks6T9JvzWzbYlkGAADNokiNxFhJ49z9and/VtJhkmZIOqhK+sMlveTux7n7\nc+5+oaTfl5YDAAAWYg0FEma2pKQWhdoFSZK7u6R7JW1WZbZNS9PT7qqRHgAALCSWaDD9ypIWl/RO\nbvw7ktatMs/QKukHmdkAd59dYZ6lJWnixIkNZg8LorOzUx0dHf2djUUKZd73KPO+R5n3rdSxc+m+\nWF+jgURfWVOS9t13337OxqKnpaWlv7OwyKHM+x5l3vco836xpqSHe3sljQYSkyXNlzQkN36IpLer\nzPN2lfTTqtRGSKHpYx9JkyTNajCPAAAsypZWCCLu6ouVNRRIuPtcM2uXtLWkWyXJzKz0+fwqsz0i\nacfcuO1K46utZ4qk6xvJGwAA+J9er4mIily1cbakQ81sPzNbT9IlkpaRdJUkmdmpZjY+lf4SSWub\n2elmtq6ZjZH0zdJyAADAQqzhPhLufmPpnhG/UGiieELS9u7+31KSoZKGpdJPMrOvSTpH0lGSXpd0\nsLvnr+QAAAALGQtXbwIAADSOZ20AAIDCCCQAAEBhTRdINPpAMFRmZj82s8fMbJqZvWNmN5vZZyqk\n+4WZvWlmM8zsHjP7dG76ADO70Mwmm9kHZvZ7M1u1777JwsvMjjezLjM7OzeeMu9BZra6mV1TKq8Z\nZvakmY3IpaHMe4iZLWZmJ5vZS6XyfMHM/l+FdJR5QWY2ysxuNbM3SvuQ0RXSLHD5mtkKZnadmXWa\n2Xtm9lszG9hofpsqkGj0gWCoaZSkCyRtImkbSUtKutvMPhYTmNmPJB0p6TuSvijpQ4XyXiq1nHMl\nfU3S7pK+Iml1SX/oiy+wMCsFwN9R2IbT4ynzHmRmy0t6SNJsSdtLWl/SDyS9l0pDmfes4yV9V9IY\nSetJOk7ScWZ2ZExAmS+wgQoXMoyRVNaRsQfL93qF/8zWpbRfkTSu4dy6e9MMkh6VdF7qsylc5XFc\nf+dtYR8Ubm/eJWnz1Lg3JY1NfR4kaaakPVKfZ0vaLZVm3dJyvtjf36lZB0nLSnpO0lclTZB0NmXe\na2V9mqQHuklDmfdsmd8m6bLcuN9Lupoy75Xy7pI0OjdugctXIYDokrRRKs32kuZJGtpIHpumRqLg\nA8FQv+UVItupkmRmaylcqpsu72mS/q6kvEcqXCKcTvOcpFfFb1LLhZJuc/e/pEdS5r1iF0mPm9mN\npSa8DjM7JE6kzHvFw5K2NrN1JMnMhkv6sqQ/lz5T5r2oB8t3U0nvufs/U4u/V+E4sUkjeWqmZ20U\neSAY6mBmplDN9Td3f6Y0eqjCBlOpvIeW3g+RNKe0kVZLgxQz20vShgp/5DzKvOetLelwhSbRUxSq\nec83s9nufo0o895wmsIZ77NmNl+hifwEd/9daTpl3rt6qnyHSno3PdHd55vZVDX4GzRTIIHec5Gk\nzyqcNaCXmNknFAK2bdx9bn/nZxGxmKTH3P2npc9PmtkGkg6TdE3/ZesjbU9Je0vaS9IzCoHzeWb2\nZil4wyKmaZo2VOyBYOiGmf1G0k6StnT3t1KT3lbog1KrvN+WtJSZDaqRBokWSatI6jCzuWY2V9IW\nko42szkKZwOUec96S9LE3LiJkj5Zes923vPOkHSau9/k7k+7+3UKdy7+cWk6Zd67eqp835aUv4pj\ncUkrqsHfoGkCidIZXHwgmKTMA8H67OEjHyWlIGJXSVu5+6vpae7+ssLGki7vQQptY7G82xU63qTT\nrKuwk6760LVF2L2SPq9whja8NDwu6VpJw939JVHmPe0hlTd9rivpFYntvJcso3DSl9al0vGEMu9d\nPVi+j0ha3sw2Si1+a4Ug5e+NZqppBkl7SJohaT+Fy4rGSZoiaZX+ztvCNig0Z7yncBnokNSwdCrN\ncaXy3UXhAHiLpP9IWiq3nJclbalwxv2QpAf7+/stLIPKr9qgzHu2fEcq9E7/saRPKVS5fyBpL8q8\n18r8SoVOeztJWkPSbgpt7b+izHusjAcqnIhsqBCkHVP6PKwny1ehg+zjkjZWaPp+TtI1Dee3vwus\nQgGOkTRJ4VKWRySN7O88LYxDaeObX2HYL5fu5wqXEs1QeHb9p3PTByjcj2JyaQd9k6RV+/v7LSyD\npL+kAwnKvFfKeCdJ/yqV59OSDqqQhjLvufIeqPD05pcV7l/wH0knSVqCMu+xMt6iyj78ip4sX4Wr\n+a6V1Klw4nmZpGUazS8P7QIAAIU1TR8JAACw8CGQAAAAhRFIAACAwggkAABAYQQSAACgMAIJAABQ\nGIEEAAAojEACAAAURiABAAAKI5AAAACFEUgAAIDC/j9jfjBsI9VrCwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11116a7b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.14663108, -0.42375968],\n",
       "       [-0.12596242, -0.45341411],\n",
       "       [-0.54578196, -1.79564317],\n",
       "       [-0.11315139, -0.33009019],\n",
       "       [ 0.19399247,  0.73282908]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.hstack([params[-1],w])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "99"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification error\n",
    "X_test = 5*np.random.randn(100,D)\n",
    "y_test = X_test.dot(w)\n",
    "y_test[y_test<=0] = 0 \n",
    "y_test[y_test>0] = 1\n",
    "\n",
    "y_inferred = sigmoid(X_test.dot(params[-1])) # Get a probability measure given X\n",
    "y_inferred[y_inferred>0.5] = 1\n",
    "y_inferred[y_inferred<=0.5] = 0\n",
    "\n",
    "np.sum(y_test==y_inferred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Tensorflow Introduction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000, 5)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(500, 5)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tf_train(X_train, y_train, batch_size=20):\n",
    "    # Dataset (inputs and labels)\n",
    "    x = tf.placeholder(tf.float32, [None, D])\n",
    "    y_ = tf.placeholder(tf.float32, [None, 1])\n",
    "    \n",
    "    # random variable\n",
    "    W = tf.Variable(tf.random_normal([D, 1],stddev=0.1))\n",
    "    # map X to inferred output function\n",
    "    a = tf.sigmoid(tf.matmul(x, W))\n",
    "\n",
    "    # Define loss and optimizer\n",
    "    cross_entropy = tf.reduce_mean(-(y_*tf.log(a)+(1-y_)*tf.log(1-a)))\n",
    "    train_step = tf.train.GradientDescentOptimizer(1e-2).minimize(cross_entropy)\n",
    "    \n",
    "    sess = tf.InteractiveSession()\n",
    "    tf.initialize_all_variables().run()\n",
    "    # Train\n",
    "    for epoch in range(1000):\n",
    "        idx = np.random.choice(len(X_train),batch_size,replace=False)\n",
    "        _,l = sess.run([train_step, cross_entropy], feed_dict={x: X_train[idx], y_: y_train[idx]})\n",
    "        if epoch%100 == 0:\n",
    "            print('loss: '+str(l))\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-42-20b4b688827a>:16: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.\n",
      "Instructions for updating:\n",
      "Use `tf.global_variables_initializer` instead.\n",
      "loss: 0.903563\n",
      "loss: 0.133423\n",
      "loss: 0.0812295\n",
      "loss: 0.0744086\n",
      "loss: 0.0868373\n",
      "loss: nan\n",
      "loss: nan\n",
      "loss: nan\n",
      "loss: nan\n",
      "loss: nan\n"
     ]
    }
   ],
   "source": [
    "tf_train(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tf_train(X_train, y_train, batch_size=20, n_epoch=1000):\n",
    "    x = tf.placeholder(tf.float32, [None, D])\n",
    "    y_ = tf.placeholder(tf.float32, [None, 1])\n",
    "    \n",
    "    W = tf.Variable(tf.random_normal([D, 1],stddev=1/np.sqrt(D)))\n",
    "\n",
    "    # Define loss and optimizer\n",
    "    z = tf.matmul(x,W)\n",
    "\n",
    "    cross_entropy = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=z,labels=y_))\n",
    "    train_step = tf.train.GradientDescentOptimizer(1e-2).minimize(cross_entropy)\n",
    "\n",
    "    sess = tf.InteractiveSession()\n",
    "    tf.initialize_all_variables().run()\n",
    "    # Train\n",
    "    for epoch in range(n_epoch):\n",
    "        idx = np.random.choice(len(X_train),batch_size,replace=False)\n",
    "        _,l = sess.run([train_step, cross_entropy], feed_dict={x: X_train[idx], y_: y_train[idx]})\n",
    "        if epoch%100 == 0:\n",
    "            print('loss: '+str(l))\n",
    "            \n",
    "    return sess.run(W)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-44-5d078cf0b8ba>:14: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.\n",
      "Instructions for updating:\n",
      "Use `tf.global_variables_initializer` instead.\n",
      "loss: 1.63057\n",
      "loss: 0.139019\n",
      "loss: 0.179242\n",
      "loss: 0.144856\n",
      "loss: 0.0849196\n",
      "loss: 0.102604\n",
      "loss: 0.0781684\n",
      "loss: 0.0877755\n",
      "loss: 0.0193359\n",
      "loss: 0.0764944\n"
     ]
    }
   ],
   "source": [
    "w_est = tf_train(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.29627892, -0.42375968],\n",
       "       [-0.34968904, -0.45341411],\n",
       "       [-1.36344194, -1.79564317],\n",
       "       [-0.23447008, -0.33009019],\n",
       "       [ 0.50962275,  0.73282908]])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.hstack([w_est,w])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "99"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification error\n",
    "X_test = 5*np.random.randn(100,D)\n",
    "y_test = X_test.dot(w)\n",
    "y_test[y_test<=0] = 0 \n",
    "y_test[y_test>0] = 1\n",
    "\n",
    "y_inferred = sigmoid(X_test.dot(w_est)) # Get a probability measure given X\n",
    "y_inferred[y_inferred>0.5] = 1\n",
    "y_inferred[y_inferred<=0.5] = 0\n",
    "\n",
    "np.sum(y_test==y_inferred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "latex_envs": {
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 0
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}