Lesson 04 - NeuralNets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning - Hidden Layers\n",
    "\n",
    "https://www.youtube.com/watch?v=UCG1FuKmIOs\n",
    "\n",
    "## Why do we stack layers \n",
    "(adapted from http://stats.stackexchange.com/questions/63152/what-does-the-hidden-layer-in-a-neural-network-compute)\n",
    "\n",
    "Let's call the input vector $x$, the hidden layer activations $h$, and the output activation $y$.  You have some function $f$ that maps from $x$ to $h$ and another function $g$ that maps from $h$ to $y$.  \n",
    "\n",
    "So the hidden layer's activation is $f(x)$ and the output of the network is $g(f(x))$.\n",
    "\n",
    "**Why have two functions ($f$ and $g$) instead of just one?**\n",
    "\n",
    "If the level of complexity per function is limited, then $g(f(x))$ can compute things that $f$ and $g$ can't do individually.  \n",
    "\n",
    "------\n",
    "\n",
    "**An example with logical functions:**\n",
    "\n",
    "For example, if we only allow $f$ and $g$ to be simple logical operators like \"AND\", \"OR\", and \"NAND\", then you can't compute other functions like \"XOR\" with just one of them.  On the other hand, we *could* compute \"XOR\" if we were willing to layer these functions on top of each other: \n",
    "\n",
    "First layer functions:\n",
    "\n",
    "* Make sure that at least one element is \"TRUE\" (using OR)\n",
    "* Make sure that they're not all \"TRUE\" (using NAND)\n",
    "\n",
    "Second layer function:\n",
    "\n",
    "* Make sure that both of the first-layer criteria are satisfied (using AND)\n",
    "\n",
    "The network's output is just the result of this second function.  The first layer *transforms the inputs* into something that the second layer can use so that the whole network can perform XOR.\n",
    "\n",
    "----\n",
    "\n",
    "**An example with images:**\n",
    "\n",
    "Slide 61 from [this talk](http://cs.nyu.edu/~fergus/tutorials/deep_learning_cvpr12/CVPR2012-Tutorial_lee.pdf) as a single image--shows (one way to visualize) what the different hidden layers in a particular neural network are looking for.\n",
    "\n",
    "![cnn](nn.png)\n",
    "\n",
    "The first layer looks for short pieces of edges in the image: these are very easy to find from raw pixel data, but they're not very useful by themselves for telling you if you're looking at a face or a bus or an elephant.\n",
    "\n",
    "The next layer composes the edges: if the edges from the bottom hidden layer fit together in a certain way, then one of the eye-detectors in the middle of left-most column might turn on.  It would be hard to make a single layer that was so good at finding something so specific from the raw pixels: eye detectors are much easier to build out of edge detectors than out of raw pixels.\n",
    "\n",
    "The next layer up composes the eye detectors and the nose detectors into faces.  In other words, these will light up when the eye detectors and nose detectors from the previous layer turn on with the right patterns.  These are very good at looking for particular kinds of faces: if one or more of them lights up, then your output layer should report that a face is present.\n",
    "\n",
    "This is useful because **face detectors are easy to build out of eye detectors and nose detectors, but really hard to build out of pixel intensities.**\n",
    "\n",
    "So each layer gets you farther and farther from the raw pixels and closer to your ultimate goal (e.g. face detection or bus detection)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgsAAAFyCAYAAAB7mplaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XmcXfP9x/HXJ+sQEiKykUVINRIimYg11oTWWioYQWjt\nFNFW24dWWy0traoWrf5Q+1haLaFqCbE0lpqRIAuyEiESZLLIOvP9/fG5t3NzzdyZe3PvPXd5Px+P\n87iZc8+95zMnydz3fM93sRACIiIiIs1pE3UBIiIiUtgUFkRERCQlhQURERFJSWFBREREUlJYEBER\nkZQUFkRERCQlhQURERFJSWFBREREUlJYEBERkZQUFkRERCQlhQURwMwGmNktZjbHzFabWZ2ZvWRm\nF5lZRdT1tYaZ9TOzhhTbZRm85yAz+6mZ9c1FzfmQcF0ujboWkWLVLuoCRKJmZkcADwJrgLuAt4EO\nwH7AtcAuwLmRFZi++4B/NbH/jQzeaxfgp8BzwPubUpSIFC+FBSlrZtYfqAbmAQeHED5JePpPZvYT\n4IgUrzegQwhhbS7rTFNtCOG+LL2XAa1ebc7MKkIIa7J0bhEpELoNIeXuB0An4NtJQQGAEMLcEMIf\n41/HmrP/YGYnm9nbeGvEYbHnNjez68zsfTNbY2azzOy7ye9pZmPM7EUz+9zMVsSOuyrpmO+Y2dtm\ntsrMPjOz/5rZSdn6ps1svpk9amb7mtmrsVsvc8zs1IRjxuMtLgCTY997vZntn/Qeh8bqWw2cHXuu\nrZn9xMxmx67FPDO7ysw6NFPHGDN7I1bHdDM7NuGYHWLnvriJ72Of2HMnZuGatLbmEWb2pJktMbMv\nzGyumd2WdMxJZva6mS2P3dJ608wu2tQaRaJiWqJaypmZfQCsCSEMbOXxDcBMYBvgRmApMCWE8KaZ\nTQIOAG4FpuEh4mjg+hDCd2Ov3wWoBaYC9wBrgZ2AkSGEg2LHnAXcgn9QPwNUALsBq0IIE1LU1g9v\nIbkC+FMThywLIdTHjp2HB50uwG3AIuBbwDBg1xDCTDPbAbgY+A5wFTAr9j5PhxCWxN5jPdA1Vu98\n4J0QwgtmdgdwWux7mAzsCYwH/hFC+GZCzfNi12Bb4M/AJ8AZwBDgsBDCpNhxLwIdQwgjk77nm4Bx\nQM/mWjQSrsv3Qgi/S3H9WqzZzLaNXYdPgP8DlgH9geNCCENix4wBngSeBv4Re/tBQI8QQtYCn0he\nhRC0aSvLDdgSaAAeTuM1DfgH5M5J+4+JPffDpP0PAhuAHWJfXwzUA1unOMc/gDcz+H76xWqojz0m\nbvV4IIkfOy+2b5+Efd2A1cC1Cfu+GTtu/ybOF3+P0Un7d4ud889J+6+NHX9AE+9xTNLfy4fA6wn7\nzood95WEfe3wD+3bWnldLk1xTKtqjv091wPDUrzX9cDnUf/71qYtm5tuQ0g56xx7XJHm6yaHEN5J\n2vd1PBT8MWn/dfjtvq/Hvl4Wezw21t+hKcuA7c1sRJp1xf0FGJ20jQFmJB03I4QwJf5FCGEp8A4w\nII1zzQshPJO073C8n8P1Sfuvw/tAJPcBWRRCeCShjhV4R9NhZtY9tvtBvAViXMLrvoa38NyTRr3N\naW3Ny2JfH21mzfX5WgZ0MrPDslCXSEFQWJBytjz2uGWar5vfxL5++IfeqqT9MxOeB3gA+A/ehL3Y\nzKrNbGxScLgGWAm8ZmbvmtmNZrZPGvW9F0J4toltZdJxTY1u+BzYOo1zzWtiX/w3+dmJO0MIi/EP\n0n5Jx8/my96NPfaPvbYOmAicnHDMOODDEMJzadTbnFbVHEJ4HvgbfqtnqZn908xOT+rXcHOs/n+Z\n2QdmdpuCgxQ7hQUpW7HfYBfh98fTsXoTzrkmhLA//tv+XcCueIB4Kh4YQgizgJ2BE4EXgeOAl8zs\np5metxn1zexvrsWjKamuRbY7RN0FDDCzvcxsC+AofJhoNrVYcwjhBGBvvBWpN3A78LqZbR57fgmw\nO95f5RHgQOAJM/trlmsVyRuFBSl3jwE7mtmem/g+C4DeZtYpaf+ghOf/J4TwXAjhe8E7xV0OHAwc\nlPD86hDCQyGEbwN9gceBy5N75udBJh/4C/CfLRt1Go3dUtiKpGuBd/BMtnPscX7Cvn/jHUrHAccC\nm5GdWxCQZs0hhNdCCD8J3uFyHB44T0p4fkMI4fEQwoUhhB3xDqCnmVk6t3hECobCgpS7a4EvgFsT\n7o//j5nt2Mohb//CO9xdmLR/At68/UTs/Zpq4p+G/zbfMXZM18QnQwgb8NsZBrRvRS3ZtCp23q3S\neM2/Yq+5JGn/d/Hw8XjS/t5JQyU7A6cCb4SE4azBR3JU4y0upwNvhRDeTqOuTan5sVhtTV2HabHH\nJv/+Yt5KPEak2GhSJilrIYS5ZnYycD8w08wSZ3DcFzgeaE3z8UR8lsOrYkMO40Mnj8KHTsbv7V8R\nm6fgcfy31R7AeXj/gZdixzxlZh/jfRsW47MoXgA81kSfiKZUmtm4JvbPCSG80orXJ5qK3674QeyD\nci0wKdYZsknBh5HeCZwdC0fP48MQT8NHnjyf9JJ38bC2B/79fhvojg9bTHYXcBHetJ/u9NWjzWyz\nJvb/oxU1vxA7dryZnY+PWJmD93c5C6ijcdbMW2OB4VlgId7v4kI8/MT7sIgUl6iHY2jTVggbsCM+\nzn8Ofh++Dv+wvhCfoTF+XD1wQzPvsTnwW+ADfA6DWcCEpGMOBB6OHbM69ng3sGPCMWfiweMTvNXj\nXeBXwBYtfA/9YvU1t92ecOxc4JEm3uM5PAwk7vsW8B6wjoRhlHjnxi+9R+y5NsCP8Q6Da/DbCb8A\n2icdNw94FO/DMTX2/U4Hjk3xfb6FD1/t1cq/25auy8mtrRnvi3BPrO4vgI+Af5IwlBK/RfJE7LnV\nsWNvArpH/e9cm7ZMN03KJCKRiU3K9FYI4eg0XlMLfBpCGJO7ykQkUVp9FszsXDObFpu+tM7MppjZ\n11Icf4B9eeW7+qbuDYuItCQ298TuwJ1R1yJSTtLts/ABPpf+e3hnoNOBR8xs99D8vbgAfIWEiW9C\nE3Pwi4g0x8wGAyOAS/HZHR9M/QoRyaa0wkIIIbkX84/N7DxgLxonn2nKkhDC8hTPi0h5CrRueObx\nwE/wfiBVIYR1Oa1KRDaScZ8FM2sDnID3FB8WfCKZ5GMOwDtMzccXw3kb+FlImGJWREREClvaQyfN\nbAjwMv7hvwLvtfyloBDzEXAO8Do+vvgsfKnbkSGEqSnOsQ0+7Gw+3itZREREWqcCH7L7ZAjh02y8\nYdotC7HFU/riS9sejweA/VMEhuTXTwYWhBCaGkMdP+Zk4N60ChMREZFE40IIWZkSPe2WheCzyc2N\nffmGmY3El909r5Vv8Ro+2U0q8wHuueceBg0a1MKhki0TJkzg+uuTF92TXNI1zz9d8/zTNc+vmTNn\ncsopp0DTi95lJBszOLYhvSlMd8dvT6SyBmDQoEEMHz4807okTV26dNH1zjNd8/zTNc8/XfPIZO02\nflphwcyuxmcmex+f5nQccABwaOz5XwG947cYzOxifPay6fg9lLPwxXI0mYqIiEiRSLdloTs+GUov\nfDrcN4FDQwjPxp7vCfRJOL4DcB2+jOsXseMPCY3zrIuIiEiBS3eehTNbeP6MpK9/A/wmg7pERESk\nQGiJavmfqqqqqEsoO7rm+adrnn+65sWvIBeSMrPhQE1NTY06xYiIiKShtraWyspKgMoQQm023lMt\nCyIiIpKSwoKIiIikpLAgIiIiKSksiIiISEoKCyIiIpKSwoKIiIikpLAgIiIiKSksiIiISEoKCyIi\nIpJSQYeFApxcUkREpOwUdFhoaIi6AhERESnosFBfH3UFIiIiorAgIiIiKRV0WNBtCBERkegVdFhQ\ny4KIiEj0FBZEREQkpYIOC7oNISIiEr2CDgtqWRAREYmewoKIiIikVNBhQbchREREolfQYUEtCyIi\nItFTWBAREZGUCjos6DaEiIhI9Ao6LKhlQUREJHoKCyIiIpJSQYcF3YYQERGJXkGHBbUsiIiIRE9h\nQURERFIq6LCg2xAiIiLRK+iwoJYFERGR6CksiIiISEoFHRZ0G0JERCR67aIuIBW1LIiIiLTe6NFw\n8MHZf9+CblnYsCHqCkRERIrHlCnw+efZf1+FBRERkRKxbh20y8E9A4UFERGREtDQ4Lfv27fP/nsX\ndFhYvz7qCkRERIpD/DNTYUFERESatG6dP5bdbQiFBRERkdYpmJYFMzvXzKaZWV1sm2JmX2vhNQea\nWY2ZrTGzd81sfGvPpz4LIiIirVNILQsfAD8AhgOVwLPAI2Y2qKmDzaw/8BgwCRgK3ADcamZjWnMy\ntSyIiIi0Ti7DQlpvGUJ4PGnXj83sPGAvYGYTLzkPmBtCuCz29Ttmth8wAXi6pfMpLIiIiLROPCxE\nfhsikZm1MbOTgM2Bl5s5bC/gmaR9TwJ7t+Ycug0hIiLSOrnss5B2Y4WZDcHDQQWwAjg2hDCrmcN7\nAouT9i0GOptZxxDC2lTnUlgQERFpnYK5DREzC+9/0AU4HrjLzPZPERgyNmnSBI4+ustG+6qqqqiq\nqsr2qURERIpOdXU11dXVACxb5vtuuKEu6+exEMKmvYHZ08DsEMJ5TTz3PFATQrg0Yd/pwPUhhK1T\nvOdwoGbs2BoefHD4JtUnIiJSDqZMgX33hYceqmXs2EqAyhBCbTbeOxvzLLQBOjbz3MvAIUn7DqX5\nPg4bUQdHERGR1sllB8e0bkOY2dXAE8D7wJbAOOAAPABgZr8CeocQ4nMp/Bm4wMyuAW7Hg8PxwOGt\nOZ/CgoiISOsUUp+F7sCdQC+gDngTODSE8Gzs+Z5An/jBIYT5ZnYEcD1wEbAQ+HYIIXmERJPUwVFE\nRKR14r9gRx4WQghntvD8GU3sewGfwCltalkQERFpnYKcZyEfFBZERERap5Cme84r3YYQERFpnbJt\nWVBYEBERaZ1c9lko6LAQT0kiIiKS2rp10KYNtG2b/fdWWBARESkBa9dCRUVu3rugw8LalCtHiIiI\nSNyaNQoLIiIikkLZhoU1a6KuQEREpDiUbVhQy4KIiEjrlG1YUMuCiIhI65RtWNiwAerro65CRESk\n8JVtWAC1LoiIiLRGWYeF1aujrkBERKTwKSyIiIhISqtXKyyIiIhICmpZEBERkZQUFkRERCQlhQUR\nERFJSWFBREREUlJYEBERkZRWrYJOnXLz3goLIiIiJUBhQURERJpVX++fl1tskZv3L+iw0LGjwoKI\niEhLvvjCH8uyZUFhQUREpGUrV/qjwoKIiIg0adUqfyzb2xDxphURERFpWlm3LFRUqGVBRESkJWXf\nsqCwICIiklo8LJRly4LCgoiISMvityHUsiAiIiJNKuuWhc02a7wAIiIi0rSVK6FNG/8lOxcKOix0\n6gQrVkRdhYiISGFbtcpvQZjl5v0LOixssQUsXx51FSIiIoVt5crc3YKAAg8LnTopLIiIiLQk3rKQ\nKwUdFjbfXGFBRESkJblccRIKPCxssYX3WQgh6kpEREQKV9nfhmho0JTPIiIiqZT1bYh4StKtCBER\nkeaVfcsCKCyIiIikopYFFBZERERSKaiWBTP7kZm9ZmbLzWyxmf3DzL7SwmsOMLOGpK3ezLq3dL74\nN66JmURERJq3bBlstVXu3j/dloVRwB+BPYHRQHvgKTPbrIXXBWAg0DO29QohfNLSydSyICIi0rJl\ny2DrrXP3/u3SOTiEcHji12Z2OvAJUAm81MLLl4QQ0vrYV1gQERFJraHBPycLqWUh2VZ4q8FnLRxn\nwFQzW2RmT5nZPq158w4dfFNYEBERadry5T4fUUGGBTMz4PfASyGEGSkO/Qg4B/gmcBzwATDZzHZv\nzXk6d1afBRERkeYsW+aPuQwLad2GSHIzsAuwb6qDQgjvAu8m7HrFzHYEJgDjWzpJ585qWRAREWlO\nwYYFM7sROBwYFUL4KIO3eI0WQgbAhAkTWLq0C/ffD9On+76qqiqqqqoyOKWIiEhpqa6u5sYbqwH4\n4Q+9r19dXV3Wz2MhzYUXYkHhGOCAEMLcjE5q9hSwPIRwfDPPDwdqampqmDBhONtvD/fem8mZRERE\nSts//wnHHgtLlkC3blBbW0tlZSVAZQihNhvnSKtlwcxuBqqAo4FVZtYj9lRdCGFN7Jirge1CCONj\nX18MzAOmAxXAWcBBwJjWnLNrV/j883SqFBERKR/x2xBduuTuHOnehjgXH/0wOWn/GcBdsT/3Avok\nPNcBuA7oDXwBvAkcEkJ4oTUn7NoV3n47zSpFRETKxLJlfvuhffvcnSPdeRZaHD0RQjgj6evfAL9J\ns67/2WYb+KylgZkiIiJlKtezN0KBrw0B3rLw6adRVyEiIlKYFBbwloVly6C+PupKRERECk+up3qG\nIggLXbv6zFTxDhwiIiLS6PPP1bLANtv4o/otiIiIfNlnnyks0LWrP6rfgoiIyJctXQrbbpvbcxR8\nWFDLgoiISPOWLFFYUMuCiIhIM+rr/Zfpsg8Lm23mm1oWRERENvbppz4IoOzDAmiuBRERkaYsWeKP\nCgv4whhLl0ZdhYiISGGJh4Vu3XJ7nqIIC927w+LFUVchIiJSWNSykKBnT4UFERGRZEuXQrt2mmcB\ngB49FBZERESSLVnityDMcnsehQUREZEilY85FqCIwsKKFfDFF1FXIiIiUjjiLQu5VhRhoWdPf1Tr\ngoiISKOPPoJevXJ/nqIICz16+KPCgoiISKOPPoLevXN/HoUFERGRIhQCLFqksPA/3bpBmzbw8cdR\nVyIiIlIYVqyAVat0G+J/2rb13p4KCyIiIu6jj/xRLQsJtt8eFi6MugoREZHCsGiRP6plIUGfPvDB\nB1FXISIiUhjiLQsKCwkUFkRERBotWgSdO8MWW+T+XAoLIiIiRWjRovy0KkCRhYUVK6CuLupKRERE\nordwIWy3XX7OVVRhAdS6ICIiAvD++9CvX37OpbAgIiJShBYsUFj4kt69fWImhQURESl3a9f63EN9\n++bnfEUTFtq1844cCgsiIlLu4vMOKSw0QSMiRERE/BYEKCw0SWFBRETEOzdCY3++XFNYEBERKTIL\nFkD37lBRkZ/zFVVY2GEHmD8f6uujrkRERCQ6c+fCgAH5O19RhYWBA2H9erUuiIhIeZs92z8T86Wo\nwsJOO/nj7NnR1iEiIhKl2bMbPxPzoajCQr9+PoTyvfeirkRERCQay5fDJ58oLDSrXTvo318tCyIi\nUr7mzPFHhYUUdtpJYUFERMpX/DNQYSGFgQMVFkREpHy99x507epbvhRdWNhpJ2+CaWiIuhIREZH8\ny3fnRijSsLB2beO82CIiIuWk4MOCmf3IzF4zs+VmttjM/mFmX2nF6w40sxozW2Nm75rZ+EwLjl8g\njYgQEZFy9N57BR4WgFHAH4E9gdFAe+ApM9usuReYWX/gMWASMBS4AbjVzMZkUC8DBkCHDjBjRiav\nFhERKV6ff+5LUw8alN/ztkvn4BDC4Ylfm9npwCdAJfBSMy87D5gbQrgs9vU7ZrYfMAF4Oq1q8eGT\nO+8M06en+0oREZHiFv/sGzw4v+fd1D4LWwEB+CzFMXsBzyTtexLYO9OTDhmisCAiIuVn+vTGX5rz\nKeOwYGYG/B54KYSQ6qZAT2Bx0r7FQGcz65jJuQcPhrffhhAyebWIiEhxevttn0KgQ4f8njet2xBJ\nbgZ2AfbNUi1fMmHCBLp06bLRvqqqKgYPrmLZMvjoI+jdO1dnFxERKSzTp298C6K6uprq6uqNjqmr\nq8v6eTMKC2Z2I3A4MCqE8FELh38M9Eja1wNYHkJYm+qF119/PcOHD//S/vikTNOnKyyIiEj5mD4d\nzj+/8euqqiqqqqo2Oqa2tpbKysqsnjft2xCxoHAMcFAI4f1WvORl4JCkfYfG9mdkhx2gosKbY0RE\nRMrBkiW+gFS+OzdC+vMs3AyMA04GVplZj9hWkXDM1WZ2Z8LL/gwMMLNrzGxnMzsfOB74XaZFt23r\nw0YUFkREpFxENRIC0m9ZOBfoDEwGFiVsJyQc0wvoE/8ihDAfOAKfl2EqPmTy2yGE5BESaRk6FKZN\n25R3EBERKR5Tp3qr+sCB+T93uvMstBguQghnNLHvBXwuhqwZPhzuuw/Wrct/r1AREZF8q6nxX5Tb\nbcrQhAwV3doQcZWVHhQ034KIiJSDmhr/7ItC0YaFoUOhTRu/eCIiIqVs1SqYNUthIW2dOsFXvwq1\ntVFXIiIikltTp/pEhE3MJpAXRRsWwBOWWhZERKTU1dR4/7woRkJAkYeF4cN9RMT69VFXIiIikju1\ntbDbbtC+fTTnL+qwMHIkrF2rIZQiIlLaXn0V9tgjuvMXdViorISOHeE//4m6EhERkdz49FPv3LjP\nPtHVUNRhoWNHGDFCYUFERErXK6/4o8LCJth3Xw8LWq5aRERK0ZQp0KOHr4sUlZIIC4sWwYIFUVci\nIiKSfVOmeKuCWXQ1FH1YiDfL6FaEiIiUmg0b4LXXYO+9o62j6MNCt26w884KCyIiUnpqa+GLL7wV\nPUpFHxagsd+CiIhIKZk0CbbYItphk1AiYWH//eGtt3x4iYiISKl49ln/jItqMqa4kggLo0f7aIhJ\nk6KuREREJDvWroWXXoJDDom6khIJC9ttB4MGwdNPR12JiIhIdrzyCqxZAwcfHHUlJRIWAMaM8bCg\n+RZERKQUPPssbLONrwkRtZIJC6NH+1wLc+ZEXYmIiMimmzQJDjoI2hTAJ3UBlJAdBx4I7drpVoSI\niBS/ujpfPKoQbkFACYWFLbeEvfZSWBARkeL31FM+IdMRR0RdiSuZsADwta95WFi7NupKREREMjdx\novdV6Ns36kpcSYWFo4+GlSvhueeirkRERCQz9fXwr3/BkUdGXUmjkgoLQ4b4qlyPPhp1JSIiIpl5\n9VWfZFBhIUfM4JhjPCxoCKWIiBSjxx7zdY9Gjoy6kkYlFRbAb0V8+CHU1ERdiYiISPoefRQOPxza\nto26kkYlFxZGjYKtt4ZHHom6EhERkfTMmAHTp8M3vxl1JRsrubDQrh0cdRQ89JBuRYiISHF56CHo\n3BkOPTTqSjZWcmEB4KST4J13YNq0qCsRERFpvQcf9L53FRVRV7KxkgwLo0f7fNrV1VFXIiIi0jrT\np/ttiBNOiLqSLyvJsNC+PRx/PNx/v25FiIhIcXjgAejSpfBuQUCJhgWAqip4/314+eWoKxEREUkt\nBLj3XjjuOOjQIepqvqxkw8J++8F228E990RdiYiISGovvghz58Lpp0ddSdNKNiy0bQvjx8N998EX\nX0RdjYiISPPuuAMGDPBfdAtRyYYFgG99y5f5/Pvfo65ERESkaStX+iiI8eOhTYF+KhdoWdmx446+\nFvitt0ZdiYiISNMefhhWrYLTTou6kuaVdFgAOPNMeOEFePfdqCsRERH5sltu8V9s+/ePupLmlXxY\nOPZYn/759tujrkRERGRjb7wBU6bABRdEXUlqJR8WKirg1FM9LKxZE3U1IiIijW66Cbbf3hdBLGQl\nHxYALrwQlizRjI4iIlI4PvvMR+yde66va1TIyiIsDBwIRx4J11+vGR1FRKQw/PWvUF8PZ50VdSUt\nK4uwADBhArz1Fjz3XNSViIhIuduwAW6+GcaOhe7do66mZWmHBTMbZWaPmtmHZtZgZinvtJjZAbHj\nErd6M8vr5TnoINhtN29dEBERidLf/uYzNl56adSVtE4mLQudgKnA+UBrG/UDMBDoGdt6hRA+yeDc\nGTODSy6Bxx6DWbPyeWYREZFGIcCvf+0LRg0fHnU1rZN2WAgh/DuEcEUI4RHA0njpkhDCJ/Et3fNm\nw8kn+3oRV18dxdlFRETgySdh2jT44Q+jrqT18tVnwYCpZrbIzJ4ys33ydN6NdOwIP/iB9z6dPTuK\nCkREpNz9+tcwciQceGDUlbRePsLCR8A5wDeB44APgMlmtnsezv0lZ54J3brBr34VxdlFRKScvfQS\nPP+8typYOm3zEbOwCWMJzawB+EYI4dE0XzcZWBBCGN/M88OBmv33358uXbps9FxVVRVVVVUZVux+\n9ztvYXjvvcKeXlNEREpHCN6aUFcHtbXZWTSqurqa6qRJhOrq6njhhRcAKkMItZt+lujCwrXAviGE\nfZt5fjhQU1NTw/Ac9P5YtQp22AGOOgpuuy3rby8iIvIlTz0Fhx0GEyf63D+5UltbS2VlJWQxLEQ1\nz8Lu+O2JSHTqBD/+sa8fPmNGVFWIiEi5CAEuvxz23huOOCLqatKXyTwLncxsaEKfgwGxr/vEnv+V\nmd2ZcPzFZna0me1oZoPN7PfAQcCNWfkOMnTOOdCvH/zoR1FWISIi5eCRR+D11+Gqq4qrr0JcJi0L\nI4A3gBp8/oTrgFrg57HnewJ9Eo7vEDvmTWAysCtwSAhhckYVZ0nHjv6X9uij3uFEREQkF9atg8su\ngzFjfILAYpT20hUhhOdJETJCCGckff0b4Dfpl5Z7J54Iv/0tfO97vkRoNjqbiIiIJLrxRpgzBx5+\nOOpKMlfWH49t2vjIiFdfhbvuiroaEREpNUuWwJVX+q3vIUOiriZzZR0WAA44AKqqvIlo2bKoqxER\nkVJyxRX+eOWV0daxqco+LIDfili9uvEvVUREZFO99hrccgv87Gc+GWAxU1gAeveGn/4UbroJ3nwz\n6mpERKTYrV8PZ58Nw4bBhRdGXc2mU1iIufhi2Hln/8utr4+6GhERKWa//z289Rb85S/QLu2hBIVH\nYSGmfXu49VZvNvr976OuRkREitX8+d5afdFF4BMpFj+FhQT77AOXXOKzO777btTViIhIsWlo8Bbq\nbt3gF7+IuprsUVhI8stfwvbbw7e/7X/pIiIirXXzzfD0095SvcUWUVeTPQoLSTbf3BeXeukluP76\nqKsREZFiMWsWfP/7cMEFcOihUVeTXQoLTdh/f/jud33diNqsrNclIiKlbP16OPVU6NsXrr026mqy\nT2GhGVdfDbvtBiedBCtXRl2NiIgUsp/9DN54A+6+21uoS43CQjM6dIDqali0yHu0ioiINOXf//Zf\nMH/xCxjeKeQuAAAWwElEQVQ5MupqckNhIYWBA30BkL/+Fe68s+XjRUSkvCxcCKecAl//OvzgB1FX\nkzsKCy0YP95HRpxzDtTURF2NiIgUivXr/Vb1Zpv5YoSlvHJxCX9r2WHmrQu77QbHHecriImIiFxy\nia9a/MADxb/2Q0sUFlqhogL+/ndfbOqkk2DDhqgrEhGRKP3pTz6nws03+4R+pU5hoZX69IGHHoLn\nn4cJEyCEqCsSEZEoPPssfOc73vn9rLOiriY/FBbScMABvjLljTdq/QgRkXI0ezaMHQsHHwzXXRd1\nNflTAmth5dc558C8eT5pU79+3o9BRERK3+LF8LWvef+EBx4ojdUkW6uMvtXsufpqX1Vs3Dh47jnY\na6+oKxIRkVxasQIOPxxWrYIpU2DrraOuKL90GyIDbdrAHXfAiBH+j+ett6KuSEREcmXdOm9Fnj3b\nJ2DaYYeoK8o/hYUMVVTAxInQvz+MGaMlrUVEStGGDb7mwwsvwCOPwNChUVcUDYWFTbDVVvDkk9C1\nK4weDQsWRF2RiIhkS329T8z397/D/ffDgQdGXVF0FBY20bbb+trl7drBIYfA++9HXZGIiGyq+no4\n/XTvyHj//XDssVFXFC2FhSzYbjsfd1tf78tbz50bdUUiIpKp+nqf5v++++Dee+H446OuKHoKC1nS\nv7/f0+rQwQPDO+9EXZGIiKRr/Xq/9XD33XDPPXDiiVFXVBgUFrKoTx+f4bFLF5/A6e23o65IRERa\n64sv/HbDgw9CdTVUVUVdUeFQWMiyXr1g8mR/HDUKXnwx6opERKQly5bBYYf53DkTJ8IJJ0RdUWFR\nWMiBbbf1wDBsmI+SeOihqCsSEZHmLFrkIx2mT4dJkzw0yMYUFnKkSxd44gn45jf9npfWkhARKTxv\nvAEjR8Knn3q/M83I2zSFhRzq2NE7yHz/+75S5QUXeOcZERGJ3qOPwn77Qc+e8NprMGRI1BUVLoWF\nHGvTBq65Bm65Bf7yF5/tccmSqKsSESlfIfiKkd/4hi8M9fzz3s9MmqewkCdnn+1zMcyYAXvsAdOm\nRV2RiEj5WbnSRzl873tw2WXep6xTp6irKnwKC3k0ahS8/jpssw3ss48PzRERkfx45x3Yc0947DEf\nHvnrX3vrr7RMlynP+vb14ZTHHgsnnwznnAOrV0ddlYhIafvHP7xVt74e/vtfGDs26oqKi8JCBDbf\n3GcHu/VWuOsuT7qzZkVdlYhI6Vm9Gi680JeYPvRQ78g4aFDUVRUfhYWImPnc4//9ry+BWlkJd9zh\nHW9ERGTTvfkmjBgBt90GN97o/RM6d466quKksBCxIUM8MJx4IpxxhqffTz6JuioRkeLV0OBz2+yx\nB7Rt633FLrjAf0mTzCgsFIBOneD22+Hhh+E//4HBg/3PIiKSnvnzfTjkhAlw3nl+22Hw4KirKn4K\nCwXk2GN98alRo3zmx1NPhc8+i7oqEZHC19DgtxqGDIGZM+Hf//bWhYqKqCsrDWmHBTMbZWaPmtmH\nZtZgZke34jUHmlmNma0xs3fNbHxm5Za+7t3h73/3jo8TJ3pHnPvuU18GEZHmvPuur/T7ne/4L1nT\np2t9h2zLpGWhEzAVOB9o8SPMzPoDjwGTgKHADcCtZjYmg3OXBTP/Bz9zpi9uMm6c/8OfMyfqykRE\nCseaNfDLX8LQofDRR75i5J/+pE6MuZB2WAgh/DuEcEUI4RGgNd1FzgPmhhAuCyG8E0K4CfgbMCHd\nc5ebXr3ggQfg8cc9OQ8ZAldfDWvXRl2ZiEi0/vUv/5n485/DRRf5yIcDD4y6qtKVjz4LewHPJO17\nEtg7D+cuCYcf7s1q3/kOXHEF7LKLTzCiWxMiUm7mzYNjjoEjjoD+/T0kXHONz18juZOPsNATWJy0\nbzHQ2cw65uH8JaFTJ7j2WnjrLfjKV3yI5SGH+H8UEZFSt3w5/OQn/stSba1P1/z005pgKV80GqLI\nDBoETzzhtyYWLYJhw+Dcc/1+nYhIqVm3zkc57LQT/Pa3cOml3p9r7FjNm5BP7fJwjo+BHkn7egDL\nQwgp775PmDCBLl26bLSvqqqKqqqq7FZYhA4/3Je7vukmv2d3111w8cW+itrWW0ddnYjIpgnBR4b9\n6Efeufv00+HKK2H77aOurLBUV1dTnbQqYV1dXdbPY2ETbnybWQPwjRDCoymO+TXw9RDC0IR99wFb\nhRAOb+Y1w4Gampoahg8fnnF95WLZMvjNb3xMcfv2HhguvljLropI8QnBW09/9jOf3fbrX/c+Cbvu\nGnVlxaO2tpbKykqAyhBCbTbeM5N5FjqZ2VAz2z22a0Ds6z6x539lZncmvOTPsWOuMbOdzex84Hjg\nd5tcvQCw1VZw1VUwdy6cdpr/J9txR7juOl+7XUSk0IXgIxz23NM7L3boAJMm+T4Fhehl0mdhBPAG\nUIPPs3AdUAv8PPZ8T6BP/OAQwnzgCGA0Pj/DBODbIYTkERKyiXr0gD/8wYdZHnkk/PCH3lv4l7/0\n1gcRkUITgvfBSgwJzzwDL74IBx8cdXUSl8k8C8+HENqEENombd+KPX9GCOHgpNe8EEKoDCFsFkIY\nGEK4O1vfgHxZ//6+/PWcOVBV5WGhXz+4/HJYsiTq6kREvOPiXXf5hEpHHukh4emnPSQccog6LxYa\njYYoYX37wh//6OOSzz4bbrjB9519NsyYEXV1IlKO6uq8j9WAATB+PPTpA88+6yFh9GiFhEKlsFAG\nevXy/5wLFsCPfwyPPearsB12mC+2osmdRCTX5s+H73/fw8Hll8Ohh/rCeY8/DgcdpJBQ6BQWysg2\n2/h/0vnz4e67YelS72k8eDD8+c+wYkXUFYpIKWlo8JENRx3lLQn/939w/vn+M+j227V0dDFRWChD\nHTrAKafA66/DCy/AV78KF1zgLRBnnw01NVFXKCLF7NNPfQKlgQN9TpiFC+Evf4EPP4Rf/xp69466\nQkmXwkIZM4NRo+DhhxubCJ94AkaMgMpK/8+t1gYRaY2GBu+gOG6cT5x0+eWw994wZYpPz3zmmZr7\npZgpLAjg9xF/+lPvDDlxoif/887z1obTTvOhTPX1UVcpIoVmzhxfs6F/f++HUFPjc7188AHcc48H\nBvVHKH75mO5Ziki7dj6M6cgj/T/7nXd6/4a77/YAccopcOqpvjSsiJSnzz/3Fsm77vJbmZ07w0kn\n+ZTMe+2lcFCK1LIgzerTx0dPzJoFr7wCxx7r8zfsuisMH+4jLObNi7pKEcmHFSvg3nvh6KN9Ariz\nzvLp5e+5xxeyu+UWtSKUMoUFaZGZz652443+Q+Gf//SezVdc4Y8jRvjc7XPnRl2piGTT6tW+mNPY\nsdC9u7csLl3qnRcXLvTbk+PGweabR12p5JrCgqSlQwc45hj42998Nsj77/d7lT//ua9HUVnpvZ1n\nzdL8DSLFaOlSuOMOb0ns1g2OP95/EbjySu8IPWUKXHSRRjSUG/VZkIxtsQWceKJvq1b55Cp/+xv8\n4he+rOxOOzX2fxg1yoOGiBSe2bPhkUd8+89/POjvtZe3Hh53nA+BlPKmsCBZ0akTnHCCb198Ac89\n56MqHnrIl87eckufMfKoo3wiqG23jbpikfK1erVPr/zkkz6L64wZUFEBY8b4kOkjj/R+CSJxCguS\ndZtv7qvHHXGE/4YydapPMT1xos8FDzBsmM8DP3q0tzpstlm0NYuUshBg5kwPB08+Cc8/D2vW+K2E\nww7zJe7HjNE8CNI8hQXJKTMPBsOG+Vjsjz+Gp57yyVvuvttHVHTsCPvu6z+sRo/2Y9u2jbpykeI2\nf76Hguef9/9vCxf6/7X99/eVaA87zKdb1ugFaQ2FBcmrnj19kqfTTvPfdmbM8B9kzzzjP8B+9CPY\naisPD6NG+TZihPo7iKQSgoeDyZM9HEye7AvHAey2m3dSPOwwDwoauSCZUFiQyJj5bzaDB8Mll/j6\n9q++6j/oXnzRO0quWuX3UvfcszE87L2394EQKVfr1sG0aT7/ycsvw0sv+SRqZrD77j6S4cAD/f9L\n165RVyulQGFBCkaHDo2BAGDDBnjjDQ8OL77oK2P+8pf+A3GXXWDkyMZt1119ghiRUrRokYeCeDio\nqfE+Bx06+ARpY8d6ONhvP9h666irlVKksCAFq1072GMP3y691Jta47NJvvaab3ff7aGiosJ/aI4c\n6ccPGwZf+Yr6PkhxCcFXZnzjDV98Kb4tXOjP9+3rQxqPP94fhw3zfggiuaawIEXDDAYN8u2MM3zf\n6tX+gzUeHiZO9KGa4AFi111h6FBvmh061O/fdu4c3fcgEldf74swTZvmgSAeEJYs8ee7dfMAPG6c\nh+C99tJESBIdhQUpapttBvvs41vcZ5/Bm2/6kM2pU+H1131BrPXr/fkBAzw07LJLY/j46lc1bExy\no6HB11B5+22YPr1xmzUL1q71Y7bbzoPB+ef747BhvsyzRipIoVBYkJLTtavfvz3wwMZ969b5OPNp\n0zxAvPWWB4gPP2w8pl+/xvAQDxI77uiT0+iHtrSkrg7ee2/jbcYM/3e3erUf06WLd+gdOdJbxwYP\n9uDavXu0tYu0RGFBykKHDn4bYuhQH7YZV1fnv+HNnNm4Pfoo3HCD/0YI3uIwYIAHh+StXz/vWyGl\nLwRfN2HBAr99EA8Es2f7Y/z2AfgthIED/fbXuHEeCoYM8dsICp5SjPRjTspaly4+LHPPPTfev2ZN\n4wfBnDm+kM6cOb7i5vz5fr8ZvANl376+9enT9Lb11vqAKAbr1/uogwUL4P33/TFxe//9xhYCgG22\n8UAwcKDPYRD/8047+VwhIqVEYUGkCfHOkbvu+uXnNmzwD445c3ybN8/HuM+bBy+84Lc24mECvGWi\nTx+/L92jh09M1dTjtttq9Ea2heBrlXz8sS+v3tQWf27p0o1XSu3WzVuO+vb19Uz69Wv8esAADVGU\n8qKwIJKmdu38w2LAAJ+iOll9vX8AffBB4/b++/6BtHChj5H/+GO/BZKoTRv/gOre3X9r7drVP5C6\ndm1623prn5xqyy1Lf4bLDRtgxQpYvrxxq6vzzqyffuof9M09xjsRxnXoAL16NW777ee3B3r18sd4\nIFCHV5FGCgsiWda2rbcibLedD3drzpo1sHhx4/bxx41//vxz/yCcPt0fP/vM98X7USTr0MGXDI+H\nh+Q/d+rk4/HjW0XFxl8nbu3be3BJ3sy+vK++vnHbsKHpP9fXexP/6tX+PSc+NrVv1aqNQ8Hy5d46\n0JzNNvOQtc02vm27rY9uSdzXo0djONBtIZH0KSyIRKSiorFpuzUaGvyDMx4cPvvMf9tesQJWrtz4\nMfHPn3zif167tult3brcfp+JKir8wz3+mPjn+GP37n7fv3Pn1NuWW3oLi9Y6EMk9hQWRItGmjXec\ny3bnuYYGDwzx8LBhg+9ragth46/btPHbMm3b+tbUn9u1861jR/1GL1KsFBZEylybNv5bfUVF1JWI\nSKFqE3UBIiIiUtgUFkRERCQlhQURERFJSWFBREREUlJYEBERkZQUFkRERCQlhQURERFJSWFBRERE\nUlJYEBERkZQUFkRERCQlhQURERFJSWFB/qe6ujrqEsqOrnn+6Zrnn6558csoLJjZBWY2z8xWm9kr\nZrZHimMPMLOGpK3ezLpnXrbkgv5D55+uef7pmuefrnnxSzssmNmJwHXAT4FhwDTgSTPrluJlARgI\n9IxtvUIIn6RfroiIiORbJi0LE4BbQgh3hRBmAecCXwDfauF1S0IIn8S3DM4rIiIiEUgrLJhZe6AS\nmBTfF0IIwDPA3qleCkw1s0Vm9pSZ7ZNJsSIiIpJ/7dI8vhvQFlictH8xsHMzr/kIOAd4HegInAVM\nNrORIYSpzbymAmDmzJlplieboq6ujtra2qjLKCu65vmna55/uub5lfDZWZGt9zRvGGjlwWa9gA+B\nvUMIrybsvwbYP4SQqnUh8X0mAwtCCOObef5k4N5WFyYiIiLJxoUQ7svGG6XbsrAUqAd6JO3vAXyc\nxvu8Buyb4vkngXHAfGBNGu8rIiJS7iqA/vhnaVakFRZCCOvNrAY4BHgUwMws9vUf0nir3fHbE82d\n51MgK2lIRESkDE3J5pul27IA8DvgjlhoeA0fHbE5cAeAmf0K6B2/xWBmFwPzgOl42jkLOAgYs6nF\ni4iISO6lHRZCCA/G5lS4Er/9MBU4LISwJHZIT6BPwks64PMy9MaHWL4JHBJCeGFTChcREZH8SKuD\no4iIiJQfrQ0hIiIiKSksiIiISEqRhIV0FqKKHX+gmdWY2Roze9fMmpyfQZqX5uJfx8Zm2vzEzOrM\nbIqZHZrPektBuv/OE163r5mtNzPNYpOmDH62dDCzq8xsfuzny1wzOz1P5ZaEDK75ODObamarYrP6\n3mZmXfNVb7Ezs1Fm9qiZfRhbmPHoVrxmkz9D8x4W0l2Iysz6A4/hU0wPBW4AbjUzjaZopQwW/9of\neAr4OjAceA6YaGZD81BuSchwwTXMrAtwJz6FuqQhw2v+ED466wzgK0AV8E6OSy0ZGfw83xf/9/1/\nwC7A8cBI4C95Kbg0dMIHFpyPL9KYUtY+Q0MIed2AV4AbEr42YCFwWTPHXwO8mbSvGvhXvmsv1i3d\na97Me7wN/Djq76VYtkyveezf9s/xH761UX8fxbRl8LPla8BnwFZR116sWwbX/LvAe0n7LgTej/p7\nKcYNaACObuGYrHyG5rVlIcOFqPbiy79lPZnieEmwCYt/Jb6HAVviP1ilBZleczM7A9gBDwuShgyv\n+VH4mjU/MLOFZvaOmf3GzLI2n34py/Cavwz0MbOvx96jBzAWeDy31Za1rHyG5vs2RKqFqHo285qe\nzRzf2cw6Zre8kpTJNU/2fbzp68Es1lXK0r7mZjYQuBqfy70ht+WVpEz+nQ8ARgGDgW8AF+PN4jfl\nqMZSk/Y1DyFMAU4BHjCzdfhMvp/jrQuSG1n5DNVoCEkptqjXT4CxIYSlUddTisysDb5w2k9DCHPi\nuyMsqVy0wZtxTw4hvB5C+DdwKTBev4jkhpntgt8z/xneH+owvDXtlgjLklbIZLrnTZHJQlQfN3P8\n8hDC2uyWV5IyXvzLzE7COx4dH0J4LjfllaR0r/mWwAhgdzOL/1bbBr8DtA44NIQwOUe1lopM/p1/\nBHwYQliZsG8mHtS2B+Y0+SqJy+Sa/xD4Twjhd7Gv3zaz84EXzezyEELyb8Cy6bLyGZrXloUQwnog\nvhAVsNFCVM0tevFy4vExh8b2SwsyvOaYWRVwG3BS7DcuaaUMrvlyYAi+wNrQ2PZnYFbsz6828RpJ\nkOG/8/8Avc1s84R9O+OtDQtzVGrJyPCabw5sSNrXgPfqV2tabmTnMzSC3psn4GtEnAZ8FW9++hTY\nNvb8r4A7E47vD6zAe3TujA8XWQeMjronarFsGVzzk2PX+Fw8gca3zlF/L8WypXvNm3i9RkPk+Jrj\n/XAWAA8Ag/Ahw+8Af476eymWLYNrPh5YG/vZsgOwL74g4ZSov5di2WL/bofiv1w0AJfEvu7TzDXP\nymdoVN/s+cB8YDWebkYkPPdX4Nmk4/fHE+xq4D3g1Kj/woptS+ea4/Mq1Dex3R7191FMW7r/zpNe\nq7CQh2uOz63wJLAyFhyuBTpG/X0U05bBNb8AeCt2zRfi8y70ivr7KJYNOCAWEpr8+Zyrz1AtJCUi\nIiIpaTSEiIiIpKSwICIiIikpLIiIiEhKCgsiIiKSksKCiIiIpKSwICIiIikpLIiIiEhKCgsiIiKS\nksKCiIiIpKSwICIiIikpLIiIiEhK/w+32RhkynI4bQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105a87278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "y = 0.5\n",
    "def loss(p,y):\n",
    "    l = -(y*np.log(p)+(1-y)*np.log(1-p))\n",
    "    return l\n",
    "\n",
    "p = np.arange(1e-3,0.999,1e-3)\n",
    "\n",
    "l = loss(p,y)\n",
    "plt.plot(p,l)\n",
    "plt.title('Cross Entropy Loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Useful terms:\n",
    "\n",
    "1. Activation\n",
    "2. Softmax\n",
    "3. Cross Entropy\n",
    "4. One Hot Encoding\n",
    "5. Gradient Descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./train-images-idx3-ubyte.gz\n",
      "Extracting ./train-labels-idx1-ubyte.gz\n",
      "Extracting ./t10k-images-idx3-ubyte.gz\n",
      "Extracting ./t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "mnist = input_data.read_data_sets('./', one_hot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x, y = mnist.train.next_batch(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20, 784)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "784"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "28*28"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20, 10)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.],\n",
       "       [ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.],\n",
       "       [ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.],\n",
       "       [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.],\n",
       "       [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.],\n",
       "       [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFfCAYAAACfj30KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJztvX+MbNtV3/nd3VXVVd33vr5vzOgZhRExeUFjNAqae5kw\nFvHgGUcicSRg/iG6ATmeaIQYwghdaRKEZI0dPFIUEHmegbwRGmVMEKElS5CBIOxHQsgPT2KceTck\n/HBiQWx+xLwXm+fbfbu7fnX3nj+6V73vWbX2PvtUV91T1b0+0tb5Uaeqzo8631pn7bXWDjFGOI7j\nOO2w1fYOOI7j3GZchB3HcVrERdhxHKdFXIQdx3FaxEXYcRynRVyEHcdxWsRF2HEcp0VchB3HcVrE\nRdhxHKdFOm3vQAjhLQC+CcDnAIza3RvHcZyl0AfwRwG8EmP8w9yGKxPhEMJfBvC/AHgrgH8N4H+O\nMf5LY9NvAvB3V7UfjuM4LfLtAH4qt8FKRDiE8OcB/DCA7wTwKQCPALwSQvjqGOMX1eafA4Cf/Mmf\nxNvf/vbKC48ePcJLL720il1snZt8bMDNPj4/ts3lWR3fpz/9aXzHd3wHcKVvOVZlCT8C8GMxxp8A\ngBDCdwH4cwD+EoAfVNuOAODtb3877t+/X3lhf39/bt1N4SYfG3Czj8+PbXNp4fhqXaxL75gLIXQB\nPADwS7IuXpZq+4cA3rHs73Mcx9lkVhEd8WUAtgG8rta/jkv/sOM4jnOFh6g5juO0yCp8wl8EcA7g\nBbX+BQCvpd706NEj7O/vV9Z95Vd+5dJ3bl14+PBh27uwUm7y8fmxbS6rOL6DgwMcHBxU1h0eHha/\nP6xiZI0QwicB/EqM8XuvlgOA3wXwf8QYf0htex/Aq6+++uqN7hBwHOf28PjxYzx48AAAHsQYH+e2\nXVV0xN8E8OMhhFfxZojaLoAfX9H3OY7jbCQrEeEY40dDCF8G4Adw6Yb4VQDfFGP8wiq+z3EcZ1NZ\nWcZcjPFlAC+v6vMdx3FuAh4d4TiO0yIuwo7jOC3iIuw4jtMiLsKO4zgt4iLsOI7TIi7CjuM4LeIi\n7DiO0yIuwo7jOC3iIuw4jtMiLsKO4zgt4iLsOI7TIi7CjuM4LeIi7DiO0yIuwo7jOC3iIuw4jtMi\nLsKO4zgt4iLsOI7TIi7CjuM4LeIi7DiO0yIuwo7jOC3iIuw4jtMiLsKO4zgt4iLsOI7TIi7CjuM4\nLeIi7DiO0yIuwo7jOC3iIuw4jtMiLsKO4zgt4iLsOI7TIi7CjuM4LeIi7DiO0yIuwo7jOC3iIuw4\njtMiLsKO4zgt4iLsOI7TIi7CjuM4LeIi7DiO0yIuwo7jOC3SaXsHHKcJMcaFt5d5a53M62142vS7\nl0EIIbsuhDBbtqZ626af76weF2HnRmAJpAhnjBEXFxeVZes1nkrjZd5mlaSEVc+HELC1tTVrepnX\nlX6m8+xxEXY2lpRFK8taUC2R5XZ+fp5tss3FxcXKjokFtq5tb28XNxZt67P4+51ni4uws5Hk3Awy\n1eKamz87Oytu5+fnKzsuy3pNret0Ouh0Ouh2u7P51Dr9GdpSBlyA28JF2NlYcv5bABWhFfHMTafT\nKSaTCabT6axZyykRbiJilkvDcjGk3AxbW1vodrvo9XrodruV+V6vh7Ozs9l8jHHuvfKkIAIs61yI\nnz0uws7GkbKC9XyMcSaw3KbT6dy6yWQya+Px2JzK/Kot4a2tLWxvb88Jp163s7ODnZ0d9Hq9yvzZ\n2Rl6vV7FbSLv3d7enutkDCG00unoXLJ0EQ4hfADAB9Tqfxtj/Jplf5dze7GiGrQIsztCxDfVJpMJ\nRqMRxuMxRqNRZV5Pz87OVnZcLJY8b63r9/uVNplM0O/35/zWIQRcXFzMBHh7e3u2XqYuxO2xKkv4\n1wG8G4A826zuV+vcWqxIB17PAqzdDWz5TqfTmfAOh8NZ42Wen06nKzumXOdap9OpLA8Gg1mbTCYY\nDAYzAZZzIR14+hwJIr4uwO2xKhE+izF+YUWf7ThJrKgIbQlb7obRaITT01OzDYfDyvJkMkl+f51P\nNSd2HPEggqunMt/pdLC7u4vxeDxzr7AAix9Ztk8J8MXFRcVH7Dx7ViXCfzyE8B8AjAD8CwDfH2P8\nvRV9l3MLScX86nhf7phjC1jcC9KGwyFOTk5m7fj4uLLMbTweV/Zl0c4sSxRFYFlsU+tYgM/OzuYE\nWCIout3u3HdvbW1V/qikg8559qxChD8J4H0A/h2ALwfwQQD/NITwX8QYT1bwfc4tIxUTbCVlaCtY\nLF/2/YoVfHx8PGtPnz4154+PjzEajZYeRSDiaYWXpULQtAALLMC9Xg+9Xq/yXTpULeWqcJ4NSxfh\nGOMrtPjrIYRPAfgdAN8G4CPL/j5nPVnkhs6lE/O8lcFmJWOwr1d8uqn509PTmQX89OnTmfBaU20J\n52ja4cWCm5vX1i2Ht2m3RafTqVi9DEdeLCLEHtJ2fVYeohZjPAwhfAbAi7ntHj16hP39/cq6hw8f\n4uHDh6vcPaclUtEN2pK1lnX2WmqduB1SUQ+8LCIsPmCxlrW/tQkcfVAqbvpY+fh0hhuHzYlIS3Yc\nf9bFxUUljG06nWJnZ6fSgcfWs95/J8/BwQEODg4q6w4PD4vfv3IRDiHcwaUA/0Ruu5deegn3799f\n9e44LZKr78DiaaUT51KLRSR1Y9cDh5lZjTvfRJw5UUNHHdShBey6QiwizEgnIwuwrhMh57bf71f+\nUHhfdAcevzd3TM4llrH4+PFjPHjwoOj9q4gT/iEAfx+XLog/AuCvAZgCOMi9z7m5WD5cprR2g5V4\nkUstziVe6KkVniav6ciDOlJi1VSILy4uEEKYJYfw58YYK/HNLMB8Xtknbh2HRGR0u93KU4d1LJ5R\ntxpWYQl/BYCfAvAWAF8A8AkA/3WM8Q9X8F3OmlPi52WxKBFaTrLgZT3PscCpxp112kUhr1vWY4o6\nkaoTYm0Fswvi7Oys8vpkMqmErmnBtM6lFcLW7XYrx8gCzfPyuS7Ey2UVHXPuxHXm4EQKa50IsSWo\nqQy3kmmTee2iYBHmjr8UpeJUIsQiwDLV54stYfHn8ms6SUX7tNkFIanOliXM++viuxq8doTzzNBC\nLNNUPC/H9eosN67nkFpOCXpO4K3vLXFHNBWolBBrS9haLzG+2gXBPmSdJagtYB3qxscn29VNneXg\nIuysjDrrV6Yp4WDr1LJU+TWr000X6rEK9+j1LNKWL9ViUUHKWcRaiLU4b21tYTKZVOo+iAXMTxRy\nLrQLQgRYoiX4GHVUhwvvanERdp4J/JhrzWvxYD+tnlphZtY6sfC4s86acnRFaj5lCV9XmCwh5icE\neV2SKmSdjoJI/ZH1ej2MRiMAMAVYrH19fNov7D7h1eEi7KwUS2C0AGsBYSs4l2iRKrIjUxGXkmaF\nw1mjcjDLEiMWYut8aSHUo2GwO4f/xDixwxLg0WiEwWAw12mXOk4X4NXgIuysHOvG1o/Xlk9YRFhE\nVhIprGXrtel0WpTUwa4G3q9UA5YfM6sjG3jK4qu31+dOZ8rJMqcx7+zsoN/vz6qv8Z8V+4R5H6x9\ndEFeDi7CTjF18b68TousFjJeZ4WIsbBaFc14vTUvlnDKqq2zcnNYomi9Xnoe60hZybKO/8i2t7dn\nYsyt1+tVag8PBoOKG4ejQdh/rIdY4nlnObgIO0WUWIhaZEutUMlY000nUFjrrXAy7WJgobWscsDO\ncCtZtqxT/XoT0c39sel1PNKGtlJ1aJu4ebhsp4zE0e12K58ly9y4JgU35/q4CDuNYIG1iuawuJaM\n63Z+fp7tbEs13UlnpRhr8bUsco0W19w0Na/X1Z3P1Lq6qBIWYF0vQjo7ZV67eE5PT2cCzCnPALCz\ns1MpGqSn2lJ2roeLsFOMtnCtUYu1b7ckwy0VetaksSXM4VaW8OrOQY0ulMPLi6zn81dyjnNRJPw5\n1nfwdeLQNbaEh8NhRYBZTC8uLtDv92clMLlxrLFbwcvDRdgpxupEs6ZStyGVIlzSdPJFyXY63rVO\nhK0ogJLG1mdJa3J+ZVrSOSj7zPvOnyHrLi4u5ixhdjmwFX1+fo7BYDDrvOv3+2a6M1dbc66Hn0mn\nGH6srxvBOGe16sSK1PDyqSHnc9toS7hOeC1ynVGLzOfOZ+oc51wodcdhWfn81GG5IPi6jsdjDAaD\nuboZYgFLfeKmHYyOjYuwU4xlCafSf6343tQ6LaC5TLe6ee0TzrkhuKVcCTwChUytdanXmvpMLR92\nzq/NxyLz1jRVcY2vqc5W1BYwxxmnsged5rgIO8WwEFg3LbsccnG8OpQsl9FWus7ahmN/ef/1vJDy\n64q46qZfs7Zt6o5gv7sWYklX1qLM7+fP4W3kumgXBCfKiKXMRYssAdYF4Z3r4SLsNCJlCesavTxS\nhTXlebnhdQhbyTQ1L+Ii1FmKTE54OZSrtDURq7qoE+5w4z8ZnXKsP0M65nTFNf1nOhqNKokbLMC9\nXq8i0M5ycBF2iuGbW9co0P5eHrNNpjL/9OnTymtSJzdnAZbMW6/p/c8tA7Y7Qotvaj4l0k3IpU3r\nJuiYYG1RiyWsoyAsAd7Z2TEFWCzgfr+fLWbkNMdF2ClGbm4rxVhnvVkDZ8pgmUdHR5WBM7lgespv\nW7ec85OWol0R2sXASQtNpk3O7/b2djLJRebrQt9YgOW9Z2dnZiccCzCHonEUhAiw5apwro+LsFOM\nvrmtOg+50YuPjo4qTdY9a6vK8tNalq9uPIoFb5Obb2oJp1wr1tTKlNN/Qizcsl7WcblLTsgIIczV\nmdjd3Z0b6slZDi7CTjGWCOesYZ3JZoWONYGFpOn7rIw2nvJjN1cf00POc4LDKizhOr83r0tFplhu\nB/4OfR11NIfu5NRuELeAl4uLsFOM5Y7gzjlLgHM1HZrczGzxcVJCyftKmhS5kUwynS3G6zjG1hJd\nPd/k/JZ0Osq8LngvHW+WAFsRE1qIZTtdY/k6MddOPS7CTiP4xtX1a7lzLmcJc02DElLug7r3p7Lc\nrHl+/JZHcJnXjevzlkRPlGKJcG6ZY63lj4EF+OzsbO7ciQBLJp2IdE6EtRXs4rtcXISdYqw44ZQ7\noqSmQwm5ONtFhDgV38u9/1Lqkac83+v1GoWoNTm/luim1g2Hw0oFNJ18oV0TljWsv19EOCXALsTL\nx0XYKSbVMZcKU8vVdSi5kUsSHUqEuE4kZdh3qbk7GAywu7s7a7wsdRVyCRr6tSbnNyW4VtMlKAFU\nBFj7pFl8rXPLImy5I9wfvBpchJ1itCVshanxI/J1fMJNMs1SQlwX88tTEeGdnR0MBgPs7e1V2u7u\n7my+3+/XWtY83+T85ixfbmdnZ2YFNL4enU7HdEdYmXYxvjmOXa5jzv3By8dF2GmEFuBcwgYX52ni\nE25ab0HekxNiS4D1yBM87I+I7p07d3D37l3cuXNn1vr9/pzg5qalsAiXNO0DFgtYBFgX6KlzQ3D8\nsLaE+Zq5AC8XF2GnGMsnrGtHpDrmUnV+NYsIML9Xf2bKEuaY35Q7QgT4ueeeq0wHg4HZuZfq/Gty\nfkvEV86j5QMej8dzo2VonzCAmRBLZtzFxcVs21zHnPuEl4+LsFNMzh1RF6Jm+YQ11xFg/gz5bI4F\nZqHUCRicFcbuCBbh/f39WRsMBqbYWstNj8myQFPrWLS5OD532FnuCEGEXJr4irUA6z9OF+Dl4iLs\nFFOXrKFD1NhCrvMJL0OA+bNYiFPZcDwqsWUJ7+3tVUT4+eefx71797C7u1scf3xdEU4tS+owVz+T\nbEUdyyxwNl0ueSVlCXuM8GpwEb7FyM3Y5IbSyRocHaFD1LjgeqrYOlAvwKnX6zr2UuJrWcEcopYS\nYRHiJiKc2jf9ByTLWnhz81IZTTpEh8OhOXindc5y5640Y66pGC/zj/Ym4SJ8y7FuotS6VM+9LsBe\n4gNmceR1dfPWskZeZ1+vCG1qWXzAu7u76Pf7MxEDMKtCJtXhACTD06xICb1f+vzW+bFF7HhcN1m2\nkkO0dSvbN/VPW64n3Q/A38N/PNY6x8ZF2AFgj+zL89oVYT0e61EuLAFmcZGKYbI+NbVExZrX6zgV\nWTexGKUNBoOKCPd6vYoIy7A/w+FwJoC5wj3y3lyccM5STgkxXxctwHVWeOmTh3WtU4k5vL/WPsjx\nuxCncRF25h5Rram2iixLWA9LlHM/sBDzOi2+1jqZz60DUIn95abXiQhLQoakJkvHFnd8yR8HR1jw\n9OLiYjYIpvXnIFgdiNZxcYiZhl0rKTFOXe+cKPL1ThUMEiHWLh/LIm8aqnfbcBF2ANhD4+hl7RvU\ngqvdEbpjR2DrzhLhlFWVEunUcrfbndWAaDqV+hBA1RIWUeaKajzuWrfbnR1nSnxEgOssYRZgy22z\niADz9a4TYqsTVnfEyn5wDDbPa/+3M4+L8C1HP+LmWl3sqm652FL9qJoT4Jwg51q32zVrP1jLIrzc\ntE+Y53u9Hs7Ozmbb6uPb2tqaxd5agsdCrF/PPdKzwJYIcJ3QptwWOX+wtoTlz0CeENh3LfvT1Cd9\nm3ARdmrFV1vBOQGWm5TdEJY7whKW0gy0OnGW9eJmKGniA9ZZdcCbljCHhJ2dnaHX6825W+T7JaPN\n6nTj82AJsKzT50nieKXpaA9LkOX65q59EyHWPuEQwswNk3I5OXlchJ0ZlvCm/INagLW1lKszwL5C\nWS6pwaAFJpeptrW1VfH16qI8eh27EXgfAcy5U7a3t83EE95HsQytYjlaiDXsUpHzJGLJQtzEHZGz\nRPVrdR1zLMRi8Xc6HTM224v+1OMi7Jg+YG3Fan9wnTtCF4hhtJVmdeqkmiXMqXW6IhoX4tFTseY4\nQ4z92TwvfuFc1Adbhxa5R3MtvtzYUrZ8scsQ4tQfr+UTFhG2BNiK6nDmcRF2ANguCUuAc8LLvefa\n0uOpZelZYpJal4vL5XUiwroq2p07d+aWt7e3Z24GOQbgTSuY12sLj0Wn0+nUpmdrUu4IvY7/KHVI\nXEmImry3VIhz2ZEyFaHl8yDnwtOcy3ARvuXUdcxpAS6xgkXALD+toF0IdeO0lQiybpz5JmKbatvb\n25VsP+BNXzCHqI1Go9lrfIxsAXe73TnLWftG9eM/r7M61qzr1CRZQ1/znCWe6ozlP1lJUddRG9on\n7kJcj4uwkxXepm4ItiJZGGRZC7MW3JL5lCjrZcsdIUV5pDylzG9tbeH09HQWGywCDFQz5obDYUWA\n+bs7nU6yUJGODKnzCVvWsL5mVqdcnSXM7091yMk0ZwlL4xBDPhdNngRuOy7CztxjbKpXW9KAucbC\n3t7erE6EiI/ceKVuhJJRi3NWcOq1nZ2dWRacjIjBoyWL2IlgcC0GXYxoOBzi9PQUJycnlegFrkMh\nYWscnqdFWAusJYba9cBTfl0nxViJMblrnltv/VGypa8blwbla1L3Z+C4CN96tFXG/jwtxlzysd/v\nY29vb+YfZAFgyzflt60T4iauiNTnS00IiQfWmXAAKhZvToBFhE9PTyvnhpM1JpPJLGyN3RHsN7VE\nmdEdpLrJa5wso7+vTojrhDEnwLroEYuwFuISH7XjIuxcYXWgiRiLqPKIxIPBoCLAbP1y2m5OLPVy\nk445KyJCz1sZc1xdjONgxeWgq8FpAT45OamkLXMRoFy1OBbQOmvYCg+0Okv191237q92f1juFn3M\nIsBaiLV7xEnjIuyYYqDDpIB5d4SIgNz08nguY5txh1ydYDZp+nNTU/nTkMaZcNoSPj8/N0cHSVnC\nug7xzs5Osni9FQKYE2Dtm9d+enGhNB0BIyeGdQJslf+U86qtYC3CTh4XYWeGjl6Qm1+W+eYbDAaz\nG1/ey6/Le3JCvIggWxEXel5HK1hNrHt+rNfF6S1rWCxhLcAykohVPc4SXr2OyXWQcoSKZQnLU4m2\ngksFmNfxebQEWLsi2Ap2IS7HRdjJ3iQixjHGik9YCzDX7u33+7P3WhZxalpiLVudiKmW8zVrSzg3\nVh5bwWIJc0lMfjLIWcI54RVKQgP5j0ML8HXcEfx7sFxFlgtGuyFYiN0fXEZjEQ4hvBPAXwHwAMCX\nA/jWGOPPqW1+AMD/COAegP8XwP8UY/yt6++us2y4Y06WrdclJEo6nrhjyYqaqLNWm7gV9Dq+sS3x\n1et5mefZx8rpuNoK1tERLMJSAIiHceIoEf6jynW2MdolUVIwKdUxxz7o1PVPrbcsYfYBixDrGhZu\nBTdjEUt4D8CvAvjbAH5GvxhC+D4A3wPgvQA+B+B/A/BKCOHtMcbJ4rvqrJLUIynfxFyyUV7XFvBg\nMJgVdmniOqgTaZ6X766bB+YFzYqJFhFuYgnLH441qrQVLpYTXiv8LGUFc/KI/r6UH7r0evNrTSzh\nXIKNW8P1NBbhGOPHAXwcAIJ9Zr8XwIdijD9/tc17AbwO4FsBfHTxXXVWBYttap2IcE6AOaUVmE/O\nsAQ4J8qpZfl+3ldrKh1YuulHe10hjDPnUj7hfr+P4XA4+9PJjSqdCjNLiaQlxFa1ulzpULlOqetd\n8puwBNjqmCvx4ztpluoTDiG8DcBbAfySrIsxHoUQfgXAO+AivLawcFlixqIrN2a328XZ2Rn6/f7c\no7G8Pye+VivZztpva5nFVRecF5GyfMJWp5yOjpCaFCLWJR1zLMAafuKwfMKp7MQ6n3CTDjreRlvD\nuRjhnDvJBbieZXfMvRVAxKXly7x+9Zqz5qTcEgBmUQUSHZDrvS8R3JIm37/IY61kwIlASkSEWMjs\nc02NGq3dEWIJSzq0dkdY4+uxuJYKcsofrP9MchlzpeFpFpYlnHJHlF5Hx8ajI24xTW4OsdTq3Aoi\n0qWCWroNr2+yz6n44pxlnbPSRcibRGzIvnBHHe/f+fn57HO4Qlnd9OnTpzg5OcHp6WnFIrdSyDW5\nJ4i6KBWe1xavC3Bzli3CrwEIAF5A1Rp+AcC/yr3x0aNH2N/fr6x7+PAhHj58uORddK6LddOJ+IpI\nNRHV1Dr9ndfdX0tUtaCkrD9rMM9cVIAWJxZgmbeOny1x3VGol4+OjipCPBwOTbdI6jzyOedpSWSK\ndZ1SonvThfjg4AAHBweVdYeHh8XvX6oIxxg/G0J4DcC7AfwbAAghPAfg6wH8rdx7X3rpJdy/f3+Z\nu+OsAHm05ikLsEwt67VOcK2behn7y/Mp69bq3ee4V/aJctqyjo3lGFndKSVukNQfTIyxkiiim/ZX\nP336FE+fPsXx8XHFGrbSya3v0+depqmnB+u4ctf5tmAZi48fP8aDBw+K3r9InPAegBdxafECwFeF\nEL4WwBsxxt8D8GEA7w8h/BYuQ9Q+BOD3Afxs0+9y1hMWYIEFmGOI627S3Do9f539rWt1ljAL8aKW\nsEy1P1imFxcXlagMnrfWiQinLGEdJVH6FJIT4Nz7VnHtbgOLWMJfB+CXcdkBFwH88NX6vwPgL8UY\nfzCEsAvgx3CZrPHPAPxZjxG+WVg3mAgwi3Bqmnst9fnL3PdUOJUlrJYlbKXqpsKzUrHJPC+CyQki\ndfMnJyezxpZwnU+4yR9SnSsidy2dMhaJE/4nALJDqMYYPwjgg4vtkrOulNxcIjr6PU1EdxU3dVNr\nuFSESyxhtnitQjscr8whcbkmlq9Ebcj60Whklhetc8tosU11PKY6Ntv4M70peHSEsxD65mKLT79e\nIrS5+WXta4kbQvuEm7gjcj5hK/nCCj+zBNYSXLF8tXWc65hLuSMs8U2dmzoreNnX7jbgIuw0RgSX\nb0CdbJC7KUtfs5YX2Vc9tfydKbeE1TEn2YO5Eo5asIA3hdhKvJD4X7FoxcWQm+qKbxxFUdoxlzsX\nJU8PTa6lY+Mi7DRCC3DqdZm3Xl903aLUWcIsPnX+YKmdUeeKYCEDqgKcyuSbTCZziSEnJyc4Pj6u\n+H+lcYKIns+FqNWdg5x/OyfCq7h2twEXYacxLCza0rQEuuSmXPWNm7PkmnbQWR1zWoz5s4H5LDix\nfjn5glOlRYSPj48rjddZ1rTOoMsla9SdA8sfbFn4LrrXw0XYKYItXF5nbbfKfVjkPaXCm/MJ5+KE\nrephOkQNQMUnzBawuBCsFGkRXYkH5qnu1LOmXHCprlkCXOKaKLkGuYputx0XYaeYJiKYK9d4nW3q\n4H1MlXjkbVPiw5+jw8u43gT7ZYfDIXZ2dirFzi8uLirZbrlMuNFoNBd2Jh1xHCMsYWhW7Y7cMeo/\nDu33lnb37l3s7e1hb2+vMkiqDGUkfzb6fJdeF6eKi7CzUnLJCYtOgbLOPqmQZlUX0+Kk3Qi8/1Y9\nXxHQ4XA4V1eX3ychY3VNLOKUAOt6xXUjaGh3iK6AZs3L9N69e9jf38fdu3dx586d2YjV/OfC/m7n\nergIOysjV0c3ty63LWNFPvBrOhxMRIu3kQQTtoT1/ms3gvhyR6PRnDDJ+2T709PT4oI8umNOIiVS\npTJT1djkPMixAW8O0soDn4p1q9v+/j729/fx3HPPzURYW8N8npzr4SLsrAxLXK2ssSbbAOUZeFZy\nhE6pFjGxLOGUALMbYjgcmhawbM+jMFuNoxokRE0Pq5SyhC0B5nMgcAlKHhMvNX3uuefw3HPPzVnC\nOzs7FYvfLeHl4CLsrATLmuUU3Vz6bm57oNrJlFvWlqy2qNmaq3NHaJfEeDyeG9RSC/B4PJ6JsK4D\nzMKrO+nY98vLsg2HnlkCzO4WyxKWYahS7c6dO7h79+7MN8yWsLb6nevjIuysDBZiFkJtmTZ5DciL\nsH4tZWXzI7vunOP9l6bjeyeTScU3ytvI6+Ku0MMR6YLsLMQpn3FdiUrG8o/zkEQiwiKwu7u7lfk7\nd+5gb29vNtWWsPuEl4uLsLNSLBeDxK9adRTqGlDN9GoaOmVFDgDllrCIJbsgAFReZ1eFDAGVi+fV\ngpxrLMKW2KbmrRGxRXClsfDK69LcJ7w6XISdlWCFdVmNhbhkHkjXPbDWyfZ6qtdpEWb3gpXpxpag\nRGGwBSy+106nUxHbumlKnPW2IsJacFPHyJbwzs7OzPoV14P4gO/evTsLS5OpNO0TdpaDi7CzUiwr\nWE9T89b2I2nAAAAgAElEQVQ6oDqKc9N5S6hyqcYsxCyILNKpiAl5dGfxzDWpAZw7J3rsOu5cFKyn\nAssSFhGWjjiJiuj3+5XoCZ66O2L5uAg7KyNlDVvCUtdYhFlYc1NpEoLGIWlagFM9/lac8HQ6raxn\nH7Fu29vb5nHkxDXXiclTjsiQ8yJT/edj+YTFHXH37l3s7+/j+eefx71792YWvB5Vmf9Y3BJeHi7C\nzspIhZ1pH6tlEaaWgfmBKFlMtQDrFGRBCzCnHKcsYe54YwHWqcs8H0KodbHwMp87nqbm+U+Fsf5k\nrI45sYT39/dx7949PP/88+j1emYmnU7Tdkt4ObgIOyshl+hQ5/PM+UgBJEXXEuVUzV+9bjQazZIk\nJEWYs9n0sPJyjFtbW7PjE6tXuzZSwms1S9hSAmtZ8tZ0e3t75tfV/l5ryn7fXF0MZzm4CDsrQfuB\nde+/Ds2yptY6YD46IueSSFmoejoej/HGG2/gyZMnODw8nFUsk4y1XH1eOV4rbIxF2HoySGFFOOh1\n2jrVA4/yPIeaSbgZZ8zpinDWH5vVoelcHxdhZ2VYoV1Wym5pNpn4YnOdbvo1bcHxMs9PJhM8efIE\nT548mQ0lL1Zxqki6FlP+42GshBH9HqY0qsOq8CY+W73MIszRDroT0aqLXBr+5yyGi7CzErQlzB1Y\ndYVscsvi/6yLE7b8w1YT0ZlMJjg6Opq1OktY9kMLqnSYaf+u7lizxJjRwmsts7XLYmq1OhHODVjq\nIrxaXISdlVBSd8Ealscq+8hTjmwoaVpMUsvT6XRWLF1q9vIw8ilLmKcswCxUOg27ToCFumPjmhC6\nOI9unBGn3RFsPYuwW+4dd0esBhdhZ2VY6b5ahLlOgq6ZoNtoNKpEAqQstJRFnJuenZ2ZY7lx3QYu\nnKNFVrCSKKwIkTq/cImVzyKs43l1syxhEWgRccsfrL9T9s1ZHi7CzkpIWcJagHXBcpnnxuvZykz5\nSnk+5zPWIsyjFuv9ko7BOuuVOT8/r7gsSjvmdAdczt+tLWGd4SbzUqSdhTjXMZf7E3CWi4uwsxKs\n6AhtCbPolU45dbl0WtIuLi6S7pFcdIS1rP8IrI641HJqv1ORHyKebAFboWe56AidhJFyP7gQrwYX\nYWcl6E45XYGMLWFuXMzcWs8dXloMtAUpU0tI9PLFxUVtqUndMSefYXXWaRGW1/X7S8LULCHmsDv2\nB3M8sGTFccW0uo45Tmqpe+JwloOLsLMyLEuYO9306MI8ogQvc5NH/BQpYU5NZZ7917m0aiv8TC+z\n8GoRtt5nRUbkLGJu2h2hq6TpMpUizpYI58aPS80718dF2FkJuWQNHtjSGmFYT3leUpebYImGtS7n\nu7X8uLnEjTpKtkl1yrElnHNHiACzFcxlKbU7gv3BqXOk98+5Pi7CzsqwUpetFGadTaetZZ6XrLll\nImJSIowpSjrZeDllkVvZfal5EVe2dFPL7B8WAbaSNGQfnGeHi7Cz1lgidh2xzH3+qj67rpOLG6ce\np+ZlylauJci5aAjtB/Z6EO3hIuxsHMsSy5SbYtmfnYtw0NPUcPTWPFu77APW8xwNYbkgOBrCefa4\nCDsbyXXFsq5zb9mfXZI6vbW1VbFU9bxelxusUw/mydlzLOTaFeE8e1yEnbWlpGNoEbEsEZtlfbbu\nVEuV1JRIB+5gs7LgdEacHoIo1VjItS/YreF2cRF2NpqmYtlEZJb12VZsr1V6stPpVDLdOOMttY4j\nIqx0ZVnPFdW0Jew1gtvFRdjZeErFchGRWcZn6/AyqwSliKNOtkgVXhcXA3e25eZ1J59OznAruD1c\nhJ21pKkg1InldQTmup+tXREsumydSsYbd67pKc+nfMXWVI8mYo2Y4SLcDi7CzsawqI94GeKy6Gen\nfMK63i+nHVuJFtZUD8aZW67rEHQRbg8XYedGocVymcKy6GfnXBEswNLRJrUeZCDO1FS7FiwXhzQO\nhdPzvOw8e1yEnY1gkQ61VYjKIp9tWcKWAFspx3fu3JkNSy9TmbdcDNYoyZyKLPuTmzrPFhdhZ6Wk\nah7kmiUi0hat19AkyqFu20X81SkB1uFmVuEdtorZEmaXQt3UBXZ9cRF2VoIVlqWrfelh5LmlKppJ\n7YiSimSpspF1UxYsS5CbWMLyJ6TLTlp1fweDwUxkcyUnJaysblRkF97NwEXYWRlsAWofqAhwTnit\nZo1ukarTaxVOLx3dwipFqT+/qU9YW8DsemAXBEdB5Or+5gbkdDYHF2FnJeQewbUFXCK+0qTgeE5k\n9TprpGNrnqupsfimIiNKhZhr/3JWHLsfSgrvWCNg+MjIm4+LsLMSdGeUtoR50EztishZxmIJpyxa\nLb7Wd4h4yrJ2P3DnW4mvuSRMLeeO4E44qwaEHgtOpxqnoh6czcBF2FkZOiyr2+3OxFQ/4qdEV4uy\nJcLWUPJWHWNe1vCIGZYQ5wTZ8iUzOXeEjoJgX7BV/5dD0lLhZh7tsFk0FuEQwjsB/BUADwB8OYBv\njTH+HL3+EQB/Ub3t4zHG91xnR53NIpWgwNaobMcCnLOApaXcCXqdtK2trcpny/cC1YE5cy6JEiyr\nWD8NpCxhCT/TNR8sd0TKCnY3xGayiCW8B+BXAfxtAD+T2OZjAN4HQH4J4wW+x9lgLJ+wJcBahHMd\ncjLVbobUVD5XhEpbwGw1a/GVednPUjGuE+KUJSzWcKqKmjUWnGX5uhhvHo1FOMb4cQAfB4CQvsLj\nGOMXrrNjzmZj+YR1NIK4K5qI8HQ6NX281jILsBYltpi1WOmsuCbWsLxfJ0dY48FpS1hEmJM49DL7\nhOWz+Xzred7GWU9W5RN+VwjhdQBfAvCPALw/xvjGir7LWVPYJ8wCrAU6xvmRjrX1K/Pb29vJjjxL\nhM/OzkyRlSZ/AimhairA/D75TD4PbAlbKcq6poSuB8HuCDmXPLXWNQmnc549qxDhjwH4aQCfBfDH\nAPx1AL8QQnhHXPQX7WwcWmhT68W9UCfCHKJmJXdYkRU6U0x32q26cA3/6bBPuNvtmtERd+/era0B\nwe4I65w7m8fSRTjG+FFa/I0Qwq8B+G0A7wLwy8v+Pmc9sUSYRYmtQ23tWtERIpzsW9aiq9fJCM6d\nTqcynU6n2N7enk2l464uppjXaXL2hXaHWMV8tL/XqvnrFc9uJisPUYsxfjaE8EUALyIjwo8ePcL+\n/n5l3cOHD/Hw4cMV76GzKizxYWtUxDmVScefI6I1nU5NX7AVdyzCPp1Oa+e5w48jKVIin4pVtmKX\n2TUynU4xmUwwHo8xHA5xcnIy63yTgT51k+/udrsVEZdzU3IdnNVxcHCAg4ODyrrDw8Pi969chEMI\nXwHgLQD+ILfdSy+9hPv37696d5xnjCXA+nVLgK1H+U6nUwlRS3XMyToW2FQTIdZuj9x8ygdtLQOY\n+bzl+8bjMUajEYbDYSUGmAf6lHZ+fo5erzc7F+xHtzoNLcF1n/BqsYzFx48f48GDB0XvXyROeA+X\nVq1c1a8KIXwtgDeu2gdw6RN+7Wq7vwHgMwBeafpdzmaTsoT1NmIJ7+zszHXg6RhbHSecC1fT8cXc\nUq81sZp1XLO2yEX4LEtYRFiiHyTkTHzFuT8kFnc+j7zORXdzWMQS/jpcuhXiVfvhq/V/B8B3A/gT\nAN4L4B6Az+NSfP/XGOP02nvrbAyW+Mq83k4ElgXHSnAQy7AuSUPmUxEW1jrxFUubTCbJeetzxK8s\nyL7w/NnZ2cwVISLMRXlCCBXLnMWcOzK73W4yjM4jIjaPReKE/wmArcwmf2bx3XFuIiy+VjKBdLbl\nLGAJ62IRtoSXW4lrQebZVzuZTJLzunNPBFhC4XT8sQ6X0z5hzn4TFwofI1AtAGRZx3reShZx1hev\nHeGsDJ1AwO6IlK84JcAifE06xeqSQLQIj8fjWRuNRpVpt9vFeDxGp9PBeDyedRKmwuBYhC2fsK6G\nJu/VIsvnYjqdotfrVV7nc81+YmvqrCcuws5K4YQFFlvdsZTy/+rGAqSjECwRTgmxXj+dTjEcDjEa\njWaugtFohF6vh9FoVEmSSI1SzMLPoqdD5sQnzMMO6c48OWf6z0i7KVxoNx8XYWclWIKgRUOaleKc\nsl614Mq8nmqXhBV6xq9NJhMMh0MMh0Ocnp5iZ2dnzmebqt2rRZ9f15bwZDKpxP3K/sr+8LlK1WHO\nWcL6PDvrj4uws1LYHaGtX46jFQEuCT3j96bmtUvCivdlMZ5MJjg9PZ0JcKpgDh+XZQHrehWWT5gF\nmF+ToZssAc5FTHin3GbjIuysDEuABRZOncQhr+um32ct83pLxFPJF5PJpJI4YZWNtI5NuzzOzs4q\nIiv7wiLMYsnuEOnkYwHe2dmZdQ7KNtoS9vC0zcZFuCVS4nFd1uUGXIce+pQ1bVna0unGlq81bhu7\nUHiZLdqcv5gLClkirIv8jEYj9Pv9Smic/IFYkSYW6/KbcGxchDcAf7xcDJ1RpmOXRch0HQeJSe73\n+2YKtQilrulgiaGuA6GjIUSYgcsnAum4sxpHb4zH47mRlvW87K+z3rgIrxm5x2vGb65ydKicDptj\nnzRXOWOLmcVbF9fRIgxULWYuwmNFREiHXAihSHwlZpk7CXk/OLzN/8DXHxfhNaHU1+n+v8VgvzQL\nLwsWW8JWvK4WYEt8AbtgvFUBTdwRsm2M0RTflCjzn8DFxcVcsXf/bWwGLsJrQF1Pv+A94c2xzhmA\nmXDJPHeGaQtYv65jhutEmPeFp3o76SDU4mtZwqPRqPKHwJ2X8h2y78564yLcMqnwKut1wWNCm5E6\nNxyVwe4IK2ONO8yk4lnKBcEREyVRHxxLbFm9lhhPJpPZ9/DQUbI/2tJ31hcX4TUhl3gA2GmpTjNy\n0QM8ICmfc91hN51OZyLMWAkiuuymVaSem/iErc45EV6eWjHDbAG7AG8GLsJrQE6Ac5awC3MZKdcN\nZ7SJv1fW64QJTqOW4uqMTtjg2sOyHMKbIz7r0DZZXye83HTUBu8319hw1hsX4TXDEuBUdpQLbz2p\n85T6I5PXuGrZ2dnZrLNOiqznXBCcAXdxcTGLARasEpdiNecsYD2f8gGfn59XhNhZb1yEW0T7fi2/\nod7OO+eaUyrEUtWM06d1Zt3FxQV2dnYAVIvu6PRjLgTPyRniq+X3S7KGdMyV+oXlGDj6gvfdBXgz\ncBFuGSs0ra6zTvfEuwDXU5dNxpEQuc4zsVa1BcwWLVdLExEGqnUi5Ds5ndnqmMtFSMj+cwQHZ9S5\nO2IzcBFeMov86Otueu71rktVbSLKN128V5XGK5EJ2v8rli93rE2n04rFK/Uh2DJmCzqXMaeb+H95\n1A0ukO/W8GbgIrxEUqFlKcs298irw5x0WqxYP9dZvukivEp0RIXUetjZ2cFgMJgJcowxmSXHFdWs\nOGPL18yCLQOfcmyzuyI2DxfhFWD5d2VepmwBWSM98LL4EVlIc1Or+LieehD/4rAPlqMnZJBOHieO\nn0z0ddeFgmQb3tbyOevx8dgF4Vbw5uEivGRS7gT9mvUIm5qKCFtFWvQ6ts70VGJgO52OW8HXxLKE\nc3V/dfzwdDo1RVi2teKOtSVcJ8DuE94MXIRXgCXC+sawfIc6UF/mpVB4qmCLXi/FyMVCs7LA3BK+\nHilLWAswX3/t960TYcsS1hEYKSF28d0cXISXjLZ6dXaU9gmmYkB5HRf7Trkb2AoWq8wak4y3dRaD\nkyLYEk6NfMHXW655qmi89fvRPmFLiLlvwXoKc9YXF+ElkrqBLKtGD38uA0xajQuFc+Usa77T6ZgW\nGVAdTNNvzuuhLeGzs7NZ+Us+35IlJ+PLyfXWY9eVWMI6DE4Ls+6Y82u8GbgIL5mUK0JHQfDw5zK6\nb6pJL3pJk6HR9TA4nIbrnTbXQ1vC3W63UqyHXw8hzAkwj1+3SHSE5Re2BNiv8WbgIrwCcgIsU31j\nihCfnJzMBpw8PT3FyclJRYS5mLi1LIKg6wroUo1+g14PtoQlldl64gghVJ549CCiljsi9yeeEmB+\n6tEdgs564yK8ZOpCjDjUiG9OGWr99PQUx8fHODk5mU1lmPRUxAOv6/V6SQtYHptdhK+PtoQtAZY0\naL7GdSIspDrlUh1zbglvLi7CKyBlyeibSURYfL9i+R4fH+P4+BhPnz7F8fHxbBBKFl49KCWXW9SP\nxLkRI5zmcPKLLn3Jo2+IL1+edPr9Pvr9fmN3hP7ztqbuE95cXISXTIn4pixhcUeIAB8dHeHo6Ghu\nJGDd5IYWoQWqgiDFyPmG9Rv0eugoEz7fksUmIiyupn6/n7SEhZLfT8oS9uiIzcRFuIYmP2Ld6Zab\njkajmcV7eno6s4YlJE3cBsD8Iy7H//JoDxKrure3h93d3exN78ka85Reax1mmLvOw+EQT58+nfn6\nh8PhLPRQivb4k8ntxkV4QaybxgrIT7XRaISTkxPz5sxFNojoSq0CjgmW2gUiwru7uxgMBrPtuBPP\nRbhKEwHWGY86uYbbcDjE0dHRzL8/HA4xGo0q19qfTG43LsIN0TcLL4uFpJMweHBGnhfxlTYej2cB\n/Vx3QMekiuCKpStNfI6DwWAmwLJNKjnAqce65qmEG32dT09PZ7596XiV13WNCed24iLcAC24el5H\nPnDoGU/1PN+4liVsDTbJosviy1NpbAlLj72LcD2WMKYsYX1dZcoizJawWM4ereK4CBdiia5lBevs\nKLkhxerV1pAePcGyjix3hIjwYDCoWL7ifmDrmP3CbgmXUfLEI64nK8xQ5k9OTkwRdkvYEVyEG6IF\nWKcqpyxhuSGlSSYcd+TwMrsjrFAzrl27u7tb8QOzn1iad8yVUyfA2h2Ru84c7y3iLJYwP/Vwco1z\nu3ARbkCdAGtLmH2/HHp2fHw8qwlh1RLWN2bOJ8wifOfOHQwGg7noCY6oECF2ES4j9QSUS7iRqBeO\nfpHfgPuEHY2LcAE5VwTHYsrNKRatTkcWEX769CmGw2G2wI/lE7YKiGsR3t3dzWbUuSVcTuq6szuC\nfcLaBSFN1wZxn7DDuAgvgL4hrY45yycsInx4eIjRaFTkZ7aKxeQs4b29vaKyly7CNqnOOD21oiP0\nU8/R0dFMhHXUhFvCjuAiXIjlJ9RWcCo6Qrsjjo6OMBwOs2O/6bHhch1zlginxpbjeSfNor5/K+Mx\nFSvuyRoO4CLciFQqKK+3fMIpS1hbrLnx4Ep9wnfu3Jnbbxfc65GyhFM+YXFHHB4e4ujoyOx41Z2w\nXu/h9uIiXIDO5+cpz/ONxqMopKZ6zDeuIaBHy9AlK3Ni7VySErSUyyHno+d1o9Fo5mqQlGQd/aBD\nDllwrWI7ACpPKPpJJvd7sMYb9BG1NwcX4RpSBVWsm1QLrdX4hpSb5OLiciBPnSFXcsO5e6GM0kSb\n1MjXvG40GuHw8BCHh4cVIdYizNmPTcaCS4mvVUlPF/X338Tm4SJciDU6hq5exb4+LcZcU0JuZrlh\ntE9Q+4GtG05bPPI+Z56cf1emVuGl1LxYwuJuSBVi0tZvrtqZYPUH6D9kq5RpyhJ21h8X4QK0Jawt\nG7mpWHxT1jCXIbTGfNOdcVYxd3ZB+A2XJxVKmOto0x1onNkoES/sjshVSSsdC84SYh3VooXYsoK5\n81U+y1lvXIRryAmwLqydE19LiMUK1paw7oxLWcL+6FmGJbq6WR2qqelwOJxZv6mMOF0ljX87OXeE\njmJJuab0b8It4c2lUS9OCOH7QwifCiEchRBeDyH8vRDCVxvb/UAI4fMhhNMQwj8IIby4vF1+9lhC\nzO4FbT3VWcBWJ03OEq4TX7/ZyrDEl63SVGihhJs9efIET548wZe+9CU8efJkFoLGVdKsZAw9NL3V\nuSvUdchZLgn3CW82TbvS3wngRwB8PYA/DaAL4BdDCAPZIITwfQC+B8B3AviTAE4AvBJC6C1lj1sg\nZQ1rIbZ8wZZfUd+U2kdZ5xP2nvBm5IQ3ZQnrcDPxAX/pS1+aiTD7hLU7Itcxl/MJA+XuiDoB9t/E\nZtDIHRFjfA8vhxDeB+A/AngA4BNXq78XwIdijD9/tc17AbwO4FsBfPSa+/vMaSLAurh3LjKixBLW\nveI5i9ixybkhLBFOJdmw++Hk5GTWAZcqUcqWcCqkUf8Ba19uXcdc6vfgoYqbxXV9wvcARABvAEAI\n4W0A3grgl2SDGONRCOFXALwDGyjCQLkQNw1R04+nQurGc4tnMUqEWNwROgWZ631wJ5wu2K8bV0lL\n+aFzlrAWVO2O4HEF/Xex2SwswuHy6n4YwCdijL95tfqtuBTl19Xmr1+9trFcV4C1PziVKWWlKusb\nLXXDOVV0PLDlkpB5XXiJB15ld8TR0RFOTk6Sach6WUSY94H3rSREja97rmPOfcKbyXUs4ZcBfA2A\nb1jSvqwl1o1bJ8SpZcsnbNUOSIWopR49+X1OFS1yKTGuK0EqIvzkyRMcHx+b8cPWulRxHmudUNox\nl6qO53/Mm8VCIhxC+FEA7wHwzhjjH9BLrwEIAF5A1Rp+AcC/yn3mo0ePsL+/X1n38OFDPHz4cJFd\nXAmp8KFcR5m+EVIdRFZ2lgi1Fm8t/rzc6XRm+2rtf245t77pDZ0Tmdx2WjBz83WWZSqpRk8vLi4w\nHA5nlq6OepBON45+sK5Hia9f+3x5eXt7ezY6ijQ9hJUeV5DrROvkDRfiZ8PBwQEODg4q6w4PD4vf\n31iErwT4WwB8Y4zxd/m1GONnQwivAXg3gH9ztf1zuIym+Fu5z33ppZdw//79pruzciyrxLJMLi4u\nspEMVvB8zsLm+hKdTsd8/O12u3OPv1qE+fv0utQ0xrj0YP86a1D7R1PzOqrACveyzmkuHVlSkVl8\nragHHXaWS75g6v60OSZcKuHJKCk8arYlxNbIKS7CzxbLWHz8+DEePHhQ9P5GIhxCeBnAQwDfDOAk\nhPDC1UuHMcbR1fyHAbw/hPBbAD4H4EMAfh/Azzb5rnUj11stN5++EXJRDKnHYbZoWYj1aBnStDBv\nb2+b1lbOArPmZZkFeVFSVqw+F6nIhdTTQ13jc8qJNdb8aDSqjHyirWArAaNUiMXKtSJceLnb7WJv\nb2/WWIAty5gFmK1hj5zZLJpawt+Fy463f6zW/w8AfgIAYow/GELYBfBjuIye+GcA/myMcXK9XW0P\nS4DF8pVpjLFxPj+HR1l+ZhHVlCVsdQqJCGvXSV3j47Ss50Wx3AV6mrJqLStX12Cw4m/1udQdolby\nzHg8nhsbjscD1EV5rO9PRT1YHa1W6/V6pginhNgaxkpbws760zROuCj4MMb4QQAfXGB/1g6rt1rH\n8AqWJZx6NKxzRehOl16vVyvAYjVbApya53V156EpdcJrnQerOp1l3aamer4uVJAjIsTtoEfHlnU8\nGkbKt5yyhMXdkHqikSGr2B0hUx5Nm90RbP3qkDW3hDcHrx1RQEqAtZikfMJa+AQtMNo640fW8Xg8\nK+heJ8ba16g7Ey8uLsxQJkuQl2kJp+Z1B1pu2eqoTFm5VuiglVDDhXlyjd0RuT8NPja2gjnGl/25\n0nZ2dmotYXZH1CVvuE94M3ARLoQFigWYLZ6S9GJBP4ZrS1gLqIjveDyuCLGOTeUC8VYTAZbv1p2G\n8posL8snbLkfdBSDti6taIZcSKCVPm41XRVNr89N5bO1u8Sa598OW8IsutzJ1u/3Z5aw5ROWkVTk\nPfy0xELvVvBm4SJcAAuwCJcIsbwOIJvFVGcJswjrG0ks4ZQQaxHWFrsWYi2EYvnKayzUy7yRLZ+p\n9UeUa6mCSXWCq7PZrOw2K8bbcmlwFlwqWsMSYRFNFmAW1sFgYFrBKUtYX19r3oV4/XERrkELcIwR\n29vbc6+HEIqC6AVLgKS8pd5eLOGc+LJ4sGUkgsrrdAiVho/1OuRE1zp+6w9Jz+esVmsdj3BszXOH\nW8q3rKdW5EbqeOU6sitC3BE8RiCPFahdEqnY4ZzLyd0Rm4OLcCFajK3XUp0juegIEZ3t7e2KEPN2\nW1tbMwHOCbFYwiK8FxcXs/2QPw/+E7FEVlwQlv+2CdZ7ciKcykbUUQ4suCUCq326KX/vZDJpFP6m\njzHVCSnXT1vCvV6vIsLStDuCRVp3zJVEvbgIrz8uwgXoDiv9qCmiV9Ixx2h3hNys0+l07vu1AKfE\nmL9bPpt92GzFW8d5cXExJ8TXJWctsi81lTVoibAlpFzNTK+X13jK83LOU52Hel0pui+BLeGdnZ2Z\nuFoWcM4V0e/3K9+hv9PZHFyEC6kL5+JHTu611mFIYgWdnZ1VBFtuHBans7MzAJeWFD+Gd7vdZL0A\nsarrEgNyr1nb1oWwaSyRTU1T6b9WujYLcEqMeb0WZkuox+NxpdBOk9+D/m1oC7TT6cxElC1bblZI\nGnfASSSFftJybgYuwgWIZSjzlhiLxWkJMD9+yiMod/Cx8MnNKxapdEaJAFs+ZvYpj0ajyudZ31En\nyqnWhFzUgJ7P1WCwLGF2PeQabyd/YvzZVgdaqRBbEShWJ2in08He3h7u3LkzczWwy8ESYBFf+d14\nEsbNxkW4Bi3A3Nki6zi6QMeDcjgS94iLC0MnULDLg0OddOSD7AuL2HR6ORJwSa+5FmG+0VPzTdC+\n3lxcbUp0rWUrzCw3r6MfrAI7TYVYRztYf1iynlORtfjWWcDa+pXr5twsXIQLYdEFMBdzK+ssS5ir\nXrElnPoO4FLEzs/PZ+t0SrKOqhBreWdnJxkjbFltvL/anaJfa0Kd+OpoiFw0Ak/rEi6sdTqChJMt\ncp2Tud+CPnc6a03mdSpySoClpaqkuSV8c3ERLkBbw4IIscxrK9gKzBcRtsKcctEDbAWxAHParXTW\nWY/KqQQOLbZWk9ebUJIFZ9V6qJvPpR6XtFSpydw1t2ARlutsTXd2duZCz7QfmIVYLGDtjuC+Axfh\nm4WLcCGpHz7H1OaEWFvCOZHSoVtiEWsLmGODe70ehsMhut1uMnbUiiVNCa+uUbuICFsZb6lprsyk\nfkG4cVcAABJFSURBVD3VaZfr0LMK6lvuCH3NU5ayjv1l4eTOtH6/X+mA05avXmYR1xXS+Lo6NwcX\n4QZYP34O57IeUbk2AIcXpSw+9gXLev5uywWhK2npiAnL9yzzqcdonpdpE1Lpx1YFtNJsOUuQLZdF\niVuD//zqrrklxJYlzE874lawoiFS0RGDwaByHb062u3ARbgQ7pSz1kkMrr45c5aw1IiQUDQRYJ5n\ni40t4JzroDSIn0XYuvH1fBOaiuwign3d+Zw7wrr2vKyTL/jaWs0SX71+MBjUuoVchG8eLsIFaLFl\n1wBPUwJs+YQ5OYPdDBwvLJEPLCBSXS0X45uKYdXLIYQ5KzpXarEJJRasJa4lUyvcrTQSw1q3qBCn\nLGFJvrDEVmfA8TqujJaK53af8M3DRbiQnCuCIyZKoiPG4/FMhLWfl9eJaIlw5WoEaH9hyVSLMPsg\nrXVNsAQ31dnG4pqat/zlqU7MXAentb7Jb4D/cPlJQqchc0ecFtzUslWUxwotdG4WLsIN0WLMyzq0\ny7KEJVRKux10zQgdvqW/T7tF6vbNWicinOrZ19MmlPhlLf9siTVr1WfI1W4oWS7F+sPVacgivnfv\n3q2IcF2T8ELLf6+XnZuDi3ANTX7wlp+VBViXQuQbSsRFxEn7AEU0uCMpJUal+xxCmEuKkJTqbreL\n6XSKXq+H6XS6sDuipLOsqfugCdaflczn/si0C4en3W7XDDPTIWhsCXPJSu6401EQ2l3E+6LXOzcD\nF+Elkuqw4fhUEVEdt6vFmDuRUo/XvL1lIZberDocTltgi1DiC85ZuiUug5J9ywmaXk65efS6Xq83\nS0PW6ci8zLG/ViqyzoSzzrmL7s3HRXjJpEKXuA4tgLkbG0iPO8dWofZvWpYxL9fdwPqzzs/PTTFo\naoE2CUXT+5D7Y5FzlyNnQVqP+fznWVfQaGtraybCVhacVYqS+wQ4C04/7Vh/DqXH7GwuLsJLRG4c\n7StkC1i2s/x7WnzFRWBZjCGE2WfmHtPrrOJcYghvUxdPq9Gda7lIh9QfTMq/a2H5w5tYuFbiSiqT\nsNfrmdXPUqnIeiw5rglhxf9arpDcNXQ2GxfhJWK5I9iS5W24l9sSQk7JtQRMi2SJxZsTYhb38/Nz\n87XccWtSEQ7W+tLoBUuMcyKlRbaulkYuPE+PimyFnVnLMjS9FXliWcJ8HKkOV+dm4SK8ZLQ7Qgsw\nR1AI3CnHFrAuOLO1tTUTYC2UHLOcQguxFjv+THmdhbMJdbG8vD7lB24SSpayHFmIdYlPntfZjdZo\nyPxaacRDv99PZiCyCOskDBfk24OL8BLRljALsI4r3d5+c7QLESMtvroGriWgYr2yX7nJjcqfo9dL\ngSKxXJtg+XlT07oOxxR1QqUFOFdHudPpzFUw49huvcxNRz3oYYisinQ8L1mO1jHoeefm4SK8RFiE\nOaRLr+/1etje3q4InLaAeQij6XRq+mlZKJmcH9gS6ZQAS5OIiZLj58+wRFav43hpPi69ru47ranl\njtDZaLKcGgE5tSwCWzeV2N9UoXwdIWO5VKx55+bgIrxkxOVguSC63e4sVEssZRZgbQFLecqcCIub\nolRw9XoWOhFI6fTLhaqV+KBTlm1d9ENunfX9qQ4tyx2RqpXMIpxLL5bG9UC4apoVCVHij7ayHUvP\ntbPZuAgvEbnh9XKn05kL0+J1UhOCLeDxeDy7sSeTyewztQujLqa3TohF3MStIfstfufcZ9dRYt3W\ndbyVREbwPvO8FjptAevi63rgTR3pwGFnKV+xFQVhRWqklvUxOTcfF+Elom8qEWDLF9rpdObcDyzC\no9GociMDdhRFygpuAouk9Vmln68tdf0dueVFSAmXbpYlrDvKdAEeHn5eT3d3d83aGtZ8p9PxDjcn\ni4vwkpGbSW783CM4+xtlLDRrXLTz8/PZ8Ea6N72JaOZC1HJT7TNOCQhb3YtYdaXblFiU7IOvK9HZ\n6XTmhp3XWXA8LyUncxXoFq3D7Nw+/BeyIvhRX5YZHZvK/kid4ry9vV0ZxNIa2FLEm4v9WPukaRLF\n0OSxuukfQaklWBr3y8kXuUL1nFSTGnKIO+L0+3XY23XcN87txEV4BbAA6w4wthStouBWjYlOp1MR\nWm69Xg/j8Xgm5tPpNLlPFlYGm17HccilAph7BLf2p4kIWxEGqbCzVGMR7nQ6lQQMa9rv9+ey3axY\nXxdipykuwktGC3DK16oz66S+BAufCHW3252J7mg0qszrcciairBV2UzC0gSr6FAq3MqKe63zgzYR\nKy2wWmxzr+VSknu9XiUMzap8pkc/1uFudW4ix7FwEV4RlhjzlAXWqjHBlrKI8Gg0ws7ODkajEUaj\n0VxnkFjMqf2x4FGLRYAlMYQtYPkM3q9UuJd0Run35ZZL0X7e3DQnzKkQtVSsL4ehiQhbGXguwE5T\nXIRXDAsvY9WYSMUW7+zsYDgcVoRA2nA4rIiJVfc31yEnoXHb29tz8chcT0I+hy3hnABaUQqWz7ip\naHEHWElkgrZYrXm2hvn8plKXLV+wW8POorgIrwDL8gUwt45rFuhHfg6dEotsOByalbhYcMbj8dy+\npIgxznzNIh78msQwaxFmEUsNh5TrsEsJcglc4yFX30FbrCmfccmxWIV3UgkXLsBOU1yEV4wVvgVU\nLWHxAVt+YhmTLlUKUXcOcUhUnRDEGCuCwi4IqWVhdbJp0bIEUAtwar6pYPFQQlaGmpWtVue/tjry\n9Pm1zjUfi3fKOYviIrwiUlERlk8YqPb6i7BxIkedALMIlwpAjHHWiaZdEOfn52bYleWO4GQHrpeg\nRUp/xqIirGs3WPUcpHqZ5S5IratruuNRn5fr+Lqd24uL8IpJibHc0LKOLWA9FJAlwtxLr0W4FLG+\neZlrWbAIy35a7gi22mWqQ7Zy803gKAYuF6nnpY6vdhXocDK9bO2ntU7OR67D0XFKcBF+Rlg3plhU\n29tvVlSzavDKgJu6GLiVKCDCXgL7qNkFIRZ4SoR1IoSOLBgMBtn4Yb2uCTqe14rplfmS4jl6P3Id\nhykrV19bF2GnCS7CS6Tpzcd+2O3tN+sL68a1hK2bn9c1sYQ58kHQZSb5j4FjaHPWaE6EUxZxKTK0\nUGpkCxZjS4StP4Gm++A4y8RFuGW0u4LX8bL2w3JcMdejWMQdwa4QXU1sd3cXp6enGA6Hc/5Xq4k1\nLFZ+TgCv444Q4dfxu/opIRed4TjrgIvwGpGLpNAizCMVs7/ZihNOIRa47lzr9/sYDofY3d3FcDjE\ncDicJYqkRpvQTQut5QteVIR1h5yIsJVGnIvGcCF21gEX4RYRV0RqXvti2Q+rx30Tf3AqY85CQtTY\nAh6PxxgMBrP0aMnO4zC5XEKDLKeiIqx1TbBC1Cx/ubaCU0LsOG3jIrwGaNG1Upy124BTilmke71e\n8ffGGOcsYF2VjSu2pTLTrGlK+FLugVL4TyMVO23F8Vrf5WLsrAMuwmuI5RPmsDB2QXC0Qq6KWgoR\nYB5Wied5OVeTV8+n/LDX9c3qzDYdsqd9wsv6XsdZFY1EOITw/QD+ewD/OYAhgH8O4PtijJ+hbT4C\n4C+qt348xviea+7rjYQt3pQ1zD5h9vlyzK6IUK6esIXUjuBCPnpelnNVyKyMMjm+lAAuIoa6ZkUq\nw027I6x9cZx1oKkl/E4APwLg/7t6718H8IshhLfHGIe03ccAvA+A/NKrBQ0ck1THHFBN7pBt2Trm\ncpSlSHKGxAbzvF6WDLrSlhLb63aOidVvFeHRiSv6jyA37zht0UiEtTUbQngfgP8I4AGAT9BL4xjj\nF669d7cE7pSTZaAqxiI+7FuVsep0EfZSJBaYs/Nyy3XZZrpDjI+lbtrkXFnpx9aUO/2u+72Osyqu\n6xO+ByACeEOtf1cI4XUAXwLwjwC8P8aot3EIFgWrEDxblyLIVpZdkwE0U0kZvMzrrAiHXMSD7gSz\n5q3luvOkvzcVCrfM73WcVbGwCIfLX/CHAXwixvib9NLHAPw0gM8C+GO4dFn8QgjhHXEZQ+zeAiyx\n4PKXeih5vdyEus/Sn5l7tNfr6o5rEXJujdw+uOA668p1LOGXAXwNgG/glTHGj9Lib4QQfg3AbwN4\nF4Bfvsb33SgWeQx3HOfmsZAIhxB+FMB7ALwzxvgHuW1jjJ8NIXwRwIvIiPCjR4+wv79fWffw4UM8\nfPhwkV10HMd5JhwcHODg4KCy7vDwsPj9oenj65UAfwuAb4wx/vuC7b8CwO8A+JYY488br98H8Oqr\nr76K+/fvN9oXx3GcdeTx48d48OABADyIMT7ObdsoZzSE8DKAbwfwFwCchBBeuGr9q9f3Qgg/GEL4\n+hDCV4YQ3g3g/wHwGQCvLHIwjuM4N5lmifvAdwF4DsA/BvB5at929fo5gD8B4GcB/DsA/xeAfwng\nv4kxNkvlchzHuQU0jRPOinaMcQTgz1xrjxzHcW4RTS1hx3EcZ4m4CDuO47SIi7DjOE6LuAg7juO0\niIuw4zhOi7gIO47jtIiLsOM4Tou4CDuO47SIi7DjOE6LuAg7juO0iIuw4zhOi7gIO47jtIiLsOM4\nTou4CDuO47SIi7DjOE6LuAg7juO0yFqLsB487yZxk48NuNnH58e2uazj8bkIt8RNPjbgZh+fH9vm\nso7Ht9Yi7DiOc9NxEXYcx2kRF2HHcZwWaTTa8oroA8CnP/3puRcODw/x+PHjZ75Dz4KbfGzAzT4+\nP7bN5VkdH+lZv27bEGNc7d7U7UAIfwHA3211JxzHcVbDt8cYfyq3wTqI8FsAfBOAzwEYtbozjuM4\ny6EP4I8CeCXG+Ie5DVsXYcdxnNuMd8w5juO0iIuw4zhOi7gIO47jtIiLsOM4TouspQiHEP5yCOGz\nIYRhCOGTIYT/qu19WgYhhA+EEC5U+82292sRQgjvDCH8XAjhP1wdxzcb2/xACOHzIYTTEMI/CCG8\n2Ma+LkLd8YUQPmJcy19oa39LCSF8fwjhUyGEoxDC6yGEvxdC+Gpju428diXHt27Xbu1EOITw5wH8\nMIAPAPgvAfxrAK+EEL6s1R1bHr8O4AUAb71qf6rd3VmYPQC/CuC7AcyF2IQQvg/A9wD4TgB/EsAJ\nLq9j71nu5DXIHt8VH0P1Wj58Nrt2Ld4J4EcAfD2APw2gC+AXQwgD2WDDr13t8V2xPtcuxrhWDcAn\nAfzvtBwA/D6Av9r2vi3h2D4A4HHb+7GC47oA8M1q3ecBPKLl5wAMAXxb2/u7pOP7CICfaXvflnBs\nX3Z1fH/qhl476/jW6tqtlSUcQugCeADgl2RdvDxr/xDAO9raryXzx68ecX87hPCTIYT/rO0dWjYh\nhLfh0rrg63gE4Fdwc64jALzr6pH334YQXg4h/Cdt79AC3MOlpf8GcCOvXeX4iLW5dmslwrj819oG\n8Lpa/zoufxibzicBvA+XGYLfBeBtAP5pCGGvzZ1aAW/F5Q//pl5H4PJx9r0A/jsAfxXANwL4hRBC\naHWvGnC1rx8G8IkYo/RN3Jhrlzg+YM2u3ToU8Lk1xBhfocVfDyF8CsDvAPg2XD4iORtCjPGjtPgb\nIYRfA/DbAN4F4Jdb2anmvAzgawB8Q9s7siLM41u3a7dulvAXAZzj0mHOvADgtWe/O6slxngI4DMA\nNqLnuQGv4dKXfyuuIwDEGD+Ly9/vRlzLEMKPAngPgHfFGP+AXroR1y5zfHO0fe3WSoRjjFMArwJ4\nt6y7ekR4N4B/3tZ+rYoQwh1cXvjsj2TTuPpRv4bqdXwOlz3WN+46AkAI4SsAvAUbcC2vBOpbAPy3\nMcbf5dduwrXLHV9i+1av3Tq6I/4mgB8PIbwK4FMAHgHYBfDjbe7UMggh/BCAv49LF8QfAfDXAEwB\nrN/AVzVc+bFfxKXVBABfFUL4WgBvxBh/D5e+uPeHEH4LlxXyPoTLKJefbWF3G5M7vqv2AQA/jUvB\nehHA38DlU80r85+2PoQQXsZlONY3AzgJIYjFexhjlCqGG3vt6o7v6rqu17VrOzwjEVby3bi8+EMA\n/wLA17W9T0s6rgNc/piHAH4XwE8BeFvb+7XgsXwjLkN/zlX7v2mbD+Iy3OkUlz/wF9ve72UcHy7L\nFH4clzfxCMC/B/B/AvhP297vguOyjukcwHvVdht57eqObx2vnZeydBzHaZG18gk7juPcNlyEHcdx\nWsRF2HEcp0VchB3HcVrERdhxHKdFXIQdx3FaxEXYcRynRVyEHcdxWsRF2HEcp0VchB3HcVrERdhx\nHKdFXIQdx3Fa5P8H+0YUaiccw+cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12c68e128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x[1].reshape((28,28)),cmap='Greys')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-24-08f9e66bca55>:14: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.\n",
      "Instructions for updating:\n",
      "Use `tf.global_variables_initializer` instead.\n",
      "loss: 2.30258\n",
      "loss: 0.389749\n",
      "loss: 0.485583\n",
      "loss: 0.222426\n",
      "loss: 0.302407\n",
      "loss: 0.286258\n",
      "loss: 0.24921\n",
      "loss: 0.282081\n",
      "loss: 0.29155\n",
      "loss: 0.299976\n"
     ]
    }
   ],
   "source": [
    "x = tf.placeholder(tf.float32, [None, 784])\n",
    "W = tf.Variable(tf.zeros([784, 10]))\n",
    "b = tf.Variable(tf.zeros([10]))\n",
    "y = tf.matmul(x,W) + b\n",
    "\n",
    "y_true = tf.placeholder(tf.float32,[None, 10])\n",
    "\n",
    "#   tf.reduce_mean(-tf.reduce_sum(y_true * tf.log(tf.nn.softmax(y)),\n",
    "#                                 reduction_indices=[1]))\n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_true, logits=y))\n",
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)\n",
    "\n",
    "sess = tf.InteractiveSession()\n",
    "tf.initialize_all_variables().run()\n",
    "\n",
    "# Training\n",
    "for i in range(1000):\n",
    "    batch_xs, batch_ys = mnist.train.next_batch(128)\n",
    "    l,_ = sess.run([cross_entropy, train_step], feed_dict={x: batch_xs, y_true: batch_ys})\n",
    "    if i%100 == 0:\n",
    "        print('loss: '+str(l))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Softmax takes a $D$ dimensional vector and squeezes them through a function such that we have $D$ outputs whos values are positive and sums to one.\n",
    "$$\n",
    "\\text{softmax}(\\mathbf{y})_d = \\frac{\\exp(-y_d)}{\\exp(-y_1)+...+\\exp(-y_D)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "[-2, 1]\n",
    "exp(-2)/(exp(-2)+exp(1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9214\n"
     ]
    }
   ],
   "source": [
    "# test the model\n",
    "correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_true,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "print(sess.run(accuracy, feed_dict={x: mnist.test.images, y_true: mnist.test.labels}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.916564\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(accuracy, feed_dict={x: mnist.train.images, y_true: mnist.train.labels}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0: loss: 2.30258 accuracy: 0.0859375\n",
      "10: loss: 0.924381 accuracy: 0.742188\n",
      "20: loss: 0.679864 accuracy: 0.84375\n",
      "30: loss: 0.598077 accuracy: 0.789062\n",
      "40: loss: 0.398598 accuracy: 0.9375\n",
      "50: loss: 0.473592 accuracy: 0.828125\n",
      "60: loss: 0.571691 accuracy: 0.8125\n",
      "70: loss: 0.475025 accuracy: 0.875\n",
      "80: loss: 0.380094 accuracy: 0.929688\n",
      "90: loss: 0.232757 accuracy: 0.96875\n",
      "100: loss: 0.496833 accuracy: 0.851562\n",
      "200: loss: 0.412172 accuracy: 0.867188\n",
      "300: loss: 0.260597 accuracy: 0.9375\n",
      "400: loss: 0.247081 accuracy: 0.929688\n",
      "500: loss: 0.294148 accuracy: 0.929688\n",
      "600: loss: 0.251191 accuracy: 0.914062\n",
      "700: loss: 0.172335 accuracy: 0.945312\n",
      "800: loss: 0.333997 accuracy: 0.914062\n",
      "900: loss: 0.207521 accuracy: 0.953125\n"
     ]
    }
   ],
   "source": [
    "sess = tf.InteractiveSession()\n",
    "tf.initialize_all_variables().run()\n",
    "\n",
    "# Training\n",
    "for i in range(1000):\n",
    "    batch_xs, batch_ys = mnist.train.next_batch(128)\n",
    "    l,_,a = sess.run([cross_entropy, train_step, accuracy], feed_dict={x: batch_xs, y_true: batch_ys})\n",
    "    if i%100 == 0 or (i<100 and i%10==0):\n",
    "        print(str(i)+': loss: '+str(l)+' accuracy: '+str(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hidden Layered Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def fully_conntected_activation(input,size_in,size_out):\n",
    "    # Initialise variables\n",
    "    W = tf.Variable(tf.truncated_normal([size_in, size_out],stddev=0.1))\n",
    "    b = tf.Variable(tf.truncated_normal([size_out], stddev=0.1))\n",
    "    # NN part\n",
    "    activation = tf.nn.relu(tf.matmul(input,W)+b)\n",
    "    \n",
    "    return activation\n",
    "\n",
    "def fully_conntected(input,size_in,size_out):\n",
    "    # Initialise variables\n",
    "    W = tf.Variable(tf.truncated_normal([size_in, size_out],stddev=0.1))\n",
    "    b = tf.Variable(tf.truncated_normal([size_out], stddev=0.1))\n",
    "    # NN part\n",
    "    activation = tf.matmul(input,W)+b\n",
    "    \n",
    "    return activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-27-fac17274bb83>:16: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.\n",
      "Instructions for updating:\n",
      "Use `tf.global_variables_initializer` instead.\n",
      "0: loss: 2.42528 accuracy: 0.0703125\n",
      "10: loss: 0.983693 accuracy: 0.71875\n",
      "20: loss: 1.03271 accuracy: 0.648438\n",
      "30: loss: 0.517626 accuracy: 0.835938\n",
      "40: loss: 0.44244 accuracy: 0.890625\n",
      "50: loss: 0.337583 accuracy: 0.90625\n",
      "60: loss: 0.440305 accuracy: 0.851562\n",
      "70: loss: 0.521924 accuracy: 0.8125\n",
      "80: loss: 0.536625 accuracy: 0.8125\n",
      "90: loss: 0.159889 accuracy: 0.960938\n",
      "100: loss: 0.384946 accuracy: 0.875\n",
      "200: loss: 0.19499 accuracy: 0.9375\n",
      "300: loss: 0.289142 accuracy: 0.914062\n",
      "400: loss: 0.241244 accuracy: 0.921875\n",
      "500: loss: 0.105217 accuracy: 0.96875\n",
      "600: loss: 0.0857907 accuracy: 0.976562\n",
      "700: loss: 0.216412 accuracy: 0.960938\n",
      "800: loss: 0.124178 accuracy: 0.945312\n",
      "900: loss: 0.107538 accuracy: 0.960938\n"
     ]
    }
   ],
   "source": [
    "x = tf.placeholder(tf.float32, [None, 784])\n",
    "y_true = tf.placeholder(tf.float32,[None, 10])\n",
    "\n",
    "hidden = fully_conntected_activation(x,784,100)\n",
    "output = fully_conntected(hidden,100,10)\n",
    "# output = fully_conntected(x,784,10)\n",
    "\n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_true, logits=output))\n",
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)\n",
    "\n",
    "# test the model\n",
    "correct_prediction = tf.equal(tf.argmax(output,1), tf.argmax(y_true,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "\n",
    "sess = tf.InteractiveSession()\n",
    "tf.initialize_all_variables().run()\n",
    "\n",
    "# Training\n",
    "for i in range(1000):\n",
    "    batch_xs, batch_ys = mnist.train.next_batch(128)\n",
    "    l,_,a = sess.run([cross_entropy, train_step, accuracy], \n",
    "                     feed_dict={x: batch_xs, y_true: batch_ys})\n",
    "    if i%100 == 0 or (i<100 and i%10==0):\n",
    "        print(str(i)+': loss: '+str(l)+' accuracy: '+str(a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9665\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(accuracy, feed_dict={x: mnist.test.images, y_true: mnist.test.labels}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "latex_envs": {
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 0
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}