02-autograd_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mRCQfI4o9f0k"
   },
   "source": [
    "Autograd: automatic differentiation\n",
    "===================================\n",
    "Central to all neural networks in PyTorch is the ``autograd`` package.\n",
    "Let’s first briefly visit this, and we will then go to training our\n",
    "first neural network.\n",
    "\n",
    "The ``autograd`` package provides automatic differentiation for all operations\n",
    "on Tensors. \n",
    "\n",
    "Let us see this in more simple terms with some examples.\n",
    "\n",
    "Tensor\n",
    "--------\n",
    "``torch.Tensor`` is the central class of the package. If you set its attribute\n",
    "``.requires_grad`` as ``True``, it starts to track all operations on it. When\n",
    "at some point you have a scalar you can call ``.backward()`` on it and the gradients \n",
    "of that scalar wrt. all tensors that require_grad are computed automatically. \n",
    "The of the scalar wrt. a tensor will be accumulated into ``.grad`` attribute of the tensor.\n",
    "\n",
    "To stop a tensor from tracking history, you can call ``.detach()`` to detach\n",
    "it from the computation history, and to prevent future computation from being\n",
    "tracked.\n",
    "\n",
    "To prevent tracking history (and using memory), you can also wrap the code block\n",
    "in ``with torch.no_grad():``. This can be particularly helpful when evaluating a\n",
    "model because the model may have trainable parameters with `requires_grad=True`,\n",
    "but for which we don't need the gradients.\n",
    "\n",
    "There’s one more class which is very important for autograd\n",
    "implementation - a ``Function``.\n",
    "\n",
    "``Tensor`` and ``Function`` are interconnected and build up an acyclic\n",
    "graph, that encodes a complete history of computation. Each tensor has\n",
    "a ``.grad_fn`` attribute that references a ``Function`` that has created\n",
    "the ``Tensor`` (except for Tensors created by the user - their\n",
    "``grad_fn is None``).\n",
    "\n",
    "If you want to compute the derivatives, you can call ``.backward()`` on\n",
    "a ``Tensor``. If ``Tensor`` is a scalar (i.e. it holds a one element\n",
    "data), you don’t need to specify any arguments to ``backward()``,\n",
    "however if it has more elements, you need to specify a ``gradient``\n",
    "argument that is a tensor of matching shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "3jf_wKDL9gfD"
   },
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CneDH3909goS"
   },
   "source": [
    "Create a tensor and set requires_grad=True to track computation with it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "yyfznPfa9gxG"
   },
   "outputs": [],
   "source": [
    "x = torch.ones(2, 2, requires_grad=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GBAz7nbO9g6K"
   },
   "source": [
    "Do an operation of tensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "IM8J-RHY9hEZ"
   },
   "outputs": [],
   "source": [
    "y = x + 2\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UBdT4uNK9hNu"
   },
   "source": [
    "``y`` was created as a result of an operation, so it has a ``grad_fn``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "cnuvDZwo9hX0"
   },
   "outputs": [],
   "source": [
    "y.grad_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fjhDPBo_9hha"
   },
   "source": [
    "Do more operations on y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "fiqw_Fab9hqf"
   },
   "outputs": [],
   "source": [
    "z = y * y * 3\n",
    "out = z.mean()\n",
    "\n",
    "out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BrhrFvj_9iGP"
   },
   "source": [
    "Gradients\n",
    "---------\n",
    "Let's backprop now\n",
    "Because ``out`` contains a single scalar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "Lcldjm8D9iQd"
   },
   "outputs": [],
   "source": [
    "out.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4PU53hhT9iZy"
   },
   "source": [
    "print gradients d(out)/dx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "bOqPHiL_9ijp"
   },
   "outputs": [],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PuZjS25g9isN"
   },
   "source": [
    "You should have got a matrix of ``4.5``.\n",
    "\n",
    "Let’s call the ``out``\n",
    "*Tensor* $o$.\n",
    "We have that $o = \\frac{1}{4}\\sum_i z_i$,\n",
    "$z_i = 3(x_i+2)^2$ and $z_i\\bigr\\rvert_{x_i=1} = 27$.\n",
    "Therefore,\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{\\partial o}{\\partial x_i} = \\frac{3}{2}(x_i+2)\n",
    "\\end{equation}\n",
    "\n",
    "hence\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{\\partial o}{\\partial x_i}\\bigr\\rvert_{x_i=1} = \\frac{9}{2} = 4.5\n",
    "\\end{equation}\n",
    "\n",
    "You can do many crazy things with autograd!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "QKfsVofm9i1k"
   },
   "outputs": [],
   "source": [
    "x0 = torch.randn(2, requires_grad=True)\n",
    "c = torch.tensor([-0.4, 0.6])\n",
    "\n",
    "x = x0\n",
    "while x.data.norm() < 1000:\n",
    "    x = torch.stack((\n",
    "        x[0] ** 2 - x[1] ** 2,\n",
    "        2 * x[0] * x[1]\n",
    "    )) + c\n",
    "     \n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "diff = ((x - torch.tensor([0.25, 0.])) ** 2).sum()\n",
    "diff.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x0.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CSuBxpvT9i-o"
   },
   "source": [
    "You can also stop autograd from tracking history on Tensors\n",
    "with ``.requires_grad``=True by wrapping the code block in\n",
    "``with torch.no_grad():``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercices:** \n",
    "\n",
    "- make a 2D parametric curve and draw its tangents in some points\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eKfV93sK9jQy"
   },
   "source": [
    "**Read Later:**\n",
    "\n",
    "Documentation of ``autograd`` and ``Function`` is at\n",
    "http://pytorch.org/docs/autograd"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "autograd_tutorial.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}