Ch09_Modern_Convolutional_Networks/Densely_Connected_Networks_(DenseNet).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Densely Connected Networks (DenseNet)\n",
    "\n",
    "ResNet significantly changed the view of how to parametrize the functions in deep networks. DenseNet\n",
    "is to some extent the logical extension of this. To understand how to arrive at it, let’s take a small detour\n",
    "to theory. Recall the Taylor $3$ expansion for functions. For scalars it can be written as "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$f(x) = f(0) + f'(x)x + \\frac{1}{2} f''(x)x^{2} + \\frac{1}{6} f(x)x^{3} + o(x^{3})$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Function Decomposition\n",
    "\n",
    "The key point is that it decomposes the function into increasingly higher order terms. In a similar vein,\n",
    "ResNet decomposes functions into\n",
    "\n",
    "                                f(x) = x + g(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is, ResNet decomposes f into a simple linear term and a more complex nonlinear one. What if we\n",
    "want to go beyond two terms? A solution was proposed by [Huang et al, 2016](https://arxiv.org/abs/1608.06993) in the form of DenseNet, an architecture that reported record performance on the ImageNet dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"92pt\" version=\"1.1\" viewBox=\"0 0 177 92\" width=\"177pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M 0.65625 0 L 0.65625 -6.4375 L 3.078125 -6.4375 C 3.566406 -6.4375 3.957031 -6.367188 4.25 -6.234375 C 4.550781 -6.109375 4.785156 -5.910156 4.953125 -5.640625 C 5.128906 -5.367188 5.21875 -5.085938 5.21875 -4.796875 C 5.21875 -4.515625 5.140625 -4.253906 4.984375 -4.015625 C 4.835938 -3.773438 4.613281 -3.578125 4.3125 -3.421875 C 4.695312 -3.304688 4.992188 -3.113281 5.203125 -2.84375 C 5.421875 -2.570312 5.53125 -2.25 5.53125 -1.875 C 5.53125 -1.570312 5.460938 -1.289062 5.328125 -1.03125 C 5.203125 -0.769531 5.046875 -0.566406 4.859375 -0.421875 C 4.671875 -0.285156 4.4375 -0.179688 4.15625 -0.109375 C 3.875 -0.0351562 3.523438 0 3.109375 0 Z M 1.515625 -3.734375 L 2.90625 -3.734375 C 3.28125 -3.734375 3.550781 -3.757812 3.71875 -3.8125 C 3.9375 -3.875 4.097656 -3.976562 4.203125 -4.125 C 4.316406 -4.28125 4.375 -4.46875 4.375 -4.6875 C 4.375 -4.90625 4.320312 -5.09375 4.21875 -5.25 C 4.113281 -5.414062 3.96875 -5.53125 3.78125 -5.59375 C 3.59375 -5.65625 3.265625 -5.6875 2.796875 -5.6875 L 1.515625 -5.6875 Z M 1.515625 -0.765625 L 3.109375 -0.765625 C 3.390625 -0.765625 3.585938 -0.773438 3.703125 -0.796875 C 3.890625 -0.828125 4.050781 -0.882812 4.1875 -0.96875 C 4.320312 -1.050781 4.429688 -1.171875 4.515625 -1.328125 C 4.597656 -1.484375 4.640625 -1.664062 4.640625 -1.875 C 4.640625 -2.113281 4.578125 -2.320312 4.453125 -2.5 C 4.328125 -2.675781 4.15625 -2.796875 3.9375 -2.859375 C 3.71875 -2.929688 3.40625 -2.96875 3 -2.96875 L 1.515625 -2.96875 Z M 1.515625 -0.765625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-2\" overflow=\"visible\">\n",
       "<path d=\"M -0.015625 0 L 2.46875 -6.4375 L 3.375 -6.4375 L 6.015625 0 L 5.046875 0 L 4.296875 -1.953125 L 1.59375 -1.953125 L 0.890625 0 Z M 1.84375 -2.640625 L 4.03125 -2.640625 L 3.359375 -4.4375 C 3.148438 -4.976562 3 -5.421875 2.90625 -5.765625 C 2.820312 -5.347656 2.703125 -4.9375 2.546875 -4.53125 Z M 1.84375 -2.640625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-3\" overflow=\"visible\">\n",
       "<path d=\"M 0.609375 1.78125 L 0.609375 -6.4375 L 2.359375 -6.4375 L 2.359375 -5.78125 L 1.40625 -5.78125 L 1.40625 1.140625 L 2.359375 1.140625 L 2.359375 1.78125 Z M 0.609375 1.78125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-4\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-5\" overflow=\"visible\">\n",
       "<path d=\"M 0.796875 0 L 0.796875 -0.90625 L 1.703125 -0.90625 L 1.703125 0 C 1.703125 0.332031 1.640625 0.597656 1.515625 0.796875 C 1.398438 1.003906 1.21875 1.164062 0.96875 1.28125 L 0.75 0.9375 C 0.914062 0.863281 1.035156 0.753906 1.109375 0.609375 C 1.191406 0.472656 1.238281 0.269531 1.25 0 Z M 0.796875 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-6\" overflow=\"visible\">\n",
       "<path d=\"M 1.921875 1.78125 L 0.171875 1.78125 L 0.171875 1.140625 L 1.125 1.140625 L 1.125 -5.78125 L 0.171875 -5.78125 L 0.171875 -6.4375 L 1.921875 -6.4375 Z M 1.921875 1.78125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-0\" overflow=\"visible\">\n",
       "<path d=\"M 4.90625 -7.984375 L 1.21875 -7.984375 L 1.21875 -0.5625 L 4.90625 -0.5625 Z M 5.515625 -8.546875 L 5.515625 -0.015625 L 0.609375 -0.015625 L 0.609375 -8.546875 Z M 5.515625 -8.546875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-1\" overflow=\"visible\">\n",
       "<path d=\"M 4.015625 -3.4375 L 4.015625 -6.078125 L 3.1875 -6.078125 L 3.1875 -3.4375 L 0.578125 -3.4375 L 0.578125 -2.625 L 3.1875 -2.625 L 3.1875 0 L 4.015625 0 L 4.015625 -2.625 L 6.625 -2.625 L 6.625 -3.4375 Z M 4.015625 -3.4375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<rect height=\"92\" style=\"fill:rgb(100%,100%,100%);fill-opacity:1;stroke:none;\" width=\"177\" x=\"0\" y=\"0\"/>\n",
       "<rect height=\"92\" style=\"fill:rgb(100%,100%,100%);fill-opacity:1;stroke:none;\" width=\"177\" x=\"0\" y=\"0\"/>\n",
       "<path d=\"M 103.5 184.5 L 166.5 184.5 L 166.5 202.5 L 103.5 202.5 Z M 103.5 184.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"28.998535\" xlink:href=\"#glyph0-1\" y=\"44.352783\"/>\n",
       "</g>\n",
       "<path d=\"M 103.5 225 L 166.5 225 L 166.5 243 L 103.5 243 Z M 103.5 225 \" style=\"fill-rule:nonzero;fill:rgb(69.802856%,85.096741%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"28.998535\" xlink:href=\"#glyph0-2\" y=\"84.852783\"/>\n",
       "</g>\n",
       "<path d=\"M 140.074219 154.976562 C 142.710938 157.613281 142.710938 161.886719 140.074219 164.523438 C 137.4375 167.160156 133.164062 167.160156 130.527344 164.523438 C 127.894531 161.886719 127.894531 157.613281 130.527344 154.976562 C 133.164062 152.339844 137.4375 152.339844 140.074219 154.976562 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.900818%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"28.70195\" xlink:href=\"#glyph1-1\" y=\"11.582001\"/>\n",
       "</g>\n",
       "<path d=\"M 135 225 L 135 208.398438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<path d=\"M 135 204.398438 L 135 208.398438 M 133.5 208.398438 L 135 204.398438 L 136.5 208.398438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<path d=\"M 135.082031 184.5 L 135.1875 172.398438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<path d=\"M 135.222656 168.398438 L 135.1875 172.398438 M 133.6875 172.386719 L 135.222656 168.398438 L 136.6875 172.414062 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<path d=\"M 135 217.082031 L 172.5 217.082031 C 174.15625 217.082031 175.5 215.738281 175.5 214.082031 L 175.5 162.75 C 175.5 161.09375 174.15625 159.75 172.5 159.75 L 147.953125 159.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<path d=\"M 143.953125 159.75 L 147.953125 159.75 M 147.953125 161.25 L 143.953125 159.75 L 147.953125 158.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<path d=\"M 207 184.5 L 270 184.5 L 270 202.5 L 207 202.5 Z M 207 184.5 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"132.498535\" xlink:href=\"#glyph0-1\" y=\"44.352783\"/>\n",
       "</g>\n",
       "<path d=\"M 207 225 L 270 225 L 270 243 L 207 243 Z M 207 225 \" style=\"fill-rule:nonzero;fill:rgb(69.802856%,85.096741%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"132.498535\" xlink:href=\"#glyph0-2\" y=\"84.852783\"/>\n",
       "</g>\n",
       "<path d=\"M 230.707031 153 L 248.242188 153 C 250.316406 153 252 156.023438 252 159.75 C 252 163.476562 250.316406 166.5 248.242188 166.5 L 230.707031 166.5 C 228.632812 166.5 226.949219 163.476562 226.949219 159.75 C 226.949219 156.023438 228.632812 153 230.707031 153 Z M 230.707031 153 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.900818%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"130.222779\" xlink:href=\"#glyph0-3\" y=\"10.602783\"/>\n",
       "  <use x=\"132.722979\" xlink:href=\"#glyph0-4\" y=\"10.602783\"/>\n",
       "  <use x=\"135.223179\" xlink:href=\"#glyph0-5\" y=\"10.602783\"/>\n",
       "  <use x=\"137.723379\" xlink:href=\"#glyph0-4\" y=\"10.602783\"/>\n",
       "  <use x=\"140.223579\" xlink:href=\"#glyph0-6\" y=\"10.602783\"/>\n",
       "</g>\n",
       "<path d=\"M 238.5 225 L 238.5 208.398438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<path d=\"M 238.5 204.398438 L 238.5 208.398438 M 237 208.398438 L 238.5 204.398438 L 240 208.398438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<path d=\"M 238.757812 184.5 L 239.109375 172.398438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<path d=\"M 239.222656 168.398438 L 239.109375 172.398438 M 237.609375 172.355469 L 239.222656 168.398438 L 240.609375 172.441406 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<path d=\"M 238.5 216.667969 L 276 216.667969 C 277.65625 216.667969 279 215.324219 279 213.667969 L 279 162.75 C 279 161.09375 277.65625 159.75 276 159.75 L 257.898438 159.75 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "<path d=\"M 253.898438 159.75 L 257.898438 159.75 M 257.898438 161.25 L 253.898438 159.75 L 257.898438 158.25 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-103,-152)\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "SVG(filename= '../img/densenet.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fig. 6.18: The main difference between ResNet (left) and DenseNet (right) in cross-layer connections:\n",
    "use of addition and use of concatenation.\n",
    "\n",
    "The key difference between ResNet and DenseNet is that in the latter case outputs are concatenated rather\n",
    "than added. As a result we perform a mapping from x to its values after applying an increasingly complex\n",
    "sequence of functions.\n",
    "            $$x → [x, f 1 (x), f 2 (x, f 1 (x)), f 3 (x, f 1 (x), f 2 (x, f 1 (x)), . . .]$$\n",
    "            \n",
    "In the end, all these functions are combined in an MLP to reduce the number of features again. In terms of\n",
    "implementation this is quite simple - rather than adding terms, we concatenate them. The name DenseNet\n",
    "arises from the fact that the dependency graph between variables becomes quite dense. The last layer of\n",
    "such a chain is densely connected to all previous layers. The main components that compose a DenseNet\n",
    "are dense blocks and transition layers. The former defines how the inputs and outputs are concatenated,\n",
    "while the latter controls the number of channels so that it is not too large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"94pt\" version=\"1.1\" viewBox=\"0 0 230 94\" width=\"230pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M 0.0625 0 L 1.765625 -2.421875 L 0.1875 -4.671875 L 1.1875 -4.671875 L 1.90625 -3.578125 C 2.03125 -3.367188 2.132812 -3.191406 2.21875 -3.046875 C 2.351562 -3.242188 2.472656 -3.414062 2.578125 -3.5625 L 3.359375 -4.671875 L 4.3125 -4.671875 L 2.703125 -2.46875 L 4.4375 0 L 3.46875 0 L 2.5 -1.453125 L 2.25 -1.84375 L 1.03125 0 Z M 0.0625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.78125 0 L 0.78125 -4.046875 L 0.078125 -4.046875 L 0.078125 -4.671875 L 0.78125 -4.671875 L 0.78125 -5.15625 C 0.78125 -5.476562 0.804688 -5.710938 0.859375 -5.859375 C 0.941406 -6.066406 1.078125 -6.234375 1.265625 -6.359375 C 1.460938 -6.484375 1.734375 -6.546875 2.078125 -6.546875 C 2.296875 -6.546875 2.539062 -6.519531 2.8125 -6.46875 L 2.6875 -5.78125 C 2.53125 -5.8125 2.378906 -5.828125 2.234375 -5.828125 C 1.992188 -5.828125 1.820312 -5.773438 1.71875 -5.671875 C 1.613281 -5.566406 1.5625 -5.375 1.5625 -5.09375 L 1.5625 -4.671875 L 2.484375 -4.671875 L 2.484375 -4.046875 L 1.5625 -4.046875 L 1.5625 0 Z M 0.78125 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-3\" overflow=\"visible\">\n",
       "<path d=\"M 3.359375 0 L 2.5625 0 L 2.5625 -5.046875 C 2.375 -4.859375 2.125 -4.671875 1.8125 -4.484375 C 1.5 -4.304688 1.222656 -4.175781 0.984375 -4.09375 L 0.984375 -4.859375 C 1.421875 -5.054688 1.804688 -5.300781 2.140625 -5.59375 C 2.472656 -5.894531 2.707031 -6.1875 2.84375 -6.46875 L 3.359375 -6.46875 Z M 3.359375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-4\" overflow=\"visible\">\n",
       "<path d=\"M 2.109375 1.890625 C 1.671875 1.335938 1.300781 0.695312 1 -0.03125 C 0.695312 -0.769531 0.546875 -1.535156 0.546875 -2.328125 C 0.546875 -3.023438 0.660156 -3.695312 0.890625 -4.34375 C 1.148438 -5.082031 1.554688 -5.816406 2.109375 -6.546875 L 2.671875 -6.546875 C 2.316406 -5.941406 2.082031 -5.507812 1.96875 -5.25 C 1.789062 -4.84375 1.648438 -4.421875 1.546875 -3.984375 C 1.421875 -3.429688 1.359375 -2.878906 1.359375 -2.328125 C 1.359375 -0.921875 1.796875 0.484375 2.671875 1.890625 Z M 2.109375 1.890625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-5\" overflow=\"visible\">\n",
       "<path d=\"M 1.109375 1.890625 L 0.546875 1.890625 C 1.421875 0.484375 1.859375 -0.921875 1.859375 -2.328125 C 1.859375 -2.878906 1.796875 -3.425781 1.671875 -3.96875 C 1.566406 -4.40625 1.425781 -4.828125 1.25 -5.234375 C 1.132812 -5.492188 0.898438 -5.929688 0.546875 -6.546875 L 1.109375 -6.546875 C 1.660156 -5.816406 2.066406 -5.082031 2.328125 -4.34375 C 2.554688 -3.695312 2.671875 -3.023438 2.671875 -2.328125 C 2.671875 -1.535156 2.519531 -0.769531 2.21875 -0.03125 C 1.914062 0.695312 1.546875 1.335938 1.109375 1.890625 Z M 1.109375 1.890625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-6\" overflow=\"visible\">\n",
       "<path d=\"M 4.53125 -0.765625 L 4.53125 0 L 0.265625 0 C 0.265625 -0.1875 0.296875 -0.367188 0.359375 -0.546875 C 0.472656 -0.835938 0.648438 -1.125 0.890625 -1.40625 C 1.128906 -1.6875 1.472656 -2.007812 1.921875 -2.375 C 2.617188 -2.957031 3.085938 -3.414062 3.328125 -3.75 C 3.578125 -4.082031 3.703125 -4.398438 3.703125 -4.703125 C 3.703125 -5.015625 3.585938 -5.273438 3.359375 -5.484375 C 3.140625 -5.703125 2.851562 -5.8125 2.5 -5.8125 C 2.113281 -5.8125 1.804688 -5.695312 1.578125 -5.46875 C 1.347656 -5.238281 1.234375 -4.921875 1.234375 -4.515625 L 0.421875 -4.609375 C 0.472656 -5.210938 0.679688 -5.671875 1.046875 -5.984375 C 1.410156 -6.304688 1.898438 -6.46875 2.515625 -6.46875 C 3.128906 -6.46875 3.613281 -6.296875 3.96875 -5.953125 C 4.332031 -5.609375 4.515625 -5.1875 4.515625 -4.6875 C 4.515625 -4.425781 4.460938 -4.171875 4.359375 -3.921875 C 4.253906 -3.671875 4.078125 -3.40625 3.828125 -3.125 C 3.585938 -2.851562 3.1875 -2.476562 2.625 -2 C 2.144531 -1.601562 1.835938 -1.332031 1.703125 -1.1875 C 1.566406 -1.039062 1.457031 -0.898438 1.375 -0.765625 Z M 4.53125 -0.765625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-7\" overflow=\"visible\">\n",
       "<path d=\"M 0.375 -1.703125 L 1.171875 -1.8125 C 1.265625 -1.363281 1.414062 -1.039062 1.625 -0.84375 C 1.84375 -0.644531 2.113281 -0.546875 2.4375 -0.546875 C 2.800781 -0.546875 3.109375 -0.671875 3.359375 -0.921875 C 3.617188 -1.179688 3.75 -1.503906 3.75 -1.890625 C 3.75 -2.253906 3.628906 -2.550781 3.390625 -2.78125 C 3.160156 -3.019531 2.863281 -3.140625 2.5 -3.140625 C 2.34375 -3.140625 2.15625 -3.109375 1.9375 -3.046875 L 2.03125 -3.75 C 2.082031 -3.738281 2.125 -3.734375 2.15625 -3.734375 C 2.488281 -3.734375 2.789062 -3.820312 3.0625 -4 C 3.332031 -4.175781 3.46875 -4.445312 3.46875 -4.8125 C 3.46875 -5.101562 3.367188 -5.34375 3.171875 -5.53125 C 2.972656 -5.71875 2.71875 -5.8125 2.40625 -5.8125 C 2.101562 -5.8125 1.847656 -5.710938 1.640625 -5.515625 C 1.441406 -5.328125 1.3125 -5.039062 1.25 -4.65625 L 0.453125 -4.796875 C 0.554688 -5.328125 0.773438 -5.738281 1.109375 -6.03125 C 1.453125 -6.320312 1.878906 -6.46875 2.390625 -6.46875 C 2.742188 -6.46875 3.066406 -6.390625 3.359375 -6.234375 C 3.660156 -6.085938 3.890625 -5.882812 4.046875 -5.625 C 4.203125 -5.363281 4.28125 -5.085938 4.28125 -4.796875 C 4.28125 -4.515625 4.203125 -4.257812 4.046875 -4.03125 C 3.898438 -3.800781 3.679688 -3.617188 3.390625 -3.484375 C 3.773438 -3.398438 4.070312 -3.21875 4.28125 -2.9375 C 4.488281 -2.664062 4.59375 -2.320312 4.59375 -1.90625 C 4.59375 -1.34375 4.382812 -0.863281 3.96875 -0.46875 C 3.5625 -0.0820312 3.046875 0.109375 2.421875 0.109375 C 1.859375 0.109375 1.390625 -0.0546875 1.015625 -0.390625 C 0.640625 -0.722656 0.425781 -1.160156 0.375 -1.703125 Z M 0.375 -1.703125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-8\" overflow=\"visible\">\n",
       "<path d=\"M 2.90625 0 L 2.90625 -1.546875 L 0.109375 -1.546875 L 0.109375 -2.265625 L 3.046875 -6.4375 L 3.703125 -6.4375 L 3.703125 -2.265625 L 4.578125 -2.265625 L 4.578125 -1.546875 L 3.703125 -1.546875 L 3.703125 0 Z M 2.90625 -2.265625 L 2.90625 -5.171875 L 0.890625 -2.265625 Z M 2.90625 -2.265625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<path d=\"M 30 30 L 60 30 L 60 60 L 30 60 Z M 30 30 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"13.75\" xlink:href=\"#glyph0-1\" y=\"80.852783\"/>\n",
       "</g>\n",
       "<path d=\"M 80 30 L 110 30 L 110 60 L 80 60 Z M 80 30 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"57\" xlink:href=\"#glyph0-2\" y=\"78.852783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"59.500488\" xlink:href=\"#glyph0-3\" y=\"82.852783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"64.505859\" xlink:href=\"#glyph0-4\" y=\"78.852783\"/>\n",
       "  <use x=\"67.502859\" xlink:href=\"#glyph0-1\" y=\"78.852783\"/>\n",
       "  <use x=\"72.002859\" xlink:href=\"#glyph0-5\" y=\"78.852783\"/>\n",
       "</g>\n",
       "<path d=\"M 130 30 L 160 30 L 160 60 L 130 60 Z M 130 30 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"107\" xlink:href=\"#glyph0-2\" y=\"78.852783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"109.500488\" xlink:href=\"#glyph0-6\" y=\"82.852783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"114.505859\" xlink:href=\"#glyph0-4\" y=\"78.852783\"/>\n",
       "  <use x=\"117.502859\" xlink:href=\"#glyph0-1\" y=\"78.852783\"/>\n",
       "  <use x=\"122.002859\" xlink:href=\"#glyph0-5\" y=\"78.852783\"/>\n",
       "</g>\n",
       "<path d=\"M 179 30 L 209 30 L 209 60 L 179 60 Z M 179 30 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"156\" xlink:href=\"#glyph0-2\" y=\"78.852783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"158.500488\" xlink:href=\"#glyph0-7\" y=\"82.852783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"163.505859\" xlink:href=\"#glyph0-4\" y=\"78.852783\"/>\n",
       "  <use x=\"166.502859\" xlink:href=\"#glyph0-1\" y=\"78.852783\"/>\n",
       "  <use x=\"171.002859\" xlink:href=\"#glyph0-5\" y=\"78.852783\"/>\n",
       "</g>\n",
       "<path d=\"M 228 30 L 258 30 L 258 60 L 228 60 Z M 228 30 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"205\" xlink:href=\"#glyph0-2\" y=\"78.852783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"207.500488\" xlink:href=\"#glyph0-8\" y=\"82.852783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"212.505859\" xlink:href=\"#glyph0-4\" y=\"78.852783\"/>\n",
       "  <use x=\"215.502859\" xlink:href=\"#glyph0-1\" y=\"78.852783\"/>\n",
       "  <use x=\"220.002859\" xlink:href=\"#glyph0-5\" y=\"78.852783\"/>\n",
       "</g>\n",
       "<path d=\"M 60 45 L 74.101562 45 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 78.101562 45 L 74.101562 43.5 L 74.101562 46.5 Z M 78.101562 45 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 110 45 L 124.101562 45 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 128.101562 45 L 124.101562 43.5 L 124.101562 46.5 Z M 128.101562 45 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 160 45 L 173.101562 45 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 177.101562 45 L 173.101562 43.5 L 173.101562 46.5 Z M 177.101562 45 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 209 45 L 222.101562 45 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 226.101562 45 L 222.101562 43.5 L 222.101562 46.5 Z M 226.101562 45 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 55.863281 30 C 64.953125 19.375 79.007812 7 95.175781 7 C 109.144531 7 121.449219 16.238281 130.339844 25.605469 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 133.003906 28.585938 L 131.457031 24.605469 L 129.21875 26.605469 Z M 133.003906 28.585938 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 106.546875 30 C 116.105469 19.375 130.671875 7 146.675781 7 C 160.4375 7 172.089844 16.152344 180.390625 25.472656 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 182.953125 28.542969 L 181.539062 24.511719 L 179.238281 26.433594 Z M 182.953125 28.542969 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 156.042969 30 C 165.230469 19.375 179.332031 7 195.175781 7 C 208.816406 7 220.601562 16.175781 229.058594 25.511719 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 231.652344 28.554688 L 230.199219 24.535156 L 227.914062 26.480469 Z M 231.652344 28.554688 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 55.082031 30 C 67.621094 13.410156 90.804688 -11 118.675781 -11 C 144.070312 -11 166.175781 9.269531 179.753906 25.421875 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 182.277344 28.523438 L 180.917969 24.476562 L 178.589844 26.367188 Z M 182.277344 28.523438 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 105.015625 30 C 117.464844 13.410156 140.492188 -11 168.175781 -11 C 193.394531 -11 215.347656 9.261719 228.839844 25.410156 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 231.351562 28.523438 L 230.003906 24.46875 L 227.671875 26.351562 Z M 231.351562 28.523438 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 54.9375 30 C 70.699219 8.28125 103.765625 -29.351562 143.675781 -29.351562 C 180.824219 -29.351562 212.332031 3.261719 229.378906 25.289062 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "<path d=\"M 231.785156 28.484375 L 230.578125 24.386719 L 228.179688 26.191406 Z M 231.785156 28.484375 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-29,33)\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SVG(filename= '../img/DenseNetDense.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fig. 6.19: Dense connections in DenseNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dense Blocks\n",
    "\n",
    "DenseNet uses the modified batch normalization, activation, and convolution architecture of ResNet (see\n",
    "the exercise in the previous section). First, we implement this architecture in the conv_block function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')\n",
    "\n",
    "import d2l\n",
    "import time\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision\n",
    "from d2l import train_ch5\n",
    "from torchvision import transforms\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "def conv_block(input_channels, num_channels):\n",
    "    layers = []\n",
    "    layers.append(nn.BatchNorm2d(input_channels))\n",
    "    layers.append(nn.ReLU())\n",
    "    layers.append(nn.Conv2d(input_channels, num_channels, kernel_size=3, padding=1))\n",
    "    blk = nn.Sequential(*layers)\n",
    "    return blk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A dense block consists of multiple conv_block units, each using the same number of output channels.\n",
    "In the forward computation, however, we concatenate the input and output of each block on the channel\n",
    "dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DenseBlock(nn.Module):\n",
    "    def __init__(self, num_convs, input_channels, num_channels, **kwargs):\n",
    "        super(DenseBlock, self).__init__(**kwargs)\n",
    "        layer = []\n",
    "        for i in range(num_convs):\n",
    "            layer.append(conv_block((num_channels * i + input_channels), num_channels))\n",
    "        self.net = nn.Sequential(*layer)\n",
    "    def forward(self, X):\n",
    "        for blk in self.net:\n",
    "            Y = blk(X)\n",
    "            # Concatenate the input and output of each block on the channel\n",
    "            # dimension\n",
    "            X = torch.cat((X, Y), dim=1)\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example, we define a convolution block with two blocks of 10 output channels. When\n",
    "using an input with 3 channels, we will get an output with the 3 + 2 × 10 = 23 channels. The number of\n",
    "convolution block channels controls the increase in the number of output channels relative to the number\n",
    "of input channels. This is also referred to as the growth rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 23, 8, 8])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "blk = DenseBlock(2, 3, 10)\n",
    "X = torch.randn(4, 3, 8, 8)\n",
    "Y = blk(X)\n",
    "Y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transition Layers\n",
    "\n",
    "Since each dense block will increase the number of channels, adding too many of them will lead to an\n",
    "excessively complex model. A transition layer is used to control the complexity of the model. It reduces\n",
    "the number of channels by using the 1 × 1 convolutional layer and halves the height and width of the\n",
    "average pooling layer with a stride of 2, further reducing the complexity of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transition_block(input_channels, num_channels):\n",
    "    layers = []\n",
    "    layers.append(nn.BatchNorm2d(input_channels))\n",
    "    layers.append(nn.ReLU())\n",
    "    layers.append(nn.Conv2d(input_channels, num_channels, kernel_size=1))\n",
    "    layers.append(nn.AvgPool2d(kernel_size=2, stride=2))\n",
    "    blk = nn.Sequential(*layers)\n",
    "    return blk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply a transition layer with 10 channels to the output of the dense block in the previous example. This\n",
    "reduces the number of output channels to 10, and halves the height and width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 10, 4, 4])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "blk = transition_block(23, 10)\n",
    "blk(Y).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DenseNet Model\n",
    "\n",
    "Next, we will construct a DenseNet model. DenseNet first uses the same single convolutional layer and\n",
    "maximum pooling layer as ResNet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Flatten(torch.nn.Module):\n",
    "    def forward(self, x):\n",
    "        return x.view(x.shape[0], -1)\n",
    "\n",
    "class Reshape(torch.nn.Module):\n",
    "    def forward(self, x):\n",
    "        return x.view(-1,1,96,96)\n",
    "\n",
    "layers = []\n",
    "layers.append(Reshape())\n",
    "layers.append(nn.Conv2d(in_channels=1, out_channels=64, kernel_size=7, stride=2, padding=3))\n",
    "layers.append(nn.BatchNorm2d(64))\n",
    "layers.append(nn.ReLU())\n",
    "layers.append(nn.MaxPool2d(kernel_size=3, stride=2, padding=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, similar to the four residual blocks that ResNet uses, DenseNet uses four dense blocks. Similar to\n",
    "ResNet, we can set the number of convolutional layers used in each dense block. Here, we set it to 4,\n",
    "consistent with the ResNet-18 in the previous section. Furthermore, we set the number of channels (i.e.\n",
    "growth rate) for the convolutional layers in the dense block to 32, so 128 channels will be added to each\n",
    "dense block.\n",
    "\n",
    "In ResNet, the height and width are reduced between each module by a residual block with a stride of 2.\n",
    "Here, we use the transition layer to halve the height and width and halve the number of channels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Num_channels: the current number of channels\n",
    "num_channels, growth_rate = 64, 32\n",
    "num_convs_in_dense_blocks = [4, 4, 4, 4]\n",
    "for i, num_convs in enumerate(num_convs_in_dense_blocks):\n",
    "    layers.append(DenseBlock(num_convs, num_channels, growth_rate))\n",
    "    # This is the number of output channels in the previous dense block\n",
    "    num_channels += num_convs * growth_rate\n",
    "    # A transition layer that haves the number of channels is added between\n",
    "    # the dense blocks\n",
    "    if i != len(num_convs_in_dense_blocks) - 1:\n",
    "        layers.append(transition_block(num_channels, num_channels // 2))\n",
    "        num_channels = num_channels // 2\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to ResNet, a global pooling layer and fully connected layer are connected at the end to produce\n",
    "the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "layers.append(nn.BatchNorm2d(num_channels))\n",
    "layers.append(nn.ReLU())\n",
    "layers.append(nn.AdaptiveMaxPool2d((1,1)))\n",
    "layers.append(Flatten())\n",
    "layers.append(nn.Linear(num_channels, 10))\n",
    "\n",
    "net = nn.Sequential(*layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Acquisition and Training\n",
    "\n",
    "Since we are using a deeper network here, in this section, we will reduce the input height and width from\n",
    "224 to 96 to simplify the computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr, num_epochs, batch_size, device = 0.1, 5, 64, d2l.try_gpu()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training on cuda:0\n",
      "epoch 1, loss 0.0070, train acc 0.835, test acc 0.833, time 340.7 sec\n",
      "epoch 2, loss 0.0043, train acc 0.897, test acc 0.882, time 346.0 sec\n",
      "epoch 3, loss 0.0036, train acc 0.913, test acc 0.870, time 354.0 sec\n",
      "epoch 4, loss 0.0032, train acc 0.925, test acc 0.825, time 371.7 sec\n",
      "epoch 5, loss 0.0029, train acc 0.933, test acc 0.916, time 392.2 sec\n"
     ]
    }
   ],
   "source": [
    "def init_weights(m):\n",
    "    if type(m) == nn.Linear or type(m) == nn.Conv2d:\n",
    "        torch.nn.init.xavier_uniform_(m.weight)\n",
    "        \n",
    "net.apply(init_weights)\n",
    "net = net.to(device)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size, resize=96)\n",
    "train_ch5(net, train_iter, test_iter, criterion, num_epochs, batch_size, device, lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "* In terms of cross-layer connections, unlike ResNet, where inputs and outputs are added together, DenseNet con-\n",
    "catenates inputs and outputs on the channel dimension.\n",
    "* The main units that compose DenseNet are dense blocks and transition layers.\n",
    "* We need to keep the dimensionality under control when composing the network by adding transition layers that\n",
    "shrink the number of channels again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n",
    "\n",
    "1) Why do we use average pooling rather than max-pooling in the transition layer?\n",
    "\n",
    "2) One of the advantages mentioned in the DenseNet paper is that its model parameters are smaller than those of\n",
    "ResNet. Why is this the case?\n",
    "\n",
    "3) One problem for which DenseNet has been criticized is its high memory consumption.\n",
    "    i) Is this really the case? Try to change the input shape to 224 × 224 to see the actual (GPU) memory con-\n",
    "    sumption.\n",
    "    ii) Can you think of an alternative means of reducing the memory consumption? How would you need to change\n",
    "    the framework?\n",
    "    \n",
    "4) Implement the various DenseNet versions presented in Table 1 of the [$ DenseNet paper^{151} $](https://arxiv.org/abs/1608.06993) .\n",
    "\n",
    "5) Why do we not need to concatenate terms if we are just interested in x and f (x) for ResNet? Why do we need this\n",
    "for more than two layers in DenseNet?\n",
    "\n",
    "6) Design a DenseNet for fully connected networks and apply it to the Housing Price prediction task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "[1] Huang, G., Liu, Z., Weinberger, K. Q., & van der Maaten, L. (2017). Densely connected convolutional networks. In\n",
    "Proceedings of the IEEE conference on computer vision and pattern recognition (Vol. 1, No. 2)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}