lectures/lecture-7/lecture-7.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 7: Graph Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Previous lecture:  Visition Transformers\n",
    "- Original idea of VIT\n",
    "- Beit (3 versions)\n",
    "- Cait\n",
    "- Cross-ViT\n",
    "- SWIN\n",
    "- Transformers for object detection\n",
    "- DINO\n",
    "- MobileVIT (3 versions)\n",
    "- XCiT\n",
    "- MLP-Mixer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Graph Neural Networks\n",
    "- Deep learning on graphs\n",
    "- Basic models\n",
    "- Applications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is a graph\n",
    "\n",
    "So far, we have discussed **images** and **sequences**. There are applications, where the data can be shown as a \n",
    "graph.\n",
    "\n",
    "\n",
    "A graph represents the relations (**edges**) between a collection of entities (**nodes** or **vertices**).\n",
    "\n",
    "Basic concepts include **adjacency matrix** of the graph.\n",
    "\n",
    "\n",
    "A sequence can be thought a particular case of the linear graph.\n",
    "\n",
    "Can we think about some particular applications of graphs? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What features can we find in graphs\n",
    "\n",
    "Graphs in Data Science comes with **different attributes**:\n",
    "\n",
    "- **Vertex (Node) attributes:** what is this node type, or something computed from the graph such as number of neighbours\n",
    "- **Edge (or link) attributes:** what is this link type (edge identity), weight of the edge\n",
    "- **Global attributes:**: number of nodes, longest path\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What kind of problems we need to solve with graphs\n",
    "\n",
    "- Graph classification (given many samples of graphs, classify them)\n",
    "- Node classification\n",
    "- Link prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Graph-level task\n",
    "\n",
    "For these tasks, our goal is to predict a property of a given graph as a whole.\n",
    "\n",
    "For example, this can be **molecule classification**: is this molecule toxic? How does it smell?\n",
    "\n",
    "The main challenge is the same as with text classification: the input data has different sizes, and the architecture should output the vector of the same size.\n",
    "\n",
    "Graph-level tasks are similar to **image classification**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Node-level\n",
    "\n",
    "We need to predict some property of each node of the graph. For example, we recommend the product or we don't recommend the product in recommender systems.\n",
    "\n",
    "\n",
    "Node-level tasks are similar to **image segmentation tasks**.\n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Edge-level\n",
    "\n",
    "We can also try to assign some labels/output to each **edge** of the graph.\n",
    "\n",
    "This can also be **link prediction**: given two nodes determine, if there is an edge between them or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Applications of GNN\n",
    "\n",
    "- social networks\n",
    "- recommendation systems\n",
    "- drug discovery\n",
    "- NP-complete optimization problems on graphs (logistics, even quantum computing).\n",
    "- Computer vision tasks, point clouds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Graph data (1)\n",
    "\n",
    "Molecules can be characterized as graphs, and now it is one of the main tools to build drug design networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Graph data (2)\n",
    "\n",
    "- Social networks as graphs\n",
    "- Citation networks as graphs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How to represent graphs\n",
    "\n",
    "- Representing the graphs using **adjacency matrix** is ok for small graphs.\n",
    "- We can apply CNN on such matrices on images\n",
    "\n",
    "1) Such approach scales badly with the number of vertices: many graphs have million of vertices!\n",
    "2) Many adjacency matrices can encode the same graph (permutation of vertices!). \n",
    "\n",
    "\n",
    "**Also, how to treat graphs of different sizes?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Spectral convolutions on graphs\n",
    "\n",
    "Let $A$ be the adjacency matrix of the graph, and $L$ be the **normalized graph Laplacian of this graph**,\n",
    "\n",
    "$$L = I - D^{-1/2} A D^{-1/2} = U \\lambda U^{\\top}, \\quad D = \\sum_j A_{ij}.$$\n",
    "\n",
    "A standard convolution operator can be thought as elementwise multiplication in the frequency domain:\n",
    "\n",
    "$$g_{\\theta} \\bigstar x = U g_{\\theta}(\\Lambda) U^{\\top},$$\n",
    "where $U$ is a **Fourier transform** and $g(\\theta)$ is applied to the eigenvalues of the normalized graph Laplacian.\n",
    "\n",
    "The transformation $U^{\\top} x$ is called **graph Fourier transform**.\n",
    "\n",
    "How we can get standard convolution using this notation? \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Two different approaches\n",
    "\n",
    "There are two different ways to build graph neural networks.\n",
    "\n",
    "1. [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering, Michaël Defferrard, Xavier Bresson, Pierre Vandergheynst](https://proceedings.neurips.cc/paper/2016/hash/04df4d434d481c5bb723be1b6df1ee65-Abstract.html)\n",
    "2. [Semi-supervised classification with graph convolutional neural networks, Kipf, Welling](https://arxiv.org/pdf/1609.02907.pdf)\n",
    "\n",
    "The first one uses spectral convolutions, the second one proposes to use adjacency matrix for **mixing** the information between different nodes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ChebNet \n",
    "\n",
    "The paper with spectral convolutions was the first. \n",
    "\n",
    "The idea is to apply **polynomial filters** as\n",
    "\n",
    "$$g_{\\theta} \\bigstar x = U g_{\\theta}(\\Lambda) U^{\\top},$$\n",
    "\n",
    "$$g_{\\theta}(\\Lambda) = \\sum_{k=0}^{K-1} \\theta_k \\Lambda^k.$$\n",
    "\n",
    "What is the analogue of the **receptive field** for such kind of convolution? \n",
    "\n",
    "<img src='chebnet.png' width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ChebNet\n",
    "Instead of monomials, one can use **Chebyshev polynomials** instead, parametrizing the convolution with the Chebyshev coefficients.\n",
    "\n",
    "$$g_{\\theta}(\\Lambda) = \\sum_{k=0}^{K-1} c_k T_k(\\Lambda).$$\n",
    "\n",
    "Application of such filter to a signal can be done in $\\mathcal{O}(K)$ matrix-by-vector products with the matrix $L$.\n",
    "\n",
    "A signal is a vector defined in each node of the graph!\n",
    "\n",
    "Then, the paper proposes to use **Pooling** by simple graph coarsening technique. \n",
    "\n",
    "\n",
    "\\begin{equation}\n",
    "\\text{Choose unmarked vertex } i, \\text{and match it with an unmarked neighbor } j \\\\\n",
    "\\text{such that } W_{ij} \\left(\\frac{1}{d_i} + \\frac{1}{d_j}\\right) \\text{ is maximized}\n",
    "\\end{equation}\n",
    "\n",
    "where $W_{ij}$ is the weight of the edge between vertices $i$ and $j$, $d_i$ and $d_j$ are the degrees of vertices $i$ and $j$, respectively.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Graph convolutional networks\n",
    "\n",
    "The work of Kimpf and Welling proposes another idea which uses adjacency matrix.\n",
    "\n",
    "\\begin{equation}\n",
    "H^{(l+1)} = \\sigma\\left(\\hat{D}^{-\\frac{1}{2}}\\hat{A}\\hat{D}^{-\\frac{1}{2}}H^{(l)}W^{(l)}\\right), \\quad \\hat{A} = I + A, \\quad\\hat{D}_i = \\sum_j \\hat{A}_{ij}.\n",
    "\\end{equation}\n",
    "\n",
    "Matrix $\\hat{A}$ is the adjacency matrix with added self-connection, $H^{(l)} \\in \\mathbb{R}^{N \\times D}$ is the matrix of activations in the $l$-th layer.\n",
    "\n",
    "This implements a mapping $f(X, A)$.\n",
    "\n",
    "We can use this for node classification!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Message passing networks\n",
    "\n",
    "The models above are **node-based**, edges are not involved.\n",
    "\n",
    "We can consider representation where the information is stored as vectors:\n",
    "\n",
    "- On the nodes \n",
    "- On the edges\n",
    "\n",
    "We can then apply MLP to node data; we can also apply MLP to the edge data. \n",
    "\n",
    "There is **no exchange** between edges and nodes!\n",
    "\n",
    "We can do simple **global pooling** (like averaging).\n",
    "\n",
    "For more complex interface a **message passing** mechanism appears."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Message passing networks (2)\n",
    "\n",
    "In has been proposed in a paper [Message passing neural networks in quantum chemistry](http://proceedings.mlr.press/v70/gilmer17a/gilmer17a.pdf)\n",
    "\n",
    "The task was predicting the energy of the molecule from its graph.\n",
    "\n",
    "During the message passing phase, hidden states $h^t_u$ \n",
    "at each node in the graph are\n",
    "updated based on messages $m^{(t+1)}_v$\n",
    "according to\n",
    "\n",
    "\\begin{align*}\n",
    "m_{t+1}^v &= \\sum_{w\\in N(v)} M_t(h_t^v, h_t^w, e_{vw}) \\\\\n",
    "h_{t+1}^v &= U_t(h_t^v, m_{t+1}^v)\n",
    "\\end{align*}\n",
    "\n",
    "Then, we can collect the features for the graph.\n",
    "\n",
    "Functions $M$, $U$ are learnable functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Attention in graphs\n",
    "\n",
    "[Graph Attention Networks\n",
    "Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio](https://arxiv.org/abs/1710.10903)\n",
    "\n",
    "Proposes graph variant of **attention mechanism**.\n",
    "\n",
    "It is very natural to look for attention mechanism, because it allows for variable-size input.\n",
    "\n",
    "The attention is computed only for the **neighbours** in a graph using the standard **self-attention** block.\n",
    "\n",
    "$$\\alpha_{ij}=\\frac{\\exp(e_{ij})}{\\sum_{k\\in N_i}\\exp(e_{ik})}$$\n",
    "\n",
    "Then we take a linear combintation with those features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Summary\n",
    "\n",
    "A typical GNN does\n",
    "\n",
    "a) Independent node transformations\n",
    "\n",
    "b) Aggregation of the node features in the form of **message passing** or **coarsening** of the graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## GIN architecture (and how powerful are GNN)\n",
    "\n",
    "[How Powerful are Graph Neural Networks?\n",
    "Keyulu Xu, Weihua Hu, Jure Leskovec, Stefanie Jegelka\n",
    "](https://arxiv.org/abs/1810.00826)\n",
    "\n",
    "This paper starts from a question: can the graph neural networks distiguish two isomorphic graphs? \n",
    "\n",
    "This will help to show **expressiveness** of GNN architecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is graph isomorphism\n",
    "\n",
    " The graph isomorphism problem asks whether two graphs are **topologically\n",
    "identical**. No polynomial-time algorithm is known for this task.\n",
    "\n",
    "**Definition**: two graphs are considered isomorphic if there is a mapping between the nodes of the graphs that preserves node adjacencies\n",
    "\n",
    "\n",
    "**Weisfeiler-Lehman (WL)** test of graph isomorphism has been proposed in 1968 is an effective and computationally efficient test that works for large set of graphs.\n",
    "\n",
    "<img src='graph-isomorphism.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Weisfeiler-Lehman test\n",
    "\n",
    "The WL algorithm produces for each graph its **canonical form**. If the canonical forms are different, the two graphs are definitely different. \n",
    "\n",
    "- For iteration  of the algorithm we will be assigning to each node a tuple $L_{i, n}$ containing the node’s old compressed label and a multiset of the node’s neighbors' compressed labels. A multiset is a set (a collection of elements where order is not important) where elements may appear multiple times.\n",
    "- At each iteration we will additionally be assigning to each node a new “compressed” label $C_{i, n}$ for that node’s set of labels. Any two nodes with the same $L_{i, n}$ will get the same compressed label.\n",
    "\n",
    "**Algorithm**:\n",
    "\n",
    "- Start with $C_{0, k} = 1$ for all nodes\n",
    "- Build $L_{i, n}$ as the multiset of the previous compressed label at this node and all neighbouring labels. All nodes with the same $L$ will get the same $C$ (algorithmically, we can store the hash of the set).\n",
    "- We stop if the number of iterations reached, or the labels do not change.\n",
    "\n",
    "Note: at the start each node will get the label equal to the degree of the vertex."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Expressive power of GNN\n",
    "\n",
    "If GNN maps graphs to different embeddings, the WL test does so as well!\n",
    "\n",
    "(So, we need to find an architecture that is as powerful as WL).\n",
    "\n",
    "**Lemma**\n",
    "\n",
    "\n",
    "Let $G_1$ and $G_2$ be any two non-isomorphic graphs. If a graph neural network $A: G \\rightarrow \\mathbb{R}^d$ maps $G_1$ and $G_2$ to different embeddings, then the Weisfeiler-Lehman graph isomorphism test also decides that $G_1$ and $G_2$ are not isomorphic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Opposite result:\n",
    "\n",
    "For a certain conditions, if WL tests gives positive results, GNN maps to different embeddings.\n",
    "\n",
    "Let $A : G \\rightarrow \\mathbb{R}^d$  be a graph neural network (GNN). With a sufficient number of GNN layers, A maps any graphs $G_1$ and $G_2$ that the Weisfeiler-Lehman test of isomorphism decides as non-isomorphic, to different embeddings if the following conditions hold: \n",
    "\n",
    "- a) A aggregates and updates node features iteratively with  \n",
    "$h^{(k)}_v = \\varphi(h^{(k-1)}_v, f(\\{h^{(k-1)}_u\\}_{u \\in N(v)})),$  where the functions f, which operates on multisets, and $\\varphi$ are injective.\n",
    "\n",
    "- b) A's graph-level readout, which operates on the multiset of node features $\\{h^{(k)}_v\\}_{v \\in G}$, is injective."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Graph Isomorphism network (GIN)\n",
    "\n",
    "The node update reads (theory say with probability one it will work, looks like ResNet-type architecture)\n",
    "\n",
    "$h^{(k)}_v = \\text{MLP}^{(k)}\\left((1-\\varepsilon_k) h^{(k-1)}_v + \\varepsilon_k \\sum_{u \\in N(v)} h^{(k-1)}_u\\right).$\n",
    "\n",
    "$h_G = \\text{CONCAT}\\left(\\text{READOUT}\\left(h^{(k)}_v \\mid v \\in G \\right)\\right)_{v \\in G, k=0,1,...,K}$\n",
    "\n",
    "We do not **average** the inbetween mode features, we stack them alltogether."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Challenges in Graph Neural Networks\n",
    "\n",
    "One of the problems in GNN is **oversmoothing**: the node representations tend to become indistinguishable if the number of layers increase.\n",
    "\n",
    "In practice, it means that deeper graph networks do not lead to better accuracy.\n",
    "\n",
    "This problem has been studied by adding [residual connections](http://proceedings.mlr.press/v119/chen20v/chen20v.pdf)\n",
    "or by the technique known as **Dropout** (in the context of graph neural networks, [DropEdge](https://arxiv.org/abs/1907.10903) ).\n",
    "\n",
    "Also, we can use [normalisation (PairNorm)](https://static.aminer.cn/upload/pdf/program/5e5e18d593d709897ce3398b_0.pdf).\n",
    "\n",
    "Still, as far a I know, depth does not help and typically 2-4 layers are used in GNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## PairNorm\n",
    "\n",
    "The idea of PairNorm is quite interesting.\n",
    "\n",
    "The graph convolution can be proposed from the following optimization problem:\n",
    "\n",
    "$$\\min_{\\hat{X}} \\sum_i \\Vert \\hat{x}_i - x_i \\Vert^2 + \\sum_{(i, j) \\in E} \\Vert \\hat{x}_i - \\hat{x}_j \\Vert^2,$$\n",
    "\n",
    "the solution gives the graph Laplacian.\n",
    "\n",
    "The problem is that after such transformation, the node embeddings become closer within a cluster. \n",
    "\n",
    "The PairNorm normalized $\\hat{X}$ in such a way that the total sum of squares of distances remains unchanged!\n",
    "\n",
    "<div style=\"display: flex\">\n",
    "    <div style=\"flex: 1; margin-right:0px; width: 100%\">  \n",
    "        <br>\n",
    "            <img src=\"pairnorm.png\" width=\"100%\">\n",
    "    </div>\n",
    "    <div style=\"flex: 1; margin-top: 20px\">\n",
    "        <img src=\"accuracy-pairnorm.png\" width=\"100%\">\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Combinatorial optimization with GNN\n",
    "\n",
    "One of the most interesting applications of GNN is **combinatorial optimization** (see, for example [review](https://arxiv.org/pdf/2102.09544.pdf))\n",
    "\n",
    "For example: **traveling salesman problem**. How we can use machine learning for it?\n",
    "\n",
    "<img src='Traveling-Salesman-Problem.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Idea\n",
    "\n",
    "Use supervised learning! \n",
    "\n",
    "The first idea was a [Pointer Network (PtrNet) by Vinyals et. al](https://proceedings.neurips.cc/paper/2015/hash/29921001f2f04bd3baee84a12e98098f-Abstract.html) \n",
    "who mapped a two-dimensional point cloud to an optimal tour.\n",
    "\n",
    "We collect **graphs** and optimal tours, and learn the mapping in a supervised way!\n",
    "\n",
    "Another idea: compute the probability of the edge to be in the tour. Then use beam search to find the optimal one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Software for GNN\n",
    "\n",
    "PyG: https://www.pyg.org/ \n",
    "\n",
    "Deep Graph Library: https://www.dgl.ai/\n",
    "\n",
    "PyG is mostly a collection of fast re-implementations of existing models (with custom sparse ops). DGL introduces a useful higher-level abstraction, allowing for auto-batching (but is somewhat slower)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## We covered the basic stuff only...\n",
    "\n",
    "Much more operations with the links to the papers can be found here:\n",
    "\n",
    "https://pytorch-geometric.readthedocs.io/en/latest/cheatsheet/gnn_cheatsheet.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "## Summary\n",
    "\n",
    "- Deep learning on graphs\n",
    "- Basic models\n",
    "- Applications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Next lecture: General tricks for efficient training:\n",
    "\n",
    "- Initialization (talked already a little bit)\n",
    "- Data augmentation\n",
    "- Neural network compression/pruning \n",
    "- Clipping \n",
    "- multi-precision training\n",
    "- simple distributed training primitives"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Слайд-шоу",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  },
  "nav_menu": {},
  "rise": {
   "scroll": true,
   "theme": "sky",
   "transition": "zoom"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}