feed.xml

<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.5.2">Jekyll</generator><link href="https://www.dgl.ai/feed.xml" rel="self" type="application/atom+xml" /><link href="https://www.dgl.ai/" rel="alternate" type="text/html" /><updated>2024-08-13T10:18:24+00:00</updated><id>https://www.dgl.ai/feed.xml</id><title type="html">Deep Graph Library</title><subtitle>Easy Deep Learning on Graphs</subtitle><entry><title type="html">GNN training acceleration with BFloat16 data type on CPU</title><link href="https://www.dgl.ai/blog/2024/08/10/bfloat16.html" rel="alternate" type="text/html" title="GNN training acceleration with BFloat16 data type on CPU" /><published>2024-08-10T00:00:00+00:00</published><updated>2024-08-10T00:00:00+00:00</updated><id>https://www.dgl.ai/blog/2024/08/10/bfloat16</id><content type="html" xml:base="https://www.dgl.ai/blog/2024/08/10/bfloat16.html">&lt;p&gt;Graph neural networks (GNN) have achieved state-of-the-art performance on
various industrial tasks. However, most GNN operations are memory-bound and
require a significant amount of RAM. To tackle this problem well known
technique to reduce tensor size by using small data type is proposed for
memory efficiency optimization of GNN training on Intel® Xeon® Scalable
processors with Bfloat16. The proposed approach could achieve outstanding
optimization on various GNN models, covering a wide range of datasets, which
speeds up the training by up to 5×.&lt;/p&gt;

&lt;h2 id=&quot;bfloat16-data-type&quot;&gt;Bfloat16 data type&lt;/h2&gt;

&lt;p&gt;Bfloat16 is a half-precision data type. It differs from the default float data
type only in mantissa length, which is 7 bits long compared to 23 bits.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2024-08-10-bfloat16/bfloat16.png&quot; alt=&quot;bfloat16&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Bfloat16 was developed by the Google Brain team and is currently used widely in
DNN and other AI applications. A lot of devices natively support bfloat16
starting from GPUs and AI accelerators and ending with CPUs. Even compilers such
as GCC, and LLVM are enabling this data type in the latest &lt;a href=&quot;https://en.cppreference.com/w/cpp/types/floating-point&quot;&gt;C/C++ standards&lt;/a&gt;.
According to the Google Brain team exponent is more valuable for training and ML
operations, so such reduction of mantissa bits will preserve the accuracy of the
model and at the same time provide the same performance as other half-precision
data types. Another advantage of using bfloat16 data type is the simplicity of
conversion between bfloat16 and float.&lt;/p&gt;

&lt;h2 id=&quot;bfloat16-cpu-acceleration&quot;&gt;Bfloat16 CPU acceleration&lt;/h2&gt;

&lt;p&gt;Starting from the 3rd Gen Intel® Xeon® Scalable processor (codenamed Cooper Lake)
x86 CPU natively supports bfloat16. It was enabled via the Intel® Advanced Vector
Extensions-512 (Intel® AVX-512): AVX512_BF16 vector instruction set, which like
other AVX512 instructions provides basic operations: dot product and conversion
functions.
In the latest 4th Gen Intel® Xeon® Scalable processors (codename Sapphire Rapids),
the Intel® &lt;a href=&quot;https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/advanced-matrix-extensions/overview.html&quot;&gt;AMX&lt;/a&gt;
instruction set was introduced to further improve 16-bit and 8-bit matrix
performance. In this instruction set “tile” instructions were added, which operate
on special “tile” 2D registers. Currently, this instruction set has only tile
matrix multiply unit (TMUL) support, that can perform matrix multiplication for
bfloat16 and int8 data types.
In the next Intel Xeon generations, starting from Granite Rapids, in addition to
bfloat16, fp16 will be supported.&lt;/p&gt;

&lt;h2 id=&quot;bfloat16-in-dgl&quot;&gt;Bfloat16 in DGL&lt;/h2&gt;

&lt;p&gt;Recently bfloat16 support was added to DGL library (starting from &lt;a href=&quot;https://github.com/dmlc/dgl/releases/tag/1.0.0&quot;&gt;DGL version 1.0.0&lt;/a&gt;
for Nvidia GPU and &lt;a href=&quot;https://github.com/dmlc/dgl/releases/tag/1.1.0&quot;&gt;DGL version 1.1.0&lt;/a&gt;
for CPU), so it is possible to use it in model training and inference for both
CPU and GPU devices.
Examples of DGL API, which will help to convert the graph and the model to
bfloat16 data type:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c&quot;&gt;# Convert graph, model, and graph features to bfloat16&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to_bfloat16&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dtype&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;bfloat16&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;model&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dtype&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;bfloat16&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;The following example trains &lt;a href=&quot;https://snap.stanford.edu/graphsage/&quot;&gt;GraphSAGE&lt;/a&gt;
with bfloat16 using the provided API:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch.nn&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;nn&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch.nn.functional&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;F&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.data&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;CiteseerGraphDataset&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.nn&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;SAGEConv&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.transforms&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AddSelfLoop&lt;/span&gt;


&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;SAGE&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;in_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hid_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;out_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;nb&quot;&gt;super&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;layers&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ModuleList&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
        &lt;span class=&quot;c&quot;&gt;# two-layer SAGE&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;layers&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;append&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;SAGEConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;in_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hid_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;gcn&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;layers&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;append&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;SAGEConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hid_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;out_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;gcn&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Dropout&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;0.5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;


    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;features&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;features&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;l&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;layer&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;enumerate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;layers&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;layer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
            &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;l&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;!=&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;layers&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;F&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;relu&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
                &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
            &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;


&lt;span class=&quot;c&quot;&gt;# Data loading&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AddSelfLoop&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;CiteseerGraphDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;


&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;int&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;train_mask&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'train_mask'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'feat'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;label&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'label'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;


&lt;span class=&quot;n&quot;&gt;in_size&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;shape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;hid_size&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;16&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;out_size&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_classes&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;model&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;SAGE&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;in_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hid_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;out_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;


&lt;span class=&quot;c&quot;&gt;# Convert model and graph to bfloat16&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to_bfloat16&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dtype&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;bfloat16&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;model&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dtype&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;bfloat16&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;


&lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;train&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;


&lt;span class=&quot;c&quot;&gt;# Create optimizer&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;optimizer&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;optim&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Adam&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;parameters&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;lr&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;1e-2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;weight_decay&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;5e-4&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;loss_fcn&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CrossEntropyLoss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;


&lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;epoch&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;range&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;100&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;logits&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;loss_fcn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;logits&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;train_mask&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;label&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;train_mask&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;


    &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;backward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;optimizer&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;step&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;


    &lt;span class=&quot;k&quot;&gt;print&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'Epoch {} | Loss {}'&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;format&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;epoch&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;item&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;h2 id=&quot;experimental-results&quot;&gt;Experimental results&lt;/h2&gt;

&lt;p&gt;The most popular examples have been tested – &lt;a href=&quot;https://arxiv.org/abs/1312.6203&quot;&gt;GCN&lt;/a&gt;,
&lt;a href=&quot;https://arxiv.org/abs/1706.02216&quot;&gt;GraphSAGE&lt;/a&gt;. For full graph training, basic
datasets were chosen, while for the mini-batch approach datasets from &lt;a href=&quot;https://ogb.stanford.edu/docs/nodeprop/&quot;&gt;OGB&lt;/a&gt;
were used, the sizes of which significantly exceed the sizes of the basic ones.
For instance, ogbn-products has around 2.5 million nodes and 61 million edges,
whereas ogbn-papers100M has 111 million nodes and 1.6 million edges. Table 1.
demonstrates accuracy which is similar for float and bfloat16 or has not been
changed significantly.&lt;/p&gt;

&lt;table style=&quot;text-align: center;&quot;&gt;
   &lt;tr&gt;
      &lt;th&gt;Model&lt;/th&gt;
      &lt;th&gt;Dataset&lt;/th&gt;
      &lt;th&gt;Test accuracy(float)&lt;/th&gt;
      &lt;th&gt;Test accuracy(bfloat16)&lt;/th&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;gcn&lt;/td&gt;
      &lt;td&gt;citeseer&lt;/td&gt;
      &lt;td&gt;71%&lt;/td&gt;
      &lt;td&gt;71%&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;gcn&lt;/td&gt;
      &lt;td&gt;cora&lt;/td&gt;
      &lt;td&gt;81%&lt;/td&gt;
      &lt;td&gt;81%&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;gcn&lt;/td&gt;
      &lt;td&gt;pubmed&lt;/td&gt;
      &lt;td&gt;79%&lt;/td&gt;
      &lt;td&gt;79%&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;graphsage&lt;/td&gt;
      &lt;td&gt;citeseer&lt;/td&gt;
      &lt;td&gt;71%&lt;/td&gt;
      &lt;td&gt;71%&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;graphsage&lt;/td&gt;
      &lt;td&gt;cora&lt;/td&gt;
      &lt;td&gt;81%&lt;/td&gt;
      &lt;td&gt;81%&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;graphsage&lt;/td&gt;
      &lt;td&gt;pubmed&lt;/td&gt;
      &lt;td&gt;78%&lt;/td&gt;
      &lt;td&gt;78%&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;gcn minibatch&lt;/td&gt;
      &lt;td&gt;ogbn-paper100M&lt;/td&gt;
      &lt;td&gt;57%&lt;/td&gt;
      &lt;td&gt;57%&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;gcn minibatch&lt;/td&gt;
      &lt;td&gt;ogbn-products&lt;/td&gt;
      &lt;td&gt;78%&lt;/td&gt;
      &lt;td&gt;78%&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;graphsage minibatch&lt;/td&gt;
      &lt;td&gt;ogbn-paper100M&lt;/td&gt;
      &lt;td&gt;62%&lt;/td&gt;
      &lt;td&gt;61%&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;graphsage minibatch&lt;/td&gt;
      &lt;td&gt;ogbn-products&lt;/td&gt;
      &lt;td&gt;76%&lt;/td&gt;
      &lt;td&gt;74%&lt;/td&gt;
   &lt;/tr&gt;
&lt;/table&gt;

&lt;p&gt;The plots below show results on &lt;a href=&quot;https://aws.amazon.com/ec2/instance-types/r6i/&quot;&gt;AWS r6i&lt;/a&gt;
powered by 3rd Generation Intel Xeon Scalable processors (codename Ice Lake),
which does not have native bfloat16 instruction, and results on &lt;a href=&quot;https://aws.amazon.com/ec2/instance-types/r7iz/&quot;&gt;AWS r7iz&lt;/a&gt;
4th Generation Intel Xeon Scalable-based (Sapphire Rapids) instances, which has
native support for both AVX512_BF16 and AMX. In both experiments, the number of
threads is limited to 16, which is the best-known number of threads for a single
run on Intel® Xeon®.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2024-08-10-bfloat16/plot_1.png&quot; alt=&quot;plot_1&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;
&lt;img src=&quot;/assets/images/posts/2024-08-10-bfloat16/plot_2.png&quot; alt=&quot;plot_2&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;The efficiency of GNN training has been enhanced on both types of Intel® Xeon®
instances with bfloat16. Notably, for basic datasets on the AWS r6i, there was
an improvement in performance of up to 32%. Similarly, for basic datasets on the
r7iz accelerated by Intel® AMX machine, the use of bfloat16 led to an improvement
in training performance of up to 92%.
When discussing the results of &lt;a href=&quot;https://ogb.stanford.edu/docs/nodeprop/&quot;&gt;OGB&lt;/a&gt;
datasets, which are notably larger in size, performance improved by up to 2.89
times on the r6i and up to 5.04 times on the r7iz. On the provided plots it was
demonstrated that all training steps experienced improvements in performance.
The most significant impact was observed in the forward pass, which was up to
12.7 times faster when utilizing bfloat16.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2024-08-10-bfloat16/plot_3.png&quot; alt=&quot;plot_3&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;
&lt;img src=&quot;/assets/images/posts/2024-08-10-bfloat16/plot_4.png&quot; alt=&quot;plot_4&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;
&lt;img src=&quot;/assets/images/posts/2024-08-10-bfloat16/plot_5.png&quot; alt=&quot;plot_5&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;
&lt;img src=&quot;/assets/images/posts/2024-08-10-bfloat16/plot_6.png&quot; alt=&quot;plot_6&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;h2 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Using the bfloat16 data type is strongly recommended for GNN training on Sapphire
Rapids and earlier generations of Intel Xeon Scalable processors to enhance
performance. Even on Ice Lake, bfloat16 enhances the efficiency of memory-bound
operations and reduces training costs.&lt;/p&gt;

&lt;p&gt;Nevertheless, certain methods within DL frameworks may not fully support or
optimally use CPU bfloat16 instructions. In such situations, we advise evaluating
both float and bfloat16 performance over a small number of epochs to determine
the optimal choice.&lt;/p&gt;</content><author><name>Ilia Taraban</name></author><category term="blog" /><category term="blog" /><summary type="html">Graph neural networks (GNN) have achieved state-of-the-art performance on various industrial tasks. However, most GNN operations are memory-bound and require a significant amount of RAM. To tackle this problem well known technique to reduce tensor size by using small data type is proposed for memory efficiency optimization of GNN training on Intel® Xeon® Scalable processors with Bfloat16. The proposed approach could achieve outstanding optimization on various GNN models, covering a wide range of datasets, which speeds up the training by up to 5×.</summary></entry><entry><title type="html">DGL 2.1: GPU Acceleration for Your GNN Data Pipeline</title><link href="https://www.dgl.ai/release/2024/03/06/release.html" rel="alternate" type="text/html" title="DGL 2.1: GPU Acceleration for Your GNN Data Pipeline" /><published>2024-03-06T00:00:00+00:00</published><updated>2024-03-06T00:00:00+00:00</updated><id>https://www.dgl.ai/release/2024/03/06/release</id><content type="html" xml:base="https://www.dgl.ai/release/2024/03/06/release.html">&lt;p&gt;We are happy to announce the release of DGL 2.1. In this release, we are making
GNN data loading lightning fast. We introduce GPU acceleration for the whole GNN
data loading pipeline in GraphBolt, including the graph sampling and feature
fetching stages.&lt;/p&gt;

&lt;h2 id=&quot;flexible-data-pipeline--customizable-stages-all-accelerated-on-your-gpu&quot;&gt;Flexible data pipeline &amp;amp; customizable stages, all accelerated on your GPU&lt;/h2&gt;

&lt;p&gt;Starting from this release, the data moving stage can now be moved earlier in
the data pipeline to enable GPU acceleration. With this in mind, the following
permutations of the core stages are now possible:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2024-03-06-release/workstream.png&quot; alt=&quot;diagram&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;To execute all of the data loading stages on the GPU, the graph and the features
need to be GPU accessible. As the GPU memory may be limited, GraphBolt offers an
in-place pinning operation to enable GPU access to the graph and features
resident in main memory.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c&quot;&gt;# Pin the graph and features in-place.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;pin_memory_&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;features&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;feature&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;pin_memory_&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;However, if GPU has sufficient memory, either graph and/or its features can be
moved to the GPU as follows:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c&quot;&gt;# Move the graph and features to the GPU.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;cuda:0&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;features&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;feature&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;cuda:0&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;It may be the case that the GPU has a large memory, however it may not be large
enough to fit all the features. In that case, it is possible to cache some part
of the features using &lt;a href=&quot;https://docs.dgl.ai/generated/dgl.graphbolt.GPUCachedFeature.html#dgl.graphbolt.GPUCachedFeature&quot;&gt;gb.GPUCachedFeature&lt;/a&gt;,
please see the &lt;a href=&quot;https://github.com/dmlc/dgl/blob/3ced3411e55bca803ed5ec5e1de6f62e1f21478f/examples/multigpu/graphbolt/node_classification.py#L288-L292&quot;&gt;GraphBolt multiGPU example&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;By placing the copy operation earlier in the pipeline enables GPU execution for
the rest of the operations. All of the GraphBolt components compose as you
expect.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c&quot;&gt;# Seed edge sampler.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;gb&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ItemSampler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;train_edge_set&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;batch_size&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1024&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;shuffle&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Copy here to execute the remaining operations on the GPU.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;copy_to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;cuda:0&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Negative sampling.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sample_uniform_negative&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;negative_ratio&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Neighbor sampling.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sample_neighbor&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;fanouts&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;15&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Fetch features.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;fetch_feature&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;features&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node_feature_keys&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;feat&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;The descriptive nature of the PyTorch datapipe lets us take a defined data
pipeline, make modifications to it to support GPU specific optimizations with no
change to the user experience. Two such examples are the &lt;code class=&quot;highlighter-rouge&quot;&gt;overlap_feature_fetch&lt;/code&gt;
and &lt;code class=&quot;highlighter-rouge&quot;&gt;overlap_graph_fetch&lt;/code&gt; arguments of &lt;a href=&quot;https://docs.dgl.ai/en/latest/generated/dgl.graphbolt.DataLoader.html&quot;&gt;gb.DataLoader&lt;/a&gt;,
where the feature fetching and graph access operations are overlapped with the
rest of the operations using a separate CUDA stream via pipeline parallelism.&lt;/p&gt;

&lt;h2 id=&quot;gpu-acceleration-speedups&quot;&gt;GPU acceleration speedups&lt;/h2&gt;

&lt;p&gt;The dgl.graphbolt doesn’t just give you flexibility, it also provides top
performance under the hood. As for the 2.1 release, almost all dgl.graphbolt
operations are GPU accelerated, except for sampling with replacement.
Additionally, the feature fetch operation now runs in parallel with everything
else, via pipeline parallelism. This has the potential to cut runtimes by up to
&lt;strong&gt;2x&lt;/strong&gt; depending on the scenario. Moreover, utilizing &lt;a href=&quot;https://docs.dgl.ai/generated/dgl.graphbolt.GPUCachedFeature.html#dgl.graphbolt.GPUCachedFeature&quot;&gt;gb.GPUCachedFeature&lt;/a&gt;
can cut feature transfer times even more, our multi-GPU benchmarks show up to
&lt;strong&gt;1.6x&lt;/strong&gt; speedup.&lt;/p&gt;

&lt;p&gt;To evaluate the performance of GraphBolt, we have tested 4 different scenarios:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Single-GPU Node Classification&lt;/li&gt;
  &lt;li&gt;Single-GPU Link Prediction&lt;/li&gt;
  &lt;li&gt;Single-GPU Heterogeneous Node Classification&lt;/li&gt;
  &lt;li&gt;Multi-GPU Node Classification&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In these scenarios, we will compare 5 different configurations. First two are
the existing baselines, and the last three are new configurations enabled by the
DGL 2.1 release:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;GraphBolt CPU backend, denoted as “dgl.graphbolt (cpu)”.&lt;/li&gt;
  &lt;li&gt;The legacy DGL dataloader with UVA on GPU, denoted as “Legacy DGL (pinned)” by
pinning the dataset in system memory.&lt;/li&gt;
  &lt;li&gt;GraphBolt GPU backend, denoted as “dgl.graphbolt (pinned)” by pinning the
dataset in system memory.&lt;/li&gt;
  &lt;li&gt;GraphBolt GPU backend, denoted as “dgl.graphbolt (pinned, 5M)” by pinning the
dataset in system memory, utilizing &lt;a href=&quot;https://docs.dgl.ai/generated/dgl.graphbolt.GPUCachedFeature.html#dgl.graphbolt.GPUCachedFeature&quot;&gt;gb.GPUCachedFeature&lt;/a&gt;
to cache 5M of the node features.&lt;/li&gt;
  &lt;li&gt;GraphBolt GPU backend, denoted as “dgl.graphbolt (cuda)” by moving the dataset
to the GPU memory.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All the experiments were run on an NVIDIA DGX-A100 system with 8 GPUs.&lt;/p&gt;

&lt;h3 id=&quot;single-gpu-node-classification&quot;&gt;Single-GPU Node Classification&lt;/h3&gt;

&lt;p&gt;We evaluate the performance of GraphBolt and the Legacy DGL dataloader when the
dataset is stored in pinned memory (UVA) against the CPU GraphBolt baseline. We
use a 3 layer GraphSage model with batch size 1024 and fanout 10 in each layer
and evaluate the performance on the ogbn-products and ogbn-papers100M datasets
using the listed baselines above.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2024-03-06-release/single-gpu-node-classification.png&quot; alt=&quot;diagram&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;As one can see, GraphBolt’s new GPU backend can get up to &lt;strong&gt;4.2x&lt;/strong&gt; speedup
compared to the GraphBolt CPU baseline while the legacy DGL dataloader can get
at most &lt;strong&gt;2.5x&lt;/strong&gt;.&lt;/p&gt;

&lt;h3 id=&quot;single-gpu-link-prediction&quot;&gt;Single-GPU Link Prediction&lt;/h3&gt;

&lt;p&gt;Here, we shift our focus to the link prediction scenario on the ogbl-citation2
dataset with a similar setting as the previous section. Here, two different
modes are evaluated, including or excluding reverse edges.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2024-03-06-release/single-gpu-link-prediction.png&quot; alt=&quot;diagram&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;We observe that GraphBolt’s new GPU backend can get up to &lt;strong&gt;5x&lt;/strong&gt; speedup
compared to its CPU baselines. Legacy DGL dataloader is slow due to missing GPU
counterparts of some operations required for link prediction dataloading.&lt;/p&gt;

&lt;h3 id=&quot;single-gpu-heterogeneous-node-classification&quot;&gt;Single-GPU Heterogeneous Node Classification&lt;/h3&gt;

&lt;p&gt;You can accelerate heterogeneous sampling on your GPU as well. The
&lt;a href=&quot;https://github.com/dmlc/dgl/blob/master/examples/sampling/graphbolt/rgcn/hetero_rgcn.py&quot;&gt;R-GCN example&lt;/a&gt;
runtime on the ogbn-mag dataset was 43.5s with the “dgl.graphbolt (cpu)”
baseline and it went down to 25.2s with the new “dgl.graphbolt (pinned)” for a
&lt;strong&gt;1.73x&lt;/strong&gt; speedup. You can expect the speedup numbers to increase as we optimize
the different use cases for the GPU.&lt;/p&gt;

&lt;h3 id=&quot;multi-gpu-node-classification&quot;&gt;Multi-GPU Node Classification&lt;/h3&gt;

&lt;p&gt;Here, we evaluate the node classification use case in the multi-GPU setting with
GraphSage model on the ogbn-papers100M dataset (111M nodes and 3.2B edges). A
similar setting is used as the single-GPU scenario except that each GPU is using
a batch size of 1024 with global batch size increasing linearly with the number
of GPUs.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2024-03-06-release/multi-gpu-node-classification.png&quot; alt=&quot;diagram&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;40GB memory on each of our A100 GPUs can be utilized by the GPU cache feature in
GraphBolt. We can achieve &lt;strong&gt;1.75x&lt;/strong&gt; improvement on a single GPU and &lt;strong&gt;2.31x&lt;/strong&gt;
improvement on 8 GPUs compared to previous state-of-the-art baseline, Legacy DGL
(pinned). Moreover, compared to the GraphBolt CPU baseline, we achieve over
&lt;strong&gt;10x&lt;/strong&gt; improvement.&lt;/p&gt;

&lt;h2 id=&quot;reduced-graph-storage-space-requirements&quot;&gt;Reduced graph storage space requirements&lt;/h2&gt;

&lt;p&gt;Many large-scale graphs and existing GNN datasets have fewer than 2 billion
nodes but more than 2 billion edges. One such example is the ogbn-papers100M
graph with its 111 million nodes and 3.2 billion edges. dgl.graphbolt uses the
CSC (Compressed Sparse Column) format to store your graph in a memory efficient
way. With our latest additions, the memory usage now scales by 4 bytes (int32)
w.r.t. # edges and 8 bytes (int64) w.r.t. # nodes, meaning close to &lt;strong&gt;2x&lt;/strong&gt; space
savings for graph storage by using mixed data types. The provided preprocessing
functionality casts the tensors in your dataset into the smallest data types
automatically for optimum space use and performance such as the edge type
information in the heterogeneous case. With these optimizations, you get &lt;strong&gt;3x&lt;/strong&gt;
space savings for the heterogenous ogb-lsc-mag240m graph compared to our
previous release.&lt;/p&gt;

&lt;h2 id=&quot;whats-more&quot;&gt;What’s more&lt;/h2&gt;

&lt;p&gt;Furthermore, dgl.graphbolt is compatible with Pytorch Geometric as well. In the
figure below, the notation in the parentheses represents where the graph and the
features are placed. “(cpu-cuda)” means that the graph is placed on the CPU
while the features are moved to the GPU. We compare our &lt;a href=&quot;https://github.com/dmlc/dgl/blob/master/examples/sampling/graphbolt/pyg/node_classification_advanced.py&quot;&gt;advanced PyG example&lt;/a&gt;
against the &lt;a href=&quot;https://github.com/pyg-team/pytorch_geometric/blob/master/examples/ogbn_products_sage.py&quot;&gt;PyG official example&lt;/a&gt;,
both using the PyG GraphSAGE model. We run the Node Classification task on the
ogbn-products dataset with [15, 10, 5] fanout.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2024-03-06-release/pyg-graphsage.png&quot; alt=&quot;diagram&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;While providing an extremely optimized Neighbor Sampler implementation, we also
offer a new drop-in replacement called Layer Neighbor Sampler from NeurIPS 2023.
One can see that we provide up to &lt;strong&gt;5.5x&lt;/strong&gt; speedup over PyG, combining GPU
acceleration, pipeline parallelism and the state-of-the-art algorithms. For more
information on the new features in DGL 2.1, please refer to our &lt;a href=&quot;https://github.com/dmlc/dgl/releases/tag/v2.1.0&quot;&gt;release notes&lt;/a&gt;.&lt;/p&gt;

&lt;h2 id=&quot;get-started-with-dgl-21&quot;&gt;Get started with DGL 2.1&lt;/h2&gt;

&lt;p&gt;You can easily install DGL 2.1 with dgl.graphbolt on any platform using &lt;a href=&quot;https://www.dgl.ai/pages/start.html&quot;&gt;pip or conda&lt;/a&gt;.
Dive into our updated &lt;a href=&quot;https://docs.dgl.ai/en/latest/stochastic_training/index.html&quot;&gt;Stochastic Training of GNNs with GraphBolt tutorial&lt;/a&gt;
and experiment with our &lt;a href=&quot;https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/stochastic_training/node_classification.ipynb&quot;&gt;node classification&lt;/a&gt;
and &lt;a href=&quot;https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/stochastic_training/link_prediction.ipynb&quot;&gt;link prediction&lt;/a&gt;
examples in Google Colab. No need to set up a local environment - just point and
click! We also updated the existing &lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/examples/sampling/graphbolt&quot;&gt;7 comprehensive single-GPU examples&lt;/a&gt;
and &lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/examples/multigpu/graphbolt&quot;&gt;1 multi-GPU example&lt;/a&gt;
with GPU Acceleration options. DGL 2.1 will be featured in the NVIDIA DGL
container 24.03 release which will be released before the end of March 2024.&lt;/p&gt;

&lt;p&gt;We welcome your feedback and are available via &lt;a href=&quot;https://github.com/dmlc/dgl/issues&quot;&gt;Github issues&lt;/a&gt; and &lt;a href=&quot;https://discuss.dgl.ai/&quot;&gt;Discuss posts&lt;/a&gt;.
Join our &lt;a href=&quot;http://slack.dgl.ai/&quot;&gt;Slack channel&lt;/a&gt; to stay updated and to connect with the community.&lt;/p&gt;</content><author><name>Muhammed Fatih Balin</name></author><category term="release" /><category term="release" /><summary type="html">We are happy to announce the release of DGL 2.1. In this release, we are making GNN data loading lightning fast. We introduce GPU acceleration for the whole GNN data loading pipeline in GraphBolt, including the graph sampling and feature fetching stages.</summary></entry><entry><title type="html">DGL 2.0: Streamlining Your GNN Data Pipeline from Bottleneck to Boost</title><link href="https://www.dgl.ai/release/2024/01/26/release.html" rel="alternate" type="text/html" title="DGL 2.0: Streamlining Your GNN Data Pipeline from Bottleneck to Boost" /><published>2024-01-26T00:00:00+00:00</published><updated>2024-01-26T00:00:00+00:00</updated><id>https://www.dgl.ai/release/2024/01/26/release</id><content type="html" xml:base="https://www.dgl.ai/release/2024/01/26/release.html">&lt;p&gt;We’re thrilled to announce the release of DGL 2.0, a major milestone in our
mission to empower developers with cutting-edge tools for Graph Neural Networks
(GNNs). Traditionally, data loading has been a significant bottleneck in GNN
training. Complex graph structures and the need for efficient sampling often
lead to slow data loading times and resource constraints. This can drastically
hinder the training speed and scalability of your GNN models. DGL 2.0 breaks
free from these limitations with the introduction of dgl.graphbolt, a
revolutionary data loading framework that supercharges your GNN training by
streamlining the data pipeline.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2024-01-26-release/diagram.png&quot; alt=&quot;diagram&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;center&gt;High-Level Architecture of GraphBolt Data Pipeline&lt;/center&gt;&lt;/p&gt;

&lt;h2 id=&quot;flexible-data-pipeline--customizable-stages&quot;&gt;Flexible data pipeline &amp;amp; customizable stages&lt;/h2&gt;

&lt;p&gt;One size doesn’t fit all - and especially not when it comes to dealing with a
variety of graph data and GNN tasks. For instance, link prediction requires
negative sampling but not node classification, some features are too large to be
stored in memory, and occasionally, we might combine multiple sampling
operations to form subgraphs. To offer adaptable operators while maintaining
high performance, dgl.graphbolt integrates seamlessly with the PyTorch datapipe,
relying on the unified “MiniBatch” data structure to connect processing stages.
The core stages are defined as:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Item Sampling&lt;/strong&gt;: randomly selects a subset (nodes, edges, graphs) from the
entire training set as an initial mini-batch for downstream computation.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Negative Sampling (for Link Prediction)&lt;/strong&gt;: generates non-existing edges as
negative examples.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Subgraph Sampling&lt;/strong&gt;: generates subgraphs based on the input nodes/edges.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Feature Fetching&lt;/strong&gt;: fetches related node/edge features from the dataset for
the given input.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Data Moving (for training on GPU)&lt;/strong&gt;: moves the data to specified device for
training.&lt;/li&gt;
&lt;/ul&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c&quot;&gt;# Seed edge sampler.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;gb&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ItemSampler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;train_edge_set&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;batch_size&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1024&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;shuffle&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Negative sampling.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sample_uniform_negative&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;negative_ratio&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Neighbor sampling.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sample_neighbor&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;fanouts&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;15&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Fetch features.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;fetch_feature&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;features&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node_feature_keys&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;feat&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Copy to GPU for training.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;copy_to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;cuda:0&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;The dgl.graphbolt allows you to plug in your own custom processing steps to
build the perfect data pipeline for your needs, for example:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c&quot;&gt;# Seed edge sampler.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;gb&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ItemSampler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;train_edge_set&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;batch_size&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1024&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;shuffle&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Negative sampling.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sample_uniform_negative&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;negative_ratio&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Neighbor sampling.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sample_neighbor&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;fanouts&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;15&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;# Exclude seed edges.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;gb&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;exclude_seed_edges&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;# Fetch features.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;fetch_feature&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;features&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node_feature_keys&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;feat&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Copy to GPU for training.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;copy_to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;cuda:0&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;The dgl.graphbolt empowers you to customize stages in your data pipelines.
Implement custom stages using pre-defined APIs, such as loading features from
external storage or adding customized caching mechanisms (e.g.
&lt;a href=&quot;https://github.com/dmlc/dgl/blob/0cb309a1b406d896311b5cfc2b5b1a1915f57c3b/python/dgl/graphbolt/impl/gpu_cached_feature.py#L11&quot;&gt;GPUCachedFeature&lt;/a&gt;),
and integrate the custom stages seamlessly without any modifications to your
core training code.&lt;/p&gt;

&lt;h2 id=&quot;speed-enhancement--memory-efficiency&quot;&gt;Speed enhancement &amp;amp; memory efficiency&lt;/h2&gt;

&lt;p&gt;The dgl.graphbolt doesn’t just give you flexibility, it also provides top
performance under the hood. It features a compact graph data structure for
efficient sampling, blazing-fast multi-threaded neighbor sampling operator and
edge exclusion operator, and a built-in option to store large feature tensors
outside your CPU’s main memory. Additionally, The dgl.graphbolt takes care of
scheduling across all hardware, minimizing wait times and maximizing efficiency.&lt;/p&gt;

&lt;p&gt;The dgl.graphbolt brings impressive speed gains to your GNN training, showcasing
over 30% faster node classification in our benchmark and a remarkable ~390%
acceleration for link prediction  in our benchmark that involve edge exclusion.&lt;/p&gt;

&lt;table style=&quot;text-align: center;&quot;&gt;
   &lt;tr&gt;
      &lt;th&gt;Epoch Time(s)&lt;/th&gt;
      &lt;th&gt;GraphSAGE&lt;/th&gt;
      &lt;th&gt;R-GCN&lt;/th&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;DGL Dataloader&lt;/td&gt;
      &lt;td&gt;22.5&lt;/td&gt;
      &lt;td&gt;73.6&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;dgl.graphbolt&lt;/td&gt;
      &lt;td&gt;17.2&lt;/td&gt;
      &lt;td&gt;64.6&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;**Speedup**&lt;/td&gt;
      &lt;td&gt;**1.31x**&lt;/td&gt;
      &lt;td&gt;**1.14x**&lt;/td&gt;
   &lt;/tr&gt;
&lt;/table&gt;
&lt;p&gt;&lt;center&gt;Node classification speedup (NVIDIA T4 GPU). GraphSAGE is tested on OGBN-Products. R-GCN is tested on OGBN-MAG&lt;/center&gt;&lt;/p&gt;

&lt;table style=&quot;text-align: center;&quot;&gt;
   &lt;tr&gt;
      &lt;th&gt;Epoch Time(s)&lt;/th&gt;
      &lt;th&gt;include seeds&lt;/th&gt;
      &lt;th&gt;exclude seeds&lt;/th&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;DGL Dataloader&lt;/td&gt;
      &lt;td&gt;37.75&lt;/td&gt;
      &lt;td&gt;135.32&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;dgl.graphbolt&lt;/td&gt;
      &lt;td&gt;15.51&lt;/td&gt;
      &lt;td&gt;27.62&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;**Speedup**&lt;/td&gt;
      &lt;td&gt;**2.43x**&lt;/td&gt;
      &lt;td&gt;**4.90x**&lt;/td&gt;
   &lt;/tr&gt;
&lt;/table&gt;
&lt;p&gt;&lt;center&gt;Link prediction speedup (NVIDIA T4 GPU) on OGBN-Citation2&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;For memory-constrained training on enormous graphs like OGBN-MAG240m, the
dgl.graphbolt also proves its worth. While both utilize mmap-based optimization,
compared to DGL dataloader, the dgl.graphbolt boasts a substantial speedup. The
dgl.graphbolt’s well-defined component API streamlines the process for
contributors to refine out-of-core RAM solutions for future optimization,
ensuring even the most massive graphs can be tackled with ease.&lt;/p&gt;

&lt;table style=&quot;text-align: center;&quot;&gt;
   &lt;tr&gt;
      &lt;th&gt;Iteration time with different RAM size (s)&lt;/th&gt;
      &lt;th&gt;128GB RAM&lt;/th&gt;
      &lt;th&gt;256GB RAM&lt;/th&gt;
      &lt;th&gt;384GB RAM&lt;/th&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;Naïve DGL dataloader&lt;/td&gt;
      &lt;td&gt;OOM&lt;/td&gt;
      &lt;td&gt;OOM&lt;/td&gt;
      &lt;td&gt;OOM&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;Optimized DGL dataloader&lt;/td&gt;
      &lt;td&gt;65.42&lt;/td&gt;
      &lt;td&gt;3.86&lt;/td&gt;
      &lt;td&gt;0.30&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;dgl.graphbolt&lt;/td&gt;
      &lt;td&gt;60.99&lt;/td&gt;
      &lt;td&gt;3.21&lt;/td&gt;
      &lt;td&gt;0.23&lt;/td&gt;
   &lt;/tr&gt;
&lt;/table&gt;
&lt;p&gt;&lt;center&gt;Node classification on OGBN-MAG240m under different RAM sizes. Optimized DGL dataloader baseline uses mmap to load features.&lt;/center&gt;&lt;/p&gt;

&lt;h2 id=&quot;whats-more&quot;&gt;What’s more&lt;/h2&gt;

&lt;p&gt;Furthermore, DGL 2.0 includes various new additions such as a hetero-relational
GCN example and several datasets. Improvements have been introduced to the
system, examples, and documentation, including updates to the CPU Docker
tcmalloc, supporting sparse matrix slicing operators and enhancements in various
examples. A set of &lt;a href=&quot;https://docs.dgl.ai/api/python/nn-pytorch.html#utility-modules-for-graph-transformer&quot;&gt;utilities&lt;/a&gt; for building graph transformer models is released
along with this version, including NN modules such as positional encoders and
layers as building blocks, and &lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/examples/core/Graphormer&quot;&gt;examples&lt;/a&gt; and &lt;a href=&quot;https://docs.dgl.ai/en/latest/graphtransformer/index.html&quot;&gt;tutorials&lt;/a&gt; demonstrating the usage of
them. Additionally, numerous bug fixes have been implemented, resolving issues
such as the cusparseCreateCsr format for cuda12, addressing the lazy device copy
problem related to DGL node/edge features e.t.c. For more information on the new
additions and changes in DGL 2.0, please refer to our &lt;a href=&quot;https://github.com/dmlc/dgl/releases/tag/v2.0.0&quot;&gt;release note&lt;/a&gt;.&lt;/p&gt;

&lt;h2 id=&quot;get-started-with-dgl-20&quot;&gt;Get started with DGL 2.0&lt;/h2&gt;

&lt;p&gt;You can easily install DGL 2.0 with dgl.graphbolt on any platform using &lt;a href=&quot;https://www.dgl.ai/pages/start.html&quot;&gt;pip or conda&lt;/a&gt;.
To jump right in, dive into our brand-new &lt;a href=&quot;https://docs.dgl.ai/en/latest/stochastic_training/index.html&quot;&gt;Stochastic Training of GNNs with GraphBolt tutorial&lt;/a&gt;
and experiment with our &lt;a href=&quot;https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/stochastic_training/node_classification.ipynb&quot;&gt;node classification&lt;/a&gt;
and &lt;a href=&quot;https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/stochastic_training/link_prediction.ipynb&quot;&gt;link prediction&lt;/a&gt;
examples in Google Colab. No need to set up a local environment - just point and
click! This first release of DGL 2.0 with dgl.graphbolt packs a punch with
&lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/examples/sampling/graphbolt&quot;&gt;7 comprehensive single-GPU examples&lt;/a&gt;
and &lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/examples/multigpu/graphbolt&quot;&gt;1 multi-GPU example&lt;/a&gt;, covering a wide range of tasks.&lt;/p&gt;

&lt;p&gt;We welcome your feedback and are available via &lt;a href=&quot;https://github.com/dmlc/dgl/issues&quot;&gt;Github issues&lt;/a&gt; and &lt;a href=&quot;https://discuss.dgl.ai/&quot;&gt;Discuss posts&lt;/a&gt;.
Join our &lt;a href=&quot;http://slack.dgl.ai/&quot;&gt;Slack channel&lt;/a&gt; to stay updated and to connect with the community.&lt;/p&gt;</content><author><name>DGLTeam</name></author><category term="release" /><category term="release" /><summary type="html">We’re thrilled to announce the release of DGL 2.0, a major milestone in our mission to empower developers with cutting-edge tools for Graph Neural Networks (GNNs). Traditionally, data loading has been a significant bottleneck in GNN training. Complex graph structures and the need for efficient sampling often lead to slow data loading times and resource constraints. This can drastically hinder the training speed and scalability of your GNN models. DGL 2.0 breaks free from these limitations with the introduction of dgl.graphbolt, a revolutionary data loading framework that supercharges your GNN training by streamlining the data pipeline.</summary></entry><entry><title type="html">DGL 1.0: Empowering Graph Machine Learning for Everyone</title><link href="https://www.dgl.ai/release/2023/02/20/release.html" rel="alternate" type="text/html" title="DGL 1.0: Empowering Graph Machine Learning for Everyone" /><published>2023-02-20T00:00:00+00:00</published><updated>2023-02-20T00:00:00+00:00</updated><id>https://www.dgl.ai/release/2023/02/20/release</id><content type="html" xml:base="https://www.dgl.ai/release/2023/02/20/release.html">&lt;p&gt;We are thrilled to announce the arrival of DGL 1.0, a cutting-edge machine
learning framework for deep learning on graphs. Over the past three years,
there has been growing interest from both academia and industry in this
technology. Our framework has received requests from various scenarios, from
academic research on state-of-the-art models to industrial demands for scaling
Graph Neural Network (GNN) solutions to large, real-world problems. With DGL
1.0, we aim to provide a comprehensive and user-friendly solution for all users
to take advantage of graph machine learning.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2023-02-20-release/request.png&quot; alt=&quot;request&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;center&gt;Different levels of user requests and what DGL 1.0 provides to fulfill them&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;DGL 1.0 adopts a layered and modular design to fulfill various user requests. The key features of DGL 1.0 include:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/examples/pytorch&quot;&gt;100+ examples&lt;/a&gt; of
state-of-the-art GNN models, &lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/examples/pytorch/ogb&quot;&gt;15+ top-ranked baselines&lt;/a&gt; on
Open Graph Benchmark (OGB), available for learning and integration&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://docs.dgl.ai/api/python/nn-pytorch.html&quot;&gt;150+ GNN utilities&lt;/a&gt;
including GNN layers, datasets, graph data transform modules, graph samplers,
etc. for building new model architectures or GNN-based solutions&lt;/li&gt;
  &lt;li&gt;Flexible and efficient message passing and sparse matrix abstraction for
developing new GNN building blocks&lt;/li&gt;
  &lt;li&gt;Multi-GPU and distributed training capability to scale to graphs of billions
of nodes and edges&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The new additions and updates in DGL 1.0 are depicted in the accompanying
figure. One of the highlights of this release is the introduction of
&lt;strong&gt;DGL-Sparse&lt;/strong&gt;, a new specialized package for graph ML models defined in sparse
matrix notations. DGL-Sparse streamlines the programming process not just for
well-established GNNs such as Graph Convolutional Networks, but also for the
latest models, including diffusion-based GNNs, hypergraph neural networks, and
Graph Transformers. In the following article, we will provide an overview of
two popular paradigms for expressing GNNs, i.e., the message passing view and
the matrix view, which motivated the creation of DGL-Sparse. We will then show
you how to get started with this new and exciting feature.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2023-02-20-release/arch.png&quot; alt=&quot;arch&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;center&gt;DGL 1.0 stack&lt;/center&gt;&lt;/p&gt;

&lt;h2 id=&quot;the-message-passing-view-vs-the-matrix-view&quot;&gt;The Message Passing View v.s. The Matrix View&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;“It’s the theory that the language you speak determines how you think and affects how you see everything.” — Louise Banks from Film Arrival&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Representing a Graph Neural Network can take two distinct forms. The first,
known as the message passing view, approaches GNN models from a &lt;em&gt;fine-grained,
local&lt;/em&gt; perspective, detailing how messages are exchanged along edges and how
node states are updated accordingly. Alternatively, due to the algebraic
equivalence of a graph to a sparse adjacency matrix, many researchers opt to
express their GNN models from a coarse-grained, global perspective, emphasizing
the operations involving the sparse adjacency matrix and dense feature tensors.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2023-02-20-release/view.png&quot; alt=&quot;view&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;These local and global perspectives are sometimes interchangeable, but more
often, provide complementary insights into the fundamentals and limitations of
GNNs. For instance, the message passing view highlights the connection between
GNNs and the Weisfeiler Lehman (WL) graph isomorphism test, which also relies
on aggregating information from neighbors (as described in &lt;a href=&quot;https://arxiv.org/abs/1810.00826&quot;&gt;Xu et al., 2018&lt;/a&gt;).
Meanwhile, the matrix view provides valuable understanding of the algebraic
properties of GNNs, leading to intriguing findings such as the &lt;em&gt;oversmoothing&lt;/em&gt;
phenomenon (as discussed in &lt;a href=&quot;https://arxiv.org/abs/1801.07606&quot;&gt;Li et al., 2018&lt;/a&gt;). In conclusion, both the message
passing view and matrix view are indispensable tools in studying and describing
GNNs, and this is precisely what motivates the key feature we will be
showcasing in DGL 1.0.&lt;/p&gt;

&lt;h2 id=&quot;dgl-sparse-sparse-matrix-abstraction-for-graph-ml&quot;&gt;DGL Sparse: Sparse Matrix Abstraction for Graph ML&lt;/h2&gt;

&lt;p&gt;In DGL 1.0, we are happy to announce the release of DGL Sparse, a new
sub-package (&lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.sparse&lt;/code&gt;) in addition to the existing message passing interface
in DGL to accomplish the support of the entire spectrum of GNN models. DGL
Sparse provides sparse matrix classes and operations specialized for Graph ML,
making it easier to program GNNs described in the matrix view. In the following
section, we will demonstrate a few examples of GNNs, showcasing their
mathematical formulation and corresponding code implementation in DGL Sparse.&lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;Graph Convolutional Network&lt;/strong&gt; (&lt;a href=&quot;https://arxiv.org/abs/1609.02907&quot;&gt;Kipf et al., 2017&lt;/a&gt;) is one of the pioneer works
in GNN modeling. GCN can be expressed in both message passing view and matrix
view. The code below compares the two different perspectives and
implementations in DGL.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2023-02-20-release/gcn.png&quot; alt=&quot;gcn&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.function&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;fn&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# DGL message passing functions&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;GCNLayer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;
    
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;X&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'X'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;X&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'deg'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;in_degrees&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;float&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;update_all&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;message&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;fn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;sum&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'m'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;'X_neigh'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;X_neigh&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'X_neigh'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;F&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;relu&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;W&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;X_neigh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
    
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;message&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;edges&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;c_ij&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;edges&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;src&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'deg'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;edges&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dst&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'deg'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;**&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;0.5&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'m'&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;edges&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;src&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'X'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;c_ij&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;p&gt;&lt;center&gt;GCN in DGL's message passing API&lt;/center&gt;&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.sparse&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dglsp&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# DGL 1.0 sparse matrix package&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;GCNLayer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;
    
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;A&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;X&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;D_invsqrt&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dglsp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;diag&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;A&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;sum&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;**&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;0.5&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;A_norm&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;D_invsqrt&lt;/span&gt; &lt;span class=&quot;err&quot;&gt;@&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;A&lt;/span&gt; &lt;span class=&quot;err&quot;&gt;@&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;D_invsqrt&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;F&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;relu&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;W&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;A_norm&lt;/span&gt; &lt;span class=&quot;err&quot;&gt;@&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;X&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;p&gt;&lt;center&gt;GCN in DGL Sparse&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Graph Diffusion-based GNNs.&lt;/strong&gt; Graph diffusion is a process of propagating or
smoothing node features/signals along edges. Many classical graph algorithms
such as PageRank belong to this category. A series of research has shown that
combining graph diffusion with neural networks is an effective and efficient
way to enhance model predictions. The equation below describe the core
computation of one representative model — &lt;em&gt;Approximated Personalized Propagation
of Neural Prediction&lt;/em&gt; (&lt;a href=&quot;https://arxiv.org/abs/1810.05997&quot;&gt;Gasteiger et al., 2018&lt;/a&gt;), which can be implemented in DGL
Sparse straightforwardly.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2023-02-20-release/appnp.png&quot; alt=&quot;appnp&quot; width=&quot;300x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;APPNP&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;A&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;X&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;Z_0&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Z&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;f_theta&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;X&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;_&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;range&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_hops&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;A_drop&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dglsp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;val_like&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;A&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;A_dropout&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;A&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;val&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;Z&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;alpha&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;A_drop&lt;/span&gt; &lt;span class=&quot;err&quot;&gt;@&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Z&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;alpha&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Z_0&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Z&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Hypergraph Neural Networks&lt;/strong&gt;. A hypergraph is a generalization of a graph in
which an edge can join any number of nodes (called an hyperedge). Hypergraphs
are particularly useful in scenarios that require capturing high-order
relations such as co-purchase behaviors in e-commerce platforms, or
co-authorship in citation networks, etc. A hypergraph is typically
characterized by its sparse incidence matrix, and thus Hypergraph Neural
Networks (HGNN) are commonly defined in sparse matrix notations. The equation
and code implementation of Hypergraph Convolution, proposed by &lt;a href=&quot;https://arxiv.org/abs/1809.09401&quot;&gt;Feng et al.,
2018&lt;/a&gt;, are presented below.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2023-02-20-release/hypergraph.png&quot; alt=&quot;hypergraph&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;HypergraphConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;H&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;X&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;d_V&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;H&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;sum&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# node degree&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;d_E&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;H&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;sum&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# edge degree&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;n_edges&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;d_E&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;shape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;D_V_invsqrt&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dglsp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;diag&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;d_V&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;**-&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;0.5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# D_V ** (-1/2)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;D_E_inv&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dglsp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;diag&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;d_E&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;**-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# D_E ** (-1)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;W&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dglsp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;identity&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;((&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;n_edges&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;n_edges&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;L&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;D_V_invsqrt&lt;/span&gt; &lt;span class=&quot;err&quot;&gt;@&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;H&lt;/span&gt; &lt;span class=&quot;err&quot;&gt;@&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;W&lt;/span&gt; &lt;span class=&quot;err&quot;&gt;@&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;D_E_inv&lt;/span&gt; &lt;span class=&quot;err&quot;&gt;@&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;H&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt; &lt;span class=&quot;err&quot;&gt;@&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;D_V_invsqrt&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Theta&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;L&lt;/span&gt; &lt;span class=&quot;err&quot;&gt;@&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;X&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Graph Transformers&lt;/strong&gt;. The Transformer has proven to be an effective learning
architecture in natural language processing and computer vision. Researchers
have begun to extend the use of Transformers to graph learning as well. One of
the pioneer work by (&lt;a href=&quot;https://arxiv.org/abs/2012.09699&quot;&gt;Dwivedi et al., 2020&lt;/a&gt;) proposed to constrain the all-pair
multi-head attention to the connected node pairs in a graph. With DGL Sparse,
implementing this new formulation is now a straightforward process, taking only
about 10 lines of code.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2023-02-20-release/gt.png&quot; alt=&quot;gt&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;GraphMHA&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;A&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;N&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;q&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;q_proj&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;reshape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;head_dim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;q&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;scaling&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;k&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;k_proj&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;reshape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;head_dim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;v&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;v_proj&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;reshape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;head_dim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

        &lt;span class=&quot;n&quot;&gt;attn&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dglsp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;bsddmm&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;A&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;q&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transpose&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# [N, N, nh]&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;attn&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;attn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;softmax&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;out&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dglsp&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;bspmm&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;attn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;v&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;out_proj&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;out&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;reshape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;h3 id=&quot;key-features-of-dgl-sparse&quot;&gt;Key Features of DGL Sparse&lt;/h3&gt;

&lt;p&gt;To handle diverse use cases in an efficient manner, DGL Sparse is designed with
two key features that set it apart from other sparse matrix libraries such as
&lt;code class=&quot;highlighter-rouge&quot;&gt;scipy.sparse&lt;/code&gt; or &lt;code class=&quot;highlighter-rouge&quot;&gt;torch.sparse&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Automatic Sparse Format Selection&lt;/strong&gt;. DGL Sparse eliminates the complexity of
choosing the right data structure for storing a sparse matrix (also known as
the &lt;em&gt;sparse format&lt;/em&gt;). Users can create a sparse matrix with a single call to
&lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.sparse.spmatrix&lt;/code&gt; and the internal DGL’s sparse matrix will automatically
select the optimal format based on the intended operation.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Scalar or Vector Non-zero Elements&lt;/strong&gt;. GNN models often associate edges with
multi-channel weight vectors, such as multi-head attention vectors, as
demonstrated in the Graph Transformer example. To accommodate this, DGL Sparse
allows non-zero elements to have vector shapes and extends common sparse
operations, such as sparse-dense-matrix multiplication (SpMM), to operate on
this new form (as seen in the &lt;code class=&quot;highlighter-rouge&quot;&gt;bspmm&lt;/code&gt; operation in the Graph Transformer
example).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;By utilizing these design features, DGL Sparse reduces code length by &lt;strong&gt;2.7 times&lt;/strong&gt;
on average when compared to previous implementations of matrix-view models with
message passing interface. The simplified code also results in &lt;strong&gt;43% less
overhead&lt;/strong&gt; in the framework. Additionally, DGL Sparse is PyTorch compatible,
making it easy to integrate with the various tools and packages available
within the PyTorch ecosystem.&lt;/p&gt;

&lt;h2 id=&quot;get-started-with-dgl-10&quot;&gt;Get started with DGL 1.0&lt;/h2&gt;

&lt;p&gt;The framework is readily available on all platforms and can be easily installed
using &lt;a href=&quot;https://www.dgl.ai/pages/start.html&quot;&gt;pip or conda&lt;/a&gt;. To get started with DGL Sparse, check out the new
&lt;a href=&quot;https://docs.dgl.ai/en/latest/notebooks/sparse/quickstart.html&quot;&gt;Quickstart tutorial&lt;/a&gt; and play with it in &lt;a href=&quot;https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/sparse/quickstart.ipynb&quot;&gt;Google Colab&lt;/a&gt; without having to set up a
local environment. In addition to the examples you’ve seen above, the first
release of DGL Sparse includes &lt;a href=&quot;https://docs.dgl.ai/en/latest/notebooks/sparse/index.html&quot;&gt;5 tutorials&lt;/a&gt; and &lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/examples/sparse&quot;&gt;11 end-to-end examples&lt;/a&gt; to help
you learn and understand the different uses of this new package.&lt;/p&gt;

&lt;p&gt;We welcome your feedback and are available via &lt;a href=&quot;https://github.com/dmlc/dgl/issues&quot;&gt;Github issues&lt;/a&gt; and &lt;a href=&quot;https://discuss.dgl.ai/&quot;&gt;Discuss posts&lt;/a&gt;.
Join our &lt;a href=&quot;http://slack.dgl.ai/&quot;&gt;Slack channel&lt;/a&gt; to stay updated and to connect with the community.&lt;/p&gt;

&lt;p&gt;For more information on the new additions and changes in DGL 1.0, please refer
to our &lt;a href=&quot;https://github.com/dmlc/dgl/releases/tag/1.0.0&quot;&gt;release note&lt;/a&gt;.&lt;/p&gt;

&lt;p align=&quot;right&quot;&gt;&lt;em&gt;(Banner image generated by Midjourney.)&lt;/em&gt;&lt;/p&gt;</content><author><name>DGLTeam</name></author><category term="release" /><category term="release" /><summary type="html">We are thrilled to announce the arrival of DGL 1.0, a cutting-edge machine learning framework for deep learning on graphs. Over the past three years, there has been growing interest from both academia and industry in this technology. Our framework has received requests from various scenarios, from academic research on state-of-the-art models to industrial demands for scaling Graph Neural Network (GNN) solutions to large, real-world problems. With DGL 1.0, we aim to provide a comprehensive and user-friendly solution for all users to take advantage of graph machine learning.</summary></entry><entry><title type="html">Improving Graph Neural Networks via Network-in-network Architecture</title><link href="https://www.dgl.ai/blog/2022/11/28/ngnn.html" rel="alternate" type="text/html" title="Improving Graph Neural Networks via Network-in-network Architecture" /><published>2022-11-28T00:00:00+00:00</published><updated>2022-11-28T00:00:00+00:00</updated><id>https://www.dgl.ai/blog/2022/11/28/ngnn</id><content type="html" xml:base="https://www.dgl.ai/blog/2022/11/28/ngnn.html">&lt;p&gt;As Graph Neural Networks (GNNs) has become increasingly popular, there is a
wide interest of designing deeper GNN architecture. However, deep GNNs suffer
from the &lt;em&gt;oversmoothing&lt;/em&gt; issue where the learnt node representations quickly
become indistinguishable with more layers. This blog features a simple yet
effective technique to build a deep GNN without the concern of oversmoothing.
The new architecture, &lt;strong&gt;Network in Graph Neural Networks (NGNN)&lt;/strong&gt; inspired by the
network-in-network architecture for computer vision, has shown superior
performance on multiple Open Graph Benchmark (OGB) leaderboards.&lt;/p&gt;

&lt;h2 id=&quot;introducing-ngnn&quot;&gt;Introducing NGNN&lt;/h2&gt;

&lt;p&gt;At a high-level, a graph neural network (MPGNN) layer can be written as a non-linear function:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;h^{(l+1)}=\sigma\left(f_w\left(\mathcal{G}, h^l\right)\right)&lt;/script&gt;

&lt;p&gt;with &lt;script type=&quot;math/tex&quot;&gt;h^{(0)}=X&lt;/script&gt; being the input node features, &lt;script type=&quot;math/tex&quot;&gt;\mathcal{G}&lt;/script&gt; being the input
graph, &lt;script type=&quot;math/tex&quot;&gt;h^L&lt;/script&gt; being the node embeddings in the last layer used by downstream
tasks, &lt;script type=&quot;math/tex&quot;&gt;L&lt;/script&gt; being the number of GNN layers. Additionally, the function
&lt;script type=&quot;math/tex&quot;&gt;f_w\left(\mathcal{G}, h^l\right)&lt;/script&gt; is determined by learnable parameters &lt;script type=&quot;math/tex&quot;&gt;w&lt;/script&gt;
and &lt;script type=&quot;math/tex&quot;&gt;\sigma(\cdot)&lt;/script&gt; is a non-linear activation function.&lt;/p&gt;

&lt;p&gt;Instead of adding many more GNN layers, NGNN deepens a GNN model by inserting
nonlinear feedforward neural network layer(s) within each GNN layer.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2022-11-28-ngnn/NGNN.png&quot; alt=&quot;ngnn&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;In essence, NGNN is just a nonlinear transformation of the original embeddings
of the nodes in the &lt;script type=&quot;math/tex&quot;&gt;l&lt;/script&gt;-th layer. Despite its simplicity, the NGNN technique is
quite powerful (we will come to that in a moment). Additionally, it does not
have large memory overhead and can work with various training methods such as
neighbor sampling or subgraph sampling.&lt;/p&gt;

&lt;p&gt;The intuition behind is straightforward. As the number of GNN layers and the
number of training iterations increases, the representations of nodes within
the same connected component will tend to converge to the same value. NGNN uses
a simple MLP after certain GNN layers to tackle the so-called oversmoothing
issue.&lt;/p&gt;

&lt;h2 id=&quot;implementing-ngnn-in-deep-graph-library-dgl&quot;&gt;Implementing NGNN in Deep Graph Library (DGL)&lt;/h2&gt;

&lt;p&gt;For better gaining insights into this trick, let us use DGL to implement a
simple NGNN, using the GCN layer as the backbone.&lt;/p&gt;

&lt;p&gt;With DGL’s builtin GCN layer &lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.nn.GraphConv&lt;/code&gt;, we can easily implement a
minimal &lt;code class=&quot;highlighter-rouge&quot;&gt;NGNN_GCN&lt;/code&gt; layer, which just applies an &lt;script type=&quot;math/tex&quot;&gt;\rm{ReLU}&lt;/script&gt; activation and a
linear transformation after a GCN layer.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.nn&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;GraphConv&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;NGNN_GCNConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;input_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hidden_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;output_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;nb&quot;&gt;super&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;NGNN_GCNConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;GraphConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;input_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hidden_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;fc&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Linear&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hidden_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;output_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;edge_weight&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;None&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;edge_weight&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;F&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;relu&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;fc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;Afterwards, you can simply stack the &lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.nn.GraphConv&lt;/code&gt; layer and the
&lt;code class=&quot;highlighter-rouge&quot;&gt;NGNN_GCN&lt;/code&gt; layer to form a multi-layer &lt;code class=&quot;highlighter-rouge&quot;&gt;NGNN_GCN&lt;/code&gt; network.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;NGNN_GCN&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;input_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hidden_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;output_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;nb&quot;&gt;super&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;NGNN_GCNConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;input_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hidden_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hidden_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;GraphConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hidden_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;output_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;input_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;input_channels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;F&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;relu&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;You can replace &lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.nn.GraphConv&lt;/code&gt; with any other graph convolution layers in
the NGNN architecture. DGL provides implementation of many popular
convolutional layers and utility modules. You can easily invoke them with one
line of code and build your own NGNN modules.&lt;/p&gt;

&lt;h2 id=&quot;model-performance&quot;&gt;Model Performance&lt;/h2&gt;

&lt;p&gt;NGNN can be used for many downstream tasks, such as Node
Classification/Regression, Edge Classification/Regression, Link prediction and
Graph Classification. In general, NGNN achieves better results than its
backbone GNN on these tasks. For instance, &lt;strong&gt;NGNN+SEAL achieves top-1
performance on the
&lt;a href=&quot;https://ogb.stanford.edu/docs/leader_linkprop/#ogbl-ppa&quot;&gt;ogbl-ppa&lt;/a&gt; leaderboard
with an improvement of Hit@100 by &lt;script type=&quot;math/tex&quot;&gt;10.91\%&lt;/script&gt; over the vanilla SEAL&lt;/strong&gt;. The table
below shows the performance improvement of NGNN over various vanilla GNN
backbones.&lt;/p&gt;

&lt;table style=&quot;text-align: center;&quot;&gt;
   &lt;tr&gt;
      &lt;th&gt;Dataset&lt;/th&gt;
      &lt;th&gt;Metric&lt;/th&gt;
      &lt;th&gt;Model&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;Performance&lt;/th&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;ogbn-proteins&lt;/td&gt;
      &lt;td&gt;ROC-AUC(%)&lt;/td&gt;
      &lt;td&gt;GraphSage+Cluster Sampling&lt;/td&gt;
      &lt;td&gt;Vanilla&lt;/td&gt;
      &lt;td&gt;67.45 ± 1.21&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;+NGNN&lt;/td&gt;
      &lt;td&gt;&lt;b&gt;68.12 ± 0.96&lt;/b&gt;&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;ogbn-products&lt;/td&gt;
      &lt;td&gt;Accuracy(%)&lt;/td&gt;
      &lt;td&gt;GraphSage&lt;/td&gt;
      &lt;td&gt;Vanilla&lt;/td&gt;
      &lt;td&gt;78.27 ± 0.45&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;+NGNN&lt;/td&gt;
      &lt;td&gt;&lt;b&gt;79.88 ± 0.34&lt;/b&gt;&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;GAT+Neighbor Sampling&lt;/td&gt;
      &lt;td&gt;Vanilla&lt;/td&gt;
      &lt;td&gt;79.23 ± 0.16&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;+NGNN&lt;/td&gt;
      &lt;td&gt;&lt;b&gt;79.67 ± 0.09&lt;/b&gt;&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;ogbl-collab&lt;/td&gt;
      &lt;td&gt;hit@50(%)&lt;/td&gt;
      &lt;td&gt;GCN&lt;/td&gt;
      &lt;td&gt;Vanilla&lt;/td&gt;
      &lt;td&gt;49.52 ± 0.70&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;+NGNN&lt;/td&gt;
      &lt;td&gt;&lt;b&gt;53.48 ± 0.40&lt;/b&gt;&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;GraphSage&lt;/td&gt;
      &lt;td&gt;Vanilla&lt;/td&gt;
      &lt;td&gt;51.66 ± 0.35&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;+NGNN&lt;/td&gt;
      &lt;td&gt;&lt;b&gt;53.59 ± 0.56&lt;/b&gt;&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;ogbl-ppa&lt;/td&gt;
      &lt;td&gt;hit@100(%)&lt;/td&gt;
      &lt;td&gt;SEAL-DGCNN&lt;/td&gt;
      &lt;td&gt;Vanilla&lt;/td&gt;
      &lt;td&gt;48.80 ± 3.16&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;+NGNN&lt;/td&gt;
      &lt;td&gt;&lt;b&gt;59.71 ± 2.45&lt;/b&gt;&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;GCN&lt;/td&gt;
      &lt;td&gt;Vanilla&lt;/td&gt;
      &lt;td&gt;18.67 ± 1.32&lt;/td&gt;
   &lt;/tr&gt;
   &lt;tr&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;&lt;/td&gt;
      &lt;td&gt;+NGNN&lt;/td&gt;
      &lt;td&gt;&lt;b&gt;36.83 ± 0.99&lt;/b&gt;&lt;/td&gt;
   &lt;/tr&gt;
&lt;/table&gt;

&lt;h2 id=&quot;further-readings&quot;&gt;Further Readings&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;NGNN Paper: &lt;a href=&quot;https://arxiv.org/abs/2111.11638&quot;&gt;https://arxiv.org/abs/2111.11638&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;NGNN+SEAL OGB submission: &lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/examples/pytorch/ogb/ngnn_seal&quot;&gt;https://github.com/dmlc/dgl/tree/master/examples/pytorch/ogb/ngnn_seal&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;NGNN+GraphSAGE OGB submission: &lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/examples/pytorch/ogb/ngnn&quot;&gt;https://github.com/dmlc/dgl/tree/master/examples/pytorch/ogb/ngnn&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;DGL built-in GNN module list: &lt;a href=&quot;https://docs.dgl.ai/api/python/nn-pytorch.html&quot;&gt;https://docs.dgl.ai/api/python/nn-pytorch.html&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;About the author:&lt;/strong&gt; &lt;a href=&quot;https://github.com/Ereboas&quot;&gt;Yakun Song&lt;/a&gt; is an undergraduate student of Shanghai Jiao Tong University.
The work was done during his internship in &lt;a href=&quot;https://www.amazonaws.cn/en/ailab/&quot;&gt;AWS Shanghai AI Lab&lt;/a&gt;.&lt;/p&gt;</content><author><name>Yakun Song</name></author><category term="blog" /><category term="blog" /><summary type="html">As Graph Neural Networks (GNNs) has become increasingly popular, there is a wide interest of designing deeper GNN architecture. However, deep GNNs suffer from the oversmoothing issue where the learnt node representations quickly become indistinguishable with more layers. This blog features a simple yet effective technique to build a deep GNN without the concern of oversmoothing. The new architecture, Network in Graph Neural Networks (NGNN) inspired by the network-in-network architecture for computer vision, has shown superior performance on multiple Open Graph Benchmark (OGB) leaderboards.</summary></entry><entry><title type="html">Accelerating Partitioning of Billion-scale Graphs with DGL v0.9.1</title><link href="https://www.dgl.ai/release/2022/09/19/release.html" rel="alternate" type="text/html" title="Accelerating Partitioning of Billion-scale Graphs with DGL v0.9.1" /><published>2022-09-19T00:00:00+00:00</published><updated>2022-09-19T00:00:00+00:00</updated><id>https://www.dgl.ai/release/2022/09/19/release</id><content type="html" xml:base="https://www.dgl.ai/release/2022/09/19/release.html">&lt;p&gt;Graphs is ubiquitous to represent relational data, and many real-world
applications such as recommendation and fraud detection involve learning from
massive graphs. As such, GNNs has emerged as a powerful family of models to
learn their representations. However, training GNNs on massive graphs is
challenging, one of the issues is high resource demand to distribute graph data
to a cluster. For example, partitioning a random graph of 1 billion nodes and 5
billion edges into 8 partitions requires a powerful AWS EC2 x1e.32xlarge
instance (128 vCPU, 3.9TB RAM) running for 10 hours to finish the job.&lt;/p&gt;

&lt;p&gt;In the latest DGL v0.9.1, we released a new pipeline for preprocess, partition
and dispatch graph of billions of nodes or edges for distributed GNN training.
At its core is a new data format called Chunked Graph Data Format (CGDF) which
stores graph data by chunks. The new pipeline processes data chunks in parallel
which not only reduces the memory requirement of each machine but also
significantly accelerates the entire procedure. For the same random graph with
1B nodes/5B edges, using a cluster of 8 AWS EC2 x1e.4xlarge (16 vCPU, 488GB RAM
each), &lt;strong&gt;the new pipeline can reduce the running time to 2.7 hours and cut down
the money cost by 3.7x.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;In this blog, we will illustrate step-by-step how to partition and distribute a
graph of billions of nodes and edges using this new feature.&lt;/p&gt;

&lt;h2 id=&quot;distributed-gnn-training-101&quot;&gt;Distributed GNN Training 101&lt;/h2&gt;

&lt;p&gt;A graph dataset typically consists of graph structure and the features
associated with nodes/edges. If the graph is heterogeneous (i.e., having
multiple types of nodes or edges), different types of nodes/edges may have
different sets of features. Training a GNN model on a multi-machine cluster
first requires users to partition their input graph, which involves two steps:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Run a graph partition algorithm (e.g., random, METIS) to assign each node to
one partition.&lt;/li&gt;
  &lt;li&gt;Shuffle and dispatch the graph structure and node/edge features to the
target machine that owns the partition.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Once the graph is partitioned and provisioned, users can then launch the
distributed training program using DGL’s launch tool, which will:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Launch one main graph server per machine that loads the local graph
partition into RAM. Graph servers provide remove process calls (RPCs) to
conduct computation like graph sampling. Optionally, users can launch more
backup graph servers that share the in-memory data of the main graph server to
increase service throughput.&lt;/li&gt;
  &lt;li&gt;Launch a key-value store (KVStore) server per machine that loads the local
node/edge features into RAM. KVStore service provides RPCs to fetch/update
node/edge features.&lt;/li&gt;
  &lt;li&gt;Launch one or more trainer processes per machine. Trainer processes are
connected with each other via PyTorch’s &lt;a href=&quot;https://pytorch.org/tutorials/intermediate/ddp_tutorial.html&quot;&gt;DistributedDataParallel&lt;/a&gt;
component. At each training iteration, they issue requests to local or remote graph servers
and KVStore servers to get a mini-batch of samples, perform gradient descent
and synchronize their gradients before the next iteration.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The figure below depicts the system architecture. For more information, check
out the &lt;a href=&quot;https://docs.dgl.ai/en/0.9.x/guide/distributed.html&quot;&gt;Distributed Training chapter&lt;/a&gt; of DGL User Guide.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2022-09-19-release/dist_train.png&quot; alt=&quot;dist_train&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Due to the complexity of graph data, graph partitioning is typically run on a
single machine, which demands the machine to have large enough memory to fit
the entire graph and features as well as the runtime usage from the partition
algorithm. For example, a random graph of 1 billion nodes and 5 billions edges
and 50 features per nodes needs 268GB when stored in DGL graph format. Using
the existing &lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.distributed.partition_graph&lt;/code&gt; API to partition this graph
requires a powerful AWS EC2 x1e.32xlarge instance (128 vCPU, 3.9TB RAM) and
runs for 10 hours — a significant bottleneck for users to train GNNs at scale.&lt;/p&gt;

&lt;p&gt;DGL v0.9.1 addressed the issue by a new distributed graph partitioning
pipeline. Specifically,&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;We designed a Chunked Graph Data Format (CGDF) for storing large graph data
in chunks to avoid loading the entire graph into a single machine.&lt;/li&gt;
  &lt;li&gt;We provided scripts to partition and dispatch chunked graph in parallel using
multiple machines to reduce the memory requirement of each machine as well as
to accelerate the procedure.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;chunked-graph-data-format&quot;&gt;Chunked Graph Data Format&lt;/h2&gt;

&lt;p&gt;Chunked graph dataset is organized as a data folder with the following data
files:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;A &lt;code class=&quot;highlighter-rouge&quot;&gt;metadata.json&lt;/code&gt; file that stores the meta information of the graph, e.g.,
graph name, node/edge types, chunk sizes, chunk file paths, etc.&lt;/li&gt;
  &lt;li&gt;A list of &lt;em&gt;edge index chunk files&lt;/em&gt; that store the source and destination node
IDs. They are typically in plain texts.&lt;/li&gt;
  &lt;li&gt;A list of &lt;em&gt;node data chunk files&lt;/em&gt;. They are typically array data stored in
&lt;a href=&quot;https://numpy.org/doc/stable/reference/generated/numpy.save.html#numpy.save&quot;&gt;NumPy array binary&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;A list of &lt;em&gt;edge data chunk files&lt;/em&gt;. They are also in NumPy array binary.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here, we illustrate the folder structure and the data files of a random social
graph (nodes being users, edges being follow relation) where nodes have two
data: “feat” and “label”. Check out the &lt;a href=&quot;https://docs.dgl.ai/en/0.9.x/guide/distributed-preprocessing.html#chunked-graph-format&quot;&gt;doc page&lt;/a&gt;
for the full specification of the format and tips for how to convert your data
to chunks.&lt;/p&gt;

&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;//data/random_graph_chunked/
  |-- metadata.json            # metadata JSON
  |-- edge_index/              # edge index chunks
    |-- user:follow:user0.txt  # user-follow-user edges chunk 0
    |-- user:follow:user1.txt  # user-follow-user edges chunk 1
    |-- user:follow:user2.txt  # user-follow-user edges chunk 2
    |-- ...
  |-- node_data/           # node data chunks
    |-- user/              # user nodes have two data: &quot;feat&quot; and &quot;label&quot;
      |-- feat0.npy        # feat chunk 0
      |-- feat1.npy        # feat chunk 1
      |-- feat2.npy        # feat chunk 2
      |-- ...
      |-- label0.npy       # label chunk 0
      |-- label1.npy       # label chunk 1
      |-- label2.npy       # label chunk 2
      |-- ...
  |-- edge_data/           # edge data chunks
    |-- user:follow:user/
       |-- ...
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;h2 id=&quot;running-distributed-partitioning--dispatching&quot;&gt;Running Distributed Partitioning &amp;amp; Dispatching&lt;/h2&gt;

&lt;p&gt;The first step is to get a cluster of machines to partition the graph. We
recommend the total RAM size of the cluster to be 2-3x larger than the graph
data size to accommodate the runtime memory needed. Next is to setup shared
workspace and software environment.&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Setup a shared folder that is accessible by each instance in the cluster
(e.g., using &lt;a href=&quot;https://wiki.archlinux.org/title/NFS&quot;&gt;NFS&lt;/a&gt;). Make sure all
instances can ssh to each other. Here, we suppose the folder is mounted to
&lt;code class=&quot;highlighter-rouge&quot;&gt;/workspace&lt;/code&gt;.&lt;/li&gt;
  &lt;li&gt;Clone and download the scripts from DGL 0.9.x branch to &lt;code class=&quot;highlighter-rouge&quot;&gt;/workspace&lt;/code&gt;:
    &lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;git clone https://github.com/dmlc/dgl.git -b 0.9.x /workspace/dgl
&lt;/code&gt;&lt;/pre&gt;
    &lt;/div&gt;
  &lt;/li&gt;
  &lt;li&gt;Copy/move the chunked graph data to &lt;code class=&quot;highlighter-rouge&quot;&gt;/workspace/&lt;/code&gt;. Here, we suppose the data
folder is at &lt;code class=&quot;highlighter-rouge&quot;&gt;/workspace/random_graph_chunked/&lt;/code&gt;.&lt;/li&gt;
  &lt;li&gt;Create an &lt;code class=&quot;highlighter-rouge&quot;&gt;/workspace/ip_config.txt&lt;/code&gt; file that contains the IP address of each
instance.
    &lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# example IP config file of a 4 machine cluster&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;172.31.19.1&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;172.31.23.205&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;172.31.29.175&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;172.31.16.98&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
    &lt;/div&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;We can then run a partition algorithm to assign each node to a partition. Here,
we choose random partitioning algorithm.&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;python /workspace/dgl/tools/partition_algo/random_partition.py &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
    --in_dir&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;/workspace/random_graph_chunked/
    --out_dir&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;/workspace/partition_assign/
    --num_partitions&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;4
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;The above script simply calculates which partition a node belongs to. We then
pass both the chunked graph and the partition assignments to the
&lt;code class=&quot;highlighter-rouge&quot;&gt;dispatch_data.py&lt;/code&gt; script to physically split the graph data into multiple pieces
and distribute them to the entire cluster.&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;python /workspace/dgl/tools/dispatch_data.py &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
    --in-dir&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;/workspace/random_graph_chunked/
    --partitions-dir&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;/workspace/partition_assign/
    --out-dir&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;/workspace/random_graph_dist/
    --ip-config&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;/workspace/ip_config.txt
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;The end result will look like the following. We can then launch distributed
training following the instructions &lt;a href=&quot;https://docs.dgl.ai/en/0.9.x/guide/distributed-tools.html&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;/workspace/random_graph_dist/
  |-- medatdata.json      # metadata JSON file
  |-- part0/              # partition 0
    |-- graph.dgl         # graph structure of partition 0 in DGL binary format
    |-- node_feat.dgl     # node feature of partition 0 in DGL binary format
    |-- edge_feat.dgl     # edge feature of partition 0 in DGL binary format
  |-- part1/              # partition 1
    |-- graph.dgl         # graph structure of partition 1 in DGL binary format
    |-- node_feat.dgl     # node feature of partition 1 in DGL binary format
    |-- edge_feat.dgl     # edge feature of partition 1 in DGL binary format
  |-- part2/
  ...
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;Note that the scripts utilize multiple machines to cooperatively partition and
process the data. Therefore, the new pipeline is significantly faster. For the
same random graph with 1B nodes/5B edges, the new pipeline can finish
partitioning in 2.7 hours (3.7x faster) using an cluster of 8 AWS EC2
x1e.4xlarge (16 vCPU, 488GB RAM).&lt;/p&gt;

&lt;h2 id=&quot;further-readings&quot;&gt;Further Readings&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://docs.dgl.ai/en/0.9.x/guide/distributed-preprocessing.html&quot;&gt;User guide for the new distributed graph partitioning pipeline&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://github.com/dmlc/dgl/tree/0.9.x/examples/pytorch/graphsage/dist&quot;&gt;Example for training a GraphSAGE model on a cluster of machines&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://github.com/dmlc/dgl/releases/tag/0.9.1&quot;&gt;Other enhancement in the latest v0.9.1 release&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;</content><author><name>DGLTeam</name></author><category term="release" /><category term="release" /><summary type="html">Graphs is ubiquitous to represent relational data, and many real-world applications such as recommendation and fraud detection involve learning from massive graphs. As such, GNNs has emerged as a powerful family of models to learn their representations. However, training GNNs on massive graphs is challenging, one of the issues is high resource demand to distribute graph data to a cluster. For example, partitioning a random graph of 1 billion nodes and 5 billion edges into 8 partitions requires a powerful AWS EC2 x1e.32xlarge instance (128 vCPU, 3.9TB RAM) running for 10 hours to finish the job.</summary></entry><entry><title type="html">v0.9 Release Highlights</title><link href="https://www.dgl.ai/release/2022/07/25/release.html" rel="alternate" type="text/html" title="v0.9 Release Highlights" /><published>2022-07-25T00:00:00+00:00</published><updated>2022-07-25T00:00:00+00:00</updated><id>https://www.dgl.ai/release/2022/07/25/release</id><content type="html" xml:base="https://www.dgl.ai/release/2022/07/25/release.html">&lt;p&gt;Six years after the first Graph Convolutional Networks paper, researchers are
actively investigating more advanced GNN architecture or training methodology.
As the developer team of DGL, we closely watch those new research trends and
release features to facilitate them. Here, we highlighted some of the new
functionalities of the recent v0.9 release.&lt;/p&gt;

&lt;h2 id=&quot;combining-graph-analytics-with-gnns-using-cugraphdgl&quot;&gt;Combining Graph Analytics with GNNs using cuGraph+DGL&lt;/h2&gt;

&lt;p&gt;Graph neural networks (GNNs) are capable of combining the feature and
structural information of graph data. Its power can be further extended when
synergistically combined with techniques of graph analytics, such as feature
augmentation.&lt;/p&gt;

&lt;p&gt;Graph analytics has been widely used for characterising graph structures, e.g.,
identifying important nodes, leading to interesting feature augmentation
methods. To exploit the synergy, we would want a fast and scalable graph
analytics engine. NVidia’s &lt;a href=&quot;https://github.com/rapidsai/cugraph&quot;&gt;RAPIDS cuGraph
library&lt;/a&gt; provides a collection of GPU
accelerated algorithms for graph analytics, such as centrality computation and
community detection. According to this
&lt;a href=&quot;https://docs.rapids.ai/api/cugraph/stable/basics/cugraph_intro.html&quot;&gt;documentation&lt;/a&gt;,
&lt;em&gt;“the latest NVIDIA GPUs (RAPIDS supports Pascal and later GPU architectures)
make graph analytics 1000x faster on average over NetworkX”&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;With collaboration with NVidia’s engineers, DGL v0.9 now allows conversion
between a DGLGraph object and a cuGraph graph object with two APIs &lt;code class=&quot;highlighter-rouge&quot;&gt;to_cugraph&lt;/code&gt;
and &lt;code class=&quot;highlighter-rouge&quot;&gt;from_cugraph&lt;/code&gt;, making it possible for DGL users to access efficient graph
analytics implementations in cuGraph.&lt;/p&gt;

&lt;h3 id=&quot;installation&quot;&gt;Installation&lt;/h3&gt;

&lt;p&gt;To install cuGraph with PyTorch and DGL, we recommend following the practice
below. Mamba is a multi-threaded version of conda.&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;conda install mamba -n base -c conda-forge

mamba create -n dgl_and_cugraph -c dglteam -c rapidsai-nightly -c nvidia -c pytorch -c conda-forge &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
    cugraph pytorch torchvision torchaudio &lt;span class=&quot;nv&quot;&gt;cudatoolkit&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;11.3 dgl-cuda11.3 tqdm

conda activate dgl_and_cugraph
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;h3 id=&quot;feature-initialization-via-cugraph&quot;&gt;Feature Initialization via cuGraph&lt;/h3&gt;

&lt;p&gt;We showcase an example of node feature initialization using the graph analytics
algorithms provided by cuGraph. Here, we consider two options:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://docs.rapids.ai/api/cugraph/stable/api_docs/api/cugraph.louvain.html&quot;&gt;Louvain algorithm&lt;/a&gt; that detects the community membership of each node based on
modularity optimization.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://docs.rapids.ai/api/cugraph/stable/api_docs/api/cugraph.core_number.html#cugraph.core_number&quot;&gt;Core number algorithm&lt;/a&gt; that calculates the maximal &lt;a href=&quot;https://en.wikipedia.org/wiki/Degeneracy_(graph_theory)&quot;&gt;k-core&lt;/a&gt; subgraph each node
belongs to. A k-core of a graph is a maximal subgraph that contains nodes of
degree k or more.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The two algorithms capture different structural characteristics of a node.
Louvain groups nodes with close spatial distance with each other, while nodes
with the same core numbers are more structurally similar with each other. The
figures below illustrate the node coloring produced by Louvain communities and
core numbers on &lt;a href=&quot;https://en.wikipedia.org/wiki/Zachary%27s_karate_club&quot;&gt;Zachary’s Karate Club Network&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2022-07-25-release/node_coloring.png&quot; alt=&quot;node_coloring&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;cuGraph offers efficient GPU implementations of these two algorithms. To call
them, we convert a &lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.DGLGraph&lt;/code&gt; to a &lt;code class=&quot;highlighter-rouge&quot;&gt;cugraph.Graph&lt;/code&gt; using the &lt;code class=&quot;highlighter-rouge&quot;&gt;to_cugraph&lt;/code&gt;
API.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;cugraph&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;louvain&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dgl_g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;cugraph_g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl_g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to_cugraph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to_undirected&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;df&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;_&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;cugraph&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;louvain&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cugraph_g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;resolution&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;c&quot;&gt;# revert the node ID renumbering by cugraph&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;df&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;cugraph_g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;unrenumber&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;df&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;'vertex'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sort_values&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'vertex'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;utils&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dlpack&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;from_dlpack&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;df&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'partition'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to_dlpack&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;())&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;long&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;core_number&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dgl_g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;cugraph_g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl_g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to_cugraph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to_undirected&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;df&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;cugraph&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;core_number&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cugraph_g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;c&quot;&gt;# revert the node ID renumbering by cugraph&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;df&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;cugraph_g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;unrenumber&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;df&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;'vertex'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sort_values&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'vertex'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;utils&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dlpack&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;from_dlpack&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;df&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'core_number'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to_dlpack&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;())&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;long&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;h3 id=&quot;training-gnn-via-dgl&quot;&gt;Training GNN via DGL&lt;/h3&gt;

&lt;p&gt;We then use the above functions to prepare node features for the ogbn-arxiv
dataset. Note that since both algorithms calculate structural categories, we
convert them to one-hot encoding and concatenate them as the initial node
features.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.transforms&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;T&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch.nn&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;nn&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch.nn.functional&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;F&lt;/span&gt;

&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.nn&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;SAGEConv&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;ogb.nodeproppred&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;DglNodePropPredDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Evaluator&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;device&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'cuda'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;DglNodePropPredDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'ogbn-arxiv'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;label&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Compose&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;([&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;AddReverse&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;AddSelfLoop&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ToSimple&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;int&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;louvain&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;core_number&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# convert to one-hot&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;F&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;one_hot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;feat1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat1&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;max&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;F&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;one_hot&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;feat2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat2&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;max&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# concat feat1 and feat2&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cat&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;([&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;feat1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dim&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;float&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;We then train a simple three layer GraphSAGE model (see complete training code
&lt;a href=&quot;https://gist.github.com/jermainewang/f316587084f73e7dd060fe203417f42a&quot;&gt;here&lt;/a&gt;).
With the help of node features initialized by graph analytics algorithms, we
are able to achieve an accuracy of about 0.6 on the test set using pure
structural information, which even outperforms an MLP model using the original
input node features. With the new DGL release, we are looking forward to seeing
more innovation on GNNs combined with graph analytics.&lt;/p&gt;

&lt;h2 id=&quot;fp16--mixed-precision-support&quot;&gt;FP16 &amp;amp; Mixed Precision Support&lt;/h2&gt;

&lt;p&gt;DGL v0.9 is now fully compatible with the &lt;a href=&quot;https://pytorch.org/docs/stable/amp.html&quot;&gt;PyTorch Automatic Mixed Precision
(AMP) package&lt;/a&gt; for mixed precision training, thus saving both training time and
GPU memory consumption.&lt;/p&gt;

&lt;p&gt;By wrapping the forward pass with torch.cuda.amp.autocast(), PyTorch
automatically selects the appropriate data type for each op and tensor. Half
precision tensors are memory efficient, most operators on half precision
tensors are faster as they leverage GPU tensorcores.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch.nn.functional&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;F&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch.cuda.amp&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;autocast&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;label&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mask&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;with&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;autocast&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;enabled&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;logit&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;F&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cross_entropy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;logit&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;mask&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;label&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;mask&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;Small gradients in &lt;code class=&quot;highlighter-rouge&quot;&gt;float16&lt;/code&gt; format have underflow problems (flush to zero).
PyTorch AMP provides a &lt;code class=&quot;highlighter-rouge&quot;&gt;GradScaler&lt;/code&gt; module to address this issue. It multiplies
the loss by a factor and invokes backward pass on the scaled loss to prevent
the underflow problem. It then unscales the computed gradients before the
optimizer updates the parameters. The scale factor is determined automatically.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch.cuda.amp&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;GradScaler&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;scaler&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;GradScaler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;backward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;scaler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;optimizer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;scaler&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;scale&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;backward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;scaler&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;step&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;optimizer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;scaler&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;update&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;Putting everything together, we have the example below.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch.nn&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;nn&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.data&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;RedditDataset&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.nn&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;GATConv&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.transforms&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AddSelfLoop&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;GAT&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;in_feats&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_classes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_hidden&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;256&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;nb&quot;&gt;super&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;GATConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;in_feats&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_hidden&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;activation&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;F&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;elu&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;GATConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_hidden&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_hidden&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;flatten&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;mean&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;device&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'cuda'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;AddSelfLoop&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;RedditDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;int&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;train_mask&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'train_mask'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'feat'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;label&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'label'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;in_feats&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;shape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;model&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;GAT&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;in_feats&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_classes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;to&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;train&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;optimizer&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;optim&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Adam&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;parameters&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;lr&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;1e-3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;weight_decay&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;5e-4&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;epoch&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;range&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;100&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;optimizer&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;zero_grad&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;label&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;train_mask&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;backward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;scaler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;loss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;optimizer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;Training GNNs using low precision or mixed precision is still an active
research topic. We hope the new v0.9 release will facilitate more research on
this topic. Check out the
&lt;a href=&quot;https://docs.dgl.ai/guide/mixed_precision.html&quot;&gt;documentation&lt;/a&gt; to know more.&lt;/p&gt;

&lt;h2 id=&quot;dgl-go-update-model-inference-and-graph-prediction&quot;&gt;DGL-Go Update: Model Inference and Graph Prediction&lt;/h2&gt;

&lt;p&gt;DGL-Go now supports training GNNs for graph property prediction tasks. It
includes two popular GNN models – Graph Isomorphism Network (GIN) and Principal
Neighborhood Aggregation (PNA). For example, to train a GIN model on the
ogbg-molpcba dataset, first generate a YAML configuration file using command:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;dgl configure graphpred --data ogbg-molpcba --model gin
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;which generates the following configuration file. Users can then manually
adjust the configuration file.&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;s&quot;&gt;version&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;0.0.2&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;pipeline_name&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;graphpred&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;pipeline_mode&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;train&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;cpu&lt;/span&gt;                     &lt;span class=&quot;c1&quot;&gt;# Torch device name, e.g., cpu or cuda or cuda:0&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;ogbg-molpcba&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;split_ratio&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;                &lt;span class=&quot;c1&quot;&gt;# Ratio to generate data split, for example set to [0.8, 0.1, 0.1] for 80% train/10% val/10% test. Leave blank to use builtin split in original dataset&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
     &lt;span class=&quot;s&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;gin&lt;/span&gt;
     &lt;span class=&quot;s&quot;&gt;embed_size&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;300&lt;/span&gt;            &lt;span class=&quot;c1&quot;&gt;# Embedding size&lt;/span&gt;
     &lt;span class=&quot;s&quot;&gt;num_layers&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;5&lt;/span&gt;              &lt;span class=&quot;c1&quot;&gt;# Number of layers&lt;/span&gt;
     &lt;span class=&quot;s&quot;&gt;dropout&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;0.5&lt;/span&gt;               &lt;span class=&quot;c1&quot;&gt;# Dropout rate&lt;/span&gt;
     &lt;span class=&quot;s&quot;&gt;virtual_node&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;false&lt;/span&gt;        &lt;span class=&quot;c1&quot;&gt;# Whether to use virtual node&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;general_pipeline&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;num_runs&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;1&lt;/span&gt;                 &lt;span class=&quot;c1&quot;&gt;# Number of experiments to run&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;train_batch_size&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;32&lt;/span&gt;        &lt;span class=&quot;c1&quot;&gt;# Graph batch size when training&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;eval_batch_size&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;32&lt;/span&gt;         &lt;span class=&quot;c1&quot;&gt;# Graph batch size when evaluating&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;num_workers&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;4&lt;/span&gt;              &lt;span class=&quot;c1&quot;&gt;# Number of workers for data loading&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;optimizer&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Adam&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;lr&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;0.001&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;weight_decay&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;0&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;lr_scheduler&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;StepLR&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;step_size&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;100&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;gamma&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;1&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;loss&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;BCEWithLogitsLoss&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;metric&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;roc_auc_score&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;num_epochs&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;100&lt;/span&gt;             &lt;span class=&quot;c1&quot;&gt;# Number of training epochs&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;save_path&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;results&lt;/span&gt;          &lt;span class=&quot;c1&quot;&gt;# Directory to save the experiment results&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;Alternatively, users can fetch model recipes of pre-defined hyperparameters for
the original experiments.&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;dgl recipe get graphpred_pcba_gin.yaml
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;To launch training:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;dgl train --cfg graphpred_ogbg-molpcba_gin.yaml
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;Another addition is a new command to conduct inference of a trained model on
some other dataset. For example, the following shows how to apply the GIN model
trained on ogbg-molpcba to ogbg-molhiv:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c&quot;&gt;# Generate an inference configuration file from a saved experiment checkpoint&lt;/span&gt;
dgl configure-apply graphpred --data ogbg-molhiv --cpt results/run_0.pth

&lt;span class=&quot;c&quot;&gt;# Apply the trained model for inference&lt;/span&gt;
dgl apply --cfg apply_graphpred_ogbg-molhiv_pna.yaml
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;It will save the model prediction in a CSV file like below&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2022-07-25-release/csv_result.png&quot; alt=&quot;csv_result&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;h2 id=&quot;further-reading&quot;&gt;Further Reading&lt;/h2&gt;

&lt;p&gt;Full release note: &lt;a href=&quot;https://github.com/dmlc/dgl/releases/tag/0.9.0&quot;&gt;https://github.com/dmlc/dgl/releases/tag/0.9.0&lt;/a&gt;&lt;/p&gt;</content><author><name>DGLTeam</name></author><category term="release" /><category term="release" /><summary type="html">Six years after the first Graph Convolutional Networks paper, researchers are actively investigating more advanced GNN architecture or training methodology. As the developer team of DGL, we closely watch those new research trends and release features to facilitate them. Here, we highlighted some of the new functionalities of the recent v0.9 release.</summary></entry><entry><title type="html">May 2022 Update Note</title><link href="https://www.dgl.ai/release/2022/05/31/release.html" rel="alternate" type="text/html" title="May 2022 Update Note" /><published>2022-05-31T00:00:00+00:00</published><updated>2022-05-31T00:00:00+00:00</updated><id>https://www.dgl.ai/release/2022/05/31/release</id><content type="html" xml:base="https://www.dgl.ai/release/2022/05/31/release.html">&lt;h2 id=&quot;synthetic-datasets-for-developing-gnn-explainability-approaches&quot;&gt;Synthetic Datasets for Developing GNN Explainability Approaches&lt;/h2&gt;

&lt;p&gt;We added the following new datasets. &lt;code class=&quot;highlighter-rouge&quot;&gt;BAShapeDataset&lt;/code&gt;, &lt;code class=&quot;highlighter-rouge&quot;&gt;BACommunityDataset&lt;/code&gt;, &lt;code class=&quot;highlighter-rouge&quot;&gt;TreeCycleDataset&lt;/code&gt;, and &lt;code class=&quot;highlighter-rouge&quot;&gt;TreeGridDataset&lt;/code&gt; were first introduced in &lt;a href=&quot;https://arxiv.org/abs/1903.03894v4&quot;&gt;GNNExplainer: Generating Explanations for Graph Neural Networks&lt;/a&gt; for node classification. &lt;code class=&quot;highlighter-rouge&quot;&gt;BA2MotifDataset&lt;/code&gt; was first introduced in &lt;a href=&quot;https://arxiv.org/abs/2011.04573&quot;&gt;Parameterized Explainer for Graph Neural Network&lt;/a&gt; for graph classification.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2022-05-31-release/syn_data.png&quot; alt=&quot;syn_data&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c&quot;&gt;# A dataset for node classification&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.data&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;BAShapeDataset&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;BAShapeDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;num_classes&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_classes&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'feat'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;label&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'label'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;# A dataset for graph classification&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.data&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;BA2MotifDataset&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;BA2MotifDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;num_classes&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_classes&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;label&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;           &lt;span class=&quot;c&quot;&gt;# Get the first graph data&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'feat'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;These synthetic graphs integrate specially designed substructures, motifs, into traditional random graph models, and assign labels based on their existence. Therefore, those substructures act as ground truth explanations for the node/graph labels, making them commonly used benchmarks for evaluating GNN explainability approaches.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;sign-diffusion-transform&quot;&gt;SIGN Diffusion Transform&lt;/h2&gt;

&lt;p&gt;We added a new data transform module &lt;code class=&quot;highlighter-rouge&quot;&gt;SIGNDiffusion&lt;/code&gt; first introduced in &lt;a href=&quot;https://arxiv.org/abs/2004.11198&quot;&gt;SIGN: Scalable Inception Graph Neural Networks&lt;/a&gt;, which diffuses node features for later use. It supports four built-in diffusion matrices, including raw adjacency matrix, random walk adjacency matrix, symmetrically normalized adjacency matrix, and personalized PageRank matrix.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.transforms&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;T&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CoraGraphDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;SIGNDiffusion&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# Diffuse for 1 &amp;amp; 2 hops&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# diffused node features will be generated as ndata&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat1&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'out_feat_1'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;       &lt;span class=&quot;c&quot;&gt;# feature diffused for 1 hop&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat2&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'out_feat_2'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;       &lt;span class=&quot;c&quot;&gt;# feature diffused for 2 hops&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;The ability to learn to aggregate neighbor information is one of the key innovation of Message Passing Neural Networks, which also brings scalability challenges due to the exponentially growing receptive field with more hops of neighbors to explore. SIGN proposed a cheap yet efficient solution that decouples model depth and receptive field size by diffusing input node features using various kinds of algorithms. Because the diffusion process is not trainable, we package it as a data transform module so that users can easily plug-in SIGN before running their own model.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;label-propagation&quot;&gt;Label Propagation&lt;/h2&gt;

&lt;p&gt;We added a new NN module &lt;code class=&quot;highlighter-rouge&quot;&gt;LabelPropagation&lt;/code&gt; first introduced in &lt;a href=&quot;http://mlg.eng.cam.ac.uk/zoubin/papers/CMU-CALD-02-107.pdf&quot;&gt;Learning from Labeled and Unlabeled Data with Label Propagation&lt;/a&gt;, which propagates node labels over a graph for inferring the labels of unlabeled nodes.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;lp&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;LabelPropagation&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;alpha&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;0.9&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CoraGraphDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;labels&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'label'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;train_mask&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'train_mask'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;new_labels&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;lp&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;labels&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;train_mask&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;The classical label propagation is a simple (non-parametric) yet effective algorithm, making it a strong baseline for many node classification datasets.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;directional-graph-network-layer&quot;&gt;Directional Graph Network Layer&lt;/h2&gt;

&lt;p&gt;We added a new NN module &lt;code class=&quot;highlighter-rouge&quot;&gt;DGNConv&lt;/code&gt; first introduced in &lt;a href=&quot;https://arxiv.org/abs/2010.02863&quot;&gt;Directional Graph Networks&lt;/a&gt;, which introduces directional aggregators in message passing based on the gradient of low-frequency eigenvectors of the graph Laplacian matrix.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# some graph&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Precompute 1 smallest non-trivial eigenvectors&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transforms&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;LaplacianPE&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_nodes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# node features&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;eig&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'PE'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;DGNConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
                      &lt;span class=&quot;n&quot;&gt;aggregators&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'dir1-av'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;'dir1-dx'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
                      &lt;span class=&quot;n&quot;&gt;scalers&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'identity'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;'amplification'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
                      &lt;span class=&quot;n&quot;&gt;delta&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;2.5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;eig_vec&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;eig&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;Directional Graph Networks (DGN) allow defining graph convolutions according to topologically-derived directional flows. It is a state-of-the-art baseline for many graph classification tasks.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;graph-isomorphism-network-layer-with-edge-features&quot;&gt;Graph Isomorphism Network Layer with Edge Features&lt;/h2&gt;

&lt;p&gt;We added a new NN module &lt;code class=&quot;highlighter-rouge&quot;&gt;GINEConv&lt;/code&gt; first introduced in &lt;a href=&quot;https://arxiv.org/abs/1905.12265&quot;&gt;Strategies for Pre-training Graph Neural Networks&lt;/a&gt;, which extends Graph Isomorphism Network (GIN) for handling edge features.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# some graph&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;xn&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_nodes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# node features&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;xe&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_edges&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# edge features&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;GINEConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Linear&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;hn&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;xn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;xe&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;Graph Isomorphism Network with edge features has been an important baseline for many graph classification tasks such as OGB graph property datasets.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;feature-masking&quot;&gt;Feature Masking&lt;/h2&gt;

&lt;p&gt;We added a new data transform module &lt;code class=&quot;highlighter-rouge&quot;&gt;FeatMask&lt;/code&gt; first introduced in &lt;a href=&quot;https://arxiv.org/abs/2010.13902&quot;&gt;Graph Contrastive Learning with Augmentations&lt;/a&gt;, which randomly masks columns of node/edge features.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.transforms&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;T&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CoraGraphDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;FeatMask&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;p&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;0.1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;node_feat_names&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'feat'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]))&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'feat'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# The node feature tensor has been randomly masked.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;Randomly masking columns of features is a simple yet useful data augmentation for graph contrastive learning.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;row-normalizer-of-features&quot;&gt;Row-Normalizer of Features&lt;/h2&gt;

&lt;p&gt;We added a new data transform module &lt;code class=&quot;highlighter-rouge&quot;&gt;RowFeatNormalizer&lt;/code&gt;, which performs row-normalization of features.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.transforms&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;T&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CoraGraphDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;RowFeatNormalizer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;node_feat_names&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'feat'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]))&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;feat&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'feat'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# The node feature tensor has been row-normalized.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;Row-normalization of raw features is a useful data pre-processing step.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;For further readings, check out the &lt;a href=&quot;https://github.com/dmlc/dgl/releases/tag/0.8.2&quot;&gt;release note&lt;/a&gt; for a complete list of new additions, improvements and bug fixes. If you have questions about DGL or GNN in general, welcome to join our &lt;a href=&quot;https://deep-graph-library.slack.com/join/shared_invite/zt-eb4ict1g-xcg3PhZAFAB8p6dtKuP6xQ&quot;&gt;Slack channel&lt;/a&gt;. If you have specific requests on what should be included in DGL next, you can submit them on our Github or fill in this &lt;a href=&quot;https://forms.gle/tgKXFuUiGo9PeYBeA&quot;&gt;survey&lt;/a&gt;.&lt;/p&gt;</content><author><name>DGLTeam</name></author><category term="release" /><category term="release" /><summary type="html">Synthetic Datasets for Developing GNN Explainability Approaches</summary></entry><entry><title type="html">April 2022 Update Note</title><link href="https://www.dgl.ai/release/2022/04/18/release.html" rel="alternate" type="text/html" title="April 2022 Update Note" /><published>2022-04-18T00:00:00+00:00</published><updated>2022-04-18T00:00:00+00:00</updated><id>https://www.dgl.ai/release/2022/04/18/release</id><content type="html" xml:base="https://www.dgl.ai/release/2022/04/18/release.html">&lt;h2 id=&quot;grouped-reversible-residual-connection-for-gnns&quot;&gt;Grouped Reversible Residual Connection for GNNs&lt;/h2&gt;

&lt;p&gt;We added a new module &lt;code class=&quot;highlighter-rouge&quot;&gt;GroupRevRes&lt;/code&gt; introduced in &lt;a href=&quot;https://arxiv.org/abs/2106.07476&quot;&gt;Training Graph Neural Networks with 1000 Layers&lt;/a&gt;. It can wrap any GNN module with grouped, reversible and residual connection (example code below).&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;GNNLayer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;in_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dropout&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;0.2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;nb&quot;&gt;super&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;GNNLayer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
        &lt;span class=&quot;c&quot;&gt;# Use BatchNorm and dropout to prevent gradient vanishing&lt;/span&gt;
        &lt;span class=&quot;c&quot;&gt;# In particular if you use a large number of GNN layers&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;norm&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;BatchNorm1d&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;in_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;GraphConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;in_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;in_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Dropout&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;norm&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# some graph&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_nodes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;32&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;reversible_conv&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;GroupRevRes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;GNNLayer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;32&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;//&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;     &lt;span class=&quot;c&quot;&gt;# 4 groups&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;y&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;reversible_conv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# forward&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;The &lt;code class=&quot;highlighter-rouge&quot;&gt;GroupRevRes&lt;/code&gt; module is reversible, meaning the backward propagation does not require storing forward activations. It can significantly reduce memory usage of GNNs, making it possible to train a very deep GNN with up to 1000 layers on a single commodity GPU.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;laplacian-positional-encoding&quot;&gt;Laplacian Positional Encoding&lt;/h2&gt;

&lt;p&gt;We added a new data transform module &lt;code class=&quot;highlighter-rouge&quot;&gt;LaplacianPE&lt;/code&gt; first introduced in &lt;a href=&quot;https://arxiv.org/abs/2003.00982&quot;&gt;Benchmarking Graph Neural Networks&lt;/a&gt;. It computes Laplacian positional encoding for a graph. Besides data transform module, we also provide a functional API. See the example of usage below:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;# data transform&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CoraGraphDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transforms&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;LaplacianPE&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat_name&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'PE'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# positional encodings will be generated as an ndata&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;pe&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'PE'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# functional API&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;pe&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;laplacian_pe&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;Laplacian positional encoding improves the expressive power of GNNs by using k-smallest non-trivial Laplacian eigenvectors as additional node features.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;random-walk-positional-encoding&quot;&gt;Random Walk Positional Encoding&lt;/h2&gt;

&lt;p&gt;We added a new data transform module &lt;code class=&quot;highlighter-rouge&quot;&gt;RandomWalkPE&lt;/code&gt; introduced in &lt;a href=&quot;https://arxiv.org/abs/2110.07875&quot;&gt;Graph Neural Networks with Learnable Structural and Positional Representations&lt;/a&gt;. It computes random-walk-based positional encoding for a graph.  Besides data transform module, we also provide a functional API. See the example of usage below:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;# data transform&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CoraGraphDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transforms&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;RandomWalkPE&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;feat_name&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'PE'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# positional encodings will be generated automatically&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;pe&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'PE'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Functional API&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;pe&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;random_walk_pe&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# functional API&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;Random walk positional encoding improves the expressive power of GNNs by using the landing probabilities of a node to itself in 1, 2, …, K steps as additional node features.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;graphsaint-samplers&quot;&gt;GraphSAINT Samplers&lt;/h2&gt;

&lt;p&gt;We added a new sampler &lt;code class=&quot;highlighter-rouge&quot;&gt;SAINTSampler&lt;/code&gt; introduced in &lt;a href=&quot;https://arxiv.org/abs/1907.04931v4&quot;&gt;GraphSAINT: Graph Sampling Based Inductive Learning Method&lt;/a&gt;. &lt;code class=&quot;highlighter-rouge&quot;&gt;SAINTSampler&lt;/code&gt; provides three strategies to extract induced subgraphs from a graph — by randomly selected node sets, randomly selected edge sets or nodes reached by random walks. See an example of usage below:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.dataloading&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;SAINTSampler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;DataLoader&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;sampler&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;SAINTSampler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;mode&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'node'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;                      &lt;span class=&quot;c&quot;&gt;# Can be 'node', 'edge' or 'walk'&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;budget&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;200&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;prefetch_ndata&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'feat'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;'label'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# optionally, specify data to prefetch&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;data_index&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;arange&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1000&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# 1000 mini-batches&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CoraGraphDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dataloader&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;DataLoader&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data_index&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sampler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_workers&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sg&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataloader&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;train&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sg&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;GraphSAINT is one of the state-of-the-art sampling methods in the family of subgraph sampling. Compared with neighbor sampling (or node-wise sampling), subgraph sampling avoids the issue of exponential neighborhood expansion, thus saving data transmission cost and enabling mini-batch training of deeper GNNs.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;en-equivariant-convolutional-layer&quot;&gt;E(n) Equivariant Convolutional Layer&lt;/h2&gt;

&lt;p&gt;We added a new GNN module &lt;code class=&quot;highlighter-rouge&quot;&gt;EGNNConv&lt;/code&gt; introduced in &lt;a href=&quot;https://arxiv.org/abs/2102.09844v3&quot;&gt;E(n) Equivariant Graph Neural Networks&lt;/a&gt;. It performs equivariant transformations on node embeddings and coordinate embeddings. See an example of usage below:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# some graph&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_nodes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;   &lt;span class=&quot;c&quot;&gt;# node features&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_nodes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;    &lt;span class=&quot;c&quot;&gt;# node coordinates&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;a&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_edges&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;    &lt;span class=&quot;c&quot;&gt;# edge features&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;EGNNConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;h&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;GNNs with the capability of equivariant transformations have wide application in real-world structure data that have coordinates (e.g., molecules, point clouds, etc.). EGNN simplified previous attempts and proposed a design that is equivariant to rotations, translations, reflections and permutations on N-dimensional coordinates while considering both node features and node coordinates.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;principal-neighbourhood-aggregation-layer&quot;&gt;Principal Neighbourhood Aggregation Layer&lt;/h2&gt;

&lt;p&gt;We added a new GNN module &lt;code class=&quot;highlighter-rouge&quot;&gt;PNAConv&lt;/code&gt; introduced in &lt;a href=&quot;https://arxiv.org/abs/2004.05718&quot;&gt;Principal Neighbourhood Aggregation for Graph Nets&lt;/a&gt;. The code below shows an example of usage:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;torch&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# some graph&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;randn&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_nodes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# node features&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PNAConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
                      &lt;span class=&quot;n&quot;&gt;aggregators&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'mean'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;'max'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;'sum'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
                      &lt;span class=&quot;n&quot;&gt;scalers&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'identity'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;'amplification'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
                      &lt;span class=&quot;n&quot;&gt;delta&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;2.5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;h&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;conv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;em&gt;Developer Recommendation:&lt;/em&gt;&lt;/strong&gt; &lt;em&gt;Principal Neighbourhood Aggregation (PNA) improves the expressive power of a GNN by combining multiple aggregation functions with degree-scalars, thus making it a state-of-the-art baseline for many graph classification tasks.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&quot;survey&quot;&gt;Survey&lt;/h2&gt;

&lt;p&gt;If there are papers for which you want to have DGL implementations or you have other feedback and suggestions, you could fill in &lt;a href=&quot;https://forms.gle/tgKXFuUiGo9PeYBeA&quot;&gt;this survey&lt;/a&gt;.&lt;/p&gt;</content><author><name>DGLTeam</name></author><category term="release" /><category term="release" /><summary type="html">Grouped Reversible Residual Connection for GNNs</summary></entry><entry><title type="html">v0.8 Release Highlights</title><link href="https://www.dgl.ai/release/2022/03/01/release.html" rel="alternate" type="text/html" title="v0.8 Release Highlights" /><published>2022-03-01T00:00:00+00:00</published><updated>2022-03-01T00:00:00+00:00</updated><id>https://www.dgl.ai/release/2022/03/01/release</id><content type="html" xml:base="https://www.dgl.ai/release/2022/03/01/release.html">&lt;p&gt;We are excited to announce the release of DGL v0.8, which brings many new
features as well as improvement on system performance. The highlights are:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;A major update of the mini-batch sampling pipeline, better customizability,
more optimizations; &lt;strong&gt;3.9x&lt;/strong&gt; and &lt;strong&gt;1.5x&lt;/strong&gt; faster for supervised and unsupervised
GraphSAGE on OGBN-Products, with only one line of code change.&lt;/li&gt;
  &lt;li&gt;Significant acceleration and code simplification of popular heterogeneous
graph NN modules (Up to &lt;strong&gt;36x&lt;/strong&gt; for RGCN convolution and &lt;strong&gt;12x&lt;/strong&gt; for HGT
convolution). 11 new off-the-shelf NN modules for building models for link
prediction, heterogeneous graph learning and GNN explanation.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;GNNLens&lt;/strong&gt;: a DGL empowered tool to visualize and understand graph data using
GNN explanation models.&lt;/li&gt;
  &lt;li&gt;New functions to create, transform and augment graph datasets, making it
easier to conduct research on graph contrastive learning or repurposing a
graph for different tasks.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;DGL-Go&lt;/strong&gt;: a new GNN model training command line tool that utilizes a simple
interface so that users can quickly apply GNNs to their problems and
orchestrate experiments with state-of-the-art GNN models.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;mini-batch-sampling-pipeline-update&quot;&gt;Mini-batch Sampling Pipeline Update&lt;/h2&gt;

&lt;p&gt;In training Neural Networks, minibatch sampling has been used to both improve
model performance and enable scaling to large datasets.  Mini-batch training in
the context of GNNs on graphs introduces new complexities, which can be broken
down into four main steps:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Extract a subgraph from the original graph.&lt;/li&gt;
  &lt;li&gt;Perform transformations on the subgraph.&lt;/li&gt;
  &lt;li&gt;Fetch the node/edge features of the subgraph.&lt;/li&gt;
  &lt;li&gt;Pass the subgraph and its features as the input to your GNN model and update parameters.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Among them, steps 1-3 are unique to GNNs and are quite costly. In v0.7, we have
released the feature to speedup step 2 by transforming subgraphs on GPU, but
the other two may continue to be the bottleneck. In this release, we further
optimized the &lt;em&gt;entire&lt;/em&gt; pipeline to reach an even better performance. We then
briefly describe our technical solutions behind that.&lt;/p&gt;

&lt;p&gt;To speed up subgraph extraction, we utilized CUDA Unified Virtual Addressing(UVA).&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2022-03-01-release/uva.png&quot; alt=&quot;uva&quot; width=&quot;400x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;(Image courtesy: &lt;a href=&quot;https://developer.download.nvidia.cn/CUDA/training/cuda_webinars_GPUDirect_uva.pdf&quot;&gt;https://developer.download.nvidia.cn/CUDA/training/cuda_webinars_GPUDirect_uva.pdf&lt;/a&gt;)&lt;/p&gt;

&lt;p&gt;CUDA UVA allows users to create in-memory data beyond the size of GPU RAM
capacity while still harnessing GPU kernels for fast computation. Storing the
entire graph structure and its features in UVA enables efficient subgraph
extraction using GPU kernels, which is effective for training large-scale GNNs
[1][2]. In this release, users can turn on the UVA mode by setting the &lt;code class=&quot;highlighter-rouge&quot;&gt;use_uva&lt;/code&gt;
flag in &lt;code class=&quot;highlighter-rouge&quot;&gt;DataLoader&lt;/code&gt;, as shown in the example below:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;                  &lt;span class=&quot;c&quot;&gt;# some DGLGraph data&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;train_nids&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;         &lt;span class=&quot;c&quot;&gt;# training node IDs&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;sampler&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dataloading&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;MultiLayerNeighborSampler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;fanout&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;15&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dataloader&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dataloading&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;DataLoader&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;train_nids&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sampler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'cuda:0'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;     &lt;span class=&quot;c&quot;&gt;# perform sampling on GPU 0&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;batch_size&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1024&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;shuffle&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;use_uva&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;         &lt;span class=&quot;c&quot;&gt;# turn on UVA optimization&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;To speed up feature fetching (step 3), DGL 0.8 supports pre-fetching node/edge
features so that the model computation can happen in parallel with data
movement. Users can specify the features as well as the labels to prefetch in
the sampler object.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;           &lt;span class=&quot;c&quot;&gt;# some DGLGraph data&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;train_nids&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# training node IDs&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;sampler&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dataloading&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;MultiLayerNeighborSampler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;fanout&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;15&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;prefetch_node_feats&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'feat'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;   &lt;span class=&quot;c&quot;&gt;# prefetch node feature 'feat'&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;prefetch_labels&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'label'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;      &lt;span class=&quot;c&quot;&gt;# prefetch node label 'label'&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dataloader&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dataloading&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;DataLoader&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;train_nids&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sampler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;device&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'cuda:0'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;     &lt;span class=&quot;c&quot;&gt;# perform sampling on GPU 0&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;batch_size&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1024&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;shuffle&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;use_uva&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;         &lt;span class=&quot;c&quot;&gt;# turn on UVA optimization&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;These optimizations bring significant speedup for both supervised and
unsupervised mini-batch training. We compared it against the original pipeline
of sampling on CPU but training on GPU for training a two-layer GraphSAGE model
on the ogbn-papers100M graph using A100 GPUs. We observed a speedup of 3.9x and
1.5x for supervised and unsupervised GraphSAGE on a single GPU respectively.
The speedup applies to multi-GPU training as well.&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt;&lt;img src=&quot;/assets/images/posts/2022-03-01-release/number1.png&quot; alt=&quot;number1&quot; /&gt;&lt;/th&gt;
      &lt;th&gt;&lt;img src=&quot;/assets/images/posts/2022-03-01-release/number2.png&quot; alt=&quot;number2&quot; /&gt;&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;Speedup of Supervised GraphSAGE&lt;/td&gt;
      &lt;td&gt;Speedup of Unsupervised GraphSAGE&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;Defining a new sampler in DGL v0.8 is also easier, with only one simple
interface &lt;code class=&quot;highlighter-rouge&quot;&gt;sample&lt;/code&gt; to follow. Optionally, users can specify how to prefetch
features for each sample. For example, the cluster sampler used by Cluster-GCN
can be implemented in just a few lines of code.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;ClusterGCNSampler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;prefetch_ndata&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;None&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;part_ids&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;metis_partition_assignment&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;c&quot;&gt;# convert partition assignment to bins of nodes&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;part_sizes&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;histogram&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;part_ids&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;float&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;k&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;int&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;node_bins&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;split&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;argsort&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;part_ids&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;part_sizes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;c&quot;&gt;# save the node feature names to be prefetched&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;prefetch_ndata&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;prefetch_ndata&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;sample&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;part_ids&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;s&quot;&gt;&quot;&quot;&quot;Sample a subgraph given a list of partition IDs.&quot;&quot;&quot;&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;node_ids&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;torch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cat&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;([&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;node_bins&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;pid&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;pid&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;part_ids&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;sg&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;subgraph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;node_ids&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# get an induced subgraph&lt;/span&gt;
        &lt;span class=&quot;c&quot;&gt;# tell which feature to pre-fetch&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;set_node_lazy_feature&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sg&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;prefetch_ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sg&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;New samplers in v0.8:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://docs.dgl.ai/generated/dgl.dataloading.ClusterGCNSampler.html#dgl.dataloading.ClusterGCNSampler&quot;&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.dataloading.ClusterGCNSampler&lt;/code&gt;&lt;/a&gt;: The sampler from &lt;a href=&quot;https://arxiv.org/abs/1905.07953&quot;&gt;Cluster-GCN: An
Efficient Algorithm for Training Deep and Large Graph Convolutional Networks&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://docs.dgl.ai/generated/dgl.dataloading.ShaDowKHopSampler.html#dgl.dataloading.ShaDowKHopSampler&quot;&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.dataloading.ShaDowKHopSampler&lt;/code&gt;&lt;/a&gt;: The sampler from &lt;a href=&quot;https://arxiv.org/abs/2012.01380&quot;&gt;Deep Graph Neural Networks with Shallow Subgraph Samplers&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This remarkable improvement would not happen without the help from the community.
We want to thank Xin Yao (@yaox12) and Dominique LaSalle (@nv-dlasalle) from
NVIDIA and David Min (@davidmin7) from UIUC for their contributions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Further reading:&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;User guide chapter for &lt;a href=&quot;https://docs.dgl.ai/guide/minibatch-custom-sampler.html&quot;&gt;customizing graph samplers&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;User guide chapter for &lt;a href=&quot;https://docs.dgl.ai/guide/minibatch-prefetching.html&quot;&gt;writing graph samplers with feature prefetching&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/examples/pytorch/cluster_gcn&quot;&gt;Example implementation&lt;/a&gt; of ClusterGCN.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;nn-module-update&quot;&gt;NN Module Update&lt;/h2&gt;
&lt;p&gt;Heterogeneous GNNs are known to be both difficult to implement as well as
difficult to optimize. In this release, we have significantly improved the
speed of dgl.nn.RelGraphConv and dgl.nn.HGTConv – two state-of-the-art NN
modules for training on heterogeneous graphs, by sometimes an order of
magnitude compared with various baselines[3]:&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt;&lt;img src=&quot;/assets/images/posts/2022-03-01-release/number3.png&quot; alt=&quot;number3&quot; /&gt;&lt;/th&gt;
      &lt;th&gt;&lt;img src=&quot;/assets/images/posts/2022-03-01-release/number4.png&quot; alt=&quot;number4&quot; /&gt;&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;Speedup of RGCN convolution&lt;/td&gt;
      &lt;td&gt;Speedup of HGT convolution&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;More importantly, writing an efficient heterogeneous graph convolution is
substantially easier. Here is a minimal implementation of RGCN convolution in
0.8 using the new &lt;a href=&quot;https://docs.dgl.ai/generated/dgl.nn.pytorch.TypedLinear.html#dgl.nn.pytorch.TypedLinear&quot;&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;nn.TypedLinear&lt;/code&gt;&lt;/a&gt; module:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;RGCNConv&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;in_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;out_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_etypes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;c&quot;&gt;# TypedLinear is a new module in 0.8!&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;linear_r&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;TypedLinear&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;in_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;out_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_etypes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;forward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;etype&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'x'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;edata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'etype'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;etype&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;update_all&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;message&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;function&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;sum&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'m'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;'h'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;g&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ndata&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'h'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;message&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;edges&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;linear_r&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;edges&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;src&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'h'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;edges&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'etype'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;This release also brings 11 new NN modules covering the most requested ones
from the community. They include but are not limited to:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;Commonly used &lt;a href=&quot;https://docs.dgl.ai/api/python/nn-pytorch.html#score-modules-for-link-prediction-and-knowledge-graph-completion&quot;&gt;edge score modules&lt;/a&gt; (e.g., TransE, TransR, etc.) for link
prediction.&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://docs.dgl.ai/api/python/nn-pytorch.html#heterogeneous-learning-modules&quot;&gt;Linear projection module and embedding module for heterogeneous graphs&lt;/a&gt;
(&lt;code class=&quot;highlighter-rouge&quot;&gt;nn.HeteroLinear&lt;/code&gt; and &lt;code class=&quot;highlighter-rouge&quot;&gt;nn.HeteroEmbedding&lt;/code&gt;).&lt;/li&gt;
  &lt;li&gt;The &lt;a href=&quot;https://docs.dgl.ai/generated/dgl.nn.pytorch.explain.GNNExplainer.html#dgl.nn.pytorch.explain.GNNExplainer&quot;&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;GNNExplainer&lt;/code&gt;&lt;/a&gt; module.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;understand-graph-via-visualisation-and-gnn-based-explanation&quot;&gt;Understand Graph via Visualisation and GNN-based Explanation&lt;/h2&gt;
&lt;p&gt;Understanding graph data using GNN-based explanation model has become an
important research topic. We partnered with the HKUST VisLab team to release
GNNLens, an interactive visualization tool for graph neural networks.&lt;/p&gt;

&lt;p&gt;To install GNNLens, &lt;code class=&quot;highlighter-rouge&quot;&gt;pip install gnnlens&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It provides Python APIs for specifying the data to be visualized. For example,
the following shows how to load DGL’s built-in Cora graph dataset and visualize
it:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.data&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;CoraGraphDataset&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;CoraGraphDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;G&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;

&lt;span class=&quot;kn&quot;&gt;from&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;gnnlens&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Writer&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;# Specify the path to create a new directory for dumping data files.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;writer&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Writer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'tutorial_graph'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;writer&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;add_graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'Cora'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cora_graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;writer&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;add_graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'Citeseer'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;graph&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;citeseer_graph&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# Finish dumping&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;writer&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;close&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;After running the script, you can then launch GNNLens with the following command:&lt;/p&gt;

&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;gnnlens --logdir tutorial_graph
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;And you will see the webpage in your browser:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2022-03-01-release/gnnlens.png&quot; alt=&quot;gnnlens&quot; width=&quot;800x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;GNNLens is not only capable of visualizing raw graph data, but also designed
for inspecting graph neural networks such as running explanation models to
explain the prediction. Please check out the tutorials in the project README:
&lt;a href=&quot;https://github.com/dmlc/gnnlens2&quot;&gt;https://github.com/dmlc/gnnlens2&lt;/a&gt;.&lt;/p&gt;

&lt;h2 id=&quot;composable-graph-data-transforms&quot;&gt;Composable Graph Data Transforms&lt;/h2&gt;

&lt;p&gt;Graph data augmentation has become an important component for graph contrastive
learning or structural prediction in general. The new release makes it easier
to compose and apply various graph augmentation and transformation algorithms
to all DGL’s built-in dataset. The new &lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.transforms&lt;/code&gt; package follows the
style of the PyTorch Dataset Transforms. Users can specify the transforms to
use with the &lt;code class=&quot;highlighter-rouge&quot;&gt;transform&lt;/code&gt; keyword argument of all DGL datasets:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl.transforms&lt;/span&gt; &lt;span class=&quot;kn&quot;&gt;as&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;T&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Compose&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;([&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;AddSelfLoop&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;T&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;GCNNorm&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CoraGraphDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transform&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;t&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;g&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# graph and features will be transformed automatically&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;p&gt;DGL v0.8 provides 16 commonly used data transform APIs. See the &lt;a href=&quot;https://docs.dgl.ai/api/python/transforms.html&quot;&gt;API
doc&lt;/a&gt; for more information.&lt;/p&gt;

&lt;p&gt;Making graph datasets easily accessible for all kinds of research is important. A common scenario is to adapt a dataset for a different task than it was originally designed for (e.g., training a link prediction model on Cora which is originally for node classification). We therefore add two dataset adapters (&lt;a href=&quot;https://docs.dgl.ai/generated/dgl.data.AsNodePredDataset.html#dgl.data.AsNodePredDataset&quot;&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.data.AsNodePredDataset&lt;/code&gt;&lt;/a&gt; and &lt;a href=&quot;https://docs.dgl.ai/generated/dgl.data.AsLinkPredDataset.html#dgl.data.AsLinkPredDataset&quot;&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;dgl.data.AsLinkPredDataset&lt;/code&gt;&lt;/a&gt;) for this purpose. We also support generating new train/val/test split and save them for later use:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;kn&quot;&gt;import&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;dgl&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;CoraGraphDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# make a Cora dataset suitable for link prediction&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;# add train/val/test split and negative samples&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dgl&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;AsLinkPredDataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dataset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;split_ratio&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;0.8&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;0.1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;0.1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;neg_ratio&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;

&lt;h2 id=&quot;one-more-thing&quot;&gt;One more thing&lt;/h2&gt;

&lt;p&gt;As GNN is still a young and blooming domain, we received many “how to start” questions from our users:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;em&gt;“I’ve heard about GNNs, how to start training a GNN model on my own datasets?”&lt;/em&gt;&lt;/li&gt;
  &lt;li&gt;&lt;em&gt;“I want to learn more about GNNs, how to start experimenting with SOTA baselines?”&lt;/em&gt;&lt;/li&gt;
  &lt;li&gt;&lt;em&gt;“I have some new research ideas, how to start building it upon existing GNN models?”&lt;/em&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To make those first steps easier, we developed &lt;strong&gt;DGL-Go&lt;/strong&gt;, a command line tool for
users to quickly access the latest GNN research progress.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/images/posts/2022-03-01-release/dglgo.png&quot; alt=&quot;dglgo&quot; width=&quot;600x&quot; class=&quot;aligncenter&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Using DGL-Go is as easy as three steps:&lt;/p&gt;
&lt;ol&gt;
  &lt;li&gt;Use &lt;code class=&quot;highlighter-rouge&quot;&gt;dgl configure&lt;/code&gt; to pick the task, dataset and model of your interests.
It generates a configuration file for later use. You could also use &lt;code class=&quot;highlighter-rouge&quot;&gt;dgl
recipe get&lt;/code&gt; to retrieve a configuration file we provided.&lt;/li&gt;
  &lt;li&gt;Use &lt;code class=&quot;highlighter-rouge&quot;&gt;dgl train&lt;/code&gt; to launch training according to the configuration and see
the results.&lt;/li&gt;
  &lt;li&gt;Use &lt;code class=&quot;highlighter-rouge&quot;&gt;dgl export&lt;/code&gt; to generate a &lt;strong&gt;self-contained, reproducible&lt;/strong&gt; Python
script for advanced customization, or try the model on custom data stored in
CSV format.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Install DGL-Go simply by &lt;code class=&quot;highlighter-rouge&quot;&gt;pip install dglgo&lt;/code&gt; and check out the project &lt;a href=&quot;https://github.com/dmlc/dgl/tree/master/dglgo&quot;&gt;README&lt;/a&gt; for more details.&lt;/p&gt;

&lt;h2 id=&quot;further-reading&quot;&gt;Further Reading&lt;/h2&gt;
&lt;p&gt;The full &lt;a href=&quot;https://github.com/dmlc/dgl/releases/tag/0.8.0&quot;&gt;release note&lt;/a&gt; of DGL v0.8.&lt;/p&gt;

&lt;h2 id=&quot;reference&quot;&gt;Reference&lt;/h2&gt;
&lt;p&gt;[1] PyTorch-Direct: Enabling GPU Centric Data Access for Very Large Graph Neural Network Training with Irregular Accesses&lt;/p&gt;

&lt;p&gt;[2] TorchQuiver: &lt;a href=&quot;https://github.com/quiver-team/torch-quiver&quot;&gt;https://github.com/quiver-team/torch-quiver&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;[3] We compared our new &lt;code class=&quot;highlighter-rouge&quot;&gt;nn.RelGraphConv&lt;/code&gt; module with multiple existing baselines from DGL and PyG. For DGL v0.7, Baseline#1 uses the old &lt;code class=&quot;highlighter-rouge&quot;&gt;nn.RelGraphConv&lt;/code&gt; module with &lt;code class=&quot;highlighter-rouge&quot;&gt;low_mem=False&lt;/code&gt;; Baseline#2 uses the old &lt;code class=&quot;highlighter-rouge&quot;&gt;nn.RelGraphConv&lt;/code&gt; with &lt;code class=&quot;highlighter-rouge&quot;&gt;low_mem=True&lt;/code&gt;; Baseline#3 uses &lt;code class=&quot;highlighter-rouge&quot;&gt;nn.HeteroGarphConv&lt;/code&gt;. For PyG, Baseline#1 uses &lt;code class=&quot;highlighter-rouge&quot;&gt;nn.RGCNConv&lt;/code&gt; while Baseline#2 uses &lt;code class=&quot;highlighter-rouge&quot;&gt;nn.FastRGCNConv&lt;/code&gt;. All the benchmarks are tested on one NVIDIA T4 GPU card.&lt;/p&gt;</content><author><name>DGLTeam</name></author><category term="release" /><category term="release" /><summary type="html">We are excited to announce the release of DGL v0.8, which brings many new features as well as improvement on system performance. The highlights are:</summary></entry></feed>