[PUBLISHER] Merge #13

* PUSH NOTE : Priors over Neural Network weights.md

* PUSH ATTACHMENT : Pasted image 20240605141546.png

* PUSH ATTACHMENT : Pasted image 20240605142432.png

* PUSH NOTE : Equivariance Initialization.md

* PUSH NOTE : Alex Flinth.md

* PUSH NOTE : Understanding Deep Learning - Chapter 10.md

* PUSH NOTE : Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolution.md

* PUSH NOTE : Optimization Dynamics of Equivariant and Augmented Neural Networks.md

docs/000 Zettelkasten/Equivariance Initialization.md

@@ -0,0 +1,8 @@
+---
+tags:
+  - dl_theory
+share: true
+---
+
+Related:
+- [[Priors over Neural Network weights|Priors over Neural Network weights]]

docs/000 Zettelkasten/Priors over Neural Network weights.md

@@ -0,0 +1,13 @@
+---
+tags:
+  - dl_theory
+  - equivariance
+share: true
+---
+From [[Understanding Deep Learning - Chapter 10|Understanding Deep Learning - Chapter 10]], 1d convolutions can be represented as weight matrices from a MLP with a specific prior where the diagonals are the same (d).
+![[Pasted image 20240605141546.png|Pasted image 20240605141546.png]]
+
+---
+
+Rotationally equivariant convolutions can be implemented by isotropic filters (a prior on the conv2d weight):
+![[Pasted image 20240605142432.png|Pasted image 20240605142432.png]]

docs/100 Reference notes/101 Literature/Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolution.md

@@ -0,0 +1,20 @@
+---
+authors:
+  - "[[Rui Wang|Rui Wang]]"
+  - "[[Elyssa Hofgard|Elyssa Hofgard]]"
+  - "[[Han Gao|Han Gao]]"
+  - "[[Robin Walters|Robin Walters]]"
+  - "[[Tess E Smidt|Tess E Smidt]]"
+year: 2024
+tags:
+  - equivariance
+  - relaxed_equivariance
+  - dl_theory
+url: https://arxiv.org/abs/2310.02299
+share: true
+---
+> [!info] Abstract
+> Modeling symmetry breaking is essential for understanding the fundamental changes in the behaviors and properties of physical systems, from microscopic particle interactions to macroscopic phenomena like fluid dynamics and cosmic structures. Thus, identifying sources of asymmetry is an important tool for understanding physical systems. In this paper, we focus on learning asymmetries of data using relaxed group convolutions. We provide both theoretical and empirical evidence that this flexible convolution technique allows the model to maintain the highest level of equivariance that is consistent with data and discover the subtle symmetry-breaking factors in various physical systems. We employ various relaxed group convolution architectures to uncover various symmetry-breaking factors that are interpretable and physically meaningful in different physical systems, including the phase transition of crystal structure, the isotropy and homogeneity breaking in turbulent flow, and the time-reversal symmetry breaking in pendulum systems.
+
+Observations:
+- "In the relaxed group convolution, the **initial relaxed (equivariant) weights** {𝑤𝑙⁢(ℎ)} in each layer are set to be the same for all ℎ, **ensuring that the model exhibits equivariance prior to being trained**. [...] we prove that these relaxed weights **only deviate from being equal when the symmetries of the input and the output are lower than that of the model.**" (Related to [[Equivariance Initialization|Equivariance Initialization]])

docs/100 Reference notes/101 Literature/Optimization Dynamics of Equivariant and Augmented Neural Networks.md

@@ -0,0 +1,26 @@
+---
+authors:
+  - "[[Alex Flinth|Alex Flinth]]"
+  - "[[Fredrik Ohlsson|Fredrik Ohlsson]]"
+year: 2023
+tags:
+  - dl_theory
+  - equivariance
+  - optimization
+url: https://arxiv.org/abs/2303.13458
+share: true
+---
+> [!info] Abstract
+> We investigate the optimization of multilayer perceptrons on symmetric data. We compare the strategy of constraining the architecture to be equivariant to that of using augmentation. We show that, under natural assumptions on the loss and non-linearities, the sets of equivariant stationary points are identical for the two strategies, and that the set of equivariant layers is invariant under the gradient flow for augmented models. Finally, we show that stationary points may be unstable for augmented training although they are stable for the equivariant models.
+
+Main observations:
+1. They show that **if the augmented model is equivariantly initialized, it will remain equivariant during training** (See [[Equivariance Initialization|Equivariance Initialization]])
+3. Compared to the equivariant approach, **augmentation introduces no new equivariant stationary points**, nor does it exclude existing ones. (See [[Multiple global minima|Multiple global minima]])
+4. The existence of a **stable equivariant minimum is not guaranteed by augmentation**. (See [[Multiple global minima|Multiple global minima]])
+
+
+Regarding [[Equivariance Initialization|Equivariance Initialization]] in this work:
+
+> We initialize ΦA with equivariant layers A0 ∈ E by drawing matrices randomly from a standard Gaussian distribution, and then projecting them orthogonally onto E. We train the network on (finite) datasets D using gradient descent in three different ways. 
+
+My intuition is that they do something like the isotropic convolution from [[Priors over Neural Network weights|Priors over Neural Network weights]]

docs/100 Reference notes/101 Literature/Understanding Deep Learning - Chapter 10.md

@@ -0,0 +1,12 @@
+---
+authors:
+  - "[[Simon J.D. Prince|Simon J.D. Prince]]"
+year: 2023
+tags:
+  - textbook
+  - dl_theory
+url: https://udlbook.github.io/udlbook/
+share: true
+---
+
+

docs/100 Reference notes/102 Authors/Alex Flinth.md

@@ -0,0 +1,5 @@
+---
+affiliation:
+  - "[[Umea University|Umea University]]"
+share: true
+---