index.xml

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Justas Dauparas on Justas Dauparas</title>
    <link>https://dauparas.github.io/</link>
    <description>Recent content in Justas Dauparas on Justas Dauparas</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en-us</language>
    <copyright>&amp;copy; 2020 Justas Dauparas</copyright>
    <lastBuildDate>Thu, 17 Dec 2020 06:44:46 -0400</lastBuildDate>
    <atom:link href="/" rel="self" type="application/rss+xml" />
    
    <item>
      <title>AlphaFold 2 &amp; Equivariance</title>
      <link>https://dauparas.github.io/post/af2/</link>
      <pubDate>Thu, 17 Dec 2020 06:44:46 -0400</pubDate>
      
      <guid>https://dauparas.github.io/post/af2/</guid>
      <description>

&lt;p&gt;&lt;a href=&#34;https://twitter.com/fabianfuchsml&#34; target=&#34;_blank&#34;&gt;Fabian Fuchs&lt;/a&gt; &amp;amp; &lt;a href=&#34;https://twitter.com/JustasDauparas&#34; target=&#34;_blank&#34;&gt;Justas Dauparas&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A few weeks ago, in the latest CASP competition for protein structure prediction (&lt;a href=&#34;https://predictioncenter.org/casp14/index.cgi&#34; target=&#34;_blank&#34;&gt;CASP14&lt;/a&gt;), DeepMind&amp;rsquo;s AlphaFold 2&lt;sup class=&#34;footnote-ref&#34; id=&#34;fnref:3&#34;&gt;&lt;a href=&#34;#fn:3&#34;&gt;1&lt;/a&gt;&lt;/sup&gt; outperformed all its competitors with an unprecedented margin. In this blog post, we aim to shed light on one of the important building blocks that distinguishes AlphaFold 2 from the other approaches and likely contributed to their success: an equivariant structure prediction module.&lt;/p&gt;

&lt;p&gt;In short: this module is a neural network that iteratively refines the structure predictions while respecting and leveraging an important symmetry of the problem, namely that of roto-translations. At this point, DeepMind has not yet published a paper so we do not know how exactly they address this. However, from their CASP14 presentation, it seems possible that their model is similar to the &lt;a href=&#34;https://arxiv.org/abs/2006.10503&#34; target=&#34;_blank&#34;&gt;SE(3)-Transformer&lt;/a&gt;. Most of this blog post is written irrespectively of the exact implementation of their SE(3)&lt;sup class=&#34;footnote-ref&#34; id=&#34;fnref:4&#34;&gt;&lt;a href=&#34;#fn:4&#34;&gt;2&lt;/a&gt;&lt;/sup&gt;-equivariant transformer module. At some points, particularly when we go into depth, we need to make some assumptions, but we will state those explicitly. We assume some basic understanding of deep learning as a prerequisite for reading this post, but not much more. We hope this blog post is valuable for both machine learners and biologists/chemists.&lt;/p&gt;

&lt;h2 id=&#34;what-is-the-protein-structure-prediction-problem-and-why-is-it-symmetric&#34;&gt;&lt;strong&gt;What is the Protein Structure Prediction Problem and Why is it Symmetric?&lt;/strong&gt;&lt;/h2&gt;

&lt;p&gt;In the protein structure prediction problem the rules of the game are that given a sequence of &lt;a href=&#34;https://en.wikipedia.org/wiki/Amino_acid&#34; target=&#34;_blank&#34;&gt;amino acids&lt;/a&gt; (alphabet = &amp;ldquo;ARNDCQEGHILKMFPSTWYV-&amp;rdquo;) you have to return XYZ coordinates of all atoms in some frame of reference. The simplified version of this problem is to return XYZ coordinates of the protein backbone atoms only, ignoring side chain atoms.&lt;/p&gt;

&lt;p&gt;What does &lt;em&gt;some frame of reference&lt;/em&gt; mean? It means that the model is expected to choose an &lt;em&gt;overall orientation&lt;/em&gt; of the protein arbitrarily. This can be broken down into the position of the centre of mass as well as the angular orientation. Assuming the protein floats freely in a solution, both of these choices do not affect the energy of the system&lt;sup class=&#34;footnote-ref&#34; id=&#34;fnref:5&#34;&gt;&lt;a href=&#34;#fn:5&#34;&gt;3&lt;/a&gt;&lt;/sup&gt;. Leveraging this property of the prediction task could prove very beneficial for the performance of the model. We will explore this later in this blog post.&lt;/p&gt;

&lt;p&gt;Traditionally, this symmetry has been dealt with in a fairly straightforward way. The first generation of AlphaFold, for example, predicted distance distributions between atoms and converted those into a potential (which is easy to make symmetric). A gradient descent algorithm was then used to find coordinates for the protein which minimise the potential. AlphaFold 2, on the other hand, is doing something different. It predicts the coordinates directly, in an end-to-end fashion. This end-to-end regression to 3D coordinates is what makes it more difficult to deal with the symmetry correctly.&lt;/p&gt;

&lt;h2 id=&#34;protein-backbone-geometry&#34;&gt;&lt;strong&gt;Protein Backbone Geometry&lt;/strong&gt;&lt;/h2&gt;

&lt;p&gt;As we mentioned earlier, the easier (but still pretty hard) version of the protein structure prediction problem is to determine the protein backbone geometry. Let&amp;rsquo;s have a look at what a protein backbone is and how it can be represented.&lt;/p&gt;

&lt;p&gt;A protein backbone is a repeating sequence (linear chain) of 3 atoms: nitrogen, carbon, and another carbon, namely&lt;/p&gt;

&lt;p&gt;$$\underbrace{N^{(1)}, C_{\alpha}^{(1)}, C^{(1)}}, \underbrace{N^{(2)}, C_{\alpha}^{(2)}, C^{(2)}},&amp;hellip;,\underbrace{N^{(L)}, C_{\alpha}^{(L)}, C^{(L)}}$$&lt;/p&gt;

&lt;p&gt;for the protein of length $L$. The middle carbon has a special name. It is called &amp;ldquo;C alpha&amp;rdquo; because that is where the side chain is connected to. The first three atoms belong to the first amino acid (first residue), the next three to the next amino acid (next residue) etc. Below we depict the protein backbone geometry of two residues.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://i.imgur.com/fZ0Sx3V.png&#34; alt=&#34;&#34; /&gt;
In the above illustration of the protein backbone, the atoms would be represented as: $N^{(1)}, C_{\alpha}^{(1)}, C^{(1)}, N^{(2)}, C_{\alpha}^{(2)},C^{(2)},N^{(3)}.$ Bond lengths are shown in angstroms and angles in degrees. For example the distance between $N^{(i)}$ and $C^{(i)}$ is about $1.47 \unicode{x212B}$ (in SI units $1.47\cdot 10^{-10}$ meters). Notice that the bond lengths and angles between neighboring backbone atoms are fixed. The only remaining degrees of freedom are the so-called torsion (dihedral) angles. Those are shown as $\psi^{(1)}, \omega^{(1)}, \phi^{(2)}, \psi^{(2)}, \omega^{(2)}$ in the diagram. A red triangle shows a plane connecting 3 atoms from one amino acid. All red triangles are of the same size except that they have different positions and different orientations. Now let&amp;rsquo;s have a look at a real protein structure visualised in PyMOL where each amino acid is only depicted by its $C_{\alpha}$ because other backbone atoms are rigidly connected to it.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://i.imgur.com/uiW9ESH.jpg&#34; alt=&#34;&#34; /&gt;
This example protein, &lt;a href=&#34;https://www.rcsb.org/structure/1qys&#34; target=&#34;_blank&#34;&gt;1QYS&lt;/a&gt;, demonstrates a relatively simple backbone configuration. On the left, adjacent $C_{\alpha}$ atoms (black dots) are connected by straight lines ($N, C$ are left out for clarity). On the right, the cartoon structure shows subsets of residues as &lt;a href=&#34;https://en.wikipedia.org/wiki/Alpha_helix&#34; target=&#34;_blank&#34;&gt;alpha helices&lt;/a&gt; and &lt;a href=&#34;https://en.wikipedia.org/wiki/Beta_sheet&#34; target=&#34;_blank&#34;&gt;beta sheets&lt;/a&gt; which are &lt;a href=&#34;https://en.wikipedia.org/wiki/Protein_secondary_structure&#34; target=&#34;_blank&#34;&gt;secondary structure&lt;/a&gt; representations. The colour changes from blue at the start to red at the end of the backbone. This protein is 92 amino acid residues long ($L=92$).&lt;/p&gt;

&lt;p&gt;Proteins come in various lengths. For example, in the CASP14 competition, the shortest target protein had 32 amino acids, whereas the longest one had 2180. This means that the computational structure prediction pipeline needs to handle variable-length inputs/outputs as well as long sequences. Furthermore, proteins can assemble into &lt;a href=&#34;https://en.wikipedia.org/wiki/Protein_complex&#34; target=&#34;_blank&#34;&gt;protein complexes&lt;/a&gt; to form larger structures which are more difficult to predict.&lt;/p&gt;

&lt;!-- **Fabian:** *I think, after the figure and before the next headline it would be nice to have a paragraph giving a little background about the relevance of this backbone representation. Is there some upper limit on how heavy a residue can be? How heavy is the overall protein? How long can the backbone be?* --&gt;

&lt;h3 id=&#34;ways-to-represent-a-protein-backbone&#34;&gt;Ways to Represent a Protein Backbone&lt;/h3&gt;

&lt;p&gt;Predicting backbone configurations in 3D space may seem to be straightforward, but is in fact challenging because atoms that are far in the sequence space (&lt;a href=&#34;https://en.wikipedia.org/wiki/Protein_structure#Primary_structure&#34; target=&#34;_blank&#34;&gt;primary protein structure&lt;/a&gt;) can be close in 3D space and they can interact. This suggests that non-local modelling is needed to accurately predict &lt;a href=&#34;https://en.wikipedia.org/wiki/Protein_structure#Tertiary_structure&#34; target=&#34;_blank&#34;&gt;tertiary structure&lt;/a&gt;. At the same time, protein backbones have a clearly defined local structure (bond lengths and angles are fixed). Choosing to focus on local and/or non-local interactions leads to different backbone representations. In the following, we will describe four common ways of representing the protein backbone. Finally, we will explain how AlphaFold 2 combined some of these and why.&lt;/p&gt;

&lt;!-- 
We are going to describe protein backbone in four different ways using XYZ coordinates of all atoms, XYZ coordinates of C alpha + orientation, torsion angles, and relative distances/orientations. --&gt;

&lt;!-- **Fabian:** *Let&#39;s add a paragraph motivating why knowing about the different representations is important? Motivating here + summarise in the end* --&gt;

&lt;h4 id=&#34;1-xyz-coordinates-of-atoms&#34;&gt;1. XYZ Coordinates of Atoms&lt;/h4&gt;

&lt;p&gt;Protein backbone atoms can be described using XYZ coordinates. The first residue is composed of $N^{(1)}, C_{\alpha}^{(1)}$, and $C^{(1)}$ each of which has an X, Y, and Z coordinate. This results in a 3 by 3 matrix that represents the entire residue. When you stack these together for the whole backbone of length $L$ you generate a tensor $X$:&lt;/p&gt;

&lt;p&gt;$$X\in \mathbb{R}^{3\times 3 \times L},$$&lt;/p&gt;

&lt;p&gt;Remember that some distances and angles between backbone atoms (within and between residues) are fixed, so these coordinates will have to satisfy those constraints to result in a valid protein structure.&lt;/p&gt;

&lt;h4 id=&#34;2-xyz-coordinates-of-c-alpha-orientation&#34;&gt;2. XYZ Coordinates of $C_\alpha$ + Orientation&lt;/h4&gt;

&lt;p&gt;This representation is based on the premise that distances and angles between $C_{\alpha}^{(i)}$ and $N^{(i)}$ and $C^{(i)}$ are fixed. Therefore it is enough to represent the XYZ coordinates of  $C_{\alpha}^{(i)}$ and the orientation of these atoms relative to $C_{\alpha}^{(i)}$. There will still be constraints on the distance and angle between adjacent residues that need to be satisfied for the local geometry to be feasible, but otherwise this representation automatically preserves intrinsic residue geometry. AlphaFold 2 used this representation in their structure module.&lt;/p&gt;

&lt;h4 id=&#34;3-protein-torsion-dihedral-angles&#34;&gt;3. Protein Torsion (Dihedral) Angles&lt;/h4&gt;

&lt;p&gt;Since bond lengths and angles between neighbouring backbone atoms are fixed, the only remaining degrees of freedom are the torsion (dihedral) angles.&lt;/p&gt;

&lt;!-- ![](https://i.imgur.com/HWuiR3Z.png) --&gt;

&lt;p&gt;&lt;img src=&#34;https://i.imgur.com/53c5aWm.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;

&lt;p&gt;A torsion (dihedral) angle is the angle between two intersecting planes, in this case it is the angle between the green and the red planes. Mathematically we define the function called $\text{Torsion}$ which takes in 4 points in a chain ($C^{(1)}, N^{(2)}, C_{\alpha}^{(2)}, C^{(2)}$) and outputs the angle between the planes created by $C^{(1)}, N^{(2)}, C_{\alpha}^{(2)}$ and $N^{(2)}, C_{\alpha}^{(2)}, C^{(2)}$ as a value between $-\pi$ and $\pi$. We write this as&lt;/p&gt;

&lt;p&gt;$$\phi^{(2)}=\text{Torsion}(C^{(1)}, N^{(2)}, C_{\alpha}^{(2)}, C^{(2)})\in [-\pi, \pi].$$
&lt;img src=&#34;https://i.imgur.com/fZ0Sx3V.png&#34; alt=&#34;&#34; /&gt;
Demonstrated in the picture above, each torsion angle has a specific name in protein literature. They are defined as&lt;/p&gt;

&lt;p&gt;$$
\begin{align}
\phi^{(i)}&amp;amp;=\text{Torsion}(C^{(i-1)}, N^{(i)}, C_{\alpha}^{(i)}, C^{(i)}),\\&lt;br /&gt;
\psi^{(i)}&amp;amp;=\text{Torsion}(N^{(i)}, C_{\alpha}^{(i)}, C^{(i)}, N^{(i+1)}),\\&lt;br /&gt;
\omega^{(i)}&amp;amp;=\text{Torsion}(C_{\alpha}^{(i)}, C^{(i)}, N^{(i+1)}, C_{\alpha}^{(i+1)}),
\end{align}
$$&lt;/p&gt;

&lt;p&gt;where $i = 1, 2, 3&amp;hellip;,L$. Notice that $\phi^{(1)}$ and $\psi^{(L)}, \omega^{(L)}$ are not defined because there are no $C^{(0)}$ and $N^{(L+1)}$ atoms. Therefore a protein backbone can be represented as a sequence of torsion angles: $$\psi^{(1)}, \omega^{(1)}, \phi^{(2)}, \psi^{(2)}, \omega^{(2)},&amp;hellip;,\phi^{(L)}.$$This representation is independent of the frame of reference and it does not have any local bond length/angle constraints, but it suffers from the lever arm effect meaning that small errors propagate along the backbone. Interestingly, AlphaFold 1 predicted torsion angles to initialise the gradient descent algorithm to optimize the relative distance potentials to obtain the protein structure. It is possible that AlphaFold 2 also predicted torsion angles to obtain initial coordinates of the structure.&lt;/p&gt;

&lt;h4 id=&#34;4-relative-distances-and-angles-between-residues&#34;&gt;4. Relative Distances and Angles between Residues&lt;/h4&gt;

&lt;p&gt;Another way to represent backbone atoms without a frame of reference is to use the pairwise distance and angle representations. One choice could be to represent relative distances between $C_{\alpha}$ coordinates and relative orientations (e.g. &lt;a href=&#34;https://en.wikipedia.org/wiki/Euler_angles&#34; target=&#34;_blank&#34;&gt;Euler angles&lt;/a&gt;) between red triangles. Denote the Euclidean distance between $C_{\alpha}^{(i)}$ and $C_{\alpha}^{(j)}$ as the distance matrix $D^{(ij)}_{\alpha}$:&lt;/p&gt;

&lt;p&gt;$$D^{(ij)}_{\alpha}=\sqrt{(X_{\alpha}^{(i)}-X_{\alpha}^{(j)})\cdot(X_{\alpha}^{(i)}-X_{\alpha}^{(j)})}=\left\lVert(X_{\alpha}^{(i)}-X_{\alpha}^{(j)})\right \rVert_2.$$&lt;/p&gt;

&lt;p&gt;The $(D^{(ij)}_{\alpha})^2$ matrix has some special properties. If we write out the dot product we get&lt;/p&gt;

&lt;p&gt;\begin{align}
(D^{(ij)}_{\alpha})^2=&amp;amp;(X_{\alpha}^{(i)}\cdot X_{\alpha}^{(i)}+X_{\alpha}^{(j)}X_{\alpha}^{(j)}-2X_{\alpha}^{(i)}X_{\alpha}^{(j)}),\\&lt;br /&gt;
(D^{(ij)}_{\alpha})^2=&amp;amp;(G_{\alpha}^{(ii)}+G_{\alpha}^{(jj)}-2G_{\alpha}^{(ij)}),\\&lt;br /&gt;
G_{\alpha}^{(ij)} :=&amp;amp; X_{\alpha}^{(i)}X_{\alpha}^{(j)},
\end{align}&lt;/p&gt;

&lt;p&gt;where $G_{\alpha}^{(ij)}$ is known as a &lt;a href=&#34;https://en.wikipedia.org/wiki/Gramian_matrix&#34; target=&#34;_blank&#34;&gt;Gram matrix&lt;/a&gt;. The Gram matrix has rank 3 because it is a product of two 3-dimensional vectors $X_{\alpha}^{(i)}$ and $X_{\alpha}^{(j)}$ and therefore the square of the distance matrix has rank $\leq 1+1+3=5$. There are requirements for the geometrically centred distance matrix to be positive semi-definite (meaning all eigenvalues are greater or equal to zero). Read more about Euclidean Distance Matrices &lt;a href=&#34;https://arxiv.org/abs/1502.07541.pdf&#34; target=&#34;_blank&#34;&gt;here&lt;/a&gt;. It is a global representation of the structure.&lt;/p&gt;

&lt;p&gt;If one has a distance matrix, there is a simple algorithm to obtain coordinates that satisfy those distances. It is called &lt;a href=&#34;https://en.wikipedia.org/wiki/Multidimensional_scaling&#34; target=&#34;_blank&#34;&gt;Multidimensional scaling (MDS)&lt;/a&gt;. Here is how it works.&lt;/p&gt;

&lt;p&gt;The squared distance matrix $D_{\alpha}^2$ is centred by the matrix $J$ to recover the Gram matrix $G_{\alpha}$ and then the Gram matrix is decomposed using eigenvalue decomposition into the diagonal matrix of eigenvalues $\Lambda$ and orthonormal matrix $U$. The obtained coordinates $X_{\alpha}$ are correct up to orthogonal transformation (rotation or reflection).&lt;/p&gt;

&lt;p&gt;\begin{align}
J =&amp;amp; I-\frac{1}{L} 11^{T},\\&lt;br /&gt;
G_{\alpha} =&amp;amp; -\frac{1}{2}JD_{\alpha}^2J,\\&lt;br /&gt;
G_{\alpha} =&amp;amp; U\Lambda U^{T}=X_{\alpha}X_{\alpha}^{T},\\&lt;br /&gt;
X_{\alpha} =&amp;amp; U\sqrt{\Lambda}.
\end{align}&lt;/p&gt;

&lt;p&gt;There are many different representations for 3D &lt;a href=&#34;https://en.wikipedia.org/wiki/Orientation_(geometry)&#34; target=&#34;_blank&#34;&gt;relative angles&lt;/a&gt; (change of frame). We are not going to discuss them here.&lt;/p&gt;

&lt;h4 id=&#34;summary&#34;&gt;Summary&lt;/h4&gt;

&lt;p&gt;Even though the final outputs need to be XYZ coordinates, it can be beneficial to use various representations in different parts of the pipeline. Most of the information for protein structure prediction comes from the inverse covariance matrix which is estimated using homologous sequences based on the premise of &lt;a href=&#34;https://en.wikipedia.org/wiki/Direct_coupling_analysis&#34; target=&#34;_blank&#34;&gt;coevolution&lt;/a&gt;. This information comes in a pairwise fashion (because it is a covariance matrix) and is used to predict the relative distances and angles (see method 4 above). Previously, AlphaFold 1 used the &lt;a href=&#34;https://en.wikipedia.org/wiki/Residual_neural_network&#34; target=&#34;_blank&#34;&gt;ResNet&lt;/a&gt; architecture to map coevolution features to binned distance probabilities.&lt;/p&gt;

&lt;p&gt;A protein can be represented as a graph where nodes are residues and edges are the connections between them. These edges include information about the relative distances and angles, but the graph is not embedded in 3D space. This is the &amp;lsquo;trunk&amp;rsquo; part of the AlphaFold 2 network, as shown later. To begin representing this information in 3D space, and obtain intitial XYZ coordinates, one could potentially use torsion angles (see method 3), but this might be too constraining. AlphaFold 2 probably used some network/algorithm to map graph features to obtain the initial XYZ coordinates of $C_\alpha$ + orientation (see method 2 above). Later in the pipeline, they improved their initial prediction by iteratively running their structure module. This is called the refinement step. Now we are going to look at the AlphaFold 2 architecture in more detail.&lt;/p&gt;

&lt;h2 id=&#34;the-alphafold-2-architecture&#34;&gt;&lt;strong&gt;The AlphaFold 2 Architecture&lt;/strong&gt;&lt;/h2&gt;

&lt;p&gt;Having introduced the problem as well as different ways to represent a protein backbone, let&amp;rsquo;s now examine what AlphaFold 2 is doing. The following slide is taken from the CASP14 conference &lt;a href=&#34;https://predictioncenter.org/casp14/doc/presentations/2020_12_01_TS_predictor_AlphaFold2.pdf&#34; target=&#34;_blank&#34;&gt;presentation&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://i.imgur.com/uhRTxUk.png&#34; alt=&#34;AlphaFold2 model&#34; /&gt;&lt;/p&gt;

&lt;p&gt;As we can see, the AlphaFold 2 architecture consists of three parts:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Embedding.&lt;/strong&gt; This is used to encode the target sequence and related sequences (MSA - &lt;a href=&#34;https://en.wikipedia.org/wiki/Multiple_sequence_alignment&#34; target=&#34;_blank&#34;&gt;multiple sequence alignment&lt;/a&gt;) as well as related protein structures called &lt;a href=&#34;https://en.wikipedia.org/wiki/Homology_modeling&#34; target=&#34;_blank&#34;&gt;templates&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Trunk.&lt;/strong&gt; This part learns the residue-residue graph edges and the sequence-residue graph edges. The residue-residue edges represent pairwise information between all residues similar to relative distances in 3D space and relative angles. The sequence-residue edges might have sequence evolutionary information. This information can be used to predict pairwise distances as shown in the slide, but more importantly, it is passed to the structure module to build XYZ coordinates of the structure.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Structure module.&lt;/strong&gt; This module uses a 3D equivariant transformer architecture to refine backbone coordinates and predict side chains. The backbone is represented as oriented red triangles, and the task of this network is to predict new positions and orientations of red triangles as well as the confidence score:&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://i.imgur.com/ZeeFc3k.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;

&lt;p&gt;From here on, we will focus on the structure module as this is where the symmetry is broken (initial 3D coordinates obtained). The &amp;lsquo;Embedding&amp;rsquo; and the &amp;lsquo;Trunk&amp;rsquo; take strings of amino acids and create matrix descriptions about potential interactions and relations. None of this is embedded in 3D space; there are no coordinates attached to any of this information. In the structure module, this bit becomes tricky. Suddenly, there are 3D coordinates. Ordinary neural networks do not know what coordinates are. By default, coordinates are just numbers. If we rotate the entire backbone, these numbers change a lot, even though the energy stays the same. This could, in principle, be learned by a neural network, but dealing with global rotations correctly in itself imposes a learning problem that is far beyond trivial. AlphaFold 2&amp;rsquo;s structure module does not have to learn this. It is baked into its design. To that end, it combines two concepts: self-attention (aka the transformer mechanism) and equivariance.&lt;/p&gt;

&lt;h2 id=&#34;what-is-a-transformer&#34;&gt;&lt;strong&gt;What is a Transformer?&lt;/strong&gt;&lt;/h2&gt;

&lt;p&gt;The words &lt;em&gt;Transformer&lt;/em&gt; and &lt;em&gt;self-attention&lt;/em&gt; are often used interchangeably and describe a mechanism in neural networks which operates on a set of objects (an object could, for example, be an atom or an amino acid) that allows for querying specific information. A self-attention layer maps from set to set or &amp;mdash; in our case &amp;mdash; graph to graph updating the features of the nodes. It focuses on one object at a time, let&amp;rsquo;s say a carbon atom, and queries the surrounding objects &lt;em&gt;conditioned&lt;/em&gt; on the information/features attached to this carbon atom. For example, given what we know already about the location of the carbon atom, it might be particularly useful to query for nitrogen atoms in its surrounding. This is in contrast to ordinary convolutions, which combine features in a weighted sum depending on their relative distance but not depending on the query point.&lt;/p&gt;

&lt;p&gt;We will first go through the attention mechanism in a general setting and then move to a graph context. If you are very familiar with graphs, but you find the first part confusing, stick with us until the end of this section before re-reading, the paragraph on graphs might make it clear for you.&lt;/p&gt;

&lt;p&gt;The following diagram gives a detailed description of what&amp;rsquo;s happening in a general non-graph context:&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://i.imgur.com/KZYyQ5S.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;

&lt;p&gt;At the beginning of the layer, each object has features $f_{in}$ attached to them. The output of the layer will be updated features $f_{out}$ for each object. Linear maps project the input features to &lt;strong&gt;q&lt;/strong&gt;ueries, &lt;strong&gt;k&lt;/strong&gt;eys, and &lt;strong&gt;v&lt;/strong&gt;alues. The query from one object is compared to the keys of all other points using a scalar product and resulting in the attention weights. These are then used to get a weighted sum of the values.&lt;/p&gt;

&lt;p&gt;Moving to a graph context, we can think of it as follows. At the beginning of a layer, we have a feature vector attached to each node. Each feature is then transformed (via a learnable, linear function) into a key, a value and a query vector. The keys and values are then propagated to all the (directed) edges going away from the node. The queries are propagated to all the incoming edges. Next, queries and keys are multiplied to get weights. For each node, a softmax over all the weights on the incoming edges is applied to normalise the attention weights. Finally, the values are multiplied with the attention weights and propagated to the nodes.&lt;/p&gt;

&lt;p&gt;The self-attention mechanism seems like a set of arbitrary choices, so why should we use this? In short, because it is known to work really well. It &amp;lsquo;transformed&amp;rsquo; (sorry) the field of &lt;a href=&#34;https://arxiv.org/abs/1706.03762&#34; target=&#34;_blank&#34;&gt;natural language processing&lt;/a&gt; and also had major successes in the fields of graph learning, image processing and relational reasoning.&lt;/p&gt;

&lt;h2 id=&#34;what-is-equivariance&#34;&gt;&lt;strong&gt;What is Equivariance?&lt;/strong&gt;&lt;/h2&gt;

&lt;p&gt;The second machine learning concept we will take a look at is equivariance. It is explained easiest by referring to CNNs (convolutional neural networks). Convolutional layers are translation equivariant, meaning that if the input image is shifted by 3 pixels to the right, the output is also shifted by 3 pixels to the right (assuming a stride of 1). Shifting the input by a few pixels to the right does not pose an entirely new problem. Making use of this &lt;em&gt;symmetry&lt;/em&gt; and treating the two inputs similarly is important: it saves parameters, reduces overfitting and speeds up learning.&lt;/p&gt;

&lt;p&gt;The following video by Daniel Worrall visualises nicely how this plays out in a CNN (full video &lt;a href=&#34;https://www.youtube.com/watch?v=qoWAFBYOtoU&#34; target=&#34;_blank&#34;&gt;here&lt;/a&gt;):&lt;/p&gt;

&lt;p&gt;&lt;p align=&#34;center&#34;&gt;&lt;iframe width=&#34;700&#34; height=&#34;400&#34; src=&#34;http://edwag.github.io/video/translation_equivariance.mp4&#34; frameborder=&#34;0&#34; allowfullscreen&gt;&lt;/iframe&gt;&lt;a href=&#34;https://dauparas.github.io/&#34; target=&#34;_blank&#34;&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Formally, a function $h(x)$ being equivariant can be written as follows&lt;sup class=&#34;footnote-ref&#34; id=&#34;fnref:1&#34;&gt;&lt;a href=&#34;#fn:1&#34;&gt;4&lt;/a&gt;&lt;/sup&gt;:&lt;/p&gt;

&lt;p&gt;$$h\big(\sigma (x)\big) = \sigma \big(h(x)\big) \quad \quad \forall \sigma, x$$&lt;/p&gt;

&lt;p&gt;Applying a transformation $\sigma$ on the input has the same effect as applying it to the output. For ordinary convolutions $\sigma$ is a translation of all pixels. For 2D images, this is often all we need. However, when we move to proteins (or other graphs / point clouds in 3D), translations alone really do not capture all the relevant transformations. One crucial symmetry group apart from translations is the special orthogonal group SO(3). This is the group which describes rotation in 3D space. Equivariance for rotations reads as follows:&lt;/p&gt;

&lt;p&gt;$$h\big(\mathbf{R} (x)\big) = \mathbf{R} \big(h(x)\big) \quad\quad \forall \,\mathbf{R} \in SO(3)$$&lt;/p&gt;

&lt;p&gt;Let&amp;rsquo;s go back to Deepmind&amp;rsquo;s slide on the structure module in AlphaFold 2. In each iteration, the structure module takes in a spatial configuration (the current best guess of what the protein looks like) and predicts corrections, i.e. local shifts and rotations of individual parts of the chain:&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://i.imgur.com/vbxEcls.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;

&lt;p&gt;The key here is: the orientation of the current protein configuration (i.e. the input to each iteration) is completely arbitrary and to some degree random. In other words, there is a symmetry we should leverage. When rotating the input protein, the corrections (which need to be expressed as vectors and/or matrices) should rotate as well. Any type of network could theoretically &lt;em&gt;learn&lt;/em&gt; this, but it would be much more elegant to bake this into the model directly &amp;mdash; with a rotation equivariant neural network. And to capture both translation &lt;em&gt;and&lt;/em&gt; rotation, we extend the requirement to roto-translation equivariance.&lt;/p&gt;

&lt;h2 id=&#34;how-to-build-a-3d-transformer-se-3-transformer&#34;&gt;&lt;strong&gt;How to build a 3D Transformer / SE(3)-Transformer&lt;/strong&gt;&lt;/h2&gt;

&lt;p&gt;So far, we established the usefulness of transformers and of roto-translation equivariance. The question is, how to actually build a roto-translation equivariant transformer network.&lt;/p&gt;

&lt;p&gt;To clean up nomenclature: the group of roto-translations in 3D is called the SE(3) group. We will assume that DeepMind uses &lt;em&gt;3D&lt;/em&gt; instead of &lt;em&gt;SE(3)&lt;/em&gt; in their slides simply because not everyone knows what SE(3) is.&lt;/p&gt;

&lt;p&gt;There are multiple possible ways of constructing an SE(3)-equivariant transformer. Currently, the only paper that we are aware of that describes this, is ours: &lt;a href=&#34;https://arxiv.org/abs/2006.10503.pdf&#34; target=&#34;_blank&#34;&gt;SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks&lt;/a&gt;.  We can only speculate whether DeepMind uses a similar approach. So at this point, a big fat &lt;strong&gt;disclaimer&lt;/strong&gt;: For the sake of this blog post (and because we are biased), we will assume that they use an approach similar to the SE(3)-Transformer, or at least an approach using irreducible representations&lt;sup class=&#34;footnote-ref&#34; id=&#34;fnref:2&#34;&gt;&lt;a href=&#34;#fn:2&#34;&gt;5&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;

&lt;p&gt;One instrumental truth in equivariance is the following: stacking multiple layers (transformations) which are equivariant with respect to the same group (e.g. SE(3)) will result in a transformation that is also equivariant with respect to that group. To start simple, let&amp;rsquo;s build a layer which takes in a vector and changes its length:&lt;/p&gt;

&lt;p&gt;$$h(\vec{x}) = \alpha \cdot \vec{x}$$&lt;/p&gt;

&lt;p&gt;This is equivariant with respect to rotations:
$$\mathbf{R} \cdot h(\vec{x}) = \mathbf{R} \cdot \alpha \cdot \vec{x} = \alpha \cdot \mathbf{R} \cdot \vec{x} = h(\mathbf{R} \cdot \vec{x})$$&lt;/p&gt;

&lt;p&gt;Clearly, we can stack many of these operations, and equivariance will still be preserved. However, that&amp;rsquo;s not very exciting. Let&amp;rsquo;s get a little bit more creative: we know that the norm of $\vec{x}$ does not change under rotation, so we can actually make $\alpha$ a function depending on the norm of $\vec{x}$ without breaking equivariance:&lt;/p&gt;

&lt;p&gt;$$h(\vec{x}) = \alpha(\left\lVert \vec{x} \right \rVert_2) \cdot \vec{x}$$&lt;/p&gt;

&lt;p&gt;where the function $\alpha$ can be quite complicated, e.g. a trainable neural network. This actually already gives us a valid non-linearity for equivariant neural networks.&lt;/p&gt;

&lt;p&gt;So far, $\alpha$ was a scalar. What if we tried to make $\alpha$ a rotation matrix? If we look at the second last equation above, there is a place where we rely on the commuting of $\alpha$ and $\mathbf{R}$:&lt;/p&gt;

&lt;p&gt;$$\mathbf{R} \cdot \alpha \cdot \vec{x} = \alpha \cdot \mathbf{R} \cdot \vec{x}$$&lt;/p&gt;

&lt;p&gt;Again, this has to hold &lt;em&gt;for all&lt;/em&gt; $\vec{x}$. In 2D, this would be fine for any rotation matrix as 2D rotation matrices commute. In 3D however, that is not the case. It turns out that this massively restricts the computations we can do. However, there is a way around it. If we &lt;em&gt;condition&lt;/em&gt; $\alpha$ on $\vec{x}$, there are non-trivial functions which fulfill the equivariance constraint. In the literature, $\alpha$ conditioned on $\vec{x}$, i.e. $\alpha(\vec{x})$, is often refered to as the kernel and denoted as $\mathbf{W}(\vec{x})$. And by choosing this kernel correctly, we can ensure that it commutes with rotations:&lt;/p&gt;

&lt;p&gt;$$\mathbf{R} \cdot \mathbf{W}(\vec{x}) \cdot \vec{x} = \mathbf{W}(\vec{x}) \cdot \mathbf{R} \cdot \vec{x}$$&lt;/p&gt;

&lt;p&gt;There are some simplifications we made so far, which would again make the types of functions we can learn overly restrictive. In the above, we assumed that the input to the kernel and the feature that is transformed are the same vector $\vec{x}$. This does not have to be the case, so let&amp;rsquo;s relax this:&lt;/p&gt;

&lt;p&gt;$$h(\vec{x}, \vec{f})=   \mathbf{W}(\vec{x}) \cdot \vec{f}$$&lt;/p&gt;

&lt;p&gt;Furthermore, we implicitly assumed that $\vec{f}$ is a 3D vector which is rotated by 3x3 rotation matrix. We want to extend this for two reasons: (1) The inputs and outputs of our neural network in most cases include information which is not best represented as 3D vector. (2) Even if we only dealt with 3D vectors in input and output space, it turns out that it is helpful to go to other representations in intermediate layers. The later has again to do with the equivariance constrains &amp;mdash; we can simply learn a more complex set of functions.&lt;/p&gt;

&lt;p&gt;This brings us to the concept of irreducible representations. We might have different types of information attached to the nodes (amino acids) in our graph. Often, we have so-called scalar information, e.g. which atoms the amino acid is composed of. Scalar information does not rotate &amp;mdash; or in other words, it is rotated by the identity matrix. We speak of information of different &lt;em&gt;types&lt;/em&gt;. 3D vectors are referred to as type 1; scalar information is referred to as type 0. There is also type 2, type 3, and so on. The theory of irreducible representations tells us that we can represent &lt;em&gt;any&lt;/em&gt; information as a concatenation of different types, the so-called irreducible representations. Moreover, we know how to rotate vectors of any type, namely by the Wigner-D matrices. The type-0 Wigner D matrix is the identity. The type one Wigner D matrices are the 3x3 rotation matrices we are familiar with. In general, type $\ell$ Wigner D matrices are of dimension $(2\ell + 1) \times  (2\ell + 1)$.&lt;/p&gt;

&lt;p&gt;Expressing everything in vectors of types $\ell_1, \ell_2, &amp;hellip;$ allows us to write down and solve the linear equivariance constraint for a transformation of point $\vec{x}$ with a feature vector $f$ attached to it. How to derive this is described in Appendix A and B of the SE(3)-Transformer &lt;a href=&#34;https://arxiv.org/abs/2006.10503.pdf&#34; target=&#34;_blank&#34;&gt;paper&lt;/a&gt;. To cut it short, it turns out that the spherical harmonics together with the Clebsch-Gordan coefficients form basis vectors for solving this constraint&lt;sup class=&#34;footnote-ref&#34; id=&#34;fnref:6&#34;&gt;&lt;a href=&#34;#fn:6&#34;&gt;6&lt;/a&gt;&lt;/sup&gt;. &lt;a href=&#34;https://arxiv.org/abs/1807.02547&#34; target=&#34;_blank&#34;&gt;3D Steerable CNNs&lt;/a&gt; and &lt;a href=&#34;https://arxiv.org/abs/1802.08219&#34; target=&#34;_blank&#34;&gt;Tensor Field Networks&lt;/a&gt; use these results to construct SE(3)-equivariant kernels for building convolutional networks. The &lt;a href=&#34;https://arxiv.org/abs/2006.10503.pdf&#34; target=&#34;_blank&#34;&gt;SE(3)-Transformer&lt;/a&gt; uses the same principles to build an equivariant attention mechanism.&lt;/p&gt;

&lt;h3 id=&#34;building-an-equivariant-transformer&#34;&gt;Building an Equivariant Transformer&lt;/h3&gt;

&lt;!-- There are obviously mutiple potential ways to build an SE(3)-equivariant transformer and we do not know how exactly DeepMind did it and whether they deviated from the SE(3)-Transformer. Time will show, but for now, here is how it&#39;s done in the SE(3)-Transformer: --&gt;

&lt;p&gt;First, we will deal with translations. By only considering relative positions between building blocks of the protein chain, the entire network, from beginning to end, is already translation invariant. This reduces the challenge to being SO(3)-equivariant, where SO is the special orthogonal group &amp;mdash; in other words, the group of rotations. Therefore, the theory above, which provides equivariance with respect to rotations, suffices.&lt;/p&gt;

&lt;p&gt;Next, we make a choice to make our lives simpler. Above, we showed that multiplying an equivariant feature with an invariant scalar results in a new equivariant feature. For the attention mechanism, this means we can (and will) choose equivariant values and invariant weights, preserving overall equivariance.&lt;/p&gt;

&lt;p&gt;We also established that we need to condition our transformations $\alpha$ on a vector to get non-trivial solutions to the equivariance constraint. For each edge in the graph, we choose this to be the relative position vector between two nodes.&lt;/p&gt;

&lt;p&gt;For each of keys, queries, and values, we can use the theory described above to build an equivariant transformation. E.g., to obtain a key for an edge going from node $j$ to node $i$, the feature $f_i$ is mapped to a key $k_{ij}$:
$$k_{ij} = W_k(x_{i} - x_{j})f_j$$
and analogously for queries and values.
$$q_{ij} = W_q(x_{i} - x_{j})f_i$$
$$v_{ij} = W_v(x_{i} - x_{j})f_j$$
where the weight matrices $W$ will be constructed from spherical harmonics, Clebsch-Gordan coefficients, and learnable radial basis functions.&lt;/p&gt;

&lt;p&gt;The values are equivariant, just as we wanted them to be. The keys and the queries now have to be combined to create invariant weights. We make use of the fact that scalar products of features of the same irreducible representation are invariant. Hence, if $k_{ij}$ and $q_{ij}$ have the same representation, $w_{ij} = k_{ij}\cdot q_{ij}$ will be invariant. The weights for node $i$ are then normalised via a softmax over all $j$.&lt;/p&gt;

&lt;p&gt;To make everything scale better, we introduce neighbourhoods. In the SE(3)-Transformer experiment on the QM9 dataset, for each atom in the molecule, those neighbourhoods were chosen to be all other atoms the atom shares a bond with. However, using the $k$ nearest neighbours is an equally valid approach.&lt;/p&gt;

&lt;p&gt;The theory described in the above paragraphs is summarised in figure 2 of the SE(3)-Transformer paper:
&lt;img src=&#34;https://i.imgur.com/6GNsG5N.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;

&lt;h3 id=&#34;what-about-edge-information&#34;&gt;What about Edge Information?&lt;/h3&gt;

&lt;p&gt;The vigilant reader might have noticed that we have only been talking about node information but have ignored edge information. An example of edge information on the input level could be the bond type between two atoms. How to incorporate this into the model depends critically on the &lt;em&gt;type&lt;/em&gt; of this information. If it is a scalar (that is the usual case), then the edge information can just be concatenated with the norm of the relative position and fed as input to the radial neural network in step 2 the figure above. If it was not a scalar but, e.g., a type-1 vector, then this procedure would break equivariance. In this case, the edge feature can be concatenated to the node features at the exact point where they are propagated to the edges. They can hence serve as input to the calculations of keys, queries, and/or values.&lt;/p&gt;

&lt;h2 id=&#34;conclusion&#34;&gt;&lt;strong&gt;Conclusion&lt;/strong&gt;&lt;/h2&gt;

&lt;p&gt;That&amp;rsquo;s it! That was our take on AlphaFold 2 with a special focus on equivariance. AlphaFold 2 is fascinating for all the right reasons. It is an impressive, interdisciplinary piece of intellectual work and will clearly have an impact beyond the scientific community. Beyond that, it might also do a great service to the equivariance community. By no means is it the first time that equivariant neural networks were used to tackle problems in the natural sciences, but it might already be the most prominent instance. We are curious where people will take this next. For more reading around equivariance and AlphaFold, we provided some references in the footnotes. If you want to play around with the code of the SE(3)-Transformer, click &lt;a href=&#34;https://github.com/FabianFuchsML/se3-transformer-public&#34; target=&#34;_blank&#34;&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Credit&lt;/strong&gt;: &lt;em&gt;A big thank you to Adam Kosiorek, Bradley Gram-Hansen, Coral Bays-Muchmore, Haobo Wang, and Adam Golinski for helpful discussions and feedback. Fabian is part of the A2I lab at Oxford and funded by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems. Justas is at the Baker Lab which is part of the University of Washington Institute for Protein Design and he is supported by The Open Philanthropy Project Improving Protein Design.&lt;/em&gt;&lt;/p&gt;
&lt;div class=&#34;footnotes&#34;&gt;

&lt;hr /&gt;

&lt;ol&gt;
&lt;li id=&#34;fn:3&#34;&gt;We suggest to read DeepMind&amp;rsquo;s blog posts for more details: &lt;a href=&#34;https://deepmind.com/blog/article/AlphaFold-Using-AI-for-scientific-discovery&#34; target=&#34;_blank&#34;&gt;AlphaFold 1&lt;/a&gt;, &lt;a href=&#34;https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology&#34; target=&#34;_blank&#34;&gt;AlphaFold 2&lt;/a&gt;. Slides from the CASP14 conference are publicly available &lt;a href=&#34;https://predictioncenter.org/casp14/doc/presentations/2020_12_01_TS_predictor_AlphaFold2.pdf&#34; target=&#34;_blank&#34;&gt;here&lt;/a&gt;.
 &lt;a class=&#34;footnote-return&#34; href=&#34;#fnref:3&#34;&gt;&lt;sup&gt;^&lt;/sup&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li id=&#34;fn:4&#34;&gt;SE(3) is the group of rotations and translations in 3D. SE is short for &lt;em&gt;special euclidean&lt;/em&gt;. DeepMind says &amp;lsquo;3D equivariant transformer&amp;rsquo; in their presentation. Most likely, they chose &lt;em&gt;3D&lt;/em&gt; just because the term is a lot more intuitive.
 &lt;a class=&#34;footnote-return&#34; href=&#34;#fnref:4&#34;&gt;&lt;sup&gt;^&lt;/sup&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li id=&#34;fn:5&#34;&gt;This is true because the gravitational forces are irrelevant compared to the electrostatic forces that govern the folding of a protein. The process of experimental measuring itself does actually introduce a frame of reference &amp;mdash; as any measuring procedure does.
 &lt;a class=&#34;footnote-return&#34; href=&#34;#fnref:5&#34;&gt;&lt;sup&gt;^&lt;/sup&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li id=&#34;fn:1&#34;&gt;This equation is actually stricter than it needs to be. The transformation $\sigma$ does not have to be the same on both sides of the equation. It just needs to be a representation of the same group.
 &lt;a class=&#34;footnote-return&#34; href=&#34;#fnref:1&#34;&gt;&lt;sup&gt;^&lt;/sup&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li id=&#34;fn:2&#34;&gt;Irreducible representations are the basis of multiple works in this area, particularly of &lt;a href=&#34;https://arxiv.org/abs/1807.02547&#34; target=&#34;_blank&#34;&gt;3D Steerable CNNs&lt;/a&gt; and &lt;a href=&#34;https://arxiv.org/abs/1802.08219&#34; target=&#34;_blank&#34;&gt;Tensor Field Networks&lt;/a&gt;. The &lt;a href=&#34;https://arxiv.org/abs/2006.10503&#34; target=&#34;_blank&#34;&gt;SE(3)-Transformer&lt;/a&gt; is heavily based on both of these. Another set of works uses regular representations. If you want to learn about these, you could read &lt;a href=&#34;https://fabianfuchsml.github.io/equivariance1of2/&#34; target=&#34;_blank&#34;&gt;this blog post&lt;/a&gt; about Taco Cohen&amp;rsquo;s seminal work from 2016, or &lt;a href=&#34;https://arxiv.org/abs/2002.12880&#34; target=&#34;_blank&#34;&gt;this 2020 paper&lt;/a&gt; from Marc Finzi.
 &lt;a class=&#34;footnote-return&#34; href=&#34;#fnref:2&#34;&gt;&lt;sup&gt;^&lt;/sup&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li id=&#34;fn:6&#34;&gt;Knowing about spherical harmonics and Clebsch-Gordan coefficients is not important for understanding the rest of the blog post. If you are still curious: the &lt;a href=&#34;https://en.wikipedia.org/wiki/Spherical_harmonics&#34; target=&#34;_blank&#34;&gt;spherical harmonics&lt;/a&gt; are a set of functions defined on the surface of a sphere. In fact, they form an orthonormal basis for functions on the sphere, so any function on the sphere can be expressed as a linear combination of spherical harmonics. The &lt;a href=&#34;https://en.wikipedia.org/wiki/Clebsch%E2%80%93Gordan_coefficients&#34; target=&#34;_blank&#34;&gt;Clebsch-Gordan&lt;/a&gt; coefficients serve as a change-of-basis matrix and are part of a particular identity when working with irreducible representations (see Eqn. 26 in the &lt;a href=&#34;https://arxiv.org/abs/2006.10503.pdf&#34; target=&#34;_blank&#34;&gt;SE(3)-Transformer&lt;/a&gt;).
 &lt;a class=&#34;footnote-return&#34; href=&#34;#fnref:6&#34;&gt;&lt;sup&gt;^&lt;/sup&gt;&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;
</description>
    </item>
    
    <item>
      <title>Learn it yourself - online resources</title>
      <link>https://dauparas.github.io/post/learning/</link>
      <pubDate>Sun, 26 Jul 2020 06:44:46 -0400</pubDate>
      
      <guid>https://dauparas.github.io/post/learning/</guid>
      <description>

&lt;p&gt;This is a short list of books and lectures with links for myself and anyone else who might find them useful. A book is worth a thousand arXiv papers!&lt;/p&gt;

&lt;h3 id=&#34;machine-learning-books&#34;&gt;Machine Learning Books&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;&lt;a href=&#34;https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf&#34; target=&#34;_blank&#34;&gt;Pattern Recognition and Machine Learning&lt;/a&gt; by Christopher M. Bishop,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.inference.org.uk/itprnn/book.pdf&#34; target=&#34;_blank&#34;&gt;Information Theory, Inference, and Learning Algorithms&lt;/a&gt; by David J.C. MacKay, &lt;a href=&#34;https://www.youtube.com/watch?v=BCiZc0n6COY&amp;amp;list=PLruBu5BI5n4aFpG32iMbdWoRVAA-Vcso6&#34; target=&#34;_blank&#34;&gt;videos&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://noiselab.ucsd.edu/ECE228/Murphy_Machine_Learning.pdf&#34; target=&#34;_blank&#34;&gt;Machine Learning: A Probabilistic Perspective&lt;/a&gt; by Kevin P. Murphy,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.Online&#34; target=&#34;_blank&#34;&gt;Bayesian Reasoning and Machine Learning&lt;/a&gt; by David Barber,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://www.deeplearningbook.org/&#34; target=&#34;_blank&#34;&gt;Deep Learning&lt;/a&gt; by Ian Goodfellow, Yoshua Bengio, and Aaron Courville,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://web.stanford.edu/~hastie/ElemStatLearn/printings/ESLII_print12.pdf&#34; target=&#34;_blank&#34;&gt;The Elements of Statistical Learning&lt;/a&gt; by Trevor Hastie, Robert Tibshirani, and Jerome Friedman,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://www.stat.cmu.edu/~larry/all-of-statistics/&#34; target=&#34;_blank&#34;&gt;All of Statistics A Concise Course in Statistical Inference&lt;/a&gt; by Larry Wasserman,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.incompleteideas.net/book/RLbook2018.pdf&#34; target=&#34;_blank&#34;&gt;Reinforcement learning: An introduction&lt;/a&gt; by Richard S. Sutton and Andrew G. Barto.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&#34;lectures-on-machine-learning&#34;&gt;Lectures on Machine Learning&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;&lt;a href=&#34;https://www.fast.ai/&#34; target=&#34;_blank&#34;&gt;fast.ai&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://web.stanford.edu/class/cs224n/index.html#schedule&#34; target=&#34;_blank&#34;&gt;Natural Language Processing with Deep Learning&lt;/a&gt;, CS224n at Stanford,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://vision.stanford.edu/teaching/cs231n/&#34; target=&#34;_blank&#34;&gt;Convolutional Neural Networks for Visual Recognition&lt;/a&gt; CS231n at Stanford,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://sites.google.com/view/berkeley-cs294-158-sp20/home&#34; target=&#34;_blank&#34;&gt;Deep Unsupervised Learning&lt;/a&gt;, CS294-158 at UC Berkeley,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://rail.eecs.berkeley.edu/deeprlcourse/&#34; target=&#34;_blank&#34;&gt;Deep Reinforcement Learning&lt;/a&gt; CS285 at UC Berkeley.&lt;/li&gt;
&lt;/ol&gt;

&lt;!-- ### Computer Science Lectures
1. [Structure and Interpretation of Computer Programs] (https://web.mit.edu/alexmv/6.037/sicp.pdf) by

### Biology Lectures
1. [Introduction to Biology - The Secret of Life](https://www.edx.org/course/introduction-to-biology-the-secret-of-life-3), --&gt;

&lt;h3 id=&#34;lectures-on-theoretical-physics-by-david-tong&#34;&gt;Lectures on Theoretical Physics by David Tong&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/relativity.html&#34; target=&#34;_blank&#34;&gt;Dynamics and Relativity&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/dynamics.html&#34; target=&#34;_blank&#34;&gt;Classical Dynamics&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/em.html&#34; target=&#34;_blank&#34;&gt;Electromagnetism&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/topicsinqm.html&#34; target=&#34;_blank&#34;&gt;Topics in Quantum Mechanics&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/solidstate.html&#34; target=&#34;_blank&#34;&gt;Solid State Physics&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/qhe.html&#34; target=&#34;_blank&#34;&gt;Quantum Hall Effect&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/statphys.html&#34; target=&#34;_blank&#34;&gt;Statistical Physics&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/kinetic.html&#34; target=&#34;_blank&#34;&gt;Kinetic Theory&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/sft.html&#34; target=&#34;_blank&#34;&gt;Statistical Field Theory&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/qft.html&#34; target=&#34;_blank&#34;&gt;Quantum Field Theory&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/gaugetheory.html&#34; target=&#34;_blank&#34;&gt;Gauge Theory&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/tasi.html&#34; target=&#34;_blank&#34;&gt;Solitons&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/gr.html&#34; target=&#34;_blank&#34;&gt;General Relativity&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/cosmo.html&#34; target=&#34;_blank&#34;&gt;Cosmology&lt;/a&gt;,&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;http://www.damtp.cam.ac.uk/user/tong/string.html&#34; target=&#34;_blank&#34;&gt;String Theory&lt;/a&gt;.&lt;/li&gt;
&lt;/ol&gt;
</description>
    </item>
    
    <item>
      <title>Theoretical Physics</title>
      <link>https://dauparas.github.io/links/physics/</link>
      <pubDate>Fri, 29 Mar 2019 11:26:20 -0400</pubDate>
      
      <guid>https://dauparas.github.io/links/physics/</guid>
      <description>&lt;p&gt;This is a link to theoretical physics.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Unified framework for multivariate distributions in biological sequences</title>
      <link>https://dauparas.github.io/publication/unified/</link>
      <pubDate>Sun, 03 Mar 2019 20:01:19 -0500</pubDate>
      
      <guid>https://dauparas.github.io/publication/unified/</guid>
      <description></description>
    </item>
    
    <item>
      <title>Depth and nonlinearity induce implicit exploration</title>
      <link>https://dauparas.github.io/publication/rl/</link>
      <pubDate>Tue, 29 May 2018 19:18:17 -0500</pubDate>
      
      <guid>https://dauparas.github.io/publication/rl/</guid>
      <description></description>
    </item>
    
    <item>
      <title>Self-organization of swimmers drives long-range fluid transport in bacterial colonies</title>
      <link>https://dauparas.github.io/publication/self_organisation/</link>
      <pubDate>Sat, 24 Feb 2018 20:25:06 -0500</pubDate>
      
      <guid>https://dauparas.github.io/publication/self_organisation/</guid>
      <description></description>
    </item>
    
    <item>
      <title></title>
      <link>https://dauparas.github.io/publication/corner_flows/</link>
      <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
      
      <guid>https://dauparas.github.io/publication/corner_flows/</guid>
      <description></description>
    </item>
    
    <item>
      <title></title>
      <link>https://dauparas.github.io/publication/flagellar_flows/</link>
      <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
      
      <guid>https://dauparas.github.io/publication/flagellar_flows/</guid>
      <description></description>
    </item>
    
    <item>
      <title></title>
      <link>https://dauparas.github.io/publication/helical_micropumps/</link>
      <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
      
      <guid>https://dauparas.github.io/publication/helical_micropumps/</guid>
      <description></description>
    </item>
    
    <item>
      <title></title>
      <link>https://dauparas.github.io/publication/thesis/</link>
      <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
      
      <guid>https://dauparas.github.io/publication/thesis/</guid>
      <description></description>
    </item>
    
    <item>
      <title>Theoretical Physics</title>
      <link>https://dauparas.github.io/talk/physics/</link>
      <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
      
      <guid>https://dauparas.github.io/talk/physics/</guid>
      <description>&lt;p&gt;This is a link to theoretical physics.&lt;/p&gt;
</description>
    </item>
    
  </channel>
</rss>