In January of this year (2020), DeepMind released a model called AlphaFold, which uses convolutional networks atop sequence-based and evolutionary features to predict protein folding structure. In particular, their model was designed to predict a distribution for how far away each pair of amino acids will be from one another in the final folded structure. Given such a trained model, you can score a candidate structure according to how likely it is under the model, and - if your process for generating candidates is differentiable, as it is in this case - you can directly optimize the structure to increase its probability.  

https://i.imgur.com/9ZBhqRo.png

The distance-prediction model takes as input two main categories of feature: 

1. Per-residue features characterizing which amino acid that residue is based on different techniques that produce one-hot amino acid type, or a distribution over amino acids. 
2. Residue pair features based on parameters of Multiple Sequence Alignment (MSA) models. I don't deeply understand the details of how the specific models here work, but at a high level: MSA features are based on the evolutionary intuition that residues that make contact within  a protein will likely evolve in a correlation with one another, and that you can simulate these evolutionary timestep correlations by comparing highly similar proteins (which were likely close in evolutionary time). 

https://i.imgur.com/h16lPwU.png

These features are stacked in a LxL grid, with the per-residue-pair features differing at each point in the grid, and the per-residue features staying constant for a full row or column (since they correspond to a given i for all j). One relevant note here is that proteins can be hundreds or thousands of residues long, and, so you can't actually construct a full LxL matrix, either on the input or output end. Instead, the notional full LxL grid is subdivided into a courser grid of 64-residue square regions, and a single one of these 64x64 regions (which could be either adjacent or far away in the protein) is passed into the model at a time. 

Given these 64x64x<features> input, the model performs several layers of dilated convolutions - which allow features at a given point in the grid to be informed by information farther away - still in a 2D arrangement. The model then outputs a 64x64 grid (one element for each [i, j] amino acid pair), where each element in the grid is a 64-deep discretized probability distribution over the distance between those two residues. When I say "discretized probability distribution," what I actually mean is "histogram". This discretization of an output distribution, where you predict how much probability mass will be in each possible distance bin, allows for flexible and finer-grained predicted distributions than, for example, you could get with a continuous Gaussian model centered around a single point. Amusingly, because the model predicts distance histograms for each residue pair, the authors term the output a "distogram". During training, the next-to-last layer of the model is also used to predict per-residue auxiliary features: the accessible surface area of the residue in the folded structure, and the secondary structure type (helix, strand, etc) that the residue will be a part of. However, these are just used to provide more signal during training, and aren't used for either protein structure optimization or calculation of test scores. 

To actually generated predicted fold structures, the authors construct a generative model of fold structure where each amino acid is assigned two torsion angles that govern its connection to its neighbor. By setting these torsion angles to different values, you can twist and reshape the protein as a whole. Given this generative model, things proceed as you might suspect: you generate a candidate, calculate the resulting inter-residue distances, calculate a likelihood of those distances under the model you've learned, and send back a gradient to change your torsion angles to make that likelihood higher. 

Empirically, the Deepmind authors evaluated on a competition dataset, and specifically compared themselves against other approaches that (like theirs) didn't make predictions for a new protein by comparing against similar templates (Template Modeling, or TM) but instead modeled from raw features (Free Modeling, or FM). AlphaFold was able to achieve high accuracy on 24 out of the 43 test domains (where a domain is a cluster of highly related proteins) compared to the next best method, which only got 14 out of the 43. Definitely still not perfect, since almost half of the test proteins were out of its reach, but fairly compelling evidence that there's value to DeepMind's approach.

They extend skip-grams for word embeddings to use character n-grams. Each word is represented as a bag of character n-grams, 3-6 characters long, plus the word itself. Each of these has their own embedding which gets optimised to predict the surrounding context words using skip-gram optimisation. They evaluate on word similarity and analogy tasks, in different languages, and show improvement on most benchmarks.

Learning to translate using two monolingual image captioning datasets and pivoting through images. The model encodes an image and generates a caption in language A, this is then encoded into the same space as language B and the representation is optimised to be similar to the correct image. The model is trained end-to-end using Gumbel-softmax.

https://i.imgur.com/lnIsFNb.png

Yuan et al. present a comprehensive survey of attacks, defenses and studies regarding the robustness and security of deep neural networks. Published on ArXiv in December 2017, it includes most recent attacks and defenses. For examples, Table 1 lists all known attacks – Yuan et al. categorize the attacks according to the level of knowledge needed, targeted or non-targeted, the optimization needed (e.g. iterative) as well as the perturbation measure employed. As a result, Table 1 gives a solid overview of state-of-the-art attacks. Similarly, Table 2 gives an overview of applications reported so far. Only for defenses, a nice overview table is missing. Still, the authors discuss (as of my knowledge) all relevant defense strategies and comment on their performance reported in the literature.

https://i.imgur.com/3KpoYWr.png
Table 1: An overview of state-of-the-art attacks on deep neural networks.

https://i.imgur.com/4eq6Tzm.png
Table 2: An overview of application sof some of the attacks in Table 1.

A joint model for training bilingual embeddings without supervision. As background information they point out that embeddings in different languages tend to be linear transforms of each other. Then they create a new network and objective that builds on CBOW training, but has two terms in the loss function: The traditional loss for the CBOW language model, and an additional loss over the mean and variance of each dimension of the embedding space. The hypothesis, I guess, is that constraining each dimension of each languages embeddings to be similar (mean and variance) will put them in the same space from the start (i.e., no transformation needed). During training, they then sample a word randomly from the source (or target) language, feed forward, and compute the distribution-based gradient using the target (or source) means as the gold, and add it to the standard gradient.
They evaluate on English-French and English-Chinese using aligned corpora, and get better, though not good, performance.

There are two issues that I can think of that might affect the (lack of) accuracy. First, the mean/variance similarity constraints might be not constrained enough. Second, the fact that the other languages in-progress embedding statistics are used as gold may cause it to get stuck in local optima? FWIW,  Zhang et al '17 (ACL) also question the distributional assumptions (that each dimension is Gaussian-distributed).