An attention mechanism and a separate encoder/decoder are two properties of almost every single neural translation model. The question asked in this paper is- how far can we go without attention and without a separate encoder and decoder? And the answer is- pretty far! The model presented preforms just as well as the attention model of Bahdanau on the four language directions that are studied in the paper.

The translation model presented in the paper is basically a simple recurrent language model. A recurrent language model receives at every timestep the current input word and has to predict the next word in the dataset. To translate with such a model, simply give it the current word from the source sentence and have it try to predict the next word from the target sentence.

Obviously, in many cases such a simple model wouldn't work. For example, if your sentence was "The white dog" and you wanted to translate to Spanish ("El perro blanco"), at the 2nd timestep, the input would be "white" and the expected output would be "perro" (dog). But how could the model predict "perro" when it hasn't seen "dog" yet?

To solve this issue, we preprocess the data before training and insert "empty" padding tokens into the target sentence. When the model outputs such a token, it means that the model would like to read more of the input sentence before emitting the next output word.

So in the example from above, we would change the target sentence to "El PAD perro blanco". Now, at timestep 2 the model emits the PAD symbol. At timestep 3, when the input is "dog", the model can emit the token "perro". These padding symbols are deleted in post-processing, before the output is returned to the user. You can see a visualization of the decoding process below:

https://i.imgur.com/znI6xoN.png

To enable us to use beam search, our model actually receives the previous outputted target token in addition to receiving the current source token at every timestep.

PyTorch code for the model is available at https://github.com/ofirpress/YouMayNotNeedAttention 

This paper presents a combination of the inception architecture
with residual networks. This is done by adding a shortcut connection
to each inception module. This can alternatively be seen as a resnet where
the 2 conv layers are replaced by a (slightly modified) inception module.
The paper (claims to) provide results against the hypothesis that adding residual
connections improves training, rather increasing the model size is what makes the difference.

Keypoint detection is an important step in various tasks such as SLAM, panorama stitching, camera calibration, and more. Efficient keypoint detectors, FAST (Features from Accelerated and Segments Test) for example, would detect keypoints where a relatively high brightness change is observed in relation to surrounding pixels. Most probably, the keypoints would be located on edges, as shown below:
https://i.imgur.com/ylC4BM3.jpg
Let's consider another image shown below. Here, while the detector is capable of detecting many keypoints, they are mostly located on trees (see subfigure (a) below). This causes redundancy and this paper focuses on solving it by selecting locally strong keypoints that are well distributed all over the image (subfigure (c)).

https://i.imgur.com/1MqZhmT.png


The algorithm requires input keypoints to be sorted in decreasing order of strength. The keypoints are then processed in that order and points that fall within the suppression range of a current keypoint are removed from the consideration. The process is repeated for the next unsuppressed keypoint in the sorted order. The process continues until no points remain. If the number of filtered points is not close enough to what we require, the suppression range is modified and the process is repeated. The suppression range is modified using binary search.

https://i.imgur.com/qryscZP.png

Binary search requires lower and upper bounds to operate. Naive initialization would be setting lower bound to 1 pixel, while upper bound - image width or height. On the other hand, this paper proposes better initialization of the suppression range that is dependent on image height $H_I$, width $W_I$, number of input keypoints $n$ as well as the number of output keypoints $m$.
* Upper bound: $\frac{H_I + W_I + 2m - \sqrt{\Delta}}{2m - 1}$ (see paper for more details)
* Lower bound: $\frac{1}{2}\sqrt{\frac{n}{m}}$

This initializing helps to decrease the number of iterations to convergence by a factor of three:

https://i.imgur.com/7NCpgbi.png

Homogenous location of keypoints is beneficial for SLAM algorithm and reduce translational and rotational errors compared to naive filtering when evaluated on KITTI:

https://i.imgur.com/4wq0kLK.png

Overall, this paper proposes a fast, efficient, and effective method to post-process noisy and redundant keypoint detections with a clear benefit to SLAM. The codes are publically available in multiple languages: C++, Python, Java, and Matlab. See https://github.com/BAILOOL/ANMS-Codes

There is no correlation between body size and cancer incidence across animal species. From a logical point of view such a correlation would be expected. If one assumes that each proliferating cell in an individual multicellular organism has an equal probability of acquiring a cancerous mutation, then organisms with significantly more cells should have a higher probability of developing a cancerous tumour. This is known as Peto's paradox. 

This paper puts Peto's paradox in the context of an evolutionary strategy to allow large multicellular organisms to live beyond reproductive age. Evolutionary theory describes the genetic instabilities (and variations) leading to the development of tumour suppression mechanisms. The evolutionary rules are clearly stated in box 1 and are reasonable considering the known heterogeneity of tumours.

I particularly like this paper because it puts some captivating numbers on the effect of tumour suppression in some animals. It takes one extreme of animal size, and a well known character in terms of tumour suppression, the blue whale, and makes back of the envelope calculations to what its cancer incidence should theoretically be. Given that the blue whale is 1000 times the size of a human the authors predict that all blue whales should have colorectal cancer by the age of 80. For information, blue whales can live to more than 100 years.

Within a species size is related to cancer risk (Hooray I knew there would be some advantage to being only 165cm) whereby a 3-4 mm increase above the average leg length results in an 80% higher risk of non smoking related cancers. All this suggests that large organisms (that also happen to live longer) have acquired mechanism to suppress cancer.

The authors describe such cancer suppression mechanisms including lower somatic mutation rates, different tissue architecture, redundancy in tumour suppressor genes and a somehow lower selective advantage of mutant cells and increased sensitivity to contact exhibition to name but a few.

A particularly tantalising theory suggested by Nagy et al is that of "hypertumours" whereby a parasitic growth from the cancerous tumour results in a lowering of the overall fitness tumour fitness. This hypothesis has yet to be tested.

My understanding is that comparing cancer suppression mechanisms across the species will lead to a better understanding of the evolutionary process involved in cancer progression and perhaps will reveal knowledge to help better develop strategies for cancer therapies in humans.

Li et al. propose an adversarial attack motivated by second-order optimization and uses input randomization as defense. Based on a Taylor expansion, the optimal adversarial perturbation should be aligned with the dominant eigenvector of the Hessian matrix of the loss. As the eigenvectors of the Hessian cannot be computed efficiently, the authors propose an approximation; this is mainly based on evaluating the gradient under Gaussian noise. The gradient is then normalized before taking a projected gradient step. As defense, the authors inject random noise on the input (clean example or adversarial example) and compute the average prediction over multiple iterations.

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).