(See also a more thorough summary in [a LaTeX PDF][1].)

This paper has some nice clear theory which bridges maximum likelihood (supervised) learning and standard reinforcement learning. It focuses on *structured prediction* tasks, where we want to learn to predict $p_\theta(y \mid x)$ where $y$ is some object with complex internal structure.

We can agree on some deficiencies of maximum likelihood learning:

- ML training fails to assign **partial credit**. Models are trained to maximize the likelihood of the ground-truth outputs in the dataset, and all other outputs are equally wrong. This is an increasingly important problem as the space of possible solutions grows.
- ML training is potentially disconnected from **downstream task reward**. In machine translation, we usually want to optimize relatively complex metrics like BLEU or TER. Since these metrics are non-differentiable, we have to settle for optimizing proxy losses that we hope are related to the metric of interest.

Reinforcement learning offers an attractive alternative in theory. RL algorithms are designed to optimize non-differentiable (even stochastic) reward functions, which sounds like just what we want. But RL algorithms have their own problems with this sort of structured output space:

- Standard RL algorithms rely on samples from the model we are learning, $p_\theta(y \mid x)$. This becomes intractable when our output space is very complex (e.g. 80-token sequences where each word is drawn from a vocabulary of 80,000 words).
- The reward spaces for problems of interest are extremely sparse. Our metrics will assign 0 reward to most of the 80^80K possible outputs in the translation problem in the paper.
- Vanilla RL doesn't take into account the ground-truth outputs available to us in structured prediction.

This paper designs a solution which combines supervised learning with a reinforcement learning-inspired smoothing method. Concretely, the authors design an **exponentiated payoff distribution** $q(y \mid y^*; \tau)$ which assigns high mass to high-reward outputs $y$ and low mass elsewhere. This distribution is used to effectively smooth the loss function established by the ground-truth outputs in the supervised data. We end up optimizing the following objective:

$$\mathcal L_\text{RML} = - \mathbb E_{x, y^* \sim \mathcal D}\left[ \sum_y q(y \mid y^*; \tau) \log p_\theta(y \mid x) \right]$$

This optimization depends on samples from our dataset $\mathcal D$ and, more importantly, the stationary payoff distribution $q$. This contrasts strongly with standard RL training, where the objective depends on samples from the non-stationary model distribution $p_\theta$. To make that clear, we can rewrite the above with another expectation:

$$\mathcal L_\text{RML} = - \mathbb E_{x, y^* \sim \mathcal D, y \sim q(y \mid y^*; \tau)}\left[ \log p_\theta(y \mid x) \right]$$

### Model details

If you're interested in the low-level details, I wrote up the gist of the math in [this PDF][1].

### Analysis

#### Relationship to label smoothing

This training approach is mathematically equivalent to label smoothing, applied here to structured output problems. In next-word prediction language modeling, a popular trick involves smoothing the target distributions by combining the ground-truth output with some simple base model, e.g. a unigram word frequency distribution. (This just means we take a weighted sum of the one-hot vector from our supervised data and a normalized frequency vector calculated on some corpus.) Mathematically, the cross entropy with label smoothing is

$$\mathcal L_\text{ML-smooth} = - \mathbb E_{x, y^* \sim \mathcal D} \left[ \sum_y p_\text{smooth}(y; y^*) \log p_\theta(y \mid x) \right]$$

(The equation above leaves out a constant entropy term.)

The gradient of this objective looks exactly the same as the reward-augmented ML gradient from the paper:

$$\nabla_\theta \mathcal L_\text{ML-smooth} = \mathbb E_{x, y^* \sim \mathcal D, y \sim p_\text{smooth}} \left[ \log p_\theta(y \mid x) \right]$$

So reward-augmented likelihood is equivalent to label smoothing, where our smoothing distribution is log-proportional to our downstream reward function.

#### Relationship to distillation

Optimizing the reward-augmented maximum likelihood is equivalent to minimizing the KL divergence $$D_\text{KL}(q(y \mid y^*; \tau) \mid\mid p_\theta(y \mid x))$$

This divergence reaches zero iff $q = p$. We can say, then, that the effect of optimizing on $\mathcal L_\text{RML}$ is to **distill** the reward function (which parameterizes $q$) into the model parameters $\theta$ (which parameterize $p_\theta$).

It's exciting to think about other sorts of more complex models that we might be able to distill in this framework. The unfortunate (?) restriction is that the "source" model of the distillation ($q$ in this paper) must admit to efficient sampling.

#### Relationship to adversarial training

We can also view reward-augmented maximum likelihood training as a data augmentation technique: it synthesizes new "partially correct" examples using the reward function as a guide. We then train on all of the original and synthesized data, again weighting the gradients based on the reward function.

Adversarial training is a similar data augmentation technique which generates examples that force the model to be robust to changes in its input space (robust to changes of $x$). Both adversarial training and the RML objective encourage the model to be robust "near" the ground-truth supervised data. A high-level comparison:

- Adversarial training can be seen as data augmentation in the input space; RML training performs data augmentation in the output space.
- Adversarial training is a **model-based data augmentation**: the samples are generated from a process that depends on the current parameters during training. RML training performs **data-based augmentation**, which could in theory be done independent of the actual training process.

---

Thanks to Andrej Karpathy, Alec Radford, and Tim Salimans for interesting discussion which contributed to this summary.

[1]: https://drive.google.com/file/d/0B3Rdm_P3VbRDVUQ4SVBRYW82dU0/view

This paper is based on an intriguing idea of combining state-space models (SSMs) and recurrent neural networks (RNNs). Ideally, it is very much needed: For the sequences which have distinct structure and high variability, probabilistic modelling is a big problem. The handcrafted and parameterised feature representations are widely used to ease the problem and it is customary to develop the probabilistic model on top of these extracted features from the signal (such as using a short-time Fourier transform and _then_ probabilistic modelling over these features).

But machines should be able to handle learning the representation part as well. So the story of the paper. Here, it is termed neural network with stochastic layers but one can safely say that the model is a state-space model with a deterministic neural network layer which is tied to both hidden variables and observations. Still, due to the complicated nonlinearities, it is not easy to understand what's going on.

One thing I found missing from the paper is the exact form of nonlinearity used for the NN layer as I am looking at it from the perspective of probabilistic modelling. But this is probably because of space reasons.

There is no correlation between body size and cancer incidence across animal species. From a logical point of view such a correlation would be expected. If one assumes that each proliferating cell in an individual multicellular organism has an equal probability of acquiring a cancerous mutation, then organisms with significantly more cells should have a higher probability of developing a cancerous tumour. This is known as Peto's paradox. 

This paper puts Peto's paradox in the context of an evolutionary strategy to allow large multicellular organisms to live beyond reproductive age. Evolutionary theory describes the genetic instabilities (and variations) leading to the development of tumour suppression mechanisms. The evolutionary rules are clearly stated in box 1 and are reasonable considering the known heterogeneity of tumours.

I particularly like this paper because it puts some captivating numbers on the effect of tumour suppression in some animals. It takes one extreme of animal size, and a well known character in terms of tumour suppression, the blue whale, and makes back of the envelope calculations to what its cancer incidence should theoretically be. Given that the blue whale is 1000 times the size of a human the authors predict that all blue whales should have colorectal cancer by the age of 80. For information, blue whales can live to more than 100 years.

Within a species size is related to cancer risk (Hooray I knew there would be some advantage to being only 165cm) whereby a 3-4 mm increase above the average leg length results in an 80% higher risk of non smoking related cancers. All this suggests that large organisms (that also happen to live longer) have acquired mechanism to suppress cancer.

The authors describe such cancer suppression mechanisms including lower somatic mutation rates, different tissue architecture, redundancy in tumour suppressor genes and a somehow lower selective advantage of mutant cells and increased sensitivity to contact exhibition to name but a few.

A particularly tantalising theory suggested by Nagy et al is that of "hypertumours" whereby a parasitic growth from the cancerous tumour results in a lowering of the overall fitness tumour fitness. This hypothesis has yet to be tested.

My understanding is that comparing cancer suppression mechanisms across the species will lead to a better understanding of the evolutionary process involved in cancer progression and perhaps will reveal knowledge to help better develop strategies for cancer therapies in humans.

This paper introduces a bunch of tricks which make sense for visual tracking. These tricks are as followed:
1. At test time, a crop with center at the previous frame's bounding box's center with size larger than the bounding box is given along with the search area in the current frame.
2. Training offline on a large set of videos (where object bounding boxes are given for a subset of frames) and images with object bounding boxes.
3. Network takes two images: i) a crop of the image/frame around the bounding box and  ii) the image centered at the center of the bounding box. Given the later, network regresses the bounding box in i).
4. Above crops are sampled such that the ground truth bounding box center in i) is not very far from the center in ii), hence network prefers smooth motion.

My take: This is very nice way to use still images to train image correlation task and hence can be used for tracking. Speed on gpu is very impressive but still not comparable on CPUs.

When high-mass (>= 8 Msun) stars end their lives in blinding explosions known as core-collapse supernovae, they can rip through the fabric of space-time and create black holes with similar masses, known as stellar-mass black holes. These vermin black holes dwarf in comparison to their big brothers, supermassive black holes that typically have masses of 10⁶-10⁹ Msun. However, as vermin usually do, they massively outnumber supermassive black holes. It is estimated that 10⁸-10⁹ stellar mass black holes are crawling around our own Milky Way, but we’ve only caught sight of a few dozens of them.

As black holes don’t emit light, we can only infer their presence from their effects on nearby objects. All stellar mass black holes detected so far reside in binary systems, where they actively accrete from their companions. Radiation is emitted as matter from the companion falls onto the accretion disk of the black hole. Isolated black holes don’t have any companions, so they can only accrete from the surrounding diffuse interstellar medium, producing a very weak signal. That is why isolated black holes, which make up the bulk of the black hole population, have long escaped our discovery. Perhaps, until now.

The authors of today’s paper turned the intense gravity of black holes against themselves. While isolated black holes do not produce detectable emission, their gravity can bend and focus light from background objects. This bending and focusing of light through gravity is known as gravitational lensing. Astronomers categorize gravitational lensing based on the source and degree of lensing: strong lensing (lensing by a galaxy or a galaxy cluster producing giant arcs or multiple images), weak lensing (lensing by a galaxy or a galaxy cluster where signals are weaker and detected statistically), and microlensing (lensing by a star or planet). During microlensing, as the lens approaches the star, the star will brighten momentarily as more and more light is being focused, up until maximum magnification at closest approach, after which the star gradually fades as the lens leaves. This effect is known as photometric microlensing (see this astrobite). Check out this microlensing simulation, courtesy of Professor Scott Gaudi at The Ohio State University: the star (orange) is located at the origin, the lens (open red circle) is moving to the right, the gray regions trace out the lensed images (blue) as the lens passes by the star, while the green circle is the Einstein radius. The Einsten radius is the radius of the annular image when the observer, the lens, and the star are perfectly aligned.

Something more subtle can also happen during microlensing, and that is the shifting of the center of light (on a telescope’s detector) relative to the true position of the source — astrometric microlensing. While photometric microlensing has been widely used to search for exoplanets and MACHOs (massive astrophysical compact halo objects), for instance by OGLE (Optical Gravitational Lensing Experiment), astrometric microlensing has not been put to good use as it requires extremely precise measurements. Typical astrometric shifts caused by stellar mass black holes are sub-milliarcsecond (sub-mas), whereas the best astrometric precision we can achieve from the ground is typically ~1 mas or more. Figure 1 shows the signal evolution of photometric and astrometric microlensing and the astrometric shifts caused by different masses.

https://astrobites.org/wp-content/uploads/2016/08/fig1-2.png

Fig. 1 – Left panel shows an example of photometric magnification (dashed line) and astrometric shift (solid line) as function of time since the closest approach between the lens and the star. Note that the peak of the astrometric shift occurs after the peak of the photometric magnification. Right panel shows the astrometric shift as a function of the projected separation between the lens and the star, in units of the Einstein radius, for different lens masses. [Figure 1 in paper]

In this paper, the authors used adaptive optics on the Keck telescope to detect astrometric microlensing signals from stellar mass black holes. Over a period of 1-2 years, they monitored three microlensing events detected by the OGLE survey. As astrometric shift reaches a maximum after the peak of photometric microlensing (see Figure 1), astrometric follow-up was started post-peak for each event. The authors fit an astrometric lensing model to their data, not all of which were taken under good observing conditions. Figure 2 shows the results of the fit: all three targets are consistent with linear motion within the uncertainties of their measurements, i.e. no astrometric microlensing. Nonetheless, as photometric microlensing is still present, the authors used their astrometric model combined with a photometric lensing model to back out various lensing parameters, the most important one being the lens masses. They found one lens to have comparable mass as a stellar-mass black hole, although verification would require future observations.

https://astrobites.org/wp-content/uploads/2016/08/fig2-2.png

Despite not detecting astrometric microlensing signals, the authors demonstrated that they achieved the precision needed in a few epochs; had the weather goddess been on their side during some critical observing periods, some signals could have been seen. This study is also the first to combine both photometric and astrometric measurements to constrain lensing event parameters, ~20 years after this technique was first conceived. For now, we’ll give stellar-mass black holes a break, but it won’t be long until we catch up.

by Suk Sien Tie