TLDR; The authors train an Generative Adversarial Network where the generator is an RNN producing discrete tokens. The discriminator is used to provide a reward for each generated sequence (episode) and to train the generator network via via Policy Gradients. The discriminator network is a CNN in the experiments. The authors evaluate their model on a synthetic language modeling task and 3 real world tasks: Chinese poem generation, speech generation and music generation. Seq-GAN outperforms competing approaches (MLE, Schedule Sampling, PG-BLEU) on the synthetic task and outperforms MLE on the real world task based on a BLEU evaluation metric.

#### Key Points

- Code: https://github.com/LantaoYu/SeqGAN
- RL Problem setup: State is already generated partial sequence. Action space is the space of possible tokens to output at the current step. Each episode is a fully generated sequence of fixed length T.
- Exposure Bias in the Maximum Likelihood approach: During decoding the model generates the next token based on a series previously generated tokens that it may never have seen during training leading to compounding errors.
- A discriminator can provide a reward when no task-specific reward (e.g. BLEU score) is available or when it is expensive to obtain such a reward (e.g. human eval).
- The reward is provided by the discriminator at the end of each episode, i.e. when the full sequence is generated. To provide feedback at intermediate steps the rest of the sequence is sampled via Monte Carlo search.
- Generator and discriminator are trained alternatively and strategy is defined by hyperparameters g-steps (# of Steps to train generator), d-steps (number of steps to train discriminator with newly generated data) and k (number of epochs to train discriminator with same set of generated data).
- Synthetic task: Randomly initialized LSTM as oracle for a language modeling task. 10,000 sequences of length 20.
- Hyperparameters g-steps, d-steps and k have a huge impact on training stability and final model performance. Bad settings lead to a model that is barely better than the MLE baseline.

#### My notes:

- Great paper overall. I also really like the synethtic task idea, I think it's a neat way to compare models.
- For the real-world tasks I would've liked to see a comparison to PG-BLEU as they do in the synthetic task. The authors evaluate on BLEU score so I wonder how much difference a direct optimization of the evaluation metric makes.
- It seems like SeqGAN outperforms MLE significantly only on the poem generation task, not the other tasks. What about the other baselines on the other tasks? What is it about the poem generation that makes SeqGAN perform so well?

## Ideas

* Find CNN topology with Q-learning and $\varepsilon$-greedy exploration and experience replay


## Evaluation

The authors seem not to know DenseNets

* CIFAR-10: 6.92 % accuracy ([SOTA](https://martin-thoma.com/sota/#image-classification) is 3.46 % - not mentioned in the paper)
* SVHN: 2.06 % accuracy ([SOTA](https://martin-thoma.com/sota/#image-classification) is 1.59% - not mentioned in the paper)
* MNIST: 0.31 % ([SOTA](https://martin-thoma.com/sota/#image-classification) is 0.21 % - not mentioned in the paper)
* CIFAR-100: 27.14 % accuracy ([SOTA](https://martin-thoma.com/sota/#image-classification) is 17.18 % - not mentioned in the paper)


## Related Work

* Google: [Neural Architecture Search with Reinforcement Learning](https://arxiv.org/abs/1611.01578) ([summary](http://www.shortscience.org/paper?bibtexKey=journals/corr/1611.01578#martinthoma))

A key question in the analysis of hippocampal memory relates to how attention modulates the encoding and long-term retrieval of spatial and nonspatial representations in this region. To address this question, we recorded from single cells over a period of 5 days in the CA1 region of the dorsal hippocampus while mice acquired one of two goal-oriented tasks. These tasks required the animals to find a hidden food reward by attending to either the visuospatial environment or a particular odor presented in shifting spatial locations. Attention to the visuospatial environment increased the stability of visuospatial representations and phase locking to gamma oscillations—a form of neuronal synchronization thought to underlie the attentional mechanism necessary for processing task-relevant information. Attention to a spatially shifting olfactory cue compromised the stability of place fields and increased the stability of reward-associated odor representations, which were most consistently retrieved during periods of sniffing and digging when animals were restricted to the cup locations. Together, these results suggest that attention selectively modulates the encoding and retrieval of hippocampal representations by enhancing physiological responses to task-relevant information.

Attention modulates the encoding and retrieval of memories, but the physiological basis of this interaction has largely been unexplored. The formation of memories which depend on the hippocampus involves the conscious recall of events that occur in specific spatial contexts, a form of memory known as episodic. To investigate the physiological consequences of the interaction between attention and memory in the hippocampus, we recorded single-cell activity and local field potentials — the local rhythmic oscillatory activity of neurons — from the same cells over several days while animals learned one of two goal-oriented tasks. In the visuospatial version of the task, mice had to associate a specific spatial location with a reward, independent of an odor cue. In the nonspatial, olfactory version, mice had to associate a specific odor with the food reward, independent of spatial location. We found that, during periods of navigation, only neurons in the visuospatially trained animals displayed long-term stable representations of space, and neuronal synchronization to so-called gamma oscillations, a mechanism of signal amplification that has been proposed to underlie attentional processes. Conversely, when animals were sniffing the odors in fixed spatial locations, only neurons in the olfactory-trained group displayed a stable increase in firing rate in response to the reward-associated odor. Our data suggest that attention modulates what is encoded and retrieved by hippocampal cells and that neuronal synchronization to gamma oscillations may underlie the mechanism whereby attention leads to stable spatial memory retrieval during navigation.

Dynamic Node Creation (DNC) is about topology learning. DNC sequentially adds single nodes to the network until the desired accuracy is achieved.

DNC uses the logistic activation function and creates layered feed-forward architectures with only one hidden layer. So basically they only added one neuron at a time to the existing hidden layer.

They expected this to be sufficient, as it was shown that networks with only one hidden layer can model "any function of interest to an arbitrary selected precision". However, the number of neurons might be really huge and much larger than if one used more layers.

## Related Work

See [Meiosis Networks summary](http://www.shortscience.org/paper?bibtexKey=conf/nips/Hanson89#martinthoma) for many topology learning papers

Since all algorithms can be modeled as multiple conditional branch operations, this paper allows you to incorporate conventional algorithms into neural networks by dynamically building the neural computation graph based on outputs of these algorithms.

They obtain near SOTA on Quora Duplicate Questions and SQuAD without heavily fine tuning the architecture to each problem.

One limitation is that the algorithm itself is not affected by the learning process and so cannot be learned.

This method provides a nice way to incorporate non-differentiable code into differentiable computation graphs which can be learned via backprop like learning mechanisms.