TLDR; The authors propose new ways to incorporate context (previous sentences) into a Recurrent Language Model (RLM). They propose 3 ways to model the context, and 2 ways to incorporate the context into the predictions for the current sentence. Context can be modeled with BoW, Sequence BoW (BoW for each sentence), and Sequence BoW with attention. Context can be incorporated using "early fusion", which gives the context as an input to the RNN, or "late fusion", which modifies the LSTM to directly incorporate the context. The authors evaluate their architecture on IMDB, BBC and Penn TreeBank corpora, and show that most approaches perform well (reducing perplexity), with Sequence BoW with attention + late fusion outperforming all others.

#### Key Points:

- Context as BoW: Compress N previous sentences into a single BoW vector
- Context as Sequential Bow: Compress each of the N previous sentences into a BoW vector and use an LSTM to "embed" them. Alternatively, use an attention mechanism.
- Early Fusion: Give the context vector as an input to the LSTM, together with the current word.
- Late Fusion: Add another gate to the LSTM that incorporates the context vector. Helps to combat vanishing gradients.
- Interestingly the Sequence BoW without attention performs very poorly. The reason here seems to be the same as for seq2seq, it's hard to compress the sentence vectors into a single fixed-length representation using an LSTM.
- LSTM models trained with 1000 units, Adadelta. Only sentences up to 50 words are considered.
- Noun phrases seem to benefit the most from the context, which makes intuitive sense.


#### Notes/Questions:

- A problem with current Language Models is that they are corpus-specific. A model trained on one corpus doesn't do well on another corpus because all sentences are treated as being independent. However, if we can correctly incorporate context we may be able to train a general-purpose LM that does well across various corpora. So I think this is important work.
- I am surprised that the authors did not try using a sentence embedding (skip-thought, paragraph-vector) to construct their context vectors. That seems like an obvious choice over using BoW.
- The argument for why the Sequence BoW without attention model performs poorly isn't convincing. In the seq2seq work the argument for attention was based on the length of the sequence. However, here the sequence is very short, so the LSTM should be able to capture all the dependencies. The performance may be poor due to the BoW representation, or due too little training data.
- Would've been nice to visualize what the attention mechanism is modeling.
- I'm not sure if I agree with the authors that relying explicit sentence boundaries is an advantage, I see it as a limiting factor.

  * They propose a two-stage GAN architecture that generates 256x256 images of (relatively) high quality.
  * The model gets text as an additional input and the images match the text.

### How
  * Most of the architecture is the same as in any GAN:
    * Generator G generates images.
    * Discriminator D discriminates betweens fake and real images.
    * G gets a noise variable `z`, so that it doesn't always do the same thing.
  * Two-staged image generation:
    * Instead of one step, as in most GANs, they use two steps, each consisting of a G and D.
    * The first generator creates 64x64 images via upsampling.
    * The first discriminator judges these images via downsampling convolutions.
    * The second generator takes the image from the first generator, downsamples it via convolutions, then applies some residual convolutions and then re-upsamples it to 256x256.
    * The second discriminator is comparable to the first one (downsampling convolutions).
    * Note that the second generator does not get an additional noise term `z`, only the first one gets it.
    * For upsampling, they use 3x3 convolutions with ReLUs, BN and nearest neighbour upsampling.
    * For downsampling, they use 4x4 convolutions with stride 2, Leaky ReLUs and BN (the first convolution doesn't seem to use BN).
  * Text embedding:
    * The generated images are supposed to match input texts.
    * These input texts are embedded to vectors.
    * These vectors are added as:
      1. An additional input to the first generator.
      2. An additional input to the second generator (concatenated after the downsampling and before the residual convolutions).
      3. An additional input to the first discriminator (concatenated after the downsampling).
      4. An additional input to the second discriminator (concatenated after the downsampling).
    * In case the text embeddings need to be matrices, the values are simply reshaped to `(N, 1, 1)` and then repeated to `(N, H, W)`.
    * The texts are converted to embeddings via a network at the start of the model.
      * Input to that vector: Unclear. (Concatenated word vectors? Seems to not be described in the text.)
      * The input is transformed to a vector via a fully connected layer (the text model is apparently not recurrent).
      * The vector is transformed via fully connected layers to a mean vector and a sigma vector.
      * These are then interpreted as normal distributions, from which the final output vector is sampled. This uses the reparameterization trick, similar to the method in VAEs.
      * Just like in VAEs, a KL-divergence term is added to the loss, which prevents each single normal distribution from deviating too far from the unit normal distribution `N(0,1)`.
      * The authors argue, that using the VAE-like formulation -- instead of directly predicting an output vector (via FC layers) -- compensated for the lack of labels (smoother manifold).
      * Note: This way of generating text embeddings seems very simple. (No recurrence, only about two layers.) It probably won't do much more than just roughly checking for the existence of specific words and word combinations (e.g. "red head").
  * Visualization of the architecture:
    * ![Architecture](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/StackGAN__architecture.jpg?raw=true "Architecture")

### Results
  * Note: No example images of the two-stage architecture for LSUN bedrooms.
  * Using only the first stage of the architecture (first G and D) reduces the Inception score significantly.
  * Adding the text to both the first and second generator improves the Inception score slightly.
  * Adding the VAE-like text embedding generation (as opposed to only FC layers) improves the Inception score slightly.
  * Generating images at higher resolution (256x256 instead of 128x128) improves the Inception score significantly
    * Note: The 256x256 architecture has more residual convolutions than the 128x128 one.
    * Note: The 128x128 and the 256x256 are both upscaled to 299x299 images before computing the Inception score. That should make the 128x128 images quite blurry and hence of low quality.
  * Example images, with text and stage 1/2 results:
    * ![Examples](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/StackGAN__examples.jpg?raw=true "Examples")
  * More examples of birds:
    * ![Examples birds](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/StackGAN__examples_birds.jpg?raw=true "Examples birds")
  * Examples of failures:
    * ![Failure Cases](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/StackGAN__failures.jpg?raw=true "Failure Cases")
    * The authors argue, that most failure cases happen when stage 1 messes up.

### **Keyword**: RNN, serialized model; non-differentiable backpropogarion; action detection in video 


**Abstract**: This paper uses an end-to-end model which is a recurrent neural network trained by REINFORCE to directly predict the temporal bounds of actions. The intuition is that people will observe moments in video and decide where to look to predict when an action is occurring. After training, Serena et al manage to achieve the state-of-art result by only observing 2% of the video frames.

**Model**: In order to take a long video and output all the instances of given action, they use two parts including an observation network and recurrent network. 
* observation network:  encode the visual representation of video frames.
     * input:  $ln$ -- the normalized location of the frame + frame $v_{ln}$
     * network: fc7 feature of finetuned VGG16 network
     * output: $on$ of 1024 dimension indicate time and frame feature 
     
* recurrent network: sequentially process the visual representations and decide where to watch next and whether to emit detection.
##### for each timestep:
     * input:  $on$ -- the representation of the frame + previous state $h_{n-1}$
     * network: $d_n = f_d(h_n; \theta_d)$. $pn = fp(h_n,\theta_p)$,$fd$ is fc. $fp$ is fc+sigmoid  
     * output: $d_n = (s_n,e_n,c_n )$as the candidate detection, where $s_n$,$e_n$ is the start and end of the detection, $c_n$ is confidence level;  $p_n$ whether $d_n$ is a valid detection. $l_{n+1}$ where to observe next. all the parameter falls in [0,1]
https://i.imgur.com/SeFianV.png


**Training**: in order to learn the supervision annotation in long videos and handle the non-differentiable components, authors use BP to train $d_n$ while use REINFORCE to train $p_n$ and $l_{n+1}$
* for $d_n$: $L(D) = \sum_n L_{cls}(d_n) + \sum_n \sum_m 1[y_{mn} = 1] L_{loc}(d_n,g_m)$
* for $p_n,l_{n+1}$: reward $J(\theta) = \sum_{a\in A} p(a) r(a)$ p(a) is the distribution of action and r(a) is the reward for the action. so training needs to maximize this.


**Summary**:
This paper uses a serialized model which first extract the feature from each frame, then use the frame feature and previous state info to generate the next observation time, detection and detection indicator. Specifically, in order to use previous information, they use RNN to store information and use REINFORCE to train $p_n$ and $l_n$, where the goal is to maximize reward for an action sequence and use Monte-Carlo sampling to numerically calculate the gradient for high dimension function.

**questions**:
1. why  $p_n$ and $l_n$ are non-differentiable components?
2. if $p_n$ and $l_n$ are non-differentiable components indeed, how do we come up with REINFORCE to compute the gradient?
3. why don't we get $p_n$ from $p_n = f_p(h_n, \theta_p)$ directly but rather use fp as the parameter in bernoulli distribution, similar question can be applied to calculation for $l_{n+1}$ in trainning time.

 
  

I enjoyed reading this paper. However I have a problem with the core statement of the problem which is described by the following sentence. 

"However, our visual experience is not at all like this. We experience the world as fully
detailed and there is currently no scientific explanation of this."

Who says we experience the world this way? I don't feel that I do at all. Let's say I'm looking at a large picture made of random pixels each of which is white or black at random. I first perceive the fact that I am looking at something with no real information. I know there are white and black pixels everywhere but I don't actually store which are white and which are black because I see no information in it and don't think it would be useful. I can focus on any small part of it and truly record what the arrangement is. If I close my eyes I can probably remember a 3x3 block accurately enough which is only 9 bits of information. If I practice for a long while, I could probably get a somewhat larger block. But I doubt I could get much higher than a 5x5 block of pixels. 

If I look at the scene around me right now, I see my familiar living room scene. It's not random like before. Again I can focus on any part of it, say a book on my bookshelf, and read the title. And there are lots of books. I can do the same for any of them but not all of them at once. I know they are have titles but I am under no illusion that I see them all at once and record the information. I know my eye and brain is not taking a 64 Megapixel picture every second like a video recorder. 

The other things in the room that show up in my peripheral vision can be summarized easily. I know there is a piano. It's large and brown and to the left side of the room. I know there are lamps and a beer glass. I can close my eyes and answer questions about the things in the room. I've compressed the useful information into a model and ignored the rest just as a JPEG or Autoencoder reduces the size of an image but keeps most of the relevant information. Perhaps my brain has approximately memorized the whole picture at low resolution. But I'm under no illusion that I'm really seeing it all at once at high resolution. 

Whether my eyes are open or closed I have this reduced model of what I'm looking at. I know from experience that there is detail everywhere without seeing it all at once. Wherever I focus my gaze I do see the detail and so I come to believe there is indeed detail everywhere even if I can't see and store all of it at once. 

The point is that there is no scientific problem here. Our brain very likely does something like what artificial neural networks do. They identify summarizing features which characterize most things at low resolution keeping at most only a small part of the field in high focus. Our brain being active can choose to query any other part of this to get higher resolution on that part. When it does it overwrites whatever it is that we were focusing on before. 

So in summary, I think the problem arises only from a confusion between "knowing that a picture has high-resolution information everywhere" and "actually recording that information". I certainly don't see any need to propose a supernatural explanation of vision. We're only beginning to understand how the brain and other artificial neural networks process information and in the latter we are often surprised how well they can do. 

#### Introduction

* Large scale natural language understanding task - predict text values given a knowledge base.
* Accompanied by a large dataset generated using Wikipedia
* [Link to the paper](http://www.aclweb.org/anthology/P/P16/P16-1145.pdf)

#### Dataset

* WikiReading dataset built using Wikidata and Wikipedia.
* Wikidata consists of statements of the form (property, value) about different items
* 80M statements, 16M items and 884 properties.
* These statements are grouped by items to get (item, property, answer) tuples where the answer is a set of values.
* Items are further replaced by their Wikipedia documents to generate 18.58M statements of the form (document, property, answer).
* Task is to predict answer given document and property.
* Properties are divided into 2 classes:
    * **Categorical properties** - properties with a small number of possible answers. Eg gender.
    * **Relational properties** - properties with unique answers. Eg date of birth.
* This classification is done on the basis of the entropy of answer distribution.
* Properties with entropy less than 0.7 are classified as categorical properties.
* Answer distribution has a small number of very high-frequency answers (head) and a large number of answers with very small frequency (tail).
* 30% of the answers do not appear in the training set and must be inferred from the document.

#### Models

##### Answer Classification

* Consider WikiReading as classification task and treat each answer as a class label.

###### Baseline
    
* Linear model over Bag of Words (BoW) features.
* Two BoW vectors computed - one for the document and other for the property. These are concatenated into a single feature vector.

###### Neural Networks Method

* Encode property and document into a joint representation which is fed into a softmax layer.

* **Average Embeddings BoW**
    
    * Average the BoW embeddings for documents and property and concatenate to get joint representation.

* **Paragraph Vectors**
    
    * As a variant of the previous method, encode document as a paragraph vector.

* **LSTM Reader**

    * LSTM reads the property and document sequence, word-by-word, and uses the final state as joint representation.

* **Attentive Reader**

    * Use attention mechanism to focus on relevant parts of the document for a given property.

* **Memory Networks**

    * Maps a property p and list of sentences x<sub>1</sub>, x<sub>2</sub>, ...x<sub>n</sub> in a joint representation by attention over the sentences in the document.

##### Answer Extraction

* For relational properties, it makes more sense to model the problem as information extraction than classification.

* **RNNLabeler**
    
    * Use an RNN to read the sequence of words and estimate if a given word is part of the answer.

* **Basic SeqToSeq (Sequence to Sequence)**

    * Similar to LSTM Reader but augmented with a second RNN to decode answer as a sequence of words.

* **Placeholder SeqToSeq**
    
    * Extends Basic SeqToSeq to handle OOV (Out of Vocabulary) words by adding placeholders to the vocabulary.
    * OOV words in the document and answer are replaced by placeholders so that input and output sentences are a mixture of words and placeholders only.

* **Basic Character SeqToSeq**

    * Property encoder RNN reads the property, character-by-character and transforms it into a fixed length vector.
    * This becomes the initial hidden state for the second layer of a 2-layer document encoder RNN.
    * Final state of this RNN is used by answer decoder RNN to generate answer as a character sequence.

* **Character SeqToSeq with pretraining**

    * Train a character-level language model on input character sequence from the training set and use the weights to initiate the first layer of encoder and decoder.

#### Experiments

* Evaluation metric is F1 score (harmonic mean of precision and accuracy).
* All models perform well on categorical properties with neural models outperforming others.
* In the case of relational properties, SeqToSeq models have a clear edge.
* SeqToSeq models also show a great deal of balance between relational and categorical properties.
* Language model pretraining enhances the performance of character SeqToSeq approach.
* Results demonstrate that end-to-end SeqToSeq models are most promising for WikiReading like tasks.