Describes how to compare classifiers when they were evaluated on multiple datasets (e.g. CIFAR 10, MNIST and SVHN). Recommends Wilcoxon signed ranks test and Friedman test with the corresponding post-hoc tests. Introduce CD (critical difference) diagrams.

* McNemar test and 5x2cv are good when comparing two classifiers on one dataset
* Describes the Wilcoxon Signed-Ranks Test in section 3.1.3 in detail

In recent years, many generative models have been proposed to learn distributed representations automatically from data. One criticism of these models are that they produce representations that are "entangled": no single component of the representation vector has meaning on its own. This paper proposes a novel neural architecture and associated learning algorithm for learning disentangled representations. The paper demonstrates the network learning visual concepts on pairs of frames from Atari Games and rendered faces.

The proposed architecture uses a gating mechanism to select an index to hidden elements that store the "unpredictable" parts of the frame into a single component. The architecture bears some similarity to other "gated" architectures, e.g. relational autoencoders, three-way RBMs, etc. in that it models input-output pairs and encodes transformations. However, these other architectures do not use an explicit mechanism to make the network model "differences". This is novel. The paper claims that the objective function is novel: "given the previous frame $x_{t-1}$ of a video and the current frame x_t, reconstruct the current frame $x_t$. This is essentially the same objective as relational autoencoders (Memisevic) and similar to gated and conditional RBMs which have been used to model pairs of frames. Therefore I would recommend de-emphasizingthe novelty of the objective.

Significance - This paper opens up many possibilities for explicit mechanisms of "relative" encodings to produce symbolic representations. There isn't much detail in the results (it's an extended abstract!) but I think the work is exciting and I'm looking forward to reading a follow up paper.

Pros 
- Attacks a major problem of current generative models (entanglement) 
- Proposes a simple yet novel solution 
- Results show visually that the technique seems to work on two non-trivial datasets

Cons 
- Experiments are really preliminary - no quantitative results 
- Doesn't mention mechanisms like dropout which attempt to prevent co-adaptation of features

  * They describe a model to locate faces in images.
  * Their model uses information from suspected face regions *and* from the corresponding suspected body regions to classify whether a region contains a face.
  * The intuition is, that seeing the region around the face (specifically where the body should be) can help in estimating whether a suspected face is really a face (e.g. it might also be part of a painting, statue or doll).

### How
  * Their whole model is called "CMS-RCNN" (Contextual Multi-Scale Region-CNN).
  * It is based on the "Faster R-CNN" architecture.
  * It uses the VGG network.
  * Subparts of their model are: MS-RPN, CMS-CNN.
  * MS-RPN finds candidate face regions. CMS-CNN refines their bounding boxes and classifies them (face / not face).
  * **MS-RPN** (Multi-Scale Region Proposal Network)
    * "Looks" at the feature maps of the network (VGG) at multiple scales (i.e. before/after pooling layers) and suggests regions for possible faces.
    * Steps:
      * Feed an image through the VGG network.
      * Extract the feature maps of the three last convolutions that are before a pooling layer.
      * Pool these feature maps so that they have the same heights and widths.
      * Apply L2 normalization to each feature map so that they all have the same scale.
      * Apply a 1x1 convolution to merge them to one feature map.
      * Regress face bounding boxes from that feature map according to the Faster R-CNN technique.
  * **CMS-CNN** (Contextual Multi-Scale CNN):
    * "Looks" at feature maps of face candidates found by MS-RPN and classifies whether these regions contains faces.
    * It also uses the same multi-scale technique (i.e. take feature maps from convs before pooling layers).
    * It uses some area around these face regions as additional information (suspected regions of bodies).
    * Steps:
      * Receive face candidate regions from MS-RPN.
      * Do per candidate region:
        * Calculate the suspected coordinates of the body (only based on the x/y-position and size of the face region, i.e. not learned).
        * Extract the feature maps of the *face* region (at multiple scales) and apply RoI-Pooling to it (i.e. convert to a fixed height and width).
        * Extract the feature maps of the *body* region (at multiple scales) and apply RoI-Pooling to it (i.e. convert to a fixed height and width).
        * L2-normalize each feature map.
        * Concatenate the (RoI-pooled and normalized) feature maps of the face (at multiple scales) with each other (creates one tensor).
        * Concatenate the (RoI-pooled and normalized) feature maps of the body (at multiple scales) with each other (creates another tensor).
        * Apply a 1x1 convolution to the face tensor.
        * Apply a 1x1 convolution to the body tensor.
        * Apply two fully connected layers to the face tensor, creating a vector.
        * Apply two fully connected layers to the body tensor, creating a vector.
        * Concatenate both vectors.
        * Based on that vector, make a classification of whether it is really a face.
        * Based on that vector, make a regression of the face's final bounding box coordinates and dimensions.
  * Note: They use in both networks the multi-scale approach in order to be able to find small or tiny faces. Otherwise, after pooling these small faces would be hard or impossible to detect.

### Results
  * Adding context to the classification (i.e. the body regions) empirically improves the results.
  * Their model achieves the highest recall rate on FDDB compared to other models. However, it has lower recall if only very few false positives are accepted.
  * FDDB ROC curves (theirs is bold red):
    * ![FDDB results](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/CMS-RCNN__fddb.jpg?raw=true "FDDB results")
  * Example results on FDDB:
    * ![FDDB examples](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/CMS-RCNN__examples.jpg?raw=true "FDDB examples")

This paper introduces an interpretation of deep residual networks as implicit ensembles of exponentially many shallow networks. For a residual block $i$, there are $2^{i-1}$ paths from input to $i$, and the input to $i$ is a mixture of  $2^{i-1}$ different distributions. The interpretation is backed by a number of experiments such as removing or re-ordering residual blocks at test time and plotting norm of gradient v/s number of residual blocks the gradient signal passes through. Removing $k$ residual blocks (for k <= 20) from a network of depth n decreases the number of paths to $2^{n-k}$ but there are still sufficiently many valid paths to not hurt classification error, whereas sequential CNNs have a single viable path which gets corrupted. Plot of gradient at input v/s path length shows that almost all contributions to the gradient come from paths shorter than 20 residual blocks, which are the effective paths. The paper concludes by saying that network 'multiplicity', which is the number of paths, plays a key role in terms of the network's expressability.

## Strengths

- Extremely insightful set of experiments. These experiments nail down the intuitions as to why residual networks work, as well as clarify the connections with stochastic depth (sampling the network multiplicity during training i.e. ensemble by training) and highway networks (reduction in number of available paths by gating both skip connections and paths through residual blocks).

## Weaknesses / Notes

- Connections between effective paths and model compression.

TLDR; The authors finetune an FR -> EN NMT model using a RL-based dual game. 1. Pick a French sentence from a monolingual corpus and translate it to EN. 2. Use an EN language model to get a reward for the translation 3. Translate the translation back into FR using an EN -> FR system. 4. Get a reward based on the consistency between original and reconstructed sentence. Training this architecture using Policy Gradient authors can make efficient use of monolingual data and show that a system trained on only 10% of parallel data and finetuned with monolingual data achieves comparable BLUE scores as a system trained on the full set of parallel data.


### Key Points

- Making efficient use of monolingual data to improve NMT systems is a challenge
- Two Agent communication game: Agent A only knows language A and agent B only knows language B. A send message through a noisy translation channel, B receives message, checks its correctness, and sends it back through another noisy translation channel. A checks if it is consistent with the original message. Translation channels are then improves based on the feedback.
- Pieces required: LanguageModel(A), LanguageModel(B), TranslationModel(A->B), TranslationModel(B->A). Monolingual Data.
- Total reward is linear combination of: `r1 = LM(translated_message)`, `r2 = log(P(original_message | translated_message)`
- Samples are based on beam search using the average value as the gradient approximation
- EN -> FR pretrained on 100% of parallel data: 29.92 to 32.06 BLEU
- EN -> FR pretrained on 10% of parallel data: 25.73 to 28.73 BLEU
- FR -> EN pretrained on 100% of parallel data: 27.49 to 29.78 BLEU
- FR -> EN pretrained on 10% of parallel data: 22.27 to 27.50 BLEU


### Some Notes

- I think the idea is very interesting and we'll see a lot related work coming out of this. It would be even more amazing if the architecture was trained from scratch using monolingual data only. Due the the high variance of RL methods this is probably quite hard to do though.
- I think the key issue is that the rewards are  quite noisy, as is the case with MT in general. Neither the language model nor the BLEU scores gives good feedback for the "correctness" of a translation.
- I wonder why there is such a huge jump in BLEU scores for FR->EN on 10% of data, but not for EN->FR on the same amount of data.