Model combination\ensembling:
Average ensembling is practical - but naive.
Combine considering each network's strengths, much better!
Moreover, let's make the networks diverse so they will have different strengths.

Wenjuan Han & Hwee Tou Ng (no twitters?)
#enough2skim #NLProc

The basic idea is quite simple:
Given some models, why would we want the average? We want to rely on each one(or group) when it is more likely to be the correct one.
This was actually introduced in our previous work (as admitted by the authors) in
aclanthology.org/W19-4414.pdf
The paper's addition:
1. Given a set of black-box models we may train at least one of them to be different from the rest with RL.
2. we can use more sophisticated NNs to combine the outputs
3. we can ignore domain knowledge for the combination (I am not sure this is a bonus) 
Results are very strong. Especially nice is that they show that the diversity training indeed helps

My criticism:
The comparisons are always to SoTA, this is meaningless. The authors propose different parts (the diversity, the combination and the combined models).
It is unclear whether ensembling after the diversity would be preferable over their's or not. 
Similarly, they compare to Kantor et al., but Kantor provided a combination method, why not compare on the same models, or combine with Kantor's method the models after the diversity training? 
To conclude, I really like the direction, and ensembling is a very practical tool that for some reason was not improved in a long time. 

 _Objective:_ Reduce learning time for [DQN](https://deepmind.com/research/dqn/)-type architectures.

They introduce a new network element, called DND (Differentiable Neural Dictionary) which is basically a dictionary that uses any key (especially embeddings) and computes the value by using kernel between keys. Plus it's differentiable.

## Architecture:

They use basically a network in two steps:

1.  A classical CNN network that computes and embedding for every image.
2.  A DND for all possible actions (controller input) that stores the embedding as key and estimated reward as value.

Also they use a buffer to store all tuples (previous image, action, reward, next image) and for training basic technique is used.

[![screen shot 2017-04-12 at 11 23 32 am](https://cloud.githubusercontent.com/assets/17261080/24951103/92930022-1f73-11e7-97d2-628e2f4b5a33.png)](https://cloud.githubusercontent.com/assets/17261080/24951103/92930022-1f73-11e7-97d2-628e2f4b5a33.png)

## Results:

Clearly improves learning speed but in the end other techniques catchup and it gets outperformed.

https://i.imgur.com/lM3EjK9.png
Problem
=============
Label map (semantic segmentation) to realistic image using GANs.

Contributions
=========
1. Coarse-to-fine generator
2. Multi-scale discriminator
3. Robust adversarial learning objective function


Coarse-to-fine Generator
=================
https://i.imgur.com/osEyGOj.png

G1 - Global generator

G2 - Local enhancer

Global Generator:
1. convolutional front-end
2. set of residual blocks
3. transposed convolutional back-end

A semantic label map is passed through the 3 components sequentially 


Local Enhancer:
1. convolutional front-end
2. set of residual blocks
3. transposed convolutional back-end


Training scheme:
1. Train standalone global generator 
2. Freeze global generator weights, train local enhancer
3. Fine-tune all weights together

Multi scale Discriminator
===================
https://i.imgur.com/hNP1cni.png

To allow for global context but work at higher resolution as well, several discriminators are applied at different image scales.


Robust adversarial learning objective function
===============================
https://i.imgur.com/j7CIbV3.png
* Compare original and generated images in feature space at different scales. 
* This is done to ensure more abstract resemblance, not just pixel-space resemblance.
* For feature extraction the discriminator is used.

Khoury and Hadfield-Menell provide two important theoretical insights regarding adversarial robustness: it is impossible to be robust in terms of all norms, and adversarial training is sample inefficient. Specifically, they study robustness in relation to the problem’s codimension, i.e., the difference between the dimensionality of the embedding space (e.g., image space) and the dimensionality of the manifold (where the data is assumed to actually live on). Then, adversarial training is shown to be sample inefficient in high codimensions.

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).

Goldblum et al. show that distilling robustness is possible, however, depends on the teacher model and the considered dataset. Specifically, while classical knowledge distillation does not convey robustness against adversarial examples, distillation with a robust teacher model might increase robustness of the student model – even if trained on clean examples only. However, this seems to depend on both the dataset as well as the teacher model, as pointed out in experiments on Cifar100. Unfortunately, from the paper, it does not become clear in which cases robustness distillation does not work. To overcome this limitation, the authors propose to combine adversarial training and distillation and show that this recovers robustness; the student model’s robustness might even exceed the teacher model’s robustness. This, however, might be due to the additional adversarial examples used during distillation.

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).