This paper studies the transferability of features learnt at different layers
of a convolutional neural network. Typically, initial layers of a CNN learn
features that resemble Gabor filter or color blobs, and are fairly general, while
the later layers are more task-specific. Main contributions:

- They create two splits of the ImageNet dataset (A/B) and explore how performance
varies for various network design choices such as
    - Base: CNN trained on A or B.
    - Selffer: first n layers are copied from a base network, and the rest of the
    network is randomly initialized and trained on the same task.
    - Transfer: first n layers are copied from a base network, and the rest of the
    network is trained on a different task.
    - Each of these 'copied' layers can either be fine-tuned or kept frozen.

- Selffer networks without fine-tuning don't perform well when the split is somewhere
in the middle of the network (n = 3-6). This is because neurons in these layers co-adapt
to each other's activations in complex ways, which get broken up when split.
    - As we approach final layers, there is lesser for the network to learn and so these
    layers can be trained independently.
    - Fine-tuning a selffer network gives it the chance to re-learn co-adaptations.

- Transfer networks transferred at lower n perform better than larger n, indicating
that features get more task-specific as we move to higher layers.
    - Fine-tuning transfer networks, however, results in better performance. They argue
    that better generalization is due to the effect of having seen the base dataset,
    even after considerable fine-tuning.

- Fine-tuning works much better than using random features.

- Features are more transferable across related tasks than unrelated tasks.
    - They study transferability by taking two random data splits, and splits of
    man-made v/s natural data.

## Strengths

- Experiments are thorough, and the results are intuitive and insightful.

## Weaknesses / Notes

- This paper only analyzes transferability across different splits of ImageNet
(as similar/dissimilar tasks). They should have reported results on transferability
from one task to another (classification/detection) or from one dataset to another
(ImageNet/MSCOCO).

- It would be interesting to study the role of dropout in preventing co-adaptations
while transferring features.