This paper introduces NEAT (NeuroEvolution of Augmenting Topologies), a genetic algorithm for finding and training neural network topologies. NEAT trains the structure and the weights.

> The weight space is explored through the crossover of network weight vectors and through the mutation of single networks’ weights.


## Evaluation

* XOR-Problem
* Reinforcement learning: balance two poles attached to a cart by moving the cart in appropriate directions to keep the pole from falling

## Topology Encoding

* Direct Encodings
  * Bit string: A bit string encodes the connection matrix / matrices. First used in Structured Genetic Algorithm (sGA). Limitations: Crossover might not be useful; fixed size of layers.
  * Graph encoding: First used in Parallel Distributed Genetic Programming (PDGP).
* Indirect Encodings
  * Cellular Encoding (CE): genomes are programs written in a specialized graph transformation language

## Glossary

For people (like me) who are new to genetic algorithms (GAs):

* [Neuroevolution](https://en.wikipedia.org/wiki/Neuroevolution) (NE): a form of machine learning that uses evolutionary algorithms to train artificial neural networks
* [crossover](https://en.wikipedia.org/wiki/Crossover_(genetic_algorithm)): a genetic operator used to vary the programming of a chromosome or chromosomes from one generation to the next
* [Speciation](https://en.wikipedia.org/wiki/Speciation): the evolutionary process by which biological populations evolve to become distinct species
* TWEANNs: Topology and Weight Evolving Artificial Neural Networks

## Realted

* 2009, K. O. Stanley , D. B. D’Ambrosio and J. Gauci: [A Hypercube-Based Indirect Encoding for Evolving Large-Scale Neural Networks.](http://www.shortscience.org/paper?bibtexKey=journals/alife/StanleyDG09): Introduces HyperNEAT

YOLOv2 is improved YOLO;
 - can change image size for varying tradeoff between speed and accuracy;
 - uses anchor boxes to predict bounding boxes;
 - overcomes localization errors and lower recall not by bigger nor ensemble but using variety of ideas from past work (batch normalization, multi-scaling and etc) to keep the network simple and fast;
 - "With batch nor-malization we can remove dropout from the model without overfitting"
 -  gets 78.6 mAP at 40 FPS.

YOLO9000;
 - uses WordTree representation which enables multi-label classification as well as making classification dataset also applicable to detection;
 - is a model trained simultaneously both for detection on COCO and classification on ImageNet;
 - is validated for detecting not labeled object classes;
 - detects more than 9000 different object classes in real-time.

The paper ([arxiv](https://arxiv.org/abs/1610.09716)) introduces DCNNs (Doubly Convolutional Neural Networks). Those are CNNs which contain a new layer type which generalized convolutional layers.

## Ideas

CNNs seem to learn many filters which are similar to other learned filters in the same layer. The weights are only slightly shifted.

The idea of double convolution is to learn groups filters where filters within each group are translated versions of each other. To achieve this, a doubly convolutional layer allocates a set of meta filters which has filter sizes that are larger than the effective filter size. Effective filters can be then extracted from each meta filter, which corresponds to convolving the meta filters with an identity kernel. All the extracted filters are then concatenated, and convolved with the input.

> We have also confirmed that replacing a convolutional layer with a doubly convolutional layer consistently improves the performance, regardless of the depth of the layer.

## Evaluation

* CIFAR-10+: 7.24% error
* CIFAR-100+: 26.53% error
* ImageNet: 8.23% Top-5 error


## Critique

The k-translation correlation is effectively a [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity). I think the authors should have mentioned that.


## Related

TODO

This paper is about finding naturally looking images for the analysis of machine learning models in computer vision. There are 3 techniques:

* **inversion**: the aim is to reconstruct an image from its representation
* **activation maximization**: search for patterns that maximally stimulate a representation component (deep dream). This does NOT use an initial natural image.
* **caricaturization**: exaggerate the visual patterns that a representation detects in an image

The introduction is nice.


## Code

The paper comes with code: [robots.ox.ac.uk/~vgg/research/invrep](http://www.robots.ox.ac.uk/~vgg/research/invrep/index.html) ([GitHub: aravindhm/deep-goggle](https://github.com/aravindhm/deep-goggle))


## Related

* 2013, Zeiler & Fergus: [Visualizing and Understanding Convolutional Networks ](http://www.shortscience.org/paper?bibtexKey=journals/corr/ZeilerF13#martinthoma)

This paper is about the analysis of CNNs. It seems to be extremely similar to what Zeiler & Fergus did. I can't see the contribution. Only cited 7 times, although it is from December 2014 -> I suggest to read the Zeiler & Fergus paper instead.

## Related

* 2013, Zeiler & Fergus: [Visualizing and Understanding Convolutional Networks ](http://www.shortscience.org/paper?bibtexKey=journals/corr/ZeilerF13#martinthoma)


## Errors

* " Section ?? provides"