Neural Architecture Search with Reinforcement Learning on ShortScience.org

arxiv.org
arxiv-vanity.com
scholar.google.com

Neural Architecture Search with Reinforcement Learning
Barret Zoph and Quoc V. Le
arXiv e-Print archive - 2016 via Local arXiv
Keywords: cs.LG, cs.AI, cs.NE
more

Summaries/Notes 3

[link] Summary by abhigenie92 7 years ago

### Main Idea:
It basically tunes the hyper-parameters of the neural network architecture using reinforcement learning. The reward signal is taken as evaluation on the validation set. The method is policy gradient as the cost function is non-differentiable. 

### Method:
 #### i.   Actions:
1. There is controller RNN which predicts some hyper-parameters of the layer conditioned on the previous predictions. This prediction is just a one-hot vector based on the previous hyper-parameter chosen. At the start for the first prediction - this vector is just all zeros.
2. Once the network is generated completely, it is trained for a fixed number of epochs, the reward signal is calculated based on the evaluation on the validation set.

#### ii.  Training:
1. Nothing fancy in the reinforcement learning approach simple policy gradients.
2. Baseline is added to reduce the variance.
3.  It takes 2-3 weeks to train it over 800 GPUs!!

#### iii. Results:
1. Use it to generate CNNs and LSTM cells. Close to state-of-art results with generated architectures.

### Possible new directions:
1. Use better techniques RL techniques like TRPO, PPO etc.
2. Right now, they generate fixed length architecture. Their reason is for variable length architectures, it is difficult to determine how much time each architecture is trained. Smaller networks are easier to train. Thus, somehow determine training time as function of the learning capacity of the network.


Code:
They haven't released the code yet. I tried to simulate it in torch.(https://github.com/abhigenie92/nn_search)

Your comment:

[link] Summary by Martin Thoma 7 years ago

Find a topology by reinforcement learning. They use REINFORCE from [Williams 1992](http://www.shortscience.org/paper?bibtexKey=Williams:92).


## Ideas

* Structure and connectivity of a Neural Network can be represented by a variable-length string.
* The RNN controller in Neural Architecture Search is auto-regressive, which means it predicts hyperparameters one a time, conditioned on previous predictions
* policy gradient method to maximize the expected accuracy of the sampled architectures
* In our experiments, the process of generating an architecture stops if the number of layers exceeds
a certain value.

## Evaluation

* Computer Vision - **CIFAR-10**: 3.65% error (State of the art are Dense-Nets with 3.46% error)
* Language - **Penn Treebank**: a test set perplexity of 62.4 (3.6 perplexity better than the previous state-of-the-art)

They had a Control Experiment "Comparison against Random Search" in which they showed that they are much better than a random exploration of the data. However, the paper lacks details how exactly the random search was implemented.

## Related Work

* [Designing Neural Network Architectures using Reinforcement Learning](https://arxiv.org/pdf/1611.02167.pdf) ([summary](http://www.shortscience.org/paper?bibtexKey=journals/corr/1611.02167))

Your comment:

[link] Summary by Léo Paillier 6 years ago

_Objective:_ Design a network that will itself find the best architecture for a given task.

_Dataset:_ [CIFAR10](https://www.cs.toronto.edu/%7Ekriz/cifar.html) and [PTB](https://catalog.ldc.upenn.edu/ldc99t42).


## Inner-workings:

The meta-network (a RNN) generates a string specifying the child network parameters. Such a child network is then trained for 35-50 epochs and its accuracy is used as the reward to train the meta-network with Reinforcement Learning.  
The RNN first generates networks with few layers (6) then this number is increased as training progresses.

## Architecture:

They develop one architecture for CNN where they predict each layers characteristic plus it's possible skip-connection:  
[![screen shot 2017-05-24 at 8 13 01 am](https://cloud.githubusercontent.com/assets/17261080/26389176/d807de42-4058-11e7-942a-8a129558e126.png)](https://cloud.githubusercontent.com/assets/17261080/26389176/d807de42-4058-11e7-942a-8a129558e126.png)

And one specific for LTSM-style:  
[![screen shot 2017-05-24 at 8 13 26 am](https://cloud.githubusercontent.com/assets/17261080/26389190/e2bfd506-4058-11e7-9168-62abd040156e.png)](https://cloud.githubusercontent.com/assets/17261080/26389190/e2bfd506-4058-11e7-9168-62abd040156e.png)

## Distributed setting:

Bellow is the distributed setting that they use with parameter servers connected to replicas (GPUs) that trained child networks.  
[![screen shot 2017-05-24 at 8 09 05 am](https://cloud.githubusercontent.com/assets/17261080/26389084/5e354456-4058-11e7-83a9-089cb2c115b7.png)](https://cloud.githubusercontent.com/assets/17261080/26389084/5e354456-4058-11e7-83a9-089cb2c115b7.png)

## Results:

Overall they trained 12800 networks on 800 GPUs but they achieve state of the art results which not human intervention except the vocabulary selection (activation type, type of cells, etc). Next step, transfer learning from one task to another for the meta-network?

Your comment:

Write your summary here (You can use $\LaTeX$ and markdown syntax):

Anon Private