A *Batch Normalization* applied immediately after fully connected layers and adjusts the values of the feedforward output so that they are centered to a zero mean and have unit variance.

It has been used by famous Convolutional Neural Networks such as GoogLeNet \cite{journals/corr/SzegedyLJSRAEVR14} and ResNet \cite{journals/corr/HeZRS15}

This is Adagrad. Adagrad is an adaptive learning rate method. Some sample code from  [[Stanford CS231n]](https://cs231n.github.io/neural-networks-3/#ada) is:

```python
# Assume the gradient dx and parameter vector x
cache += dx**2
x += - learning_rate * dx / (np.sqrt(cache) + eps)
```

Adam is like RMSProp with momentum. The (simplified) update [[Stanford CS231n]](https://cs231n.github.io/neural-networks-3/#ada) looks as follows:

```
m = beta1*m + (1-beta1)*dx
v = beta2*v + (1-beta2)*(dx**2)
x += - learning_rate * m / (np.sqrt(v) + eps)
```

This paper proposes a method to send messages between cell phones over Bluetooth by using the device name field. This allows devices to communicate directly with each other without pairing. 

The proposed Randomout algorithm randomly restarts filter weights in a CNN when it has a low gradient.