This paper introduces a neural networks module that can learn input-dependent
spatial transformations and can be inserted into any neural network. It supports
transformations like scaling, cropping, rotations, and non-rigid deformations.
- The spatial transformer network consists of the following:
- Localization network that regresses to the transformation parameters
given the input.
- Grid generator that uses the transformation parameters to produce a
grid to sample from the input.
- Sampler that produces the output feature map sampled from the input
at the grid points.
- Differentiable sampling mechanism
- The sampling is written in a way such that sub-gradients can be defined
with respect to grid coordinates.
- This enables gradients to be propagated through the grid generator and
localization network, and for the network to jointly learn the spatial
transformer along with rest of the network.
- A network can have multiple STNs
- at different points in the network, to model incremental transformations
at different levels of abstraction.
- in parallel, to learn to focus on different regions of interest. For example,
on the bird classification task, they show that one STN learns to be a head detector,
while the other focuses on the central part of the body.
- Their attention (and by extension transformation) mechanism is differentiable
as opposed to earlier works on non-differentiable attention mechanisms that used
reinforcement learning (REINFORCE). It also supports a richer variety of
transformations as opposed to earlier works on learning transformations, like DRAW.
- State-of-the-art classification performance on distorted MNIST, SVHN, CUB-200-2011.
## Weaknesses / Notes
This is a really nice way to generalize spatial transformations in a differentiable
manner so the model can be trained end-to-end. Classification performance, and more
importantly, qualitative results of the kind of transformations learnt on larger datasets
(like ImageNet) should be evaluated.