Neural Machine Translation with Recurrent Attention Modeling
arXiv e-Print archive - 2016 via Local arXiv
First published: 2016/07/18 (4 years ago) Abstract: Knowing which words have been attended to in previous time steps while
generating a translation is a rich source of information for predicting what
words will be attended to in the future. We improve upon the attention model of
Bahdanau et al. (2014) by explicitly modeling the relationship between previous
and subsequent attention levels for each word using one recurrent network per
input word. This architecture easily captures informative features, such as
fertility and regularities in relative distortion. In experiments, we show our
parameterization of attention improves translation quality.
TLDR; The standard attention model does not take into account the "history" of attention activations, even though this should be a good predictor of what to attend to next. The authors augment a seq2seq network with a dynamic memory that, for each input, keep track of an attention matrix over time. The model is evaluated on English-German and Englih-Chinese NMT tasks and beats competing models.
- How expensive is this, and how much more difficult are these networks to train?
- Sequentiallly attending to neighboring words makes sense for some language pairs, but for others it doesn't. This method seems rather restricted because it only takes into account a window of k time steps.