Dynamic Evaluation of Neural Sequence Models
Ben Krause
and
Emmanuel Kahembwe
and
Iain Murray
and
Steve Renals
arXiv e-Print archive - 2017 via Local arXiv
Keywords:
cs.NE, cs.CL
First published: 2017/09/21 (7 years ago) Abstract: We present methodology for using dynamic evaluation to improve neural
sequence models. Models are adapted to recent history via a gradient descent
based mechanism, causing them to assign higher probabilities to re-occurring
sequential patterns. Dynamic evaluation outperforms existing adaptation
approaches in our comparisons. Dynamic evaluation improves the state-of-the-art
word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1
and 44.3 respectively, and the state-of-the-art character-level cross-entropies
on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char
respectively.
