Data Noising as Smoothing in Neural Network Language Models
Wang, Sida I.
Ng, Andrew Y.
arXiv e-Print archive - 2017 via Local Bibsonomy
The paper investigates better noising techniques for RNN language models.
A noising technique from previous work would be to randomly replace words in the context or replace them with a blank token. Here they investigate ways of choosing better which words to replace and choosing the replacements from a better distribution, inspired by methods in n-gram smoothing. They show improvement on language modeling (PTB and text8) and machine translation (English-German).