Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation

vladimir karpukhin1, Omer Levy2, Jacob Eisenstein3, Marjan Ghazvininejad2
1Facebook Artificial Intelligence Research, 2Facebook AI Research, 3Georgia Institute of Technology


Abstract

Contemporary machine translation systems achieve greater coverage by applying subword models such as BPE and character-level CNNs, but these methods are highly sensitive to orthographical variations such as spelling mistakes. We show how training on a mild amount of random synthetic noise can dramatically improve robustness to these variations, without diminishing performance on clean text. We focus on translation performance on natural typos, and show that robustness to such noise can be achieved using a balanced diet of simple synthetic noises at training time, without access to the natural noise data or distribution.