Adding Google 1 Billion Benchmark Dataset to PyTorch dataset

[h56cho]

Hello,

Currently, for language modelling, PyTorch has 3 built-in datasets (WikiText103, WikiText2, Penn Treebank). Would it be possible to add the Google 1 Billion Benchmark dataset as one of the PyTorch language modelling built-in data?

The link to the Google 1 Billion Benchmark dataset is below:

http://www.statmt.org/lm-benchmark/

Thanks,

[zhangguanheng66]

Just for future contributors, the raw files are quite large compared with current built-in language modeling datasets. However, it's still useful to include this benchmark dataset for Torchtext users. My suggestion is to start with the one "News Crawl corpus (2011 only)"