Code

Migrating EnWik9 to datapipes #1511#1512

Code

Merged

Nayef211 merged 6 commits intopytorch:mainpytorch/text:mainfrom Nayef211:feature/migrate_enwik9_datasetNayef211/text:feature/migrate_enwik9_datasetCopy head branch name to clipboardJan 20, 2022

+39-20

Commits

Commits on Jan 13, 2022

• Migrating enwik9 dataset to use torchdata

  

  nayef211

  committed
  ebda77b

  Copy full SHA for ebda77b

Commits on Jan 14, 2022

• Added typing to params

  

  nayef211

  committed
  9546731

  Copy full SHA for 9546731

Commits on Jan 19, 2022

• Merge branch 'main' into feature/migrate_enwik9_dataset

  

  nayef211

  committed
  20d59e3

  Copy full SHA for 20d59e3

• Fixed PR comments. Updated to data_dp

  

  nayef211

  committed
  0f23a7e

  Copy full SHA for 0f23a7e

• Added caching for extracted files

  

  nayef211

  committed
  a8e54d9

  Copy full SHA for a8e54d9

Commits on Jan 20, 2022

• Moved FileOpener after ondiskcache datapipe

  

  nayef211

  committed
  ae957ae

  Copy full SHA for ae957ae