prepare datasets for new encoding kwarg.

[erip]

This may need to wait for appropriate nightlies to be published, but @ejguan suggested this feature will make it into pytorch 1.11 so that's a good sign.

[ejguan]

Note: The PR with the change of encoding has already been merged into release/1.11 branch.

[Nayef211]

This may need to wait for appropriate nightlies to be published, but @ejguan suggested this feature will make it into pytorch 1.11 so that's a good sign.

I think we're using the latest main branch for torchdata for our unit tests so our tests should not be failing if the encoding kwarg is already present in the torchdata repo.

Here are the code pointers to where we are installing torchdata

• linux unittests
• windows unittests

[erip]

The encoding kwarg is in pytorch proper, not torchdata.

[Nayef211]

The encoding kwarg is in pytorch proper, not torchdata.

Ahh understood, thanks for clarifying!

[Nayef211]

Btw can we fix the merge conflicts with some of the datasets?

[erip]

Oops -- yep!

[parmeet]

Great to see this PR @erip! Hopefully we should soon be in position to land these changes :)

[erip]

This is a good sign. Once nightlies with Windows (and OS X) are published, this should be gtg -- Linux nightlies have propagated and tests pass in those cases.

[parmeet]

Overall the changes LGTM! We can merge once after the windows tests are passing. Thanks.

[erip]

New windows and OS X nightlies landed, so this should hopefully pass and be good to go. 🎆

[codecov]

Codecov Report

Merging #1616 (af4da8a) into main (16acc71) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1616   +/-   ##
=======================================
  Coverage   85.25%   85.25%           
=======================================
  Files          58       58           
  Lines        2483     2483           
=======================================
  Hits         2117     2117           
  Misses        366      366           

Impacted Files	Coverage Δ
torchtext/datasets/ag_news.py	95.23% <100.00%> (ø)
torchtext/datasets/amazonreviewfull.py	96.15% <100.00%> (ø)
torchtext/datasets/amazonreviewpolarity.py	96.00% <100.00%> (ø)
torchtext/datasets/cc100.py	95.83% <100.00%> (ø)
torchtext/datasets/conll2000chunking.py	95.83% <100.00%> (ø)
torchtext/datasets/dbpedia.py	96.00% <100.00%> (ø)
torchtext/datasets/enwik9.py	95.45% <100.00%> (ø)
torchtext/datasets/imdb.py	97.29% <100.00%> (ø)
torchtext/datasets/iwslt2016.py	89.83% <100.00%> (ø)
torchtext/datasets/iwslt2017.py	92.72% <100.00%> (ø)
... and 12 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 16acc71...af4da8a. Read the comment docs.