Fix handling of end of file while reading vocab from file

[abhinavarora]

Issue

As we were onboarding pre-commit for torchtext in #1546, we added end of file newline rule. With this rule it was discovered that our experimental vocab incorrectly handled new line at the end of vocab file. Vocab simply added an empty string to the vocab due to which our unit tests failed.

Fix

Vocab builder should ignore tokens that are empty.

Testing

pytest -v test/experimental/test_with_asset.py

[Nayef211]

This LGTM. Thanks for catching and fixing this bug @abhinavarora!

As a small nit, I would avoid combining formatting changes with features/bug fixes since it makes it hard to differentiate between the 2 types of changes when reviewing the PR or when referring to it in the future in case the PR introduces a bug.

[Nayef211]

What exactly is the change for this line? On the Github UI, the diff looks the same to me

[abhinavarora]

Hey @Nayef211 this PR does not make any formatting change. The change in vocab_test.txt is to add a new line at the end of the file, so that the test is also able to test this condition.

[abhinavarora]

This LGTM. Thanks for catching and fixing this bug @abhinavarora!

As a small nit, I would avoid combining formatting changes with features/bug fixes since it makes it hard to differentiate between the 2 types of changes when reviewing the PR or when referring to it in the future in case the PR introduces a bug.

The PR does not add any formatting changes :-) Those are in a different PR #1546,