fix OBO error while reading vocab files with empty lines in BERT tokenizer

[parmeet]

fixes #1840

[Nayef211]

Overall LGTM. I was wondering if we considered modifying the existing _load_vocab_from_file implementation to add a flag to include newline characters? I know that the existing implementation has the options for multithreading as well as other perf optimizations (which I guess may not be too relevant to load in a smaller vocab file as might be the case for the BERT vocab).

[parmeet]

I was wondering if we considered modifying the existing _load_vocab_from_file implementation to add a flag to include newline characters?

This is special handling just to take into consideration vocab files from HF, so i have created explicit reading utility inside BERTTokenizer. I wasn't so sure if in general newlines are part of Vocab and if we should add it to _load_vocab_from_file?