Switch export_model to use AutoModel and AutoTokenizer

[jeswan]

This PR refactors export_model to use AutoModel and AutoTokenizer. It must be approved after the tests for export_model have been merged (#1259). This PR reduces is part of the effort to reduce the if-else statements regarding model types in our codebase. model_tokenizer_path has been removed and replaced by hf_pretrained_model_name_or_path.

[zphang]

If I'm reading this right, this PR needs to be updated to pass the new tests, right? export_model has a different signature I think.

[codecov]

Codecov Report

Merging #1260 (1b9848c) into master (6e6c2e3) will increase coverage by 0.00%.
The diff coverage is 85.36%.

@@           Coverage Diff           @@
##           master    #1260   +/-   ##
=======================================
  Coverage   55.51%   55.52%           
=======================================
  Files         149      149           
  Lines       10903    10878   -25     
=======================================
- Hits         6053     6040   -13     
+ Misses       4850     4838   -12     

Impacted Files	Coverage Δ
jiant/scripts/download_data/constants.py	100.00% <ø> (ø)
jiant/shared/model_setup.py	64.86% <ø> (+2.22%)	⬆️
jiant/shared/model_resolution.py	33.60% <37.50%> (-0.82%)	⬇️
jiant/proj/main/export_model.py	89.65% <90.00%> (+7.83%)	⬆️
jiant/proj/main/modeling/model_setup.py	37.93% <100.00%> (+1.24%)	⬆️
jiant/proj/main/modeling/taskmodels.py	39.06% <100.00%> (ø)
jiant/proj/main/runscript.py	73.04% <100.00%> (ø)
jiant/proj/main/tokenize_and_cache.py	86.76% <100.00%> (+0.19%)	⬆️
jiant/proj/simple/runscript.py	80.64% <100.00%> (+0.31%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6e6c2e3...1b9848c. Read the comment docs.

[zphang]

Nothing jumps out to me, also relatively low risk to merge now, I think.

Potentially one thing you might want to look at is if updating to transformers 4+ will be complicated by any of this.

[jeswan]

@zphang A few notes on how this PR will be affected by Transformers v4:

• AutoTokenizer will default to using the fast (rust) tokenizers. I am currently planning on switching to fast tokenizers as part of the update to Transformers v4.
• slow versions of the following tokenizers will not be available since SentencePiece is removed from required dependencies:
  • XLNetTokenizer
  • AlbertTokenizer
  • CamembertTokenizer
  • MBartTokenizer
  • PegasusTokenizer
  • T5Tokenizer
  • ReformerTokenizer
  • XLMRobertaTokenizer
• Side note: Model forward will have return_dict set to True as default and will require jiant forward passes to use False if using tuple unpacking.

[zphang]

Sounds good, moving remaining discussion to other PR.