Natural Questions (MRQA), NewsQA, Quoref

[zphang]

Relevant hyperparameters:

• max_seq_length=384
• batch_size=4
• learning_rate=1e-5
• epochs=3

---

• Natural Questions (MRQA version):
  • jiant (RoBERTA-large): EM=68.0 / F1=79.7
  • Longpre et al. (XLNET+Augmentation): F1=~81-82
  • Takahashi et al.(BERT-large): EM=67.9 / F1=79.8
• NewsQA
  • jiant (RoBERTA-large): EM=53.2 / F1=69.3
  • Trischler et al. (mLSTM): EM=34.4 / F1=49.6
• Quoref
  • jiant (RoBERTA-large): EM=78.6 / F1=83.9
  • "RoBERTa-MT" on Leaderboard, Test (RoBERTa): EM=72.6, F1=80.7

[codecov]

Codecov Report

Merging #1207 into master will decrease coverage by 1.33%.
The diff coverage is 26.56%.

@@            Coverage Diff             @@
##           master    #1207      +/-   ##
==========================================
- Coverage   57.44%   56.11%   -1.34%     
==========================================
  Files         138      136       -2     
  Lines        9861     9975     +114     
==========================================
- Hits         5665     5597      -68     
- Misses       4196     4378     +182     

Impacted Files	Coverage Δ
...cripts/download_data/datasets/hf_datasets_tasks.py	100.00% <ø> (ø)
jiant/tasks/constants.py	100.00% <ø> (ø)
jiant/tasks/evaluate/core.py	37.15% <0.00%> (ø)
...iant/scripts/download_data/datasets/files_tasks.py	7.48% <4.73%> (-8.00%)	⬇️
jiant/tasks/lib/mrqa_natural_questions.py	48.93% <48.93%> (ø)
jiant/tasks/lib/quoref.py	65.62% <65.62%> (ø)
jiant/tasks/lib/newsqa.py	70.00% <70.00%> (ø)
jiant/scripts/download_data/constants.py	100.00% <100.00%> (ø)
jiant/tasks/lib/templates/squad_style/core.py	30.79% <100.00%> (ø)
jiant/tasks/retrieval.py	100.00% <100.00%> (ø)
... and 13 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1b14c1...52c0d32. Read the comment docs.