[T5] T5 in ParlAI #3519

klshuster · 2021-03-12T21:26:39Z

Patch description

Port of HuggingFace's T5 model for ParlAI. Tagging people I thought might be interested, This PR serves a dual purpose:

Get T5 in ParlAI
Demonstrate an effective way of using HF models in ParlAI

The second point is relevant in that I tried writing a generalized version of a DictionaryAgent that wraps a HF tokenizer.

On that note - I am seeking feedback on whether this would be more appropriate in a utils file (e.g. parlai.utils.huggingface)

Testing steps
Extensive CI; HF had some nice integration tests I copied over, and I used similar integration tests as the ones used for BART.

$ pytest test_t5.py
=====starts =====

test_t5.py ........                                                                                                                                                            [100%]

====slowest 10 durations ====
38.64s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_t5_ft
33.27s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_t5_model_parallel
18.95s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_t5_gen
14.99s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_small
12.57s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_summarization
12.21s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_translation_en_to_fr
10.35s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_translation_en_to_ro
9.90s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_translation_en_to_de
0.05s teardown tests/nightly/gpu/test_t5.py::TestT5Model::test_translation_en_to_ro
0.03s teardown tests/nightly/gpu/test_t5.py::TestT5Model::test_t5_model_parallel
====8 passed, 1 warning in 155.14s (0:02:35) ====

klshuster · 2021-03-12T22:22:25Z

does anyone know why circle can't find tests to run? did i mess something up?

tests/nightly/gpu/test_t5.py

EricMichaelSmith

Cool, seems reasonable at a quick glance, but will defer to others with more context for approval

parlai/agents/t5/modules.py

klshuster · 2021-03-16T17:47:22Z

I'm re-requesting reviews because the model now lives in parlai/agents/hugging_face; I refactored the HFDictionaryAgent as well, but verified with CI that gpt2 and dialogpt tests still pass:

$ pytest test_dialogpt.py
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-1.0.0.dev0
rootdir: /private/home/kshuster/ParlAI, configfile: pytest.ini
plugins: hydra-core-1.0.0, requests-mock-1.8.0, regressions-2.1.1, datadir-1.3.1
collected 4 items

test_dialogpt.py ....                                                                                                                                                         [100%]

====slowest 10 durations ====
708.84s call     tests/nightly/gpu/test_dialogpt.py::TestDialogptModel::test_dialogpt
62.30s call     tests/nightly/gpu/test_dialogpt.py::TestDialogptModel::test_batchsize
5.53s call     tests/nightly/gpu/test_dialogpt.py::TestDialogptModel::test_nospecialtok
0.01s call     tests/nightly/gpu/test_dialogpt.py::TestDialogptModel::test_start_token

(6 durations < 0.005s hidden.  Use -vv to show these durations.)
==== 4 passed, 6 warnings in 778.03s (0:12:58) ====


$ pytest test_gpt2.py
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-1.0.0.dev0
rootdir: /private/home/kshuster/ParlAI, configfile: pytest.ini
plugins: hydra-core-1.0.0, requests-mock-1.8.0, regressions-2.1.1, datadir-1.3.1
collected 5 items

test_gpt2.py .....                                                                                                                                                            [100%]

====slowest 10 durations ====
119.10s call     tests/nightly/gpu/test_gpt2.py::TestDistributed::test_distributed
116.12s call     tests/nightly/gpu/test_gpt2.py::TestGpt2::test_batchsize
12.81s call     tests/nightly/gpu/test_gpt2.py::TestHuggingFaceDict::test_custom_special_tokens
8.67s setup    tests/nightly/gpu/test_gpt2.py::TestGpt2::test_nospecialtok
6.02s call     tests/nightly/gpu/test_gpt2.py::TestGpt2::test_nospecialtok
0.01s call     tests/nightly/gpu/test_gpt2.py::TestGpt2::test_start_token

(4 durations < 0.005s hidden.  Use -vv to show these durations.)
====5 passed, 7 warnings in 263.99s (0:04:23) ====

klshuster · 2021-03-17T15:40:58Z

$ pytest test_gpt2.py test_dialogpt.py test_t5.py
=====test session starts =====
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-1.0.0.dev0
rootdir: /private/home/kshuster/ParlAI, configfile: pytest.ini
plugins: hydra-core-1.0.0, requests-mock-1.8.0, regressions-2.1.1, datadir-1.3.1
collected 17 items

test_gpt2.py .....                                                                                                                                                            [ 29%]
test_dialogpt.py ....                                                                                                                                                         [ 52%]
test_t5.py ........                                                                                                                                                           [100%]

====slowest 10 durations ====
494.52s call     tests/nightly/gpu/test_dialogpt.py::TestDialogptModel::test_dialogpt
50.78s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_t5_ft
49.19s call     tests/nightly/gpu/test_dialogpt.py::TestDialogptModel::test_batchsize
40.02s call     tests/nightly/gpu/test_gpt2.py::TestGpt2::test_batchsize
32.69s call     tests/nightly/gpu/test_gpt2.py::TestDistributed::test_distributed
32.33s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_t5_model_parallel
19.26s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_t5_gen
12.63s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_summarization
12.50s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_small
11.41s call     tests/nightly/gpu/test_t5.py::TestT5Model::test_translation_en_to_fr
====17 passed, 8 warnings in 789.71s (0:13:09) ====

klshuster · 2021-03-17T23:41:36Z

Running all gpu tests locally, I get:

======== short test summary info ========
FAILED tests/nightly/gpu/test_bert.py::TestBertModel::test_biencoder - ImportError: BERT rankers needs pytorch-pretrained-BERT installed.
FAILED tests/nightly/gpu/test_bert.py::TestBertModel::test_crossencoder - ImportError: BERT rankers needs pytorch-pretrained-BERT installed.
FAILED tests/nightly/gpu/test_style_gen.py::TestClassifierOnGenerator::test_simple - AssertionError
======== 3 failed, 72 passed, 356 warnings in 4435.44s (1:13:55) ========

we can ignore the first two; the third might be an issue with my downloaded model for that? (the assertion that failed was assert 'beam_size' in opt_from_disk

klshuster · 2021-03-19T18:48:49Z

all tests pass!! PR is ready for review

spencerp

Seems reasonable to me. Thanks for adding this!

spencerp · 2021-03-19T19:22:03Z

parlai/agents/hugging_face/dict.py

        self.override_special_tokens(opt)
-        for i in range(self.tokenizer.vocab_size):
-            token = self.tokenizer._convert_id_to_token(i)
-            self.add_token(token)
-            self.freq[token] = 1
+        self._unk_token_idx = self.hf_tokenizer.unk_token_id


Should these lines be swapped? Right now _unk_token_idx from override_special_tokens is overridden.

spencerp · 2021-03-19T19:26:32Z

parlai/agents/hugging_face/hugging_face.py

 except ImportError:
    raise ImportError('Please run `pip install transformers`.')


+version = float('.'.join(transformers.__version__.split('.')[:2]))


nit; Can we identify this as a global more distinctly somehow? So that we don't end up mixing this up with some other variable named "version". Perhaps just all caps (VERSION)?

yeah i'll make it more explicit (HF_VERSION)

klshuster · 2021-03-19T20:44:33Z

failing gpu test is flaky

klshuster added 3 commits March 12, 2021 15:39

t5

2bbd634

lint

769fe29

slight cleanup

a7748c4

klshuster requested review from stephenroller, spencerp, EricMichaelSmith and emilydinan March 12, 2021 21:26

facebook-github-bot added the CLA Signed label Mar 12, 2021

stephenroller reviewed Mar 12, 2021

View reviewed changes

tests/nightly/gpu/test_t5.py Outdated Show resolved Hide resolved

tests/nightly/gpu/test_t5.py Outdated Show resolved Hide resolved

klshuster added 3 commits March 12, 2021 19:17

overfit task

2a5cb49

Merge branch 'master' into t5

fbfcfb8

try catch

a4bf50e

EricMichaelSmith reviewed Mar 15, 2021

View reviewed changes

parlai/agents/t5/modules.py Outdated Show resolved Hide resolved

klshuster added 5 commits March 16, 2021 10:35

install transformers for cpu

31815e9

Merge branch 'master' into t5

5cab403

always install transformers

73c9310

migrate t5 to hf

3e8625b

black

5808e45

klshuster requested review from EricMichaelSmith and stephenroller March 16, 2021 17:45

klshuster added 7 commits March 16, 2021 13:48

import again

6ba7739

change to import error

828e06f

set device

4dc650e

bump tf versions

555a29b

get version right

fa9e9fb

Merge branch 'master' into t5

a0853a9

handle gpt2

dfde14d

klshuster added 2 commits March 17, 2021 16:07

Merge branch 'master' into t5

71a08c9

add tear down

91f12af

klshuster added 5 commits March 18, 2021 11:22

Merge branch 'master' into t5

4f65c31

change vectorize

d675ede

update comment for vectorize; move version check to

3f86978

reduce bsz for dialogpt

4547549

Merge branch 'master' into t5

c6d8b6a

spencerp approved these changes Mar 19, 2021

View reviewed changes

address spencer comments

58eb4d6

klshuster merged commit a8fe17c into master Mar 19, 2021

klshuster deleted the t5 branch March 19, 2021 20:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[T5] T5 in ParlAI #3519

[T5] T5 in ParlAI #3519

klshuster commented Mar 12, 2021

klshuster commented Mar 12, 2021

EricMichaelSmith left a comment

klshuster commented Mar 16, 2021

klshuster commented Mar 17, 2021

klshuster commented Mar 17, 2021

klshuster commented Mar 19, 2021

spencerp left a comment

spencerp Mar 19, 2021

klshuster Mar 19, 2021

spencerp Mar 19, 2021

klshuster Mar 19, 2021

klshuster commented Mar 19, 2021

[T5] T5 in ParlAI #3519

[T5] T5 in ParlAI #3519

Conversation

klshuster commented Mar 12, 2021

Patch description

klshuster commented Mar 12, 2021

EricMichaelSmith left a comment

Choose a reason for hiding this comment

klshuster commented Mar 16, 2021

klshuster commented Mar 17, 2021

klshuster commented Mar 17, 2021

klshuster commented Mar 19, 2021

spencerp left a comment

Choose a reason for hiding this comment

spencerp Mar 19, 2021

Choose a reason for hiding this comment

klshuster Mar 19, 2021

Choose a reason for hiding this comment

spencerp Mar 19, 2021

Choose a reason for hiding this comment

klshuster Mar 19, 2021

Choose a reason for hiding this comment

klshuster commented Mar 19, 2021