Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

[RAG] Handle Special Toks #3833

Merged
merged 2 commits into from
Jul 22, 2021
Merged

[RAG] Handle Special Toks #3833

merged 2 commits into from
Jul 22, 2021

Conversation

klshuster
Copy link
Contributor

Patch description
Handle specifying special tokens in a RAG setup by implementing _resize_token_embeddings

Testing steps
Updates test_rag.py, ran locally:

$ pytest -k TestOtherOptions
===== test session starts =====
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-1.0.0.dev0
rootdir: /private/home/kshuster/ParlAI, configfile: pytest.ini
plugins: hydra-core-1.0.7, requests-mock-1.8.0, regressions-2.1.1, datadir-1.3.1
collected 118 items / 116 deselected / 2 selected

test_rag.py ..                                                                                                                                                                                                                                                                                                                                                      [100%]
===== slowest 10 durations =====
12.15s call     tests/nightly/gpu/test_rag.py::TestOtherOptions::test_n_positions
8.56s call     tests/nightly/gpu/test_rag.py::TestOtherOptions::test_resize_embs

mojtaba-komeili
mojtaba-komeili reviewed Jul 21, 2021
View reviewed changes
parlai/agents/rag/rag.py Outdated
],
dim=0,
)
if init_weight.size(0) <= opt['n_positions'] + opt['n_extra_positions']:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should it be < or <=. I am guessing for the equal condition we do not need to resize.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

mojtaba-komeili
mojtaba-komeili reviewed Jul 21, 2021
View reviewed changes
Copy link
Contributor

@mojtaba-komeili mojtaba-komeili left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@klshuster
Copy link
Contributor Author

klshuster commented Jul 21, 2021 •
edited by mojtaba-komeili
Loading

LGTM

@mojtaba-komeili did you mean to approve?

Yeah, sorry I thought I had clicked on approval.

@klshuster klshuster merged commit eae6d9c into master Jul 22, 2021
@klshuster klshuster deleted the rag_resize_embs branch July 22, 2021 00:35
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@mojtaba-komeili mojtaba-komeili mojtaba-komeili approved these changes

@spencerp spencerp Awaiting requested review from spencerp

Assignees
No one assigned
Labels
CLA Signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@klshuster @mojtaba-komeili @facebook-github-bot