Generation: fix handling of special tokens #31254
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
I had to fix the test by passing special token in config because previously all special tokens were updated inside generation strategy code blocks. I will see how to make it BC, not ready for merge yet |
|
I think it's better to just raise warning and ask users to provide attention masks, which I already commited. Another option would be to prepare special tokens twice, once in generate and the second time juts before pre-fill. I prefer the first options as it's less redundant code and more intuitive, lmk if we want full BC even with repeating code |
src/transformers/generation/utils.py
Outdated
| generation_config.eos_token_id, self.generation_config.eos_token_id, device=device | ||
| ) | ||
| pad_token_id = _tensor_or_none( | ||
| generation_config.pad_token_id, self.generation_config.eos_token_id, device=device |
There was a problem hiding this comment.
Should be self.generation_config.pad_token_id
* fix special tokens in generatioon * fix test * add warning * fix the check * warn once * fix
* fix special tokens in generatioon * fix test * add warning * fix the check * warn once * fix
What does this PR do?
Fixes #31251. For backwards compatibility we have to check all special tokens firstly in kwargs, and then in self.config. Earlier we fixed that for
decoder_start_token_id, this PR extends it for all special tokens.Verified that it works with the provided script and the script from last fix on
decoder_start_token_id