Document and validate typical_p in generation #19128
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
src/transformers/generation_utils.py
Outdated
| elif do_sample is False: | ||
| raise ValueError( | ||
| "Decoder argument `typical_p` must be used in sampling mode. " | ||
| "Make sure that `do_sample` is set to `True`." | ||
| ) |
There was a problem hiding this comment.
This is a problem for several other options, such as top_k or length_penalty, which have no effect in some generation modes.
Despite it being a correct check, we are assessing how to best handle it. We would like to avoid having an if/else branch for each input variable, as it would make generate even longer and less readable. Since it is harmless to have typical_p set with do_sample=False and we don't want to promote further checks of this kind, I'm going to ask to remove it for now.
There was a problem hiding this comment.
OK, I do hope that the do_sample check can happen in the future because receiving typical_p but doing greedy or non-sampled beam decoding is, in my view, silently failing.
Removed this and committed requested changes ✅
mapmeld
force-pushed
the
validate_genparams
branch
from
c58f6f7 to
189b228
Compare
What does this PR do?
Throws a
ValueErrorwhentypical_pargument is provided to text-generation, but its value ordo_sample=Falseprevent typical decoding from happening as intended. Adds a line documenting typical decoding.Most arguments to generate were previously covered in #18261 , but not
typical_p.