MarianMTModel no longer has postprocess_next_token_scores function

[dsvilarkovic]

Hello I just saw that in file https://github.com/huggingface/transformers/blob/v4.19.2/src/transformers/data/test_generation_utils.py you mention calling the postprocess_next_token_scores() method from MarianMTModel, which doesn't seem to exist anymore in it's source file anymore: https://github.com/huggingface/transformers/blob/v4.19.2/src/transformers/models/marian/modeling_marian.py and neither exist as a PreTrainedModel method anymore, and neither in GenerationMixin model (in GenerationMixin it used to be most recently in https://huggingface.co/transformers/v3.3.1/_modules/transformers/generation_utils.html).

I would be really happy if you could tell what is the alternative to this function in the most recent library version!

[LysandreJik]

cc @patil-suraj

[patil-suraj]

Hi @dsvilarkovic !

This functionality still exists, but the generate method is refactored to support this more cleanly. This is now implemented as LogitsProcessor. cf #6949

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.