Paligemma - fix slow tests, add bf16 and f16 slow tests

[molbap]

What does this PR do?

As per PR title.
cc @ydshieh

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

Thanks @molbap . Some nits.

Let's try the run-slow feature to make sure everything is passing on CI runner.

[molbap]

OOO but the models are gated @ydshieh , the CI bot needs access for testing, not sure how to do it! (the run-slow works apart from that 💯 )

[younesbelkada]

@molbap for that you can use require_read_token context manager:

transformers/tests/models/llama/test_modeling_llama.py

Line 475 in 977ce58

@require_read_token

:D !

[molbap]

cc @ydshieh should be good to go!

[ydshieh]

Thank you @molbap ! LGTM.

I am not sure I understand the to replace what is used part in this comment

Let' s make sure we test the preprocessing to replace what is used

(what is what and what is used here).

I mean it might be good to be more precise - but maybe I lack some context

[molbap]

Apparently it comes from Llava code - I have no idea what this comment means! maybe @younesbelkada ?

[ydshieh]

In this case, we can merge, and once @younesbelkada has answered, we can update the comments in another PR (if needed)

[younesbelkada]

Thanks !
About your question, honestly I don't know, I think we can safely ignore it

[ydshieh]

If that comment is that confusing, let's just remove it in another PR (both here and in LLava testing file). WDTY?