Contrastive search doesn't work on Gemma3

[jood-canva]

System Info

• transformers version: 4.53.3
• Platform: Linux-6.8.0-1036-aws-x86_64-with-glibc2.35
• Python version: 3.11.10
• Huggingface_hub version: 0.34.3
• Safetensors version: 0.4.3
• Accelerate version: 1.6.0
• Accelerate config: not found
• DeepSpeed version: not installed
• PyTorch version (accelerator?): 2.7.1+cu126 (CUDA)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: No
• Using GPU in script?: Yes
• GPU type: NVIDIA L40S

Who can help?

Hi @zucchini-nlp and @gante

I think there is a bug with the new way contrastive search has been implemented. Following #40428 we now have to use the community package https://huggingface.co/transformers-community/contrastive-search . However I am getting the error

  File "/home/coder/work/canva/tools/build/python/third_party/.venv/lib/python3.11/site-packages/transformers/generation/utils.py", line 2363, in generate
    custom_generate_function = self.load_custom_generate(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/coder/work/canva/tools/build/python/third_party/.venv/lib/python3.11/site-packages/transformers/generation/utils.py", line 417, in load_custom_generate
    is_local_code = os.path.exists(pretrained_model_name_or_path)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 19, in exists
TypeError: stat: path should be string, bytes, os.PathLike or integer, not function

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run the following snippet:

import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
ckpt = "google/gemma-3-4b-it"
model = Gemma3ForConditionalGeneration.from_pretrained(
    ckpt, device_map="auto", torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained(ckpt)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/spaces/big-vision/paligemma-hf/resolve/main/examples/password.jpg"},
            {"type": "text", "text": "What is the password?"}
        ]
    }
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

gen_out = model.generate(
    **inputs,
    custom_generate="transformers-community/contrastive-search",
    penalty_alpha=0.6,
    top_k=4,
    max_new_tokens=128,
    trust_remote_code=True,
)

Expected behavior

It should be able to generate? I don't understand why the custom generate function throws an error here.

Sorry if I'm missing something obvious!

[zucchini-nlp]

@jood-canva hey! The custom generation had issues in the past which was resolved in #38304. Updating transformers version will help (pip install -U transformers)

I am seeing another error in contrastive search which comes from custom code, will submit a PR on the hub

[zucchini-nlp]

cc @gante , I didn't merge directly to main. LMK if you agree 🤗

https://huggingface.co/transformers-community/contrastive-search/discussions/1

[gante]

@zucchini-nlp -- @manueldeprada is tracking that issue, not sure if he has a different fix in mind 🤔

[manueldeprada]

Indeed, merged!! thanks a lot for the fix @zucchini-nlp !!! I am quite busy these days.

Dola needs more work, will get to that tomorrow