4.39.3; ZeroShotClassificationPipeline broken.

[FanaticPythoner]

System Info

• transformers version: 4.39.3
• Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.35
• Python version: 3.10.14
• Huggingface_hub version: 0.21.4
• Safetensors version: 0.4.2
• Accelerate version: 0.28.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.2.2+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: Yes.
• Using distributed or parallel set-up in script?: Using device_map and manually/algorithmically constructing max_memory.

Who can help?

@Narsil @ArthurZucker @gante

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

When upgrading from Transformers 4.38.2 to the latest release (4.39.3), the below error can be observed, wherein "discriminatorPipeline" is a zero shot classification pipeline (pl = pipeline("zero-shot-classification", model=model, tokenizer=model)). "model" is a AutoModelForCausalLM, obtained with:

    def load_model(self):
        model_path = "ybelkada/Mixtral-8x7B-Instruct-v0.1-bnb-4bit"
        model_class_to_use = AutoModelForCausalLM
        if self.model_config is None:
            self.model_config = AutoConfig.from_pretrained(HuggingFaceModelHandler.model_path, trust_remote_code=True)

        sortedMaxMem = dict(sorted(HuggingFaceModelHandler.maxMemory.items(), key=lambda item: item[0] if item[0] != "cpu" else 10000000))

        dtype = torch.bfloat16
        bnb_config = BitsAndBytesConfig(
                load_in_4bit=True,
                bnb_4bit_use_double_quant=True,
                bnb_4bit_quant_type="nf4",
                bnb_4bit_compute_dtype=dtype
            )

        model = model_class_to_use.from_pretrained(
                model_path,
                trust_remote_code=True,
                use_auth_token=True,
                torch_dtype=dtype,
                quantization_config=bnb_config,
                max_memory=sortedMaxMem,
                device_map="auto"
            )

        model.eval()

        return model

"tokenizer" is a AutoTokenizer, obtained with:

    def load_tokenizer(self):
        tokenizer_path = 'ybelkada/Mixtral-8x7B-Instruct-v0.1-bnb-4bit'
        tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, trust_remote_code=True)
        
        if tokenizer.pad_token is None:
            tokenizer.pad_token_id = tokenizer.unk_token_id

        return tokenizer

And finally, the pipeline is called by doing:

        ...
        discriminatorPipeline_res = self.mph.get_free_resource("discriminatorPipeline") # Returns the zero shot classification pipeline resource proxy
        discriminatorPipeline = discriminatorPipeline_res.get() # Returns the raw hf zero shot classification pipeline object

        try:
            outputs =  discriminatorPipeline(prompt,
                                    candidate_labels=candidate_labels, # list of 2 strings
                                    do_sample=do_sample,
                                    temperature=temperature+temperatureModifier,
                                    top_k=top_k,
                                    top_p=top_p,
                                    multi_label=False,
                                    eos_token_id=eos_token_id,
                                    repetition_penalty=repetition_penalty,
                                    length_penalty=length_penalty,
                                    encoder_repetition_penalty=encoder_repetition_penalty,
                                    use_cache=use_cache)
        except Exception as e:
            print(e)
            raise e
        finally:
            discriminatorPipeline_res.release()
        ...

The error didn't happen at all in 4.38.2. Reverting back from 4.39 to 4.38 fixes the issue. However, I'd like to use the /c4ai-command-r-plus model, and it seems to requires 4.39.

Traceback (most recent call last):
File "/home/user/project/HuggingFaceModelHandler.py", line 848, in discriminatorPipelineExec
outputs = discriminatorPipeline(prompt,
File "/home/user/anaconda3/envs/cvaitmp/lib/python3.10/site-packages/transformers/pipelines/zero_shot_classification.py", line 206, in call
return super().call(sequences, **kwargs)
File "/home/user/anaconda3/envs/cvaitmp/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1198, in call
return next(
File "/home/user/anaconda3/envs/cvaitmp/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 124, in next
item = next(self.iterator)
File "/home/user/anaconda3/envs/cvaitmp/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 266, in next
processed = self.infer(next(self.iterator), **self.params)
File "/home/user/anaconda3/envs/cvaitmp/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1112, in forward
model_outputs = self._forward(model_inputs, **forward_params)
TypeError: ZeroShotClassificationPipeline._forward() got an unexpected keyword argument 'pad_token_id'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/home/user/anaconda3/envs/conda_env_name/lib/python3.10/site-packages/transformers/pipelines/zero_shot_classification.py", line 206, in call
return super().call(sequences, **kwargs)
File "/home/user/anaconda3/envs/conda_env_name/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1198, in call
return next(
File "/home/user/anaconda3/envs/conda_env_name/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 124, in next
item = next(self.iterator)
File "/home/user/anaconda3/envs/conda_env_name/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 266, in next
processed = self.infer(next(self.iterator), **self.params)
File "/home/user/anaconda3/envs/conda_env_name/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1112, in forward
model_outputs = self._forward(model_inputs, **forward_params)
TypeError: ZeroShotClassificationPipeline._forward() got an unexpected keyword argument 'pad_token_id'

Expected behavior

The above code should work as is, yet it does not in the newest version.

[FanaticPythoner]

Anyone? It's definitely a big blocker. It prevents us from using newer models.

[gante]

cc @zucchini-nlp -- this issue is probably related to #29614