Passing tokenizer call kwargs (like truncation) in pipeline

[BramVanroy]

System Info

• transformers version: 4.32.1
• Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.34
• Python version: 3.11.4
• Huggingface_hub version: 0.16.4
• Safetensors version: 0.3.3
• Accelerate version: 0.22.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.0.1+cu117 (True)

Who can help?

@ArthurZucker for the tokenizers and @Narsil for the pipeline.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I am trying to figure out how I can truncate text in a pipeline (without explicitly writing a preprocessing step for my data). I've looked at the documentation and searched the net. A lot of people seem to be asking this question (both on forums and Stack Overflow), but any solution that I could find does not work anymore. Below I try a number of them but none of them work. How can I enable truncation in the pipeline?

from transformers import pipeline

model_name = "bert-base-cased"
text = "Luke, I am not your [MASK]. " * 512  # Make sure text is longer than max model length

# As-is (error, size mismatch -- no truncation seems to happen)
pipe = pipeline("fill-mask", model=model_name)
result = pipe([text])

# truncation in init (error, unrecognized keyword)
pipe = pipeline("fill-mask", model=model_name, truncation=True)
result = pipe([text])

# truncation in call (error, unrecognized keyword)
pipe = pipeline("fill-mask", model=model_name)
result = pipe([text], truncation=True)

# truncation as tokenizer kwargs in tuple (error, size mismatch)
tokenizer_tuple = (model_name, {"truncation": True})
pipe = pipeline("fill-mask", model=model_name, tokenizer=tokenizer_tuple)
result = pipe([text])

# Truncation as tokenize_kwargs (https://github.com/huggingface/transformers/issues/21971#issuecomment-1456725779)
# Unexpected keyword error
pipe = pipeline("fill-mask", model=model_name, tokenize_kwargs={"truncation": True})
result = pipe([text])

Expected behavior

A fix if this is currently not implemented or broken but definitely also a documentation upgrade to clarify how tokenizer kwargs should be passed to a pipeline - both init and call kwargs!

[Narsil]

Indeed it's not implemented:

https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/fill_mask.py#L96

The name would be tokenizer_kwargs though to be consistent with the rest.

[nmcahill]

Hello! I'm going to take a crack at this one if that's cool.

[BramVanroy]

Have at it! Would be great to have this implemented. @nmcahill

[thedamnedrhino]

Need the same functionality in "text-generation" pipeline. Would like to take a go!

[mirix]

Any solution for this?

[ArthurZucker]

I am pretty sure the solution is to add the kwargs to the pipeline in that case text-generation . #28362 fixed is so closing

[mirix]

The problem is that max_length (tokenizer) gets mistaken with max_new_tokens (generator).

So the pipeline complains about a duplicated argument and informs that max_new_tokens will take precedence.

Anyway, I have implemented a preprocessing function, so the issue is workarounded.