[Tasks] Mismatch between input specs and task pipelines expected generation

[hanouticelina]

Description

There's a mismatch between the task input spec and the expected generation parameters in task pipelines for automatic-speech-recognition and image-to-text tasks, and possibly others. Specifically, the input schema defines a parameter named generate for generation parameters. However, the _sanitize_parameters() method implemented in these task pipelines (eg: AutomaticSpeechRecognitionPipeline. _sanitize_parameters()) expects a parameter named generate_kwargs instead.

This mismatch causes errors when calling InferenceAPI according to the specified input specs.

Reproduction

If we follow the input specs defined here for automatic-speech-recognition:

import requests
import base64

API_URL = "https://api-inference.huggingface.co/models/openai/whisper-large-v3"
headers = {"Authorization": f"Bearer {os.environ.get('HF_TOKEN')}"}

with open("sample-3.flac", "rb") as f:
    data = f.read()
payload = {
    "inputs": base64.b64encode(data).decode("utf-8"),
    "parameters": {
        "return_timestamps": True,
        "generate": {
            "max_new_tokens": 100,
            "temperature": 0.7,
        },
    },
}

response = requests.post(
    API_URL,
    headers=headers,
    json=payload,
)
print(response.json())

this returns an output with an error:

{'error': 'unknown error', 'warnings': ["There was an inference error: unknown error: AutomaticSpeechRecognitionPipeline._sanitize_parameters() got an unexpected keyword argument 'generate'"]}

When replacing generate with generate_kwargs:

import requests
import base64

API_URL = "https://api-inference.huggingface.co/models/openai/whisper-large-v3"
headers = {"Authorization": f"Bearer {os.environ.get('HF_TOKEN')}"}

with open("sample-3.flac", "rb") as f:
    data = f.read()
payload = {
    "inputs": base64.b64encode(data).decode("utf-8"),
    "parameters": {
        "return_timestamps": True,
        "generate_kwargs": {
            "max_new_tokens": 100,
            "temperature": 0.7,
        },
    },
}

response = requests.post(
    API_URL,
    headers=headers,
    json=payload,
)
print(response.json())

No error in the output:

{'text': ' Thank you.', 'chunks': [{'timestamp': [0.0, 1.0], 'text': ' Thank you.'}]}

We have the same issue for image-to-text task.

I wanted to open this issue first before proposing any PR to have your opinion first on how to better align the task input specs with the task pipelines.

[coyotte508]

To be clear this is a report regarding the json-schema definitions of tasks in @huggingface/tasks in https://github.com/huggingface/huggingface.js/tree/main/packages/tasks/src/tasks/automatic-speech-recognition/spec and such

cc @SBrandeis

[SBrandeis]

cc @Wauplin as well

[Wauplin]

Thanks for flagging @hanouticelina This is quite annoying indeed. I personally don't like the idea of having an API parameter called generate_kwargs (which feels python-centric).

@ArthurZucker @LysandreJik is there a world where transformers pipelines could accept either generate or generate_parameters as kwargs instead of generate_kwargs? Since these are used for Inference API and Inference Endpoint, it would be nice to rename them.

[ArthurZucker]

generate_parameters does not look bad, generate is meaningless so no!

[Wauplin]

So generate_parameters it is then :)

[Wauplin]

cc @Rocketknight1 who started to work on integrating officially the specs to transformers pipelines in huggingface/transformers#33730.

@hanouticelina let's update the specs to generate_parameters and ship it in huggingface_hub. Then transformers will be able to make the change.

[hanouticelina]

Perfect! I will take care of updating the specs now :)

[pcuenca]

generate_parameters does not look bad, generate is meaningless so no!

Big +1

Or generation_parameters, if we are moving away from the Python-centric generate_kwargs.

[LysandreJik]

Agree with @pcuenca that generation_parameters sounds better -> generate_parameters makes it sound like it's the parameters that will be passed to the generate() method in transformers (or to the /generate path of an API).

generation_parameters seems like the most agnostic

[pcuenca]

Exactly my thinking, sorry for not explaining. Thanks @LysandreJik!

[hanouticelina]

+1 @pcuenca. It also aligns with the naming used in the schema here

[Wauplin]

Even better! Agree on generation_parameters :)