Add validation for maximum sequence length in modeling_whisper.py

[AmirMohammadFakhimi]

What does this PR do?

Added a validation check to ensure that the sequence length of labels does not exceed the maximum allowed length of 448 tokens. If the sequence length exceeds this limit, a ValueError is raised with a descriptive error message.

This change prevents the model from encountering errors or unexpected behavior due to excessively long sequences during training or fine-tuning, ensuring consistent input dimensions and improving overall robustness.

While training, I encountered the following error multiple times:

../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [1167,0,0], thread: [0,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [1167,0,0], thread: [1,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.

Eventually, this led to the following error message:

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging, consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

It took me almost a week to diagnose and understand the root cause of this problem. This simple validation can save others from facing similar debugging challenges and wasted time.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

speech models: @sanchit-gandhi

[LysandreJik]

cc @ylacombe and @eustlb when back from leave!

[ylacombe]

Hey @AmirMohammadFakhimi, many thanks for opening this PR!

Do you think you could give a minimum code snippet to reproduce the issue you have? It doesn't have to be a training code, I think a forward pass with handmade labels should do the trick!

Thanks for your help

cc @eustlb for visibility

[ylacombe]

Hey @AmirMohammadFakhimi, thanks for providing the code snippet!

So, according to Whisper configuration, there's two hard limits on tokens and spectrograms lengths: max_source_positions and max_target_positions. The latter is where you can find the hard 448 token limits.

If you wanted to modify that limit, you'd have to modify the architecture, and thus to re-train the model from scratch.

Your PR is justified, I'd just modify the mention to 448 to config.max_target_positions, e.g:

-            if labels.shape[1] > 448:
+           if labels.shape[1] > config.max_target_positions:

[ylacombe]

Could you also add a test here that makes sure the model works with labels sequence length smaller than the limit, and that makes sure it throws an error if not ?

[AmirMohammadFakhimi]

Could you also add a test here that makes sure the model works with labels sequence length smaller than the limit, and that makes sure it throws an error if not ?

I added three tests to the file. I also merged branch 'huggingface:main' into patch-1.
For testing, I made a hardcode here because I think it is hardcoded in the whisper model, too. Do you have any better idea?

And finally, just for info, there are these lines in test_modeling_whisper.py:

config, inputs_dict = self.prepare_config_and_inputs()

config, input_dict = self.prepare_config_and_inputs()

config, inputs = self.model_tester.prepare_config_and_inputs()

I used the second format. I think having a convention for it will help.

[ylacombe]

Thanks for iterating ! Let's not hardcode the max length !

[ylacombe]

For example here it would be labels_length = config.max_target_positions

[ylacombe]

Thanks for iterating!
This is a bit complicate, you can do it like propose in this review!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ylacombe]

Thanks for iterating quickly on this !

cc @LysandreJik for review!

[jugal-sheth]

try below

# Initialize parameters and tokenizer
transcripts_folder = "path_to_transcripts"
max_target_length = 448
show_target_length = 400
tokenizer = InitializeTokenizer("model_name", language="en", task="transcribe")

# Define the function to check and delete faulty transcripts
function check_and_delete_transcripts():
    files = ListAllFiles(transcripts_folder, extension=".txt")  # Get all .txt files

    for file in ProgressBar(files, description="Checking transcripts"):
        content = ReadFile(file)  # Read the transcript content
        tokenized = TokenizeContent(tokenizer, content)  # Tokenize the content
        token_length = GetTokenLength(tokenized)  # Get the length of tokens

        if token_length > max_target_length:
            Print(f"Faulty file: {file}, Tokens: {token_length}")
            DeleteFile(file)  # Delete if too long
        elif token_length > show_target_length:
            Print(f"File exceeds show limit: {file}")
            PrintContent(content)  # Optionally delete or inspect further

# Execute the function
check_and_delete_transcripts()

this will help you get rid of faulty data then finetune the model