Add V3 Support

[Oscaarjs]

This is a continuation of #548 so credit to its contributors and author @stillmatic.

I've tried to adress some of the comments recieved on that PR.

Potential TODOs:

• Dynamic _LANGUAGE_CODES in tokenizer.py depending on whether V3 (which supports "yue") is loaded.

[Oscaarjs]

@nguyendc-systran , would be great if you could have a look

[trungkienbkhn]

@Oscaarjs , hello. We have generated the Systran Whisper large-v3 conversion model with new file preprocessor_config.json in the HF-to-ct2 script, could you please update this info also in README.md file of your PR.
Example: ct2-transformers-converter --model openai/whisper-large-v3 --output_dir whisper-large-v3-ct2
--copy_files tokenizer.json preprocessor_config.json --quantization float16

[Oscaarjs]

@trungkienbkhn Ah yes! Did the changes, please check if I made them correctly

[trungkienbkhn]

Sounds good to me. Hopefully there will be some benchmark tests for the faster whisper large-v3 soon.

[salahzoubi]

@Oscaarjs im guessing this still doesn't allow for batch transcribe (which is built-in large-v3)?

[Oscaarjs]

@Oscaarjs im guessing this still doesn't allow for batch transcribe (which is built-in large-v3)?

What do you mean by built-in? Afaik there's nothing changed with v3 and v2 that changes that part of the model. Or are you referring to HFs pipeline implementation of it? (which do support batching)

[funboarder13920]

@Oscaarjs im guessing this still doesn't allow for batch transcribe (which is built-in large-v3)?

Your question was off topic in the other PR, it is still off topic in this PR.
Batch inference has nothing to do with large-v3