Generate, translate and format subtitles in one liner

[baxtree]

Hiya, would like to share that whisper has been integrated into subaligner’s subtitle generation process. This integration, along with existing converters and other translation models, enables us to create near-production quality subtitles in a wide range of formats and languages.

Thank you for keeping those fantastic models accessible to everyone.

[dgoryeo]

Hi @baxtree , this looks quite interesting. One of the weaknesses of Whisper models is that a subtitle line can start seconds before the actual speech --the timings are not as accurate. Would Subligner be able to address that --correct the start/end times for each subtitle line? Thanks.

[ryanheise]

@dgoryeo finally got a chance to work on this today. I've created a PR #1461 to address this. Let me know how it goes.

[baxtree]

The funny thing is after upgrading Whisper from v20230124 to v20230314, I can't see substantial start-time issues for the audio clip shared by @dgoryeo earlier. No additional VAD-based alignments were applied and all arguments of transcribe() used the default values except for language.

[dgoryeo]

Thanks @ryanheise. I noticed that f572f21 was merged into openai:main 4 days ago. I will test it later this week to check.

[baxtree]

Upgraded to v20230918 and got the same time codes as those with v20230314, better than v20230124. srt-subtitles.zip
pip install openai-whisper==20230918
subaligner -m transcribe -v 'ItouJunji-ja-WhisperLarge+SileroVAD-0.25.wav' -ml jpn -mr whisper -mf large -o 'ItouJunji-ja-WhisperLarge-v20230918.srt'

[baxtree]

Update on v20231106 and model large-v3: Some degradation in transcription quality has been observed, at least for jpn as discussed around the above sample. Sticking to -mf large-v2 after upgrading to v20231106 brings better outcomes for this use case.
subaligner -m transcribe -v 'ItouJunji-ja-WhisperLarge+SileroVAD-0.25.wav' -ml jpn -mr whisper -mf large-v2 -o 'ItouJunji-ja-WhisperLarge-v20231106-large-v2.srt'
srt-subtitles.zip

[phineas-pta]

how do you deal with hallucinations from whisper?

[baxtree]

Good question. This is indeed a missing jewel atm and we plan to rely on the future releases of Whisper or its variants to do the right thing. Until then, QA personnel need to be involved in reviewing subtitles before their distribution. Apart from hallucinations, there are other aspects that need to be addressed to meet the editorial guideline anyway.

[lslatr23]

Is it possible to install Subaligner directly in Windows? When running the pip commands in installation guide I receive some errors, so I'am wondering if it is even worth tryin to install under Windows.

I have Linux but only in VMware as virtual machine, and the problem is that the guest OS (Linux) in VMware cannot access the GPU to accelerate AI workloads.

[phineas-pta]

when u have problem, instead of wondering, do:

• google / chatgpt at 1st
• if not work then raise issue in repo with full logs

concerning linux dev in windows, if u dont need gui then dont use virtual machine, use WSL2