Simultaneous translation

[RomanKoshkin]

Hi,
Do I understand correctly that this script:

python3 simulstreaming_whisper.py \
   some.wav \
   --language de  \
   --task translate \
   --vac \
   --logdir path/to/logs \
   --log-level WARNING

is NOT designed for en-X translation, but can only work for X-en (as Whisper was trained for X-en)? In other words, running the above script will still output just an English transcript, right? If so, could you provide an example of end-to-end speech-to-text translation?

[Gldkslfmsd]

hi,
yes, you're right. Whisper can do only xx-en translation or ASR in many languages including de. There's no direct en-de translation in SimulStreaming at this moment, but:

• it's planned
• there is EuroLLM in a cascade following Whisper ASR. See translate dir, and translate-server branch.

[RomanKoshkin]

Thanks, but it says here https://github.com/ufal/SimulStreaming/tree/translate-server/translate, in the "Usage" section:

- input can be either text file with timestamps in specific format
...

- or *instance.log format of IWSLT2025 
...

So, am I right that the system cannot translate audio directly to a text? The way it's done now is

1. I need to run Whisper to get some timestamped transcripts
2. then translate the trancript with EuroLLM by running a separate script (with the path to the timestamps provided as a CLI parameter to that second script)?

I just thought that since you submitted to IWSLT25, wasn't it supposed to be some self-contained docker image that would do everything in one pass to get BLEU and StreamLAAL? Sorry if I misunderstand something. Thanks!

[Gldkslfmsd]

Yes, it's cascade solution, intended for several parallel processes on 2 GPUs. It's not documented yet, therefore this branch is not merged.

There are two processing options:

• simulation from file: first process audio with simulstreaming_whisper.py, save output as text file, then process it with simulstreaming_translate.py .
• real-time from microphone: start simulstreaming_whisper_server.py and simulstreaming_translate_server.py (on 2 GPUs, possibly on two machines), then arecord -> netcat to whisper server -> netcat to translate server . As -> I mean a bash pipe or this pipeliner for complex pipeline with forks for multi-targetting. One translate server on one GPU serves one target language.

The code for the translate server is not cleaned yet, I need to change the message format to json to send end-of-voice tags in other channel than the content. That's another reason for unmerged branch.

In IWSLT25, I chose to submit only the logs, no docker container. But I made it working in an end-to-end demo: https://elitr.eu/iwslt25/ . It's down at this moment but I you can ping me to start it.

[RomanKoshkin]

Thanks Dominik. I understand. I just thought to use your system as a baseline (just plug and play it into simuleval, give it some audio files and reference translations to get BLEU and (Stream)LAAL, but it looks a bit more complicated for now.

[Gldkslfmsd]

it should work, it should be documented enough so that you can do it. Feel free to specify what exactly should I fix.