reproduce Ultravox0.4.1

[Gpwner]

the weights you share on huggingface need to change the stack_factor to 10 https://huggingface.co/fixie-ai/ultravox-v0_4_1-llama-3_1-8b/blob/main/ultravox_config.py#L103

[farzadab]

Hmm, why do you say that? The stack_factor we use is 8, not 10.

[Gpwner]

I noticed that the README mentions using whisper/medium(which is the reason i think it should be 10), but the release log actually states that openai/whisper-large-v3-turbo is being used. Also, I would like to ask if the release_config.yaml should be modified like this?

text_model: "meta-llama/Meta-Llama-3.1-8B-Instruct"
audio_model: "openai/whisper-large-v3-turbo"

loss_config:
  loss_function: "KL_Divergence"

train_sets:
  - name: librispeech-clean-transcription # ASR
  - name: librispeech-other-transcription # ASR
  - name: librispeech-clean-continuation # continuation
  - name: librispeech-other-continuation # continuation
  - name: commonvoice-en-transcription # ASR
  - name: commonvoice-ar-transcription # ASR
  - name: commonvoice-de-transcription # ASR
  - name: commonvoice-es-transcription # ASR
  - name: commonvoice-fr-transcription # ASR
  - name: commonvoice-it-transcription # ASR
  - name: commonvoice-ja-transcription # ASR
  - name: commonvoice-pt-transcription # ASR
  - name: commonvoice-ru-transcription # ASR
  - name: commonvoice-hi-transcription # ASR
  - name: commonvoice-tr-transcription # ASR
  - name: commonvoice-sv-transcription # ASR
  - name: commonvoice-uk-transcription # ASR
  - name: commonvoice-en-continuation # continuation
  - name: commonvoice-ar-continuation # continuation
  - name: commonvoice-de-continuation # continuation
  - name: commonvoice-es-continuation # continuation
  - name: commonvoice-fr-continuation # continuation
  - name: commonvoice-it-continuation # continuation
  - name: commonvoice-ja-continuation # continuation
  - name: commonvoice-pt-continuation # continuation
  - name: commonvoice-ru-continuation # continuation
  - name: commonvoice-hi-continuation # continuation
  - name: commonvoice-tr-continuation # continuation
  - name: commonvoice-sv-continuation # continuation
  - name: commonvoice-uk-continuation # continuation

  - name: peoplespeech-clean-transcription # ASR
  - name: peoplespeech-clean-continuation # continuation

# Temporarily remove heysquad_human from val_sets as it causes the training to fail.
val_sets:
  - name: covost2-en-de
  - name: covost2-zh-en
  - name: peoplespeech-clean-transcription

[farzadab]

Yes, you're right that the release_config should be updated to include the datasets and and the audio model. Hopefully we can do that on the next release. I doubt we're using any of the commonvoice-*-transcription tasks however.

[Gpwner]

thanks.i will give a try to reploduce ultravox0.4.1 base on the release_config.yaml:

text_model: "meta-llama/Meta-Llama-3.1-8B-Instruct"
audio_model: "openai/whisper-large-v3-turbo"

loss_config:
  # Choose from ["KL_Divergence", "CrossEntropy"], default is "KL_Divergence"
  loss_function: "KL_Divergence"

train_sets:
  - name: librispeech-clean-transcription # ASR
  - name: librispeech-other-transcription # ASR
  - name: librispeech-clean-continuation # continuation
  - name: librispeech-other-continuation # continuation
  - name: commonvoice-en-continuation # continuation
  - name: commonvoice-ar-continuation # continuation
  - name: commonvoice-de-continuation # continuation
  - name: commonvoice-es-continuation # continuation
  - name: commonvoice-fr-continuation # continuation
  - name: commonvoice-it-continuation # continuation
  - name: commonvoice-ja-continuation # continuation
  - name: commonvoice-pt-continuation # continuation
  - name: commonvoice-ru-continuation # continuation
  - name: commonvoice-hi-continuation # continuation
  - name: commonvoice-tr-continuation # continuation
  - name: commonvoice-sv-continuation # continuation
  - name: commonvoice-uk-continuation # continuation

  - name: peoplespeech-clean-transcription # ASR
  - name: peoplespeech-clean-continuation # continuation

# Temporarily remove heysquad_human from val_sets as it causes the training to fail.
val_sets:
  - name: covost2-en-de
  - name: covost2-zh-en
  - name: peoplespeech-clean-transcription

[Gpwner]

@farzadab hi, when download commonvoice-hi-continuation i got some error,can you help? thanks

datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=275394930.996, num_examples=9378, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=124766946, num_examples=4689, shard_lengths=None, dataset_name='common_voice_17_0')}, {'expected': SplitInfo(name='validation', num_bytes=145392985.176, num_examples=4856, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validation', num_bytes=70206799, num_examples=2428, shard_lengths=None, dataset_name='common_voice_17_0')}, {'expected': SplitInfo(name='test', num_bytes=220164125.264, num_examples=6308, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='test', num_bytes=102224721, num_examples=3154, shard_lengths=None, dataset_name='common_voice_17_0')}, {'expected': SplitInfo(name='other', num_bytes=253400896.056, num_examples=8088, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='other', num_bytes=139869493, num_examples=4044, shard_lengths=None, dataset_name='common_voice_17_0')}, {'expected': SplitInfo(name='invalidated', num_bytes=53706876.0, num_examples=1550, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='invalidated', num_bytes=26878451, num_examples=775, shard_lengths=None, dataset_name='common_voice_17_0')}, {'expected': SplitInfo(name='validated', num_bytes=721036368.28, num_examples=20658, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validated', num_bytes=298616650, num_examples=10329, shard_lengths=None, dataset_name='common_voice_17_0')}]

[Gpwner]

What is the approximate final loss ? thanks @farzadab @zqhuang211

[zqhuang211]

Depending on the training data, choice of whisper encoder, and number of epochs, the training loss can be anywhere 0.1 and 0.05. We haven’t seen cases where training loss drops below 0.1 and then goes up significantly.

[Gpwner]

@zqhuang211 Could you help point out where exactly the issue is in the current code? I'd like to try fixing it.

[Gpwner]

@zqhuang211 When you train Ultravox based on a text-based LLM, do you first need to generate a continuation dataset(transcribe the audio into text first) using the text-based LLM? For example, when training Ultravox based on meta-llama/Meta-Llama-3.1-8B-Instruct, do you need to generate the continuation dataset with the text-based LLM first before using it to train Ultravox?

[zqhuang211]

The issue with the ST dataset (CoVoST 2) is that some audio samples are empty, which can trigger an exception in the Hugging Face dataloader. Although this exception is handled to allow training to continue, it results in only a limited subset of ST dataset samples being used for training.

[zqhuang211]

yes, our current approach relies on continuations generated by a text-based LLM (the same LLM used in ultravox or a different LLM).

[carter54]

@zqhuang211 Hi~ When training an ultravox model based on llama3.1-8B-instruct, is it required that the responses in the continuation dataset are generated using llama3.1-8B-instruct? If not required, will it introduce additional difficulty to the training?

[Gpwner]

@farzadab I'm excited that you've release the latest Ultravox v0.5 weights. I was wondering if you could share the release_config.yaml used during training.Thanks.