Roadmap and tasks [OPEN FOR CONTRIBUTORS]

[akshhack]

Some areas where we expect the community to help in growing MARS5's capabilities are:
---> Improving inference stability and consistency
---> Speed/performance optimizations
---> Improving reference audio selection when given long references.
---> Benchmark performance numbers for MARS5 on standard speech datasets.

Specific tasks

• Profile the GPU and CPU memory and runtime speed metrics of the current model, add to readme.
• Port model operations not supported by MPS to equivalents to speed up apple mac inference. E.g. site-packages/torch/nn/functional.py:4840: UserWarning: The operator 'aten::col2im' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications.
• Cleanly add more performant ODE samplers to DDPM inference code (even just DPM++2M would be great).
• Make demo/ user-interface program to rapidly collect human preference ratings between two audio samples, one generated by the model, and one ground truth.
• Implement a way to do long-form generation. E.g. one possibility is to chunk long intput text into smaller pieces and then synthesize the codes each in turn, concatenating them, and vocoding the final result.

OPEN FOR CONTRIBUTIONS

[matbee-eth]

"Implement a way to do long-form generation. E.g. one possibility is to chunk long input text into smaller pieces and then synthesize the codes each in turn, concatenating them, and vocoding the final result.

Given a sentence+source audio is there any way to determine, roughly, the duration that the generated audio would be?

[RF5]

Hmm, not easily unfortunately. You could get an estimate by trying it a few times and come up with a trend, but it would be quite stochastic I think, and vary greatly depending on your inference parameters.

[agi-dude]

Also the possibility to train in other languages

[matus-pikuliak]

Implement a way to do long-form generation. E.g. one possibility is to chunk long intput text into smaller pieces and then synthesize the codes each in turn, concatenating them, and vocoding the final result.

I can look into this. I think the best approach would be to split the input into multiple chunks, optimally based on sentences so that the speech seems continuous, and then always feed the previous chunk as a reference for the model. There is also the problem of leading silences, which do not always seem to be filtered out with the trim.

[akshhack]

Great @matus-pikuliak I'll consider that you'll pick this up. Please keep us in the loop with your findings and please feel free to reach out to me directly on our Discord -- we can collaborate more closely there. Thanks!

[RF5]

Hi @matus-pikuliak , thanks for your interest! The trim() function has a top_db setting which controls how harsh the silence trimming is, it might be a simple test of searching for an optimal trim threshold. I think your plan of a 'sliding window' type approach could work quite well. Either that, or a 'fixed window' kind of thing where, for every sentence chunk, you use the same fixed reference audio as the first chunk.

[matus-pikuliak]

I have created a PR #44 for this problem. There are some design decisions that might need to be addressed there.

[elsatch]

I am quite interested into the "Make demo/ user-interface program to rapidly collect human preference ratings between two audio samples, one generated by the model, and one ground truth."

Are you looking to compare MARS5 TTS generated outputs against other TTS generated outputs (similar to the lmsys chatbot arena)?
Are you looking to compare human generated samples against MARS5 generated samples? Do you have any dataset located to test against?

[RF5]

We're mostly looking to get preference data to make a good fine tuning set with e.g. DPO or RHLF, so that mars could be fine-tuned to stop making silences, be more stable, etc.... So either human generated samples vs mars generated samples, or good vs bad mars samples. So long as we have tuples of (human-rated good sample, human-rated relatively worse sample), I think we can make it work

[AndreyMay]

• get added to TTS Arena for comparison against other solutions
  https://huggingface.co/spaces/TTS-AGI/TTS-Arena

[SiriuslySirius]

Possibility of being capable of streaming similar to what Coqui AI XTTSV2 can do?

Here is an example from their docs: https://docs.coqui.ai/en/latest/models/xtts.html#streaming-manually

This is very effective with LLM chatbots for real-time conversations.