How to improve inference latency performance?

[twocode]

2024-10-22 03:26:36.033 | INFO     | app:generate_audio:73 - Refined text: ['but since [uv_break] 波 卡 [uv_break] like [uv_break] like 里 法, like pocari sweat, [uv_break] the drink. [uv_break], and [uv_break] 东 方 民 族, [uv_break] eastern cultures and peoples, are super different,']
2024-10-22 03:26:36.033 | INFO     | app:generate_audio:78 - Start voice inference.
text:  16%|█▌        | 62/384(max) [00:01, 56.04it/s]
code:  30%|██▉       | 606/2048(max) [00:10, 55.61it/s]
2024-10-22 03:26:48.069 | INFO     | app:generate_audio:91 - Inference completed.

This simple sentence took 12 seconds on Nvidia Tesla T4. Is it correct to assume ChatTTS is not suitable for situations that require low "Time To First Audio(TTFA)"?

[medemi68]

I second this. I think there is some major improvement needed to get the Time to first audio down pat. I was able to do some optimization personally by reducing the chunk size and setting the stream speed. I was running it on a 4090 which has 80 TFLOPS or so, and I was able to get the inference speed a lot faster. But definitely you would need to use streams to get a fast time to first byte, disable the refine text portion too.