Inconsistent Transcription with Whisper Turbo on Kubernetes Using NVIDIA T4 GPU

[abhijith-zupaloop]

I am experiencing inconsistent transcription results when running the Whisper Turbo model on a Kubernetes node equipped with an NVIDIA T4 (16GB) GPU. The same model, when run on my laptop with an RTX 2000 Ada (8GB) GPU, produces accurate transcripts without any issues.

On the Kubernetes setup, the transcripts often contain incorrect mappings, repeated segments, or are even blank at times. However, on my laptop, the transcriptions are accurate and as expected.

Steps Taken

• Verified the MD5 checksums of input files to ensure no data corruption.
• Confirmed the same model version and input data are used in both environments.
• No errors or warnings were observed in the logs during inference on either environment.

Environment Details
Laptop:

• GPU: RTX 2000 Ada (8GB)
• Model: Whisper Turbo
• Transcriptions: Accurate

Kubernetes Node:

• GPU: NVIDIA T4 (16GB)
• Model: Whisper Turbo
• Transcriptions: Inconsistent (incorrect mappings, repetitions, or blanks)

Expected Behavior
The transcription results from the Kubernetes node should match the accuracy and quality of the results produced on the laptop GPU.

Actual Behavior
The transcripts on the Kubernetes node are inconsistent and often incorrect.

Request
I would appreciate any guidance on:

1. Possible differences in GPU behavior/configuration that could cause this issue.
2. Debugging steps or tools to further investigate the problem.
3. Any known issues with running Whisper Turbo on NVIDIA T4 GPUs or within Kubernetes environments.

Thank you for your help!

[edsu]

That's interesting that you observed better results on your laptop. I've also observed non-deterministic output across identical runs using T4 (in AWS Batch) and the large-v3 model.

Just out of curiosity, are your Python environments, and CUDA versions equivalent on your laptop and the T4 machine?