Text Normalizer & Reproducing Results on FLEURS

[roudimit]

Hi, thanks for open-sourcing the model / code!

I am trying to reproduce the zero-shot results on FLEURS. Below is a summary of my results.
My results with Large-V2 are pretty close, but the Small and Tiny models deviate quite significantly.
I found the best results are when the text normalizer is applied to both the model’s predictions and the reference text. Is that correct, or should it only be applied to either the prediction or the reference?
Also, I found the results were better with the text from FLEURS that was normalized by Google, and then applying the Whisper normalizer.

I think the reported Japanese WER results should be compared to our CER results since spaces were added in the Whisper evaluation.

Would you have any other suggestions?
Here's my code. My label file contains the space-separated chars from FLEURS.
Ie. the first row in en_us/test.tsv is
1980 12741024238657315067.wav Unfortunately, studying traffic flow is difficult because driver behavior cannot be predicted with one-hundred percent certainty. unfortunately studying traffic flow is difficult because driver behavior cannot be predicted with one-hundred percent certainty u n f o r t u n a t e l y | s t u d y i n g | t r a f f i c | f l o w | i s | d i f f i c u l t | b e c a u s e | d r i v e r | b e h a v i o r | c a n n o t | b e | p r e d i c t e d | w i t h | o n e - h u n d r e d | p e r c e n t | c e r t a i n t y | 202240 MALE
And I use this as the label: u n f o r t u n a t e l y | s t u d y i n g | t r a f f i c | f l o w | i s | d i f f i c u l t | b e c a u s e | d r i v e r | b e h a v i o r | c a n n o t | b e | p r e d i c t e d | w i t h | o n e - h u n d r e d | p e r c e n t | c e r t a i n t y |

import os
import sys
import whisper
from whisper.normalizers import EnglishTextNormalizer, BasicTextNormalizer
import editdistance
from tqdm import tqdm

wav_scp = sys.argv[1]
text_tok = sys.argv[2]
lang = sys.argv[3]

print('Loading model')
model = whisper.load_model("tiny")

if lang == 'english' or 'English':
    std = EnglishTextNormalizer()
else:
    std = BasicTextNormalizer()

utt2txt = {}
with open(text_tok) as f:
    for i, line in enumerate(f):
        utt2txt[i] = line.rstrip()

c_err, c_len, w_err, w_len = 0, 0, 0, 0

with open(wav_scp) as f:
    for i, line in enumerate(tqdm(f)):
        if i == 0:
            audio_root = line.rstrip() 
            continue
        wav, _ = line.rstrip().split('\t')

        pred = model.transcribe(os.path.join(audio_root, wav), language=lang)['text']
        pred = std(pred) # normalize the prediction
        pred_words = pred.split()
        pred_units = pred.replace(' ', '|').replace('', ' ').split()

        # normalize the reference for whisper, labels are chars-space separated
        gt_words = utt2txt[i - 1].replace(' ', '').replace('|', ' ').rstrip() # convert to regular words
        gt_words = std(gt_words) # normalize
        gt_units = gt_words.replace(' ', '|').replace('', ' ').split() # get the chars-space separated again
        gt_words = gt_words.split()

        c_err += editdistance.eval(pred_units, gt_units)
        c_len += len(gt_units)

        w_err += editdistance.eval(pred_words, gt_words)
        w_len += len(gt_words)
        print('HYPO: {}'.format(pred_words))
        print('REF: {}'.format(gt_words))

wer = 100. * w_err/w_len
print("WER: %.4f" % wer)
cer = 100. * c_err/c_len
print("CER: %.4f" % cer)```

[jongwook]

Hi! You're correct, for the languages that don't typically use spaces, we used character error rates. This is mentioned in Appendix C, but I realize it should've been clearer in the tables as well.

[roudimit]

@jongwook thanks! Do you have any idea about why my reproduced results are worse than the results in the paper (across all of the languages in my table)? My results with all of the models except Large are noticeably worse.

[jongwook]

The main difference seems that in our evaluation we used the transcription column without needing to handle the | characters and such, but I'm not totally sure if that was the culprit.

[roudimit]

Got it, thanks. That is also the text that I'm using for the evaluation, so maybe there's some other difference. Is the Whisper text normalizer applied to both the model's predictions and the reference text?

[roudimit]

Update: I think the python API doesn't set beam_size=5, best_of=5, which are used in the paper - would be good to double check if those are set!

[DiwakarBasnet]

Hey sorry this is totally not related to the topic but I am confused and had a question, is the model pre trained on fleurs or is it just used for evaluation purpose? Can I fine tune large-v3 on fleurs and common voice?