Sentencepiece

[gandroz]

I wanted to add the sentencepiece featurizer from Google instead of using subword on tensorflow when training a conformer.

[nglehuy]

Please get the newest code and resolve conflicts 😄

[nglehuy]

Hi @gandroz, thank you for your contribution, I have some suggestions:

Firstly,
These lines:

def __init_upoints(self):
    text = [""]
    for idx in range(1, self.num_classes):
        text.append(self.model.decode_ids([idx]))
    self.upoints = tf.strings.unicode_decode(text, "UTF-8")
    self.upoints = self.upoints.to_tensor()  # [num_classes, max_subword_length]

Are used when the blank = 0 (which is the text = [""]). Can you either update the blank to 0 or update this function please.

Secondly, why do you repeat the class SentencePieceFeaturizer twice?

[gandroz]

Hmmm the duplicate was apparently due to a merge conflict, sorry.
I also corrected the blank to use the PAD tag of sentencepiece. I noticed the blank is hard-coded to 0, maybe it could be configurable of class-dependent ?
Finally, I added some tests to be sure the featurizer is consistent with the subword featurizer.

[nglehuy]

@gandroz The blank is not suppose to be hard-coded to 0 (yeah the SubwordFeaturizer has the blank hard-coded to 0 so that the blank matches upoint's blank), only the unicode points have the blank is always 0, which is the NULL character \000. The upoints is an array which indices are the ids, if the blank is the id 3 then the upoints[3] = 0. The purpose of upoints is to support tflite to decode to Unicode Points instead of ids or string (tf.string is not supported yet in tflite) and in the future it might be used for multi-languages ASR.
Furthermore, according to the author of warp-transducer, we should use blank=0 for better performance.

[nglehuy]

About this file. You should name it as the corpus that you create subwords from 😄

[nglehuy]

@gandroz Please change the name of the vocabularies/conformer.subwords 😄

[gandroz]

@usimarit actually, it is your file which stands in your Drive that I used to perform some (unit)tests

[nglehuy]

@gandroz Oh I see 😄 But that file contains subwords from test-clean too, so it's not quite true, you should use the vocabularies/librispeech_train_4_1030.subwords and remove that vocabularies/conformer.subwords

[gandroz]

@usimarit ok no problem. Anyway, that's just for some tests ;) not for performances

[nglehuy]

@gandroz I just want to remove unnecessary components to keep the repo clean 😄