# unpacked small.tar should go here:
+datadir = Path('/mnt/')
+# you can download it downloaded from
+# https://github.com/facebookresearch/libri-light/blob/main/data_preparation/README.md
{'__key__': './dev-clean/1272/128104/1272_128104_000001_000000',
+ '__url__': '/data2/libritts-r-raw-000000.tar',
+ 'normalized.txt': "A 'JOLLY' ART CRITIC",
+ 'original.txt': "A 'JOLLY' ART CRITIC",
+ 'wav': (tensor([[ 0.0000, 0.0000, 0.0000, ..., -0.0036, -0.0038, -0.0050]]),
+ 24000)}
+
+
+
+
# test it locally
+input:str='https://huggingface.co/datasets/collabora/librilight-webdataset/resolve/main/librilight-large-wo6454-flac-000002.tar'
+output:str=input.rsplit("/", 1)[1].replace('flac', 'vad') +".gz"
+
+ds = load_dataset(input)
+vad_model = whisperx.vad.load_vad_model('cuda')
+
+with wds.TarWriter(output) as sink:
+for s in progress_bar(ds, total=10):
+ audio, sr = s['audio']
+assert(sr ==16000)
+ sink.write({
+"__key__": s['__key__'],
+"vad.npy": np.array(segment_audio(vad_model, audio), dtype=np.float32)
+ })
+
+!ls -lh {output}
+!tar tf {output}
+
+
Lightning automatically upgraded your loaded checkpoint from v1.5.4 to v2.0.2. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint --file ../../../.cache/torch/whisperx-vad-segmentation.bin`
+
+
+
Model was trained with pyannote.audio 0.0.1, yours is 2.1.1. Bad things might happen unless you revert pyannote.audio to 0.x.
+Model was trained with torch 1.10.0+cu102, yours is 2.0.1+cu118. Bad things might happen unless you revert torch to 1.x.
+-rw-r--r-- 1 root root 7.5K Sep 21 08:51 librilight-large-wo6454-vad-000002.tar.gz
+large/10089/five_minutes_stories_1508_librivox_64kb_mp3/5minutesstories_03_molesworth_64kb.vad.npy
+large/10089/five_minutes_stories_1508_librivox_64kb_mp3/5minutesstories_04_molesworth_64kb.vad.npy
+large/10089/five_minutes_stories_1508_librivox_64kb_mp3/5minutesstories_08_molesworth_64kb.vad.npy
+large/10089/five_minutes_stories_1508_librivox_64kb_mp3/5minutesstories_09_molesworth_64kb.vad.npy
+large/10089/five_minutes_stories_1508_librivox_64kb_mp3/5minutesstories_10_molesworth_64kb.vad.npy
+large/10089/five_minutes_stories_1508_librivox_64kb_mp3/5minutesstories_11_molesworth_64kb.vad.npy
+large/10089/goodcheerstories_1511_librivox_64kb_mp3/goodcheerstories_13_dickinson_64kb.vad.npy
+large/10089/goodcheerstories_1511_librivox_64kb_mp3/goodcheerstories_30_dickinson_64kb.vad.npy
+large/10089/mothers_nursery_tales_1512_librivox_64kb_mp3/mothers_nursery_tales_16_pyle_64kb.vad.npy
+large/10089/mothers_nursery_tales_1512_librivox_64kb_mp3/mothers_nursery_tales_25_pyle_64kb.vad.npy
+
+
+
+
+
+
+
+
+
+ 100.00% [10/10 00:10<00:00]
+
+
+
+
+
+
Batch processing
+
Let’s put everything above together.
+
+
# for reference, this was the performance on a single 4090:
+process_shard('https://huggingface.co/datasets/collabora/librilight-webdataset/resolve/main/librilight-small-flac-000000.tar')
+
+
Lightning automatically upgraded your loaded checkpoint from v1.5.4 to v2.0.2. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint --file ../../../.cache/torch/whisperx-vad-segmentation.bin`
+
+
+
Model was trained with pyannote.audio 0.0.1, yours is 2.1.1. Bad things might happen unless you revert pyannote.audio to 0.x.
+Model was trained with torch 1.10.0+cu102, yours is 2.0.1+cu118. Bad things might happen unless you revert torch to 1.x.
+
+
+
+
+
+
+
+
+
+ 100.00% [335/335 03:30<00:00]
+
+
+
+
+
+
for x in wds.WebDataset('/data2/libritts-r-vad-000000.tar').decode(): break
+x['__key__'].split('/')
Precompute Whisper transcriptions for VQ bottleneck distilation
+
Doing transcription means sampling from the Whisper auto-regresive decoder. This is too slow to do for each training batch. Fortunately the trainscriptions are small text snippets so we can precompute them once for the whole dataset.
+
We use segments from Voice Activity Detection to reduce any boundary issues, the we use webdataset to yields multiple chunks from a FLAC file we only load once. The VAD segments are merged into longer chunks to make Whisper processing more efficent (it always processes 30s at a time)
You can pass in either a URL or a local file name. Either way it will expect a vad file in the local directory. The result will go into a file in the current directory named after the source file but replacing flac with txt.
(['cc/7 Sec Riddles/[-hIfETsPxPg] New TYPE Of Riddles： Can You Ace Our New Game？ ？_023',
+ 'cc/Bon Appetit/[jmntzm5yBYY] Melissa Makes Chicken Afritada ｜ From the Home Kitchen ｜ Bon Appétit_037',
+ 'cc/7 Sec Riddles/[-hIfETsPxPg] New TYPE Of Riddles： Can You Ace Our New Game？ ？_009',
+ 'cc/Bon Appetit/[jmntzm5yBYY] Melissa Makes Chicken Afritada ｜ From the Home Kitchen ｜ Bon Appétit_010',
+ 'cc/7 Sec Riddles/[-hIfETsPxPg] New TYPE Of Riddles： Can You Ace Our New Game？ ？_004',
+ 'cc/Bon Appetit/[jmntzm5yBYY] Melissa Makes Chicken Afritada ｜ From the Home Kitchen ｜ Bon Appétit_049',
+ 'cc/Bon Appetit/[jmntzm5yBYY] Melissa Makes Chicken Afritada ｜ From the Home Kitchen ｜ Bon Appétit_000',
+ 'cc/Bon Appetit/[jmntzm5yBYY] Melissa Makes Chicken Afritada ｜ From the Home Kitchen ｜ Bon Appétit_024',
+ 'cc/7 Sec Riddles/[-hIfETsPxPg] New TYPE Of Riddles： Can You Ace Our New Game？ ？_033',
+ 'cc/Bon Appetit/[jmntzm5yBYY] Melissa Makes Chicken Afritada ｜ From the Home Kitchen ｜ Bon Appétit_034',
+ 'cc/Bon Appetit/[jmntzm5yBYY] Melissa Makes Chicken Afritada ｜ From the Home Kitchen ｜ Bon Appétit_012',
+ 'cc/Bon Appetit/[jmntzm5yBYY] Melissa Makes Chicken Afritada ｜ From the Home Kitchen ｜ Bon Appétit_052',
+ 'cc/Bon Appetit/[jmntzm5yBYY] Melissa Makes Chicken Afritada ｜ From the Home Kitchen ｜ Bon Appétit_007',
+ 'cc/7 Sec Riddles/[-hIfETsPxPg] New TYPE Of Riddles： Can You Ace Our New Game？ ？_030',
+ 'cc/Bon Appetit/[jmntzm5yBYY] Melissa Makes Chicken Afritada ｜ From the Home Kitchen ｜ Bon Appétit_055',
+ 'cc/Bon Appetit/[jmntzm5yBYY] Melissa Makes Chicken Afritada ｜ From the Home Kitchen ｜ Bon Appétit_006'],
+ tensor([[-0.0154, -0.0289, -0.0376, ..., 0.0000, 0.0000, 0.0000],
+ [-0.0035, -0.0058, -0.0082, ..., 0.0000, 0.0000, 0.0000],
+ [-0.0082, -0.0150, -0.0179, ..., 0.0000, 0.0000, 0.0000],
+ ...,
+ [-0.0018, 0.0017, 0.0005, ..., 0.0000, 0.0000, 0.0000],
+ [ 0.0014, 0.0021, 0.0019, ..., 0.0000, 0.0000, 0.0000],
+ [ 0.0055, 0.0106, 0.0086, ..., 0.0000, 0.0000, 0.0000]]),
+ tensor([ 5.1536, 8.5666, 2.2867, 22.5939, 1.7406, 22.4744, 2.2355, 2.3549,
+ 2.0307, 18.0717, 6.4505, 2.1843, 1.6382, 5.5461, 2.6450, 29.1297]))
Precompute Whisper transcriptions for VQ bottleneck distilation
+
Doing transcription means sampling from the Whisper auto-regresive decoder. This is too slow to do for each training batch. Fortunately the trainscriptions are small text snippets so we can precompute them once for the whole dataset.
+
We use segments from Voice Activity Detection to reduce any boundary issues, the we use webdataset to yields multiple chunks from a FLAC file we only load once. The VAD segments are merged into longer chunks to make Whisper processing more efficent (it always processes 30s at a time)
You can pass in either a URL or a local file name. Either way it will expect a vad file in the local directory. The result will go into a file in the current directory named after the source file but replacing flac with txt.
+
+
+
+
The autoreload extension is already loaded. To reload it, use:
+ %reload_ext autoreload
# load some VAD ouputs
+ds = wds.WebDataset(
+ vad.flac_to_vad_name(flac_url)
+).decode().to_tuple('vad.npy')
+chunks = [x[0] for x in progress_bar(ds, total='noinfer')]
+
+
+
+
+
+
+
+
+ 100.00% [335/335 00:00<00:00]
+
+
+
+
+
+
# quick test
+len(chunks[0]), len(chunk_merger(chunks[0]))
+
+
(46, 28)
+
+
+
+
plt.hist([te-ts for x in chunks for ts,te in x])
+plt.title('Segment length distribution straight out of the VAD algorithm');
+
+
+
+
+
+
+
+
+
+
plt.hist([te-ts for x in chunks for ts,te in chunk_merger(x)]);
+plt.title('Chunk length distribution after greedy merging');
+
+
+
+
+
+
+
+
+
+
(np.array([te-ts for x in chunks for ts,te in chunk_merger(x)]) <10).mean()
+
+
0.03671825647504738
+
+
+
In the above distribution only 3,7% of the samples have < 10 seconds. We noticed that this limits the ability of the T2S model to generate short sequences reliably.
+
It does not seem to matter for quantizing Whisper so we can keep this distribution (it uses less compute for training).
+
For T2S we can add some more shorter chunks at random:
+
+
plt.hist([te-ts for x in chunks for ts,te in chunk_merger(x, random_cutter)])
+plt.title('Chunk length distribution after randomized merging');
+
+
+
+
+
+
+
+
+
+
+
Merge the FLAC and VAD datasets
+
First we want to merge the VAD dataset with the FLAC audio data.
for x in progress_bar(txt_ds, total=10):
+ IPython.display.display(IPython.display.Markdown(f"#### {x['__key__']} chunk {x['i']} of {x['imax']}"))
+ fname =f"test-{x['i']}.ogg"
+ torchaudio.save(fname, x['samples'][None,:int((x['tend']-x['tstart'])*s['sample_rate'])], s['sample_rate'])
+ IPython.display.display(IPython.display.Audio(url=fname, rate=x['sample_rate']))
+ IPython.display.display(IPython.display.Markdown(x['txt']))
+
+
+
+
+
+
+
+
+ 100.00% [10/10 00:02<00:00]
+
+
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_000 chunk 0 of 115
+
+
+
+
+
+
+
+
Rozdział 22. Stare mięśca, nowi ludzie. Stierfort i ja zabawiliśmy dwa tygodnie w tamtej okolicy.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_001 chunk 1 of 115
+
+
+
+
+
+
+
+
Byliśmy prawie ciągle razem. Czasem tylko rozstawaliśmy się na kilka godzin. Styrford, bowiem, był zawołanym żeglarzem. Ja zaś nie smakowałem zbytnia o w tego rodzaju rozrywkach. To też gdy przyjaciel mój puszczał się w towarzystwie pana PegoTi na morze, pozostawałem zwykle na lądzie. Korzystanie z pokoiku u PegoTi krępowało mnie nieco.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_002 chunk 2 of 115
+
+
+
+
+
+
+
+
Wiedząc, jak dalece przez dzień cały jest zajęta do oglądaniem chorego męża, wracałem wcześniej bieczorem, gdy Steelfort, będąc panem swego czasu, niczym się nie krempował. To wiedziałem się też, że gdy już spał w najlepsze, on podejmował rybaków w ulubionej winiarni pana PegoT pod dobrą chęcią, lub wyrybaczkim odzieniu spędzał całe księżycowe nocy na morzu.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_003 chunk 3 of 115
+
+
+
+
+
+
+
+
Wiedziałem, że jego żywa gorąca natura potrzebuje rozrywek i niebezpieczeństw i postępowanie jego wcale mnie nie dziwiło. Rostawaliśmy się i z tego jeszcze powodu, że Stirforta nie mogły pociągać tak jak mnie wycieczki do Blanderstone. To też czasem, że gnaliśmy się po wczesnym śniadaniu, a schodzi i dopiero późno na obiad. Nie miałem pojęcia co robił, czym się wówczas zajmował.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_004 chunk 4 of 115
+
+
+
+
+
+
+
+
Wiedziałem tylko, że znany był i lubiany przez wszystkich, a posiadał darż, szczególny wynajdwanie rozrywek i zajęcia nawet tam, gdzie inny na jego miejscu nic będzie nie znalazł. Co do mnie przebiegając drogę do Blanderstom, przeżywałem w pamięci każdy dzień z przeszłości i to mi wypełniało myśl i serce. Przypominało mi się każde niegdyś tu przeżyte wrażenie.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_005 chunk 5 of 115
+
+
+
+
+
+
+
+
nadmogiło pod drzewami, gdzie spoczywali moi rodzice, zamienioną przez pegoty w kwietnik, na którą gdybyła tylko jeszcze miejscem wiecznego z poczynku megajca, spoglądałem z takim żalem, i którą widział otwartą na przyjęcie z włog, mojej pięknej, kochanej matki jej dzieciątka, długie, spędzałem godziny. Lężała ona na uboczu, wrogu cmentarze.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_006 chunk 6 of 115
+
+
+
+
+
+
+
+
Przechodząc z drogą, czytać mogłem wypisane na kamieniu nazwiska, a dzwonkościelny zdawał się być głosem pożegnania. W godzinach tych, myśląc o nich.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_007 chunk 7 of 115
+
+
+
+
+
+
+
+
Myślałem zarazem o tym zawsze, jakie miejsce zajmę w życiu, jakich wielkich lub dobrych dokonam czynów. Ech okroków moich nie odbijało nud innych, te tylko, jak gdybym krocząc, ubokużyjącej jeszcze matki, nad powietrzne budował zamki. Stary dom nasz zmienił się niedopoznania.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_008 chunk 8 of 115
+
+
+
+
+
+
+
+
Znikły bez śladu, powiżone przez wrony dawno opuszczone gniazda, adrzewa, strzyżone i ścinane utraciły, dawny kształt, ogród zdziczał, połowa okien była zabita. Dom zamieszkany został przez jakiegoś chorego umysłowo-gentelmena ich, tych, co go doglądali.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_009 chunk 9 of 115
+
+
+
+
+
+
+
+
Chory przesiadawał zwykle w okniem, i niech ktoś pokójku i spoglądał na cmentarz. Ciekawy byłem, czy też jego myśli, kreślą te same obrazy, co moje, gdy w czeznym rankiem, w nocnej koszulce wyglądałem tym okienkiem, witając pasące się o wschodzie słońca trzody. Dawni nasi sąsiedzi, Państwu Grraper,
+
+
+
+
for x in progress_bar(txt_ds, total=10):
+ IPython.display.display(IPython.display.Markdown(f"#### {x['__key__']} chunk {x['i']} of {x['imax']}"))
+ fname =f"test-{x['i']}.ogg"
+ torchaudio.save(fname, x['samples'][None,:int((x['tend']-x['tstart'])*s['sample_rate'])], s['sample_rate'])
+ IPython.display.display(IPython.display.Audio(url=fname, rate=x['sample_rate']))
+ IPython.display.display(IPython.display.Markdown(x['txt']))
+
+
+
+
+
+
+
+
+ 100.00% [10/10 00:02<00:00]
+
+
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_000 chunk 0 of 115
+
+
+
+
+
+
+
+
Rozdział dwudziesty drugi Stare miejsca, nowi ludzie Styrford i ja zabawiliśmy dwa tygodnie w tamtej okolicy.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_001 chunk 1 of 115
+
+
+
+
+
+
+
+
Byliśmy prawie ciągle razem. Czasem tylko rozstawaliśmy się na kilka godzin. Styrford bowiem był zawołanym żeglarzem, ja zaś nie smakowałem zbytnio w tego rodzaju rozrywkach. Toteż gdy przyjaciel mój puszczał się w towarzystwie pana Pegoty na morze, pozostawałem zwykle na lądzie. Korzystanie z pokoiku u Pegoty krępowało mnie nieco.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_002 chunk 2 of 115
+
+
+
+
+
+
+
+
Wiedząc, jak dalece przez dzień cały jest zajęta doglądaniem chorego męża, wracałem wcześniej wieczorem, gdy Stilford, będąc panem swego czasu, niczym się nie krępował. Dowiedziałem się też, że gdybym już spał w najlepsze, on podejmował rybaków w ulubionej winiarni pana Pegoty pod dobrą chęcią lub w rybackim odzieniu spędzał całe księżycowe noce na morzu.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_003 chunk 3 of 115
+
+
+
+
+
+
+
+
Wiedziałem, że jego żywa, gorąca natura potrzebuje rozrywek i niebezpieczeństw i postępowanie jego wcale mnie nie dziwiło. Rozstawaliśmy się i z tego jeszcze powodu, że z Tyrforda nie mogły pociągać, tak jak mnie, wycieczki do Blunderstown. To też czasem żegnaliśmy się po wczesnym śniadaniu, a schodzili dopiero późno na obiad. Nie miałem pojęcia, co robił, czym się wówczas zajmował.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_004 chunk 4 of 115
+
+
+
+
+
+
+
+
Wiedziałem tylko, że znany był i lubiany przez wszystkich, a posiadał dar szczególny wynajdywanie rozrywek i zajęcia nawet tam, gdzie inny na jego miejscu nic by nie znalazł. Co do mnie, przebiegając drogę do Blunderstone, przeżywałem w pamięci każdy dzień z przeszłości i to mi wypełniało myśl i serce. Przypominało mi się każde niegdyś tu przeżyte wrażenie.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_005 chunk 5 of 115
+
+
+
+
+
+
+
+
Nad mogiłą pod drzewami, gdzie spoczywali moi rodzice, zamienioną przez pegot i w kwietnik, na którą, gdy była tylko jeszcze miejscem wiecznego spoczynku mego ojca, spoglądałem z takim żalem i którą widział otwartą na przyjęcie zwłok mojej pięknej, kochanej matki i jej dzieciątka. Długie spędzałem godziny. Leżała ona na uboczu, w rogu cmentarza.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_006 chunk 6 of 115
+
+
+
+
+
+
+
+
Przechodząc drogą, czytać mogłem wypisane na kamieniu nazwiska, a dzwoń kościelny zdawał się być głosem pożegnania. W godzinach tych, myśląc o nich…
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_007 chunk 7 of 115
+
+
+
+
+
+
+
+
Myślałem zarazem o tym zawsze, jakie miejsce zajmę w życiu, jakich wielkich lub dobrych dokonam czynów. Echo kroków moich nie odbijało nut innych, te tylko, jak gdybym krocząc u boku żyjącej jeszcze matki nadpowietrzne budował zamki. Stary dom nasz zmienił się nie do poznania.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_008 chunk 8 of 115
+
+
+
+
+
+
+
+
Znikły bez śladu powichrzone przez wrony dawno opuszczone gniazda, a drzewa strzyżone i ścinane utraciły dawny kształt. Ogród zdziczał. Połowa okien była zabita. Dom zamieszkany został przez jakiegoś chorego umysłowo dżentelmena i tych, co go doglądali.
+
+
+
./dickens-dawid-copperfield-t1/dickens-dawid-copperfield-t1_022_tom-i-rozdzial-xxii_009 chunk 9 of 115
+
+
+
+
+
+
+
+
Chory przesiadywał zwykle w oknie mego niegdyś pokoiku i spoglądał na cmentarz. Ciekawy byłem, czy też jego myśli kreślą te same obrazy co moje, gdy wczesnym rankiem w nocnej koszulce wyglądałem tym okienkiem, witając pasące się o wschodzie słońca trzody. Dawni nasi sąsiedzi, państwo Graper.
from whisperspeech import wh_transcribe
+import IPython
+
+
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/pyannote/audio/core/io.py:43: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call.
+ torchaudio.set_audio_backend("soundfile")
+
+
+
+
Prepare the dataset
+
+
shards = [str(x) for x in Path('/data/whisperspeech-wds/').glob('librilight-*.tar')]
chunks = [len(x['vad.npy'][1:-1]) for x in progress_bar(ds, total='noinfer')]
+
+
+
+
+
+
+
+
+ 100.00% [3411/3411 00:01<00:00]
+
+
+
+
+
+
sum(chunks)
+
+
203078
+
+
+
+
for x in progress_bar(ds2, total=5):
+ IPython.display.display(IPython.display.Markdown(f"## {x['__key__']} from {x['__url__']}\n{x['txt']}"))
+ IPython.display.display(IPython.display.Audio(x['samples'], rate=16000))
+
+
+
+
+
+
+
+
+ 100.00% [5/5 00:01<00:00]
+
+
+
+
+
large/6454/kaffirkangarooklondiketales_1611_librivox_64kb_mp3/kaffirkangaroo_03_leavitt_64kb_006 from /data/whisperspeech-wds/librilight-large-6454-flac-000007.tar
+
Physically I was incapable of complying with the command, and mentally I had not the slightest intention of departing. In an outhouse devoted to storing melees, sheepskins, and harness, an old man was sitting on the doorstep, compounding a mixture which I recognized as a sheep remedy.
+
+
+
+
+
+
+
+
large/6454/kaffirkangarooklondiketales_1611_librivox_64kb_mp3/kaffirkangaroo_03_leavitt_64kb_009 from /data/whisperspeech-wds/librilight-large-6454-flac-000007.tar
+
The following day I was the most surprised man in South Africa when I learned that my preparation was working a marvelous cure. I was invited to remain with the bore the balance of the season as an honoured guest. Day after day I tramped the hills, returning at night as wise and as rich as when I set out. There were unmistakable indications that gold should be found in the vicinity, but the stubborn fact remained that I could not find it.
+
+
+
+
+
+
+
+
large/6454/kaffirkangarooklondiketales_1611_librivox_64kb_mp3/kaffirkangaroo_03_leavitt_64kb_001 from /data/whisperspeech-wds/librilight-large-6454-flac-000007.tar
+
I was one of the first prospectors in the Transvaal to search for gold and a precious dance it led me. At that time, but few Englishmen had ventured into the Boer country, and such was the jealousy with which they were regarded that it was impossible to secure any information which would assist in the search. Footsoir and weary, I tramped from farm to farm, content
+
+
+
+
+
+
+
+
large/6454/kaffirkangarooklondiketales_1611_librivox_64kb_mp3/kaffirkangaroo_03_leavitt_64kb_032 from /data/whisperspeech-wds/librilight-large-6454-flac-000007.tar
+
Dead, more than twenty years. In fact, before I was married and came to live here, for he was my husband’s father. Did you know him? Yes, but I was only a little girl at the time. Why have the clothes been kept?
+
+
+
+
+
+
+
+
large/6454/kaffirkangarooklondiketales_1611_librivox_64kb_mp3/kaffirkangaroo_03_leavitt_64kb_004 from /data/whisperspeech-wds/librilight-large-6454-flac-000007.tar
+
Fortunately, I had acquired some knowledge of sheep in Australia else I believe that I should have starved. When all else failed, I became a sheep doctor and then did a compound whose virtues would have done credit to the most widely advertised path and medicine nostrum.
# convert the final checkpoint
+model = make_model('base.en-2d-512c-dim64').load_checkpoint('vq_stoks-epoch=3-step=28582-val_loss=11.42.ckpt')
+model.save_model(f'vqmodel-512c-dim64-4e-hyptuned-32gpu.model')
# convert the final checkpoint
+model = make_model('medium-2d-512c-dim64').load_checkpoint('../vq_stoks-epoch=0-step=9776-val_loss=0.00.ckpt')
+model.save_model(f'vqmodel-medium-en+pl-512c-dim64.model')
+
+
+
# convert the final checkpoint
+model = make_model('base-2d-512c-dim64').load_checkpoint('../vq_stoks--2-24696-acc=0.91.ckpt')
+model.save_model(f'vqmodel-base-en+pl-512c-dim64.model')
+
+
+
# convert the final checkpoint
+model = make_model('medium-2d-1024c-dim64').load_checkpoint('../vq_stoks-chad_gold-3-32813-acc=0.96.ckpt')
+model.save_model(f'vqmodel-medium-v2-en+pl-1024c-dim64.model')
/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 10 batches x 32 samples, 1.9 hours) was reported to be 10 (when accessing len(dataloader)), but 11 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
+/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 10 batches x 32 samples, 1.9 hours) was reported to be 10 (when accessing len(dataloader)), but 12 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 6290 batches x 32 samples, 1307.9 hours) was reported to be 6290 (when accessing len(dataloader)), but 6291 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 10 batches x 32 samples, 1.9 hours) was reported to be 10 (when accessing len(dataloader)), but 11 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
+/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 10 batches x 32 samples, 1.9 hours) was reported to be 10 (when accessing len(dataloader)), but 12 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
+/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 6290 batches x 32 samples, 1307.9 hours) was reported to be 6290 (when accessing len(dataloader)), but 6291 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 10 batches x 32 samples, 1.9 hours) was reported to be 10 (when accessing len(dataloader)), but 11 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
+/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 10 batches x 32 samples, 1.9 hours) was reported to be 10 (when accessing len(dataloader)), but 12 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
+/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 6280 batches x 32 samples, 1306.1 hours) was reported to be 6280 (when accessing len(dataloader)), but 6281 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
+
+
+
+
+
+
+
+
+
+
+
# base.en whisper with learned positional embeddings, out_blocks after positional, mlp before vq
+# cleaned dataset (removed 1st and last segments)
+vqmodel = RQBottleneckTransformer(codebook_dim=32, vq_codes=4096, q_depth=1, n_head=8, depth=1,
+ downsample=2, threshold_ema_dead_code=0, use_cosine_sim=True, whisper_model_name="base.en").cuda()
+train("svq", vqmodel, train_ds, val_ds, bs=14, epochs=5, lr=3e-3, warmup_steps=2000,
+ run_valid_every_iters=10000, table_row_every_iters=50000, dl_workers=4, visual_class=RQVisual)
+vqmodel.save_model('vq-base.en-2d-4096c-cosine32-padfix-premlp-learnpos-5e-cleaned.model')
/tmp/ipykernel_90303/1747892456.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_90303/1747892456.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
+/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 6132 batches x 32 samples, 1277.7 hours) was reported to be 6132 (when accessing len(dataloader)), but 6133 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
/tmp/ipykernel_90303/1747892456.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_90303/1747892456.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
/tmp/ipykernel_94907/1747892456.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_94907/1747892456.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
+/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 6132 batches x 32 samples, 1277.7 hours) was reported to be 6132 (when accessing len(dataloader)), but 6133 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
/tmp/ipykernel_94907/1747892456.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_94907/1747892456.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
/tmp/ipykernel_94907/1747892456.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_94907/1747892456.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
/tmp/ipykernel_94907/1747892456.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_94907/1747892456.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
/tmp/ipykernel_100642/1747892456.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_100642/1747892456.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
+IOPub message rate exceeded.
+The notebook server will temporarily stop sending output
+to the client in order to avoid crashing it.
+To change this limit, set the config variable
+`--NotebookApp.iopub_msg_rate_limit`.
+
+Current values:
+NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
+NotebookApp.rate_limit_window=3.0 (secs)
+
/tmp/ipykernel_129075/774804256.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_129075/774804256.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
+/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 6132 batches x 32 samples, 1277.7 hours) was reported to be 6132 (when accessing len(dataloader)), but 6133 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
/tmp/ipykernel_133489/774804256.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_133489/774804256.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
+/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 9933 batches x 32 samples, 1275.0 hours) was reported to be 9933 (when accessing len(dataloader)), but 9934 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
/tmp/ipykernel_100642/1747892456.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_100642/1747892456.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
+/opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:645: UserWarning: Length of IterableDataset Dataset: 6132 batches x 32 samples, 1277.7 hours) was reported to be 6132 (when accessing len(dataloader)), but 6133 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
+ warnings.warn(warn_msg)
/tmp/ipykernel_94907/1747892456.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_94907/1747892456.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
/tmp/ipykernel_94907/1747892456.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_94907/1747892456.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
/tmp/ipykernel_94907/1747892456.py:43: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.clear()
+/tmp/ipykernel_94907/1747892456.py:46: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
+ loss_p.set_xlim(10000, self.total_steps)
0
+1 Chapter
+2 Chapter 5 of the
+3 Chapter 5 of the things in our garden.
+4 Chapter 5 of the Things in Our Garden by Arthur Rachael.
+5 Chapter 5 of the things in our garden by Arthur Ransom.
+6 Chapter 5 of the Things in Our Garden by Arthur Ransom. This LibraVox Recordings.
+7 Chapter 5 of the Things in Our Garden by Arthur Ransom. This LibraVox recording is in the public.
+8 Chapter 5 of the Things in Our Garden by Arthur Ransom. This LibraVox recording is in the public domain.
+9 Chapter 5 of the Things in Our Garden by Arthur Ransom. This LibraVox recording is in the public domain. Chapter 5
+10 Chapter 5 of the Things in Our Garden by Arthur Ransom. This LibraVox recording is in the public domain. Chapter 5, their own garden.
+11 Chapter 5 of the Things in Our Garden by Arthur Ransom. This LibraVox recording is in the public domain. Chapter 5, Their Own Gardens.
+12 Chapter 5 of the Things in Our Garden by Arthur Ransom. This Libra-Vox recording is in the public domain. Chapter 5, Their Own Gardens.
+13 Chapter 5 of the Things in Our Garden by Arthur Ransom. This Libra-Vox recording is in the public domain. Chapter 5, their own gardens, close by the wood at the
+14 Chapter 5 of the Things in Our Garden by Arthur Ransom. This Libra box recording is in the public domain. Chapter 5, Their Own Gardens, Close by the wood at the bottom of the garden.
+
+
+
+
test_incremental('base.en')
+
+
0
+1 Chapter 4
+2 Chapter 5 of the
+3 Chapter 5 of the Things in our Guard.
+4 Chapter 5 of The Things in Our Garden by Arthur Raffy
+5 Chapter 5 of The Things in Our Garden by Arthur Ransom.
+6 Chapter 5 of The Things in Our Garden by Arthur Ransom.
+7 CHAPTER V.
+8 CHAPTER V.
+9 CHAPTER V.
+10 CHAPTER V.
+11 CHAPTER V.
+12 CHAPTER V. Their Own Gardens.
+13 CHAPTER V. Their Own Gardens.
+14 CHAPTER V.
+
+
+
+
test_incremental('large-v2')
+
+
0
+1 Chapter 4.
+2 Chapter 5 of the
+3 Chapter 5 of The Things in Our Garden
+4 V. THE THINGS IN OUR GARDEN
+5 V. THE THINGS IN OUR GARDEN.
+6 CHAPTER V
+7 V. THE THINGS IN OUR GARDEN
+8 CHAPTER V
+9 CHAPTER V
+10 V. THEIR OWN GARDEN
+11 V. THEIR OWN GARDENS
+12 V. THEIR OWN GARDENS
+13 V. THEIR OWN GARDENS CLOSE BY THE WOOD
+14 V. THEIR OWN GARDENS CLOSE BY THE WOOD AT THE BOTTOM OF THE GARDEN
# 4096 tokens, we later found out that tokens from this model do carry speaker information
+calc_model_entropy(make_test_ds(), "vqmodel-4e-hyptuned-32gpu.model")
WER: 6.9
+GT: I WISH I HADN'T CRIED SO MUCH SAID ALICE AS SHE SWAM ABOUT TRYING TO FIND HER WAY OUT
+GEN: "'I wish I hadn't cried so much,' said Alice, as she swam about, trying to find her way out. "'I wish I hadn't cried so much,' said Alice, as she swam about, trying to find her way out. "'I wish I hadn't cried so much,' said Alice, as she swam about, trying to find her way out. "'I wish I hadn't cried so much,' said Alice, as she swam about, trying to find her way out. "'I wish I hadn't cried so much,' said Alice, as she swam about, trying to find her way out. "'I wish I hadn't cried so much,' said Alice, as she swam about, trying to find her way out. "'I wish I hadn't cried so much,' said Alice, as she swam about, trying to find her way out. "'I wish I hadn't cried so much,' said Alice, as she swam about, trying to find her
+
+
+
+
show_stat(_18, 819)
+
+
WER: 1.0
+GT: SCUSE ME SAID TROT I NEGLECTED TO TELL YOU THAT YOU'RE NOT THE BOOLOOROO ANY MORE
+GEN: Thanks for watching!
+
+
+
+
show_stat(_13, 114)
+
+
WER: 1.0
+GT: YES IN REALITY THOSE WERE HAPPY TIMES
+GEN: Thank you.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/3D. Split out validation.html b/3D. Split out validation.html
new file mode 100644
index 0000000..122c043
--- /dev/null
+++ b/3D. Split out validation.html
@@ -0,0 +1,1036 @@
+
+
+
+
+
+
+
+
+
+WhisperSpeech
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Lightning automatically upgraded your loaded checkpoint from v1.5.4 to v2.1.0. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../.cache/torch/whisperx-vad-segmentation.bin`
+
+
+
Model was trained with pyannote.audio 0.0.1, yours is 2.1.1. Bad things might happen unless you revert pyannote.audio to 0.x.
+Model was trained with torch 1.10.0+cu102, yours is 2.1.0+cu121. Bad things might happen unless you revert torch to 1.x.
+Benchmarking run of 1024 samples (64 batches)
Benchmarking run of 1024 samples (64 batches)
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+ - Avoid using `tokenizers` before the fork if possible
+ - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Benchmarking run of 1024 samples (64 batches)
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+ - Avoid using `tokenizers` before the fork if possible
+ - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Benchmarking run of 1024 samples (1024 batches)
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+ - Avoid using `tokenizers` before the fork if possible
+ - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
/opt/conda/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
+ warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
/opt/conda/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:149: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
+ warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
+
+
+
+
+
+
+
+
+
+
+
# encoder loss barely helps, probably because the RoPE cross-attention bias is already helping a lot
+model = make_model('micro', quantizers=4, frozen_embeddings_model='vqmodel-medium-en+pl-512c-dim64.model',
+ tunables=Tunables(causal_encoder=True)).cuda()
+train(f"s2a-new", model, train_ds, val_ds, half=True, bs=32, lr=model.tunables.lr0, epochs=1, warmup_steps=model.tunables.warmup_steps,
+ table_row_every_iters=25000, run_valid_every_iters=5000, visual_class=CMLMVisual)
/opt/conda/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:149: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
+ warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
+
+
+
+
+
+
+
+
+
+
+
# we can prioritize the loss for the first quantizer
+# we'd have to compare generations to really know if it helps though
+model = make_model('micro', quantizers=4, frozen_embeddings_model='vqmodel-medium-en+pl-512c-dim64.model',
+ tunables=Tunables(q0_loss_mult=5)).cuda()
+train(f"s2a-new", model, train_ds, val_ds, half=True, bs=32, lr=model.tunables.lr0, epochs=1, warmup_steps=model.tunables.warmup_steps,
+ table_row_every_iters=25000, run_valid_every_iters=5000, visual_class=CMLMVisual)
/opt/conda/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:149: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
+ warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/5B. Multi-lang text to semantic token modeling_files/figure-html/cell-13-output-4.png b/5B. Multi-lang text to semantic token modeling_files/figure-html/cell-13-output-4.png
new file mode 100644
index 0000000..51943ae
Binary files /dev/null and b/5B. Multi-lang text to semantic token modeling_files/figure-html/cell-13-output-4.png differ
diff --git a/5B. Multi-lang text to semantic token modeling_files/figure-html/cell-15-output-5.png b/5B. Multi-lang text to semantic token modeling_files/figure-html/cell-15-output-5.png
new file mode 100644
index 0000000..98091d2
Binary files /dev/null and b/5B. Multi-lang text to semantic token modeling_files/figure-html/cell-15-output-5.png differ
diff --git a/5b. multi-lang text to semantic token modeling.html b/5b. multi-lang text to semantic token modeling.html
new file mode 100644
index 0000000..04554c8
--- /dev/null
+++ b/5b. multi-lang text to semantic token modeling.html
@@ -0,0 +1,1170 @@
+
+
+
+
+
+
+
+
+
+WhisperSpeech - Text to semantic tokens model
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Text to semantic tokens model
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
from whisperspeech.wer_metrics import*
+from whisperspeech.train import*
+
+from fastprogress import master_bar
+import webdataset as wds
/opt/conda/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:149: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
+ warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
Applies a linear transformation to the incoming data: :math:y = xA^T + b.
+
This module supports :ref:TensorFloat32<tf32_on_ampere>.
+
On certain ROCm devices, when using float16 inputs this module will use :ref:different precision<fp16_on_mi200> for backward.
+
Args: in_features: size of each input sample out_features: size of each output sample bias: If set to False, the layer will not learn an additive bias. Default: True
+
Shape: - Input: :math:(*, H_{in}) where :math:* means any number of dimensions including none and :math:H_{in} = \text{in\_features}. - Output: :math:(*, H_{out}) where all but the last dimension are the same shape as the input and :math:H_{out} = \text{out\_features}.
+
Attributes: weight: the learnable weights of the module of shape :math:(\text{out\_features}, \text{in\_features}). The values are initialized from :math:\mathcal{U}(-\sqrt{k}, \sqrt{k}), where :math:k = \frac{1}{\text{in\_features}} bias: the learnable bias of the module of shape :math:(\text{out\_features}). If :attr:bias is True, the values are initialized from :math:\mathcal{U}(-\sqrt{k}, \sqrt{k}) where :math:k = \frac{1}{\text{in\_features}}
Applies a linear transformation to the incoming data: :math:y = xA^T + b.
+
This module supports :ref:TensorFloat32<tf32_on_ampere>.
+
On certain ROCm devices, when using float16 inputs this module will use :ref:different precision<fp16_on_mi200> for backward.
+
Args: in_features: size of each input sample out_features: size of each output sample bias: If set to False, the layer will not learn an additive bias. Default: True
+
Shape: - Input: :math:(*, H_{in}) where :math:* means any number of dimensions including none and :math:H_{in} = \text{in\_features}. - Output: :math:(*, H_{out}) where all but the last dimension are the same shape as the input and :math:H_{out} = \text{out\_features}.
+
Attributes: weight: the learnable weights of the module of shape :math:(\text{out\_features}, \text{in\_features}). The values are initialized from :math:\mathcal{U}(-\sqrt{k}, \sqrt{k}), where :math:k = \frac{1}{\text{in\_features}} bias: the learnable bias of the module of shape :math:(\text{out\_features}). If :attr:bias is True, the values are initialized from :math:\mathcal{U}(-\sqrt{k}, \sqrt{k}) where :math:k = \frac{1}{\text{in\_features}}
Applies Layer Normalization over a mini-batch of inputs.
+
This layer implements the operation as described in the paper Layer Normalization <https://arxiv.org/abs/1607.06450>__
+
.. math:: y = * +
+
The mean and standard-deviation are calculated over the last D dimensions, where D is the dimension of :attr:normalized_shape. For example, if :attr:normalized_shape is (3, 5) (a 2-dimensional shape), the mean and standard-deviation are computed over the last 2 dimensions of the input (i.e. input.mean((-2, -1))). :math:\gamma and :math:\beta are learnable affine transform parameters of :attr:normalized_shape if :attr:elementwise_affine is True. The standard-deviation is calculated via the biased estimator, equivalent to torch.var(input, unbiased=False).
+
.. note:: Unlike Batch Normalization and Instance Normalization, which applies scalar scale and bias for each entire channel/plane with the :attr:affine option, Layer Normalization applies per-element scale and bias with :attr:elementwise_affine.
+
This layer uses statistics computed from input data in both training and evaluation modes.
+
Args: normalized_shape (int or list or torch.Size): input shape from an expected input of size
+
.. math::
+ [* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1]
+ \times \ldots \times \text{normalized\_shape}[-1]]
+
+ If a single integer is used, it is treated as a singleton list, and this module will
+ normalize over the last dimension which is expected to be of that specific size.
+eps: a value added to the denominator for numerical stability. Default: 1e-5
+elementwise_affine: a boolean value that when set to ``True``, this module
+ has learnable per-element affine parameters initialized to ones (for weights)
+ and zeros (for biases). Default: ``True``.
+bias: If set to ``False``, the layer will not learn an additive bias (only relevant if
+ :attr:`elementwise_affine` is ``True``). Default: ``True``.
+
Attributes: weight: the learnable weights of the module of shape :math:\text{normalized\_shape} when :attr:elementwise_affine is set to True. The values are initialized to 1. bias: the learnable bias of the module of shape :math:\text{normalized\_shape} when :attr:elementwise_affine is set to True. The values are initialized to 0.
Applies a linear transformation to the incoming data: :math:y = xA^T + b.
+
This module supports :ref:TensorFloat32<tf32_on_ampere>.
+
On certain ROCm devices, when using float16 inputs this module will use :ref:different precision<fp16_on_mi200> for backward.
+
Args: in_features: size of each input sample out_features: size of each output sample bias: If set to False, the layer will not learn an additive bias. Default: True
+
Shape: - Input: :math:(*, H_{in}) where :math:* means any number of dimensions including none and :math:H_{in} = \text{in\_features}. - Output: :math:(*, H_{out}) where all but the last dimension are the same shape as the input and :math:H_{out} = \text{out\_features}.
+
Attributes: weight: the learnable weights of the module of shape :math:(\text{out\_features}, \text{in\_features}). The values are initialized from :math:\mathcal{U}(-\sqrt{k}, \sqrt{k}), where :math:k = \frac{1}{\text{in\_features}} bias: the learnable bias of the module of shape :math:(\text{out\_features}). If :attr:bias is True, the values are initialized from :math:\mathcal{U}(-\sqrt{k}, \sqrt{k}) where :math:k = \frac{1}{\text{in\_features}}
All datasets that represent an iterable of data samples should subclass it. Such form of datasets is particularly useful when data come from a stream.
+
All subclasses should overwrite :meth:__iter__, which would return an iterator of samples in this dataset.
+
When a subclass is used with :class:~torch.utils.data.DataLoader, each item in the dataset will be yielded from the :class:~torch.utils.data.DataLoader iterator. When :attr:num_workers > 0, each worker process will have a different copy of the dataset object, so it is often desired to configure each copy independently to avoid having duplicate data returned from the workers. :func:~torch.utils.data.get_worker_info, when called in a worker process, returns information about the worker. It can be used in either the dataset’s :meth:__iter__ method or the :class:~torch.utils.data.DataLoader ’s :attr:worker_init_fn option to modify each copy’s behavior.
+
Example 1: splitting workload across all workers in :meth:__iter__::
+
>>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_DATALOADER)
+>>> # xdoctest: +SKIP("Fails on MacOS12")
+>>> class MyIterableDataset(torch.utils.data.IterableDataset):
+... def __init__(self, start, end):
+... super(MyIterableDataset).__init__()
+... assert end > start, "this example code only works with end >= start"
+... self.start = start
+... self.end = end
+...
+... def __iter__(self):
+... worker_info = torch.utils.data.get_worker_info()
+... if worker_info is None: # single-process data loading, return the full iterator
+... iter_start = self.start
+... iter_end = self.end
+... else: # in a worker process
+... # split workload
+... per_worker = int(math.ceil((self.end - self.start) / float(worker_info.num_workers)))
+... worker_id = worker_info.id
+... iter_start = self.start + worker_id * per_worker
+... iter_end = min(iter_start + per_worker, self.end)
+... return iter(range(iter_start, iter_end))
+...
+>>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6].
+>>> ds = MyIterableDataset(start=3, end=7)
+
+>>> # Single-process loading
+>>> print(list(torch.utils.data.DataLoader(ds, num_workers=0)))
+[tensor([3]), tensor([4]), tensor([5]), tensor([6])]
+
+>>> # xdoctest: +REQUIRES(POSIX)
+>>> # Mult-process loading with two worker processes
+>>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6].
+>>> # xdoctest: +IGNORE_WANT("non deterministic")
+>>> print(list(torch.utils.data.DataLoader(ds, num_workers=2)))
+[tensor([3]), tensor([5]), tensor([4]), tensor([6])]
+
+>>> # With even more workers
+>>> # xdoctest: +IGNORE_WANT("non deterministic")
+>>> print(list(torch.utils.data.DataLoader(ds, num_workers=12)))
+[tensor([3]), tensor([5]), tensor([4]), tensor([6])]
+
Example 2: splitting workload across all workers using :attr:worker_init_fn::
+
>>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_DATALOADER)
+>>> class MyIterableDataset(torch.utils.data.IterableDataset):
+... def __init__(self, start, end):
+... super(MyIterableDataset).__init__()
+... assert end > start, "this example code only works with end >= start"
+... self.start = start
+... self.end = end
+...
+... def __iter__(self):
+... return iter(range(self.start, self.end))
+...
+>>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6].
+>>> ds = MyIterableDataset(start=3, end=7)
+
+>>> # Single-process loading
+>>> print(list(torch.utils.data.DataLoader(ds, num_workers=0)))
+[3, 4, 5, 6]
+>>>
+>>> # Directly doing multi-process loading yields duplicate data
+>>> print(list(torch.utils.data.DataLoader(ds, num_workers=2)))
+[3, 3, 4, 4, 5, 5, 6, 6]
+
+>>> # Define a `worker_init_fn` that configures each dataset copy differently
+>>> def worker_init_fn(worker_id):
+... worker_info = torch.utils.data.get_worker_info()
+... dataset = worker_info.dataset # the dataset copy in this worker process
+... overall_start = dataset.start
+... overall_end = dataset.end
+... # configure the dataset to only process the split workload
+... per_worker = int(math.ceil((overall_end - overall_start) / float(worker_info.num_workers)))
+... worker_id = worker_info.id
+... dataset.start = overall_start + worker_id * per_worker
+... dataset.end = min(dataset.start + per_worker, overall_end)
+...
+
+>>> # Mult-process loading with the custom `worker_init_fn`
+>>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6].
+>>> print(list(torch.utils.data.DataLoader(ds, num_workers=2, worker_init_fn=worker_init_fn)))
+[3, 5, 4, 6]
+
+>>> # With even more workers
+>>> print(list(torch.utils.data.DataLoader(ds, num_workers=12, worker_init_fn=worker_init_fn)))
+[3, 4, 5, 6]
+
+
# validate that we don't reset the datasets on each `iter`
+# this is important with webdatasets since sample shuffling is very bad initially, unless num_workers << num_shards
+from itertools import islice
+ds = join_datasets(["ABCDEFG"])
+for x in islice(ds, 3):
+print(x)
+for x in islice(ds, 5):
+print(x)
+
+
A
+B
+C
+D
+E
+F
+G
+
+
+
+
# will stop as soon as it exhausts one iterator
+for x in join_datasets(['ABCDEFG', 'abcdefg', range(20)]):
+print(x)
Merge a dataset into the current one returning samples with the union of keys. Pass in a function that takes a URL of a sample and returns a dataset for it (called everytime the URL changes).
+
It requires (and validates) that both datasets have the same ordering of keys so you have to use it before any sample shuffling. Shard shuffling is ok.
This language list is comming straigh from openai-whisper. The upstream file is here: https://github.com/openai/whisper/blob/main/whisper/tokenizer.py but we are freezing this to the openai-whisper==20230918 version right now.
To get these 3 data representations we have to run the audio data through several models. The first two steps are always the same, the rest depend on the model we want to run.
To train the text to semantic token model we need to transcribe the audio with Whisper small.en and extract the semantic tokens (5A. T2S dataset preparation).
+
To train the semantic to acoustic model we need to extract the semantic tokens and compress the audio with Encodec for the semantic to acoustic model (4A. S2A dataset preparation).
+
+
These three steps are all independent since they require different chunking of speech data. For quantizing Whisper and S2A training we greedily merge the VAD segments from the same speaker into (at most) 30 second chunks to improve training performance (more uniform chunks mean less computation time is spent on padding). For T2S we randomly truncate when merging the VAD segments so the model also learns how to work with shorter texts. The code to perform this is in 1C. VAD merging.
+
+
+
TL;DR example – give me the codes!
+
In this example we will convert a single split from the Multilingual Libri Speech dataset.
+
+
Prepare the webdataset shards
+
The first, most time-consuming, step is to convert the data from it’s original form into the webdataset format. If you want to skip this section and still follow along, the results can be downloaded from Hugging Face at datasets/collabora/multilingual-librispeech-webdataset.
+
First we need tarp which is a tool that helps create and manipulate the webdataset tar files more effectively. You can check out more about it in the official tarp README
+
go install -v github.com/collabora/tarp/tarp@latest
+
Afterwards, we download and unpack the original dataset files:
Next, we’ll need to convert each line in the transcripts.txt file:
+
10065_10039_000000 ses vêtements devinrent tout brillants de lumière et blancs comme la neige en sorte qu'il n'y a point de foulon sur la terre qui puisse en faire d'aussi blancs
+
into a tarp script:
+
train/10065_10039_000000.opus file:mls_french_opus/train/audio/10065/10039/10065_10039_000000.opus
+train/10065_10039_000000.txt text:ses vêtements devinrent tout brillants de lumière et blancs comme la neige en sorte qu'il n'y a point de foulon sur la terre qui puisse en faire d'aussi blancs
+
We can achieve this using a short Python script (saved as make-script.py):
Once we have this, we can run the conversion process. The python script outputs data sample descriptions which are fed to tarp create that archives them into a tar stream (a bit similar to tar -T -). The tarp split will then cut the incoming stream into 2GB shards and save them to separate files, making sure to split on sample boundaries.
+
The 2GB size was chosen as a good compromise between the shard count and shard transcription time for mp3/opus files with mutlilingual speech. For LibriLight (English compressed with FLAC) the magic number was 5GB because we FLAC compresses less and we can also use a smaller model for transcribing English speech.
We do the sharding mainly to be able to effectively process data on many GPUs but for the sake of simplicity we will use a single GPU here. The process stays the same, but different tools would be used to schedule the jobs. For reference, below the commands, we have specified their approximate runtimes on a RTX 4090 for the French subset of MLS.
After we have all the samples we may want to extract some validation sets. There are many ways to do it but here we’ll manually choose some speakers we’ll later skip completely during training.
+
We start by dumping all the sample ids:
+
parallel tar tf {} ::: stoks/*-atoks-3kbps-*.tar.gz |sed-e's/\.atoks\.npy//'> all-samples-maxvad
+parallel tar tf {} ::: stoks/*-small.en-txt-*.tar.gz |sed-e's/\.txt//'> all-samples-eqvad
+wc-l all-samples-maxvad
+
Because the sample ids (which are the original file paths) have speaker ids in them we can make a quick histogram:
From the result we can copy and paste 10 speaker ids of around 50 samples each to get 512 validation samples. We’ll exclude them from the training set because we want to validate on unseen speakers. We have to repeat this process for both splits (maxvad and eqvad since they have’ll different sample counts and ids):
Once we have all the ids we can rescan the whole dataset once and split out the validation samples to separate webdataset shards to make validation fast:
All WhisperSpeech training and preproc code got reorganized around webdatasets. Webdatasets are just simple tar files that store all our data samples (files) but they are great for working with very large datasets. Inside these tar files we can store multiple files per sample in any format we want (e.g. the speech mp3/flac/wav files, the text transcripts, tokens in numpy arrays). For example from the data used to train the S2A model we have:
The name of the file is the same as the file name of the original dataset sample and the extensions tell us what kind of value they hold and in which format.
+
Furthermore we can split the whole dataset into fixed-size tar files called shards and load them on demand without unpacking. It turns out that this is exactly what we need for both AI training and data preprocessing:
+
+
for training we start a multiple CPU workers in parallel, open different shards in each, stream the data sequentially from disk (fast), decode it independently and them shuffle the samples we receive from each worker to create varied training batches
+
for preprocessing we independently send each shard to a worker and save all the results in a new webdataset shard
+
+
Reading samples sequentialy allows us to simply compress the whole file with gzip and offers best performance even on spinning or network disks.
+
+
+
+
+
+
+Note
+
+
+
+
For the Juwels cluster there is another crucial benefit. There is a pretty low limit on the total number of files on network disks (inodes to be precise) so there is a strong preference to keep data in a few large files. The network file system performance is also better if we don’t have to open too many files.
+
+
+
Keeping each shard around 5GB seems to work great (the processed shards will likely be a lot smaller but it’s a lot easier to keep a 1-to-1 shard mapping). For the almost 4TB LibriLight dataset this translates to 625 files.
+
We found it quite useful to also keep all the data in some splits. This is data dependent but for LibriLight we followed the original split (small, medium, large) but also extracted the 6454 speaker from the large split because it is was the largest single speaker dataset and it allowed us to use it during development without downloading the full 4TB.
+
+
+
+
+
+
+Caution
+
+
+
+
The sample file names should not have dots in them, otherwise the WebDataset code gets confused which files go together into one sample. This can be worked around later but it’s easiest if we just do .replace('.', '_') when storing the initial raw dataset.
+
+
+
+
+
Joins on WebDatasets
+
One novel functionality we developed for this project is the capability to join multiple preprocessed webdatasets. This mechanism relies on keeping a constant ordering of samples in a shard and ensuring 1-to-1 correspondence between the input and output shards during preprocessing.
+
Example usage:
+
ds = wds.WebDataset([str(x) for x in Path('librilight/').glob('*.tar')]).compose( # load all audio shards
+ wds.decode(wds.torch_audio), # decode the audio data
+ vq_stoks.merge_in( # merge another WebDataset
+# for each audio (`raw`) shard, find the path and name of a corresponding `vad` shard
+ vq_stoks.derived_dataset('librilight-processed/', 'vad')
+ ),
+)
+
derived_dataset creates for us a helper function that returns an opened derived dataset given the original shard file name:
This will generate a file named guttenberg-audiobooks-vad-000000.tar.gz in the guttenberg-preproc directory.
+
On the cluster we can run multiple jobs in parallel (24 in this case), each processing one input shard. Since each job is pretty short (around 30 minutes) it’s easier for the scheduler to squeeze these between longer and higher-priority jobs.
The prepare_s2a_dataset script is taking raw audio data from the input file, automatically finding corresponding shards with VAD results in ../librilight-preproc and writing the results to the whisperspeech-s2a-512c-dim64 directory.
Right now we are using the VAD model from WhisperX that is enough to avoid cutting audio in the middle of a word which would hurt automated transcriptions quite a lot. For more fancy datasets with multiple speakers we could use pyannote for it’s detection of multiple people speaking at once and diarization capability.
+
We later merge the VAD segments into longer chunks for more efficient training (less padding == higher efficiency). The code and histogram plots can be found in 2A. Whisper quantization dataset preparation
For training the TTS model (T2S) we are using running batches of chunked speech segments though FasterWhisper. We use the small.en model since there seems to be little benefit from using the larger models on English speech. For multilingual TTS we would probably want to switch to large-v2.
+
+
+
+
+
+
+Note
+
+
+
+
Right now we extract both semantic tokens and transcriptions in one go. Doing the transcriptions is very time consuming are the result is unlikely to change. OTOH we may want to regenerate the semantic tokens if we train different quantized Whisper models. Because of that we may want to split this into two separate steps and only merge the results just before we generate the training dataset.
This is basically the same as T2S above but with Encodec instead of Whisper.
+
+
+
Train/validation split
+
We create validation splits differently for each dataset. For example for LibriLight we use the speaker labels to create a common and unseen speakers splits. Once we have a list of samples we want to use we extract them from the full dataset into a new shard while keeping a list of IDs to skip during training. This way we avoid copying the training samples.
+
This has the downside of delaying all shuffling until training. This is especially problematic for smaller datasets with not enough shards since multiple workers may read the same shard and initially (before the shuffling buffer is filled) deliver the same samples multiple times. This causes overfitting. This is not a problem early in training (the model is too random to overfit) and we make sure we don’t reset the dataloaders between epochs but it is causing issues when resuming training from a checkpoint. The workaround is to preload the shuffling bufferwith a lot of samples (.shuffle(initial=20000)). Unfortunately it has the downside of putting a lot of load on the filesystem and adding a significant delay before training can start.
+If you have questions or you want to help you can find us in the #audio-generation channel on the LAION Discord server.
+
An Open Source text-to-speech system built by inverting Whisper. Previously known as spear-tts-pytorch.
+
We want this model to be like Stable Diffusion but for speech – both powerful and easily customizable.
+
We are working only with properly licensed speech recordings and all the code is Open Source so the model will be always safe to use for commercial applications.
+
Currently the models are trained on the English LibreLight dataset. In the next release we want to target multiple languages (Whisper and EnCodec are both multilanguage).
We were able to do this with frozen semantic tokens that were only trained on English and Polish. This supports the idea that we will be able to train a single semantic token model to support all the languages in the world. Quite likely even ones that are not currently well supported by the Whisper model. Stay tuned for more updates on this front. :)
+
+
+
Progress update [2024-01-18]
+
We spend the last week optimizing inference performance. We integrated torch.compile, added kv-caching and tuned some of the layers – we are now working over 12x faster than real-time on a consumer 4090!
+
We can mix languages in a single sentence (here the highlighted English project names are seamlessly mixed into Polish speech):
+
+
To jest pierwszy test wielojęzycznego Whisper Speech modelu zamieniającego tekst na mowę, który Collabora i Laion nauczyli na superkomputerze Jewels.
We also added an easy way to test voice-cloning. Here is a sample voice cloned from a famous speech by Winston Churchill (the radio static is a feature, not a bug ;) – it is part of the reference recording):
You can test all of these on Colab (we optimized the dependencies so now it takes less than 30 seconds to install). A Huggingface Space is coming soon.
+
+
+
Progress update [2024-01-10]
+
We’ve pushed a new SD S2A model that is a lot faster while still generating high-quality speech. We’ve also added an example of voice cloning based on a reference audio file.
Another trio of models, this time they support multiple languages (English and Polish). Here are two new samples for a sneak peek. You can check out our Colab to try it yourself!
+
English speech, female voice (transferred from a Polish language dataset):
We encourage you to start with the Google Colab link above or run the provided notebook locally. If you want to download manually or train the models from scratch then both the WhisperSpeech pre-trained models as well as the converted datasets are available on HuggingFace.
+
+
+
Roadmap
+
+
+
+
+
+
+
+
+
Architecture
+
The general architecture is similar to AudioLM, SPEAR TTS from Google and MusicGen from Meta. We avoided the NIH syndrome and built it on top of powerful Open Source models: Whisper from OpenAI to generate semantic tokens and perform transcription, EnCodec from Meta for acoustic modeling and Vocos from Charactr Inc as the high-quality vocoder.
+
We gave two presentation diving deeper into WhisperSpeech. The first one talks about the challenges of large scale training:
+
+
+
+Tricks Learned from Scaling WhisperSpeech Models to 80k+ Hours of Speech - video recording by Jakub Cłapa, Collabora
+
+
+
The other one goes a bit more into the architectural choices we made:
We utilize the OpenAI Whisper encoder block to generate embeddings which we then quantize to get semantic tokens.
+
If the language is already supported by Whisper then this process requires only audio files (without ground truth transcriptions).
+
+
+
+Using Whisper for semantic token extraction diagram
+
+
+
+
+
+
EnCodec for modeling acoustic tokens
+
We use EnCodec to model the audio waveform. Out of the box it delivers reasonable quality at 1.5kbps and we can bring this to high-quality by using Vocos – a vocoder pretrained on EnCodec tokens.
+
+
+
+EnCodec block diagram
+
+
+
+
+
Appreciation
+
+
This work would not be possible without the generous sponsorships from:
We gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding part of this work by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Booster at Jülich Supercomputing Centre (JSC), with access to compute provided via LAION cooperation on foundation models research.
+
We’d like to also thank individual contributors for their great help in building this model:
+
+
inevitable-2031 (qwerty_qwer on Discord) for dataset curation
+
+
+
+
Consulting
+
We are available to help you with both Open Source and proprietary AI projects. You can reach us via the Collabora website or on Discord ( and )
+
+
+
Citations
+
We rely on many amazing Open Source projects and research papers:
+
@article{SpearTTS,
+title = {Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision},
+url = {https://arxiv.org/abs/2302.03540},
+author = {Kharitonov, Eugene and Vincent, Damien and Borsos, Zalán and Marinier, Raphaël and Girgin, Sertan and Pietquin, Olivier and Sharifi, Matt and Tagliasacchi, Marco and Zeghidour, Neil},
+publisher = {arXiv},
+year = {2023},
+}
+
@article{MusicGen,
+title={Simple and Controllable Music Generation},
+url = {https://arxiv.org/abs/2306.05284},
+author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez},
+publisher={arXiv},
+year={2023},
+}
+
@article{Whisper
+title = {RobustSpeechRecognitionviaLarge-ScaleWeakSupervision},
+ url = {https://arxiv.org/abs/2212.04356},
+ author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
+ publisher = {arXiv},
+ year = {2022},
+}
+
@article{EnCodec
+title = {HighFidelityNeuralAudioCompression},
+ url = {https://arxiv.org/abs/2210.13438},
+ author = {Défossez, Alexandre and Copet, Jade and Synnaeve, Gabriel and Adi, Yossi},
+ publisher = {arXiv},
+ year = {2022},
+}
 model_files/figure-html/cell-27-output-7.png)
 model_files/figure-html/cell-28-output-7.png)
 model_files/figure-html/cell-29-output-6.png)
 model_files/figure-html/cell-30-output-6.png)
 model_files/figure-html/cell-31-output-7.png)
 model_files/figure-html/cell-32-output-6.png)
 model_files/figure-html/cell-33-output-7.png)
 model_files/figure-html/cell-34-output-7.png)
 model_files/figure-html/cell-35-output-7.png)
 model_files/figure-html/cell-36-output-7.png)
 model_files/figure-html/cell-37-output-7.png)
 model_files/figure-html/cell-38-output-7.png)
 model_files/figure-html/cell-39-output-7.png)
 model_files/figure-html/cell-40-output-7.png)
 model_files/figure-html/cell-41-output-7.png)
 model_files/figure-html/cell-42-output-5.png)
 model_files/figure-html/cell-43-output-5.png)
 model_files/figure-html/cell-44-output-7.png)
 model_files/figure-html/cell-45-output-7.png)
 model_files/figure-html/cell-46-output-7.png)
 model_files/figure-html/cell-47-output-7.png)
 model_files/figure-html/cell-48-output-7.png)
 evaluation_files/figure-html/cell-57-output-1.png)
 evaluation_files/figure-html/cell-64-output-1.png)
 evaluation_files/figure-html/cell-67-output-1.png)
 evaluation_files/figure-html/cell-68-output-1.png)
 evaluation_files/figure-html/cell-69-output-1.png)
