Size of initial minibatch

[danpovey]

Piotr, in our snowfall eg with mini_librispeech, the 1st minibatch is 16 seconds long which seems on the long side.
Is that typical of the data, or are they arranged from longest to shortest?
.. because if we want them in a nonrandom order we probably want shortest to longest, which would be better for convergence.

[pzelasko]

They are randomly shuffled. There are also multiple cuts packed into a single batch example which might make the duration excessive. Do you think some curriculum learning like option would be useful (sorting by length, possibly for a number of first epochs)?

…

Wiadomość napisana przez Daniel Povey ***@***.***> w dniu 11/22/20, o godz. 08:33:  Piotr, in our snowfall eg with mini_librispeech, the 1st minibatch is 16 seconds long which seems on the long side. Is that typical of the data, or are they arranged from longest to shortest? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[danpovey]

I assume the length of the example is the length of the longest individual cut?

…

On Sun, Nov 22, 2020 at 10:36 PM Piotr Żelasko ***@***.***> wrote: They are randomly shuffled. There are also multiple cuts packed into a single batch example which might make the duration excessive. Do you think some curriculum learning like option would be useful (sorting by length, possibly for a number of first epochs)? > Wiadomość napisana przez Daniel Povey ***@***.***> w dniu 11/22/20, o godz. 08:33: > >  > Piotr, in our snowfall eg with mini_librispeech, the 1st minibatch is 16 seconds long which seems on the long side. > Is that typical of the data, or are they arranged from longest to shortest? > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub, or unsubscribe. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#146 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLOYOXZGR56UXZHSKNPDSREOXFANCNFSM4T6PLHFA> .

[pzelasko]

By default it's twice of that (it helps the heuristic pack cuts better), but can be adjusted; see: https://github.com/lhotse-speech/lhotse/blob/master/lhotse/dataset/speech_recognition.py#L136

[danpovey]

I think we should make it that by default, not twice of that, because for many model types the time taken could be much more sensitive to the sequence length than to the total number of frames in the sequence. For attention models it can even take time quadratic in the num-frames.

…

On Sun, Nov 22, 2020 at 11:11 PM Piotr Żelasko ***@***.***> wrote: By default it's twice of that (it helps the heuristic pack cuts better), but can be adjusted; see: https://github.com/lhotse-speech/lhotse/blob/master/lhotse/dataset/speech_recognition.py#L136 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#146 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLO7HI4M4Q5H2S4VW7CLSRES3DANCNFSM4T6PLHFA> .

[pzelasko]

Ok, will do. I wonder though, if it makes sense to concatenate the cuts for the attention models at all (unless these are subsequent utterances from the same recording/conversation).

…

Wiadomość napisana przez Daniel Povey ***@***.***> w dniu 11/22/20, o godz. 10:20:  I think we should make it that by default, not twice of that, because for many model types the time taken could be much more sensitive to the sequence length than to the total number of frames in the sequence. For attention models it can even take time quadratic in the num-frames. On Sun, Nov 22, 2020 at 11:11 PM Piotr Żelasko ***@***.***> wrote: > By default it's twice of that (it helps the heuristic pack cuts better), > but can be adjusted; see: > https://github.com/lhotse-speech/lhotse/blob/master/lhotse/dataset/speech_recognition.py#L136 > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#146 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAZFLO7HI4M4Q5H2S4VW7CLSRES3DANCNFSM4T6PLHFA> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.