math.sqrt(0.5)

[PetrochukM]

Hey There!

The Wavenet paper never mentioned multiplying skip connections with math.sqrt(0.5). What is up with that?

wavenet_vocoder/wavenet_vocoder/wavenet.py

Line 224 in 1b571a7

skips *= math.sqrt(0.5)

[r9y9]

https://github.com/pytorch/fairseq/blob/4973d05ac7651f46e1c7edc8060600b44baf9e71/fairseq/models/fconv.py#L157

It comes from Deep Voice 3 and https://arxiv.org/abs/1705.03122. The reason is that I wrote code based on https://github.com/r9y9/deepvoice3_pytorch. Probably work without it.

Same for weight normalization:

wavenet_vocoder/hparams.py

Lines 63 to 64 in 1b571a7

	# If True, apply weight normalization as same as DeepVoice3
	weight_normalization=True,

.

[PetrochukM]

Thank you!

Anything else to watch out for?

Working with the Tacotron 2 authors to reimplement their paper. This means, for example, I am not normalizing my spectrogram to [0, 1]. I did not apply any decibel clipping. Etc.

What else is different in your Wavenet from the original Wavenet? For example, the ReLU during upsampling that I mentioned earlier.

[tuan3w]

Hi @r9y9 ,
Currently, I'm not able to access to my GPU to test my hypothesis but I think we should discuss about this number. As I understand, the target of this multiplication is to keep output has same variance as its inputs [1] [2]. As from your code:

        skips = None
        for f in self.conv_layers:
            x, h = f(x, c, g_bct)
            if skips is None:
                skips = h
            else:
                skips += h
                if self.legacy:
                    skips *= math.sqrt(0.5)

        x = skips

In order to maintain the same variance, I think it should be:

        skips = None
        for f in self.conv_layers:
            x, h = f(x, c, g_bct)
            if skips is None:
                skips = h
            else:
                skips += h

        x = skips / math.sqrt(len(self.conv_layers))

As in transformer network paper, the authors claim that it helps when feature dimension is large. As in our case, it will help if number of layers is large. I haven't tested it yet, but I think it will improve the performance of model.
Thanks.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.