Initializing segment embeddings on Transformer Encoder

[wade3han]

Hello! I figured out that the code is not initializing the segment embeddings on Transformer encoder: https://github.com/facebookresearch/ParlAI/blob/master/parlai/agents/transformer/modules/encoder.py#L201-L202

I found that initializing the segment embeddings induces the model to converge faster, when fine-tuning the seq2seq model (such as Blender) with the usage of segment embeddings. (Note that the pre-trained model didn't use the segment embeddings)
I guess adding the initialization code as below will be helpful.

        if self.n_segments >= 1:
            self.segment_embeddings = nn.Embedding(self.n_segments, self.dim)
            nn.init.normal_(self.segment_embeddings.weight, 0, self.dim ** -0.5)

Anyway, it may be an intended feature, then is there reason you didn't apply the initialization for the segment embeddings?

[stephenroller]

Nah that was overlooked. That's a good point. I'd happily accept a patch.