Some doubts about SublayerConnection

[watersounds]

According to what you wrote：
“That is, the output of each sub-layer is $\mathrm{LayerNorm}(x + \mathrm{Sublayer}(x))$, where $\mathrm{Sublayer}(x)$ is the function implemented by the sub-layer itself. We apply dropout (cite) to the output of each sub-layer, before it is added to the sub-layer input and normalized.”
I think the rentun value should be self.norm(x + self.dropout(sublayer(x))) rather than x + self.dropout(sublayer(self.norm(x))).

Look forward to your reply.

[StellaAthena]

Where do we write x + self.dropout(sublayer(self.norm(x)))? That's not what the passage you quote says.

[Bruising6802]

In the_annotated_transformer.py on line 357. In the function documentation it even says that the norm was moved.

[lvXiangwei]

I have the same question as you. The explanation can be found in #92 .

[Bruising6802]

Maybe it's best to mention this issue in the notebook, because it causes confusion for many.

[watersounds]

Where do we write x + self.dropout(sublayer(self.norm(x)))? That's not what the passage you quote says.

annotated-transformer/the_annotated_transformer.py

Line 357 in debc9fd

return x + self.dropout(sublayer(self.norm(x)))

[kuraga]

So, close as duplicate of #92?