About key_padding_mask in multihead self attention

[YuchenHui22314]

Hi!

Thank you for your implementation!

I would like to know if there are particular reasons why https://github.com/pmixer/SASRec.pytorch/blob/master/model.py#L83 this line for key_padding_mask is commented? It seems that this mask is necessary to prevent from attending to paddings?

Thanks again,

Sincerely

Yuchen

[pmixer]

Hi!

Thank you for your implementation!

I would like to know if there are particular reasons why https://github.com/pmixer/SASRec.pytorch/blob/master/model.py#L83 this line for key_padding_mask is commented? It seems that this mask is necessary to prevent from attending to paddings?

Thanks again,

Sincerely

Yuchen

Thx for the question, I could hardly recall the exact details in short time, well, generally:

As padding item is 0, corresponding to all 0 embeddings as initialized in https://github.com/pmixer/SASRec.pytorch/blob/master/model.py#L36, attending to 0 would not effect the output

Also, for pytorch api, there's the statement If both attn_mask and key_padding_mask are supplied, their types should match. currently, I may encountered the issue when there was the statement attn_mask and key_padding_mask can not be used at the same time when using pytorch 1.16 for implementation about 3 years ago, thus had one of them commented.

Pls feel free to uncomment the line to check how would key mask affect model training and inference.

[YuchenHui22314]

Thanks for your prompt reply!

Oh ok I see, I will try! After careful consideration, I think, conceptuallly, that even letting all paddings to 0 (what the code does now) will still influence the attention mechanism, since before attention softmax, the original attention score could be negative, 0 or positive. Therefore zero does not mean sth special for softmax function (should be -inf).

Thanks again!

Yuchen

[pmixer]

attention

Thx, for attention score, yes, -inf is required as you described. Well, here I mean attending to 0 equals attending to nothing actually, I didn't mean attention score being zero in former reply. When someone used K dot Q to get attention scores, then use attention scores to do the weighted sum of V, V=0 will not effect the final output theoritically.

BTW, some of the lectures by Prof. Lee may help further clarifying these details about multi-head attention, pls consider checking https://www.youtube.com/@HungyiLeeNTU/search?query=attention if you are interested.

[WindSighiii]

However, it will affect the values when the value is not entirely zeros, because softmax will still assign some weights to the padding positions, which in turn reduces the weights assigned to the non-padding positions.

[pmixer]

o the non-padding positions.

You're right, and that's what I claimed "When the embedding values of the padding item are not initialized as
0, it may work more like a bias term instead of padding." in https://arxiv.org/html/2504.09596v1.

Keeping embeddings and paddings tidy(make those should be 0 be 0 instead of random values) is necessary but not yet in current versions. Pls DIY in your experiments when needed.