This repository has been archived by the owner on Oct 17, 2023. It is now read-only.

About Layer Weights αk #33

Open

lbdbj opened this issue Nov 25, 2022 · 1 comment

lbdbj commented Nov 25, 2022

大佬您好，您的论文中有提到将每层embedding的权重设置为1/(K+1)，但在代码中并没有体现，请问这是怎么回事

zbt78 commented Apr 4, 2023

在model.py的 107行
light_out = torch.mean(embs, dim=1)
作者好像是对这三层求平均值了.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.