Biases are excluded from BYOL moving average update

[wjn0]

Based on the code of the BYOLMAWeightUpdate, bias terms are excluded from the moving average update. I don't think this matches the original paper. I did not see any documentation on this point, but it seems intentional. Is there a source or reference for this difference (assuming I haven't misunderstood the original paper)?

The relevant code is here, excerpted:

# apply MA weight update
for (name, online_p), (_, target_p) in zip(online_net.named_parameters(), target_net.named_parameters()):
    if 'weight' in name:
        target_p.data = self.current_tau * target_p.data + (1 - self.current_tau) * online_p.data

To see some of the parameters which are excluded from the MA update with this condition in place (when using a ResNet-50 as the backbone, for example), you can run the following snippet:

from torchvision.models import resnet50
print([param_name for param_name in list(dict(resnet50().named_parameters()).keys()) if 'weight' not in param_name])

[github-actions]

Hi! thanks for your contribution!, great first issue!

[akihironitta]

@annikabrundyn Would you mind having a look?

[sidhantls]

Yeah, the paper explicitly mentions that bias parameters are omitted from optimization with LARS and weight decay but there's no mention about omission for the moving average update

[wjn0]

I think this issue should be looked at rather than closed (and ideally fixed if it indeed wasn't an intentional modification) - not that it seems to affect performance much. The PR would be super simple, and I'm happy to do if the original contributor of this implementation is too busy?

[annikabrundyn]

hey @wjn0 I've just checked the paper again and I think you're right - thanks for spotting this! if you'd like you can submit a PR and ping me for review? also happy to make the change myself this weekend