Removing L2-norm in contrastive loss (L2-norm already present in CosSim)

[Lucas-rbnt]

Description

The forward method of the ContrastiveLoss performs L2-normalization before computing cosine similarity. The torch.nn.functional.cosine_similarity method already handles this pre-processing to make sure that input and target lie on the surface of the unit hypersphere. This step involves an unnecessary cost and, thus, can be removed.

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.

[wyli]

/build

[wyli]

Hi @Lucas-rbnt, when the L2 norms are large values (e.g. for high dimensional embeddings) do you think this might be less stable numerically, have you tested this PR in end-to-end trainings?

cc @finalelement

[wyli]

Hi @Lucas-rbnt, when the L2 norms are large values (e.g. for high dimensional embeddings) do you think this might be less stable numerically, have you tested this PR in end-to-end trainings?

cc @finalelement

seems to be addressed in pytorch 1.12 for the same pytorch/pytorch@9e137ee

[wyli]

Thanks, it looks good to me. (probably less stable for early versions of pytorch but more efficient for the recent versions)

[wyli]

Thanks, it looks good to me. (probably less stable for early versions of pytorch but more efficient for the recent versions)

[Lucas-rbnt]

Hi @wyli!
You were faster than me! I found also this commit. In torchmetrics for instance, the normalization comes before the calculation also.

I am currently running a SimCLR training on BraTS21 data centered on tumor to compare before and after the commit.
I will post here results and insights. The change is so minor that the difference will also be minor but will see! First results estimated in 3 hours :)

[wyli]

/build