From e26b0e2f92ce8c2f1aa921833433f4758d690e67 Mon Sep 17 00:00:00 2001 From: Jordan Stomps Date: Wed, 12 Jul 2023 12:13:57 -0400 Subject: [PATCH 1/2] adding details to NTXentLoss documentation --- docs/losses.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/losses.md b/docs/losses.md index 2af1f905..641da746 100644 --- a/docs/losses.md +++ b/docs/losses.md @@ -787,6 +787,13 @@ This is also known as InfoNCE, and is a generalization of the [NPairsLoss](losse - [Representation Learning with Contrastive Predictive Coding](https://arxiv.org/pdf/1807.03748.pdf){target=_blank} - [Momentum Contrast for Unsupervised Visual Representation Learning](https://arxiv.org/pdf/1911.05722.pdf){target=_blank} - [A Simple Framework for Contrastive Learning of Visual Representations](https://arxiv.org/pdf/2002.05709.pdf){target=_blank} + +In the equation below, loss is computed for each positive pair, `k_+`, in a batch, normalized by all pairs in the batch, `k_i in K`. +For each `embeddings` with `labels` and `ref_emb` with `ref_labels`, positive pair `(embeddings[i], ref_emb[j])` are defined when `labels[i] == ref_labels[j]`. +When `embeddings` and `ref_emb` are augmented versions of each other (e.g. SimCLR), `labels[i] == ref_labels[i]` (see [SelfSupervisedLoss](losses.md#selfsupervisedloss)). +Note that multiple positive pairs can exist if the same label is present multiple times in `labels` and/or `ref_labels`. + +Instead of passing labels (`NTXentLoss(embeddings, labels, ref_emb=ref_emb, ref_labels=ref_labels)`), `indices_tuple` could be passed (see [`pytorch_metric_learning.utils.loss_and_miner_utils.get_all_pairs_indices](https://github.com/KevinMusgrave/pytorch-metric-learning/blob/master/src/pytorch_metric_learning/utils/loss_and_miner_utils.py)). ```python losses.NTXentLoss(temperature=0.07, **kwargs) ``` @@ -799,6 +806,16 @@ losses.NTXentLoss(temperature=0.07, **kwargs) * **temperature**: This is tau in the above equation. The MoCo paper uses 0.07, while SimCLR uses 0.5. +**Other info:** + +For example, consider `labels = ref_labels = [0, 0, 1, 2]`. Two losses will be computed: + +* Positive pair of indices `[0, 1]`, with negative pairs of indices `[0, 2], [0, 3]`. + +* Positive pair of indices `[1, 0]`, with negative pairs of indices `[1, 2], [1, 3]`. + +Labels `1`, and `2` do not have positive pairs, and therefore the negative pair of indices `[2, 3]` will not be used. + **Default distance**: - [```CosineSimilarity()```](distances.md#cosinesimilarity) From fac0fe4afafc0128ad7dc587f8497a45284a8e8a Mon Sep 17 00:00:00 2001 From: Kevin Musgrave Date: Thu, 20 Jul 2023 21:40:07 -0400 Subject: [PATCH 2/2] Minor rewording and reorganization of NTXentLoss docs --- docs/losses.md | 34 +++++++++++++++++++--------------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/docs/losses.md b/docs/losses.md index 641da746..af3cd76e 100644 --- a/docs/losses.md +++ b/docs/losses.md @@ -788,12 +788,26 @@ This is also known as InfoNCE, and is a generalization of the [NPairsLoss](losse - [Momentum Contrast for Unsupervised Visual Representation Learning](https://arxiv.org/pdf/1911.05722.pdf){target=_blank} - [A Simple Framework for Contrastive Learning of Visual Representations](https://arxiv.org/pdf/2002.05709.pdf){target=_blank} -In the equation below, loss is computed for each positive pair, `k_+`, in a batch, normalized by all pairs in the batch, `k_i in K`. -For each `embeddings` with `labels` and `ref_emb` with `ref_labels`, positive pair `(embeddings[i], ref_emb[j])` are defined when `labels[i] == ref_labels[j]`. -When `embeddings` and `ref_emb` are augmented versions of each other (e.g. SimCLR), `labels[i] == ref_labels[i]` (see [SelfSupervisedLoss](losses.md#selfsupervisedloss)). -Note that multiple positive pairs can exist if the same label is present multiple times in `labels` and/or `ref_labels`. +??? "How exactly is the NTXentLoss computed?" + + In the equation below, a loss is computed for each positive pair (`k_+`) in a batch, normalized by all positive and negative pairs in the batch that have the same "anchor" embedding (`k_i in K`). + + - What does "anchor" mean? Let's say we have 3 pairs specified by batch indices: (0, 1), (0, 2), (1, 0). The first two pairs start with 0, so they have the same anchor. The third pair has the same indices as the first pair, but the order is different, so it does not have the same anchor. + + Given `embeddings` with corresponding `labels`, positive pairs `(embeddings[i], embeddings[j])` are defined when `labels[i] == labels[j]`. Now let's look at an example loss calculation: + + Consider `labels = [0, 0, 1, 2]`. Two losses will be computed: + + * A positive pair of indices `[0, 1]`, with negative pairs of indices `[0, 2], [0, 3]`. + + * A positive pair of indices `[1, 0]`, with negative pairs of indices `[1, 2], [1, 3]`. + + Labels `1`, and `2` do not have positive pairs, and therefore the negative pair of indices `[2, 3]` will not be used. + + Note that an anchor can belong to multiple positive pairs if its label is present multiple times in `labels`. + + Are you trying to use `NTXentLoss` for self-supervised learning? Specifically, do you have two sets of embeddings which are derived from data that are augmented versions of each other? If so, you can skip the step of creating the `labels` array, by wrapping `NTXentLoss` with [`SelfSupervisedLoss`](losses.md#selfsupervisedloss). -Instead of passing labels (`NTXentLoss(embeddings, labels, ref_emb=ref_emb, ref_labels=ref_labels)`), `indices_tuple` could be passed (see [`pytorch_metric_learning.utils.loss_and_miner_utils.get_all_pairs_indices](https://github.com/KevinMusgrave/pytorch-metric-learning/blob/master/src/pytorch_metric_learning/utils/loss_and_miner_utils.py)). ```python losses.NTXentLoss(temperature=0.07, **kwargs) ``` @@ -806,16 +820,6 @@ losses.NTXentLoss(temperature=0.07, **kwargs) * **temperature**: This is tau in the above equation. The MoCo paper uses 0.07, while SimCLR uses 0.5. -**Other info:** - -For example, consider `labels = ref_labels = [0, 0, 1, 2]`. Two losses will be computed: - -* Positive pair of indices `[0, 1]`, with negative pairs of indices `[0, 2], [0, 3]`. - -* Positive pair of indices `[1, 0]`, with negative pairs of indices `[1, 2], [1, 3]`. - -Labels `1`, and `2` do not have positive pairs, and therefore the negative pair of indices `[2, 3]` will not be used. - **Default distance**: - [```CosineSimilarity()```](distances.md#cosinesimilarity)