How to use NTXentLoss as in CPC?

[vgaraujov]

Hello! Thanks for this incredible contribution.

I want to know how to use the NTXentLoss as in CPC model. I mean, I have a positive sample and N-1 negative samples.

Thank you for your help in this matter.

[KevinMusgrave]

If you have just a single positive pair in your batch:

from pytorch_metric_learning.losses import NTXentLoss
loss_func = NTXentLoss()

# in your training loop
batch_size = data.size(0)
embeddings = your_model(data)
labels = torch.arange(batch_size)
# The assumption here is that data[0] and data[1] are the positive pair
# And there are no other positive pairs in the batch
labels[1] = labels[0]
loss = loss_func(embeddings, labels)
loss.backward()

If your batch size is N, and you have N/2 positive pairs:

from pytorch_metric_learning.losses import NTXentLoss
loss_func = NTXentLoss()

# in your training loop
batch_size = data.size(0)
embeddings = your_model(data)
# The assumption here is that data[0] and data[1] are a positive pair
# data[2] and data[3] are the next positive pair, and so on
labels = torch.arange(batch_size)
labels[1::2] = labels[0::2]
loss = loss_func(embeddings, labels)
loss.backward()

Basically you need to create labels such that positive pairs share the same label.

[vgaraujov]

Thank you for your answer @KevinMusgrave
To be sure how it works...

Regarding this assumption, data[0] and data[1] are positive pairs, so the rest (data[2:]) will be used as negatives samples?

If you have just a single positive pair in your batch:

from pytorch_metric_learning.losses import NTXentLoss
loss_func = NTXentLoss()

# in your training loop
batch_size = data.size(0)
embeddings = your_model(data)
labels = torch.arange(batch_size)
# The assumption here is that data[0] and data[1] are the positive pair
# And there are no other positive pairs in the batch
labels[1] = labels[0]
loss = loss_func(embeddings, labels)
loss.backward()

Does it mean that all examples that are not positive samples in the batch automatically used as negative samples?

[KevinMusgrave]

Yes, data[2:] will be used as negative samples, because their labels are different from data[0] and data[1]. And data[0] and data[1] are the only positive pair because there are no other labels that occur more than once.

Regarding this assumption, data[0] and data[1] are positive pairs, so the rest (data[2:]) will be used as negatives samples?

To be clear, data[0] is not a pair by itself. It forms a positive pair with data[1]. Similarly, data[0] forms a negative pair with data[2], data[3]...data[N].

[vgaraujov]

Thank you so much @KevinMusgrave !

[YounkHo]

I'm still confused that if I have a batch of randomly sampled image and their corresponding label. How can I use NTXentLoss?

from pytorch_metric_learning.losses import NTXentLoss
loss_func = NTXentLoss()

# in your training loop
batch_size = data.size(0)
embeddings = your_model(data)
labels = torch.arange(batch_size)
# The assumption here is that data[0] and data[1] are the positive pair
# And there are no other positive pairs in the batch
labels[1] = labels[0]
loss = loss_func(embeddings, labels)
loss.backward()

From code provided, why can we regard data[0] and data[1] as positive sample while other pairs are negative?

[KevinMusgrave]

That assumption was in response to the original question. If you have labels, then you can ignore the above discussion and just do:

loss = loss_func(embeddings, labels)

[YounkHo]

Then how can NTXentLoss distinguish positive sample or negative? Images are labeled with their own labels.

[KevinMusgrave]

Images with the same label form positive pairs, and images with different labels form negative pairs.

For example, if the labels in a batch are [0, 0, 1, 1, 1] then:

• the positive pairs will be formed by indices [0, 1], [1, 0], [2, 3], [2, 4], [3, 2], [3, 4], [4, 2], [4, 3]
• the negative pairs will be formed by indices [0, 2], [0, 3], [0, 4], [1, 2], [1, 3], [1, 4], [2, 0], [2, 1], [3, 0], [3, 1], [4, 0], [4, 1]

[YounkHo]

What if there is no same label pairs as batch are sampled randomly?

[KevinMusgrave]

Then NTXentLoss will return 0, because it requires positive pairs to compute an actual loss.

[KevinMusgrave]

You can try using MPerClassSampler to ensure there are positive pairs in every batch.