Visualisation of Loss value

[aretius]

Thanks for sharing such an awesome working.

Since you have used weighted GCL, I wonder how the loss values actually looked like. Usually in contrastive losses there are lot of stability issues - was wondering how the train and validation loss curve looked like in your cases.
If you can't share that - would be happy to know how were the instability handled?

[alanzty]

Thanks for your interest in our work. We didn't do anything extra to stabilize the training process. Did you try using a large batchsize? The loss is aggregated over the batch, so larger batchsize would lower the loss variance.

[aretius]

I am using gradient accumulation to a batch size of 4096.
But my train and validation loss are quite high on the order of ~2. It does decrease from 2.7 to 2 finally after a few epochs but then kind of saturates.
In my previous experiments with open_clip I have usually seen good performance if my loss is around 0.2-0.4.

[alanzty]

Which code and data did you use to train?

[aretius]

Since there are no training scripts in the repo, I followed the procedure mentioned in the paper - parameters in similar range and used the open_clip implementation.
I was trying it on one of the fashion datasets - DeepFashion.
If you want i can share the code here

[alanzty]

If you train on DeepFashion, what ranking score do you use? It is possible for the validation loss to stay high, because if the ranking is arbitrary, then it doesn't generalize from training set to validation set. But the training loss should converge.

[aretius]

I currently created a custom ranker using LLM based on product categories. I just wanted to test on product categories and ranked them based on similarity to the text.
My training loss keeps on decreasing but validation loss unfortunately plateaus. My train loss goes to 0.7 but validation loss keeps hovering around 1.5. The loss starts at ~3.5 for both.

• Have you observed something similar?
• Also since you trained GCL on Google shopping queries did you do random negative sample or ensured that in a batch there are only unique queries?

[alanzty]

The gap between validation and training loss looks quite normal. Tho i found it is often possible to get much lower training loss than 0.7.

That is a good question. I implemented something to avoid query collision but haven't found a reliable way to solve this problem yet.

[aretius]

Understood thanks for the insight.
So in your experience query collision is not a good thing right? Ideally i should sample in a batch all different query items?