About bad test performance.

[huangzy-55]

Hi, I tried to run and test the code, but here are some issues, can you help me?
Cause my GPU is not enough, I tried to save the representations after coca and then load them to train the model. But after doing it, the test performance is very bad of novel tasks and novel steps, but is good as the article saying of novel scenes and abstraction. For what reason this may happen?

[RishiHazra]

Hi @huangzy-55, to get some clarity, can you tell me exactly which baseline you're trying to reproduce or modify? From what I understand, you're using the CoCa representations for training the NSG model? Is that the case?

[huangzy-55]

Yes. I am using vis_feat='coca',text_feat='coca',context_encoder='mha' this config to train NSG. And coca is not finetuned by me. I just download it from openclip of the given link.

[RishiHazra]

@huangzy-55
Can you tell me how many epochs you trained it for? Based on the training details in the paper (Sec 10.1), it should be
--epochs 50.

If you've already tried that, can you try training with frames per segment = 12 instead of 20 (default value) like so --fp_seg 12 ? If the problem persists, I can try replicating it on my end.

[huangzy-55]

I have trained 80 epochs. But I didn't try to change fp_seg, will it make any difference? For what reason?

[RishiHazra]

fp_seg = 12 should idelally increase your Novel Tasks performance, and likely decrease your Novel Steps performance. I'm just trying to narrow down the error. If it doesn't change as expected, there could be an issue with the loading of features.

[huangzy-55]

OK. I will train this way to see the test performance.

[huangzy-55]

Hi, I am still working on this, could you please upload the tsv file for CTV t5 pre-training? I really need it.

[huangzy-55]

And, what is the groundtruth of train and test files in CTV you uploaded, and how to use the json files in meta folder ?

[RishiHazra]

@bc2754 Hi Brian, could you take a look at this? Thanks