where is " [LSEP]" located in?

[jiaxiansen123]

When running the command of "Model Evaluation",my generated resulet only contains "". Could you help me find the relevant code about " [LSEP]".Thank you very much.

[ybai-nlp]

Hi, I used the '[unused5]' token to represent '[LSEP]' in BERT dictionary, hence the id of '[LSEP]' should be "5" in the model.

This modification don't need code change in model architecture, I directly modified data file to construct the training data.

However, it is required to distinguish the position of the '[LSEP]' token during inference stage, this part of code is in the file "src/models/predictor.py".

[jiaxiansen123]

Thanks for your response sincerely!
When running NCLS+MS model, the total loss is " loss_ncls +loss_ms". However,I can't find some relevant code to represent loss_ncls or loss_ms respectively ,especially the total computation "+" to get the total loss. Also,there is same situation as loss_MCLAS.Maybe my understanding is not enough.
I'd appreciate it if you can give more explaination.

[ybai-nlp]

Hi, thanks for asking!

My implementation is pretty personal and perhaps not elegant enough.

For the NCLS+MS scenario, since Presumm used the sharded_compute_loss trick and it's hard to modify it, I directly concat the two outputs of the monolingual decoder and the cross-lingual decoder into one tensor. This part of code is in L291 model_builder.py (produce the monolingual output), and L271 in trainer.py (concatenate the outputs).

For the MCLAS loss, the output is S^A + S^B, the ground truth label is also S^A + S^B, so calculating the normal NLLloss between them is just fine.

Hope this can help you!