BertForMultiLabelSequenceClassification not initialized

[DraculaXly]

11/28/2019 20:03:03 - INFO - transformers.modeling_utils - loading weights file chinese_L-12_H-768_A-12\pytorch_model.bin
11/28/2019 20:03:05 - INFO - transformers.modeling_utils - Weights of BertForMultiLabelSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
11/28/2019 20:03:05 - INFO - transformers.modeling_utils - Weights from pretrained model not used in BertForMultiLabelSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias'

When I use MultiLabels, I got the message above.
First, I have downloaded the pytorch_model.bin, vocab.txt and config.json.
Second, Should the label value be only 0 or 1? Because each of my label has four values [-2, -1, 0 ,1].

[zampierimatteo91]

I believe the labels should be binarized.

[adiv5]

Getting the same issue about initializations.any update?

[muzier]

11/28/2019 20:03:03 - INFO - transformers.modeling_utils - loading weights file chinese_L-12_H-768_A-12\pytorch_model.bin
11/28/2019 20:03:05 - INFO - transformers.modeling_utils - Weights of BertForMultiLabelSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
11/28/2019 20:03:05 - INFO - transformers.modeling_utils - Weights from pretrained model not used in BertForMultiLabelSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias'

When I use MultiLabels, I got the message above.
First, I have downloaded the pytorch_model.bin, vocab.txt and config.json.
Second, Should the label value be only 0 or 1? Because each of my label has four values [-2, -1, 0 ,1].

Where do you put the pretrained model config file in the package? Thanks~

[yunju63]

It is because the weights which are specific for fine-tuning do not(and of course can not) come from pertained model. Also, the weights specific for pre-training are not used in fine-tuning.

Refer to this issue