MRPC hyperparameters question

[ethanjperez]

When describing how you reproduced the MRPC results, you say:
"Our test ran on a few seeds with the original implementation hyper-parameters gave evaluation results between 82 and 87."
and you link to the SQuAD hyperparameters (https://github.com/google-research/bert#squad).

Is the link a mistake? Or did you use the SQuAD hyperparameters for tuning on MRPC? More generally, I'm wondering if there's a reason the MRPC dev set accuracy is slightly lower (in [82, 87] vs. [84, 88] reported by Google)

[thomwolf]

Hi Ethan,
Thanks we used the MRPC hyper-parameters indeed, I corrected the README.
Regarding the dev set accuracy, I am not really surprised there is a slightly lower accuracy with the PyTorch version (even though the variance is high so it's hard to get something significant). That is something that is generally observed (see for example the work of Remi Cadene) and we also experienced that with our TF->PT port of the OpenAI GPT model.
My personal feeling is that there are slight differences in the way the backends of TensorFlow and PyTorch handle the operations and these differences make the pre-trained weights sub-optimal for PyTorch.

[ethanjperez]

Great, thanks for clarifying that. Regarding the slightly lower accuracy, that makes sense. Thanks for your help and for releasing this!

[ethanjperez]

Maybe it would help to train the Tensorflow pre-trained weights for e.g. one epoch in PyTorch (using the MLM and next-sentence objective)? That may help transfer to other tasks, depending on what the issue is

[thomwolf]

Hi @ethanjperez, actually the weight initialization fix (tf. truncated_normal_initializer(stddev=0.02) was translated in weight.data.normal_(0.02) instead of weight.data.normal_(mean=0.0, std=0.02) fixed in 2a97fe2) has brought us back to the TensorFlow results on MRPC (between 84 and 88%).
I am closing this issue.

[ethanjperez]

@thomwolf Great to hear - thanks for working to fix it!