Transformer models do not use their corresponding pretrained poolers

[zphang]

The way that we are currently using Transformers models involves taking the based encoder, and extracting the full set of hidden activations (across all layers). See link. We later separately pull out only the top layer, and extract the first token representation if we're doing a single-vector task such as classification.

Because of this workflow, we do not end up using the pretrained pooler layers in the respective models, e.g. BERT, ALBERT. RoBERTa also inherits from BERT.

On the other hand, we do not expect this to be a major issue, as we have seen good results from tuning with this format across several works, e.g. https://arxiv.org/abs/1812.10860, https://arxiv.org/abs/1905.00537

[HaokunLiu]

Adding this as additional reference.

https://arxiv.org/pdf/1903.05987.pdf found diagnostic classifier on finetuned BERT layers achieves similar performance in layer 9-12 (MRPC), and layer 5-12 (STS-B). See figure 1 in the linked pdf.
This suggests pertrained layers on the top may not be that helpful to downstream pair sentence classification tasks.

[zphang]

This is an automatically generated comment.

As we update jiant to v2.x, jiant v1.x has been migrated to https://github.com/nyu-mll/jiant-v1-legacy. As such, we are closing all issues relating to jiant v1.x in this repository.

If this issue is still affecting you in jiant v1.x, please follow up at nyu-mll/jiant-v1-legacy#1062.

If this issue is still affecting you in jiant v2.x, reopen this issue or create a new one.