Look into domain adaptive pre-training

[bruffridge]

The last thing is "domain adaptive pre-training". If you're using SciBERT you're most of the way there already, so it might not give you that much. But the idea is to take vanilla BERT, fine-tune it as a language model on your entire corpus of domain-specific documents, and then fine-tune again for the classification task. Dirk