Adding BERT model and protocol = transformer encoder model + masked language modeling (MLM)

[prihoda]

This is a very worthwhile effort. Are you considering adding the BERT transformer encoder model and the associated masked language modeling task for pre-training?

The task is actually the same as ResidueClassificationSolver, but it would only accept one sequence file (the output) and generate the randomly masked input on the fly. This could be done by a special type of Dataset, that's how fairseq implements this: https://github.com/facebookresearch/fairseq/blob/main/fairseq/data/mask_tokens_dataset.py

One issue I realized though is that the data might not fit into memory, so you would need to rewrite some of the logic. But at least for finetuning existing language models (which might be the main usecase) it would work even in memory.