Fix bug in BEiT adapters

[jannik-brinkmann]

This pull request fixes a bug in the BEiT adapter implementation, which I contributed in the first place. Although the tests were successful and the model converged on the test data set, during latest experimentation I became aware that the model in Adapter-Transformers, when it was not equipped with adapters, did output different embeddings than the original HF-Transformers model, which is obviously not intended. The bug fix contains two changes:

• Move the attention adapters call to the first residual connection, instead of calling it before lambda_1.
• Move the output adapters call into BeitLayer. In contrast to e.g. ViT it cannot be called in Output, due to the necessary lambda_2 and drop_path calls in BeitLayer before adapters need to be called.

This ensures that the model without adapters has the same output as the original HF-Transformers model it is built upon. In addition, I observed a significantly faster convergence on the same training data set as before (EuroSAT): previously it achieved 88 % accuracy after 10 epochs, now it achieves 97.4 % accuracy after 1 epoch and 98.2 % accuracy after 3 epochs. The newly observed convergence rate is similar to that of ViT with adapters.

[calpt]

Looks good, thanks for the detailed investigation on this and the bug fix!