Using ViTDet with a pretrained model that has a cls_token

[dgcnz]

Currently, the ViT used in the ViTDet project doesn't have a cls_token and the SimpleFeaturePyramid doesn't support cls_tokens as it expects patch-level features of shape (batch, hidden_dim, patch_size, patch_size). If I wanted to try a pretrained model like dinov2 which has a cls_token, what would be the correct way to do so?

1. Ignore the cls_token from the start, that is, drop it after the pos embedding and before the transformer blocks.
2. Ignore the cls_token after the transformer blocks and before the final layer norm.

Thanks :^)