With Vision Transformer

[wangxinghangcnn]

Hello, I seem to have encountered an issue that has been bothering me for two weeks. I would greatly appreciate it if you could reply. During the training process of VIT, I used the Clamelyon16 dataset to predict the presence or absence of cancer, but the accuracy of the training set is often about 10 points higher than that of the validation set. I tried increasing the maxNumBlocks to 1000 or 1500, but the final accuracy compared to the training set was still 7-9 points lower. Since I need to modify the VIT model, I downloaded the official VIT from Github and used the base_32_model, just like you did, with 50% of the weights frozen. However, my model is prone to overfitting, and modifying the drop_path_ratio and drop_ratio did not have a significant effect. In between, I also added cosine annealing learning rate adjustment, and now I don't know where the problem lies.