[Question] GPU Memory Related

[chagmgang]

In your research, it is hard to train model with long sequence such as 768 in gpu.
However, I can't find any special way to reduce gpu memory in your code.
I want to know about your technique for training long sequence tokens with vision transformer.

[ronghanghu]

Hi @chagmgang, in our experiments, we used Google Cloud TPUs for pretraining. For GPUs, I think one can use gradient accumulation (with --accum_iter) and activation checkpointing to reduce memory usage