del tokenizer doesn't work for me in OOM

[goodstudent9]

          > > > Are you sure? It seems that the deepcopy is ok there. The tokenizer will be removed after the code run out of the scope of the preprocess_qwen

It was definitely a source of memory leakage for me. Give it a try, sadly I'm still unable to prepare the MR :(

We add:

    del tokenizer

and it is solved. Hardly know why, but it works. Ahhh, I guess I have to say "amazing".

That seems like del tokenizer doesn't work for me. The memory usage still becomes higher and higher......
DO you know why?
I added the del tokeizer in the end of the preprocess_qwen(). But at first, the memory usage is 40G, but after 7 hours, it becomes 60G, so every new epoch will add 10GB in memory......
Why, that is so wired!

Originally posted by @goodstudent9 in #196 (comment)