-
-
Notifications
You must be signed in to change notification settings - Fork 218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Encoding 1.5GB corpus file runs out of RAM #213
Comments
I have run into an issue where specific files, for whatever reason, can crash the tokenizer. I've had a 15kb XML file swallow 30GBs of RAM. I'm not really sure why some files cause this, but perhaps that's the issue you're running into? |
kaggle gives you 16gigs of ram and you would need like 100 or more gigs of ram to encode it (edit this is super old and it should use cpu and not ram) |
I got basic setup of training tokenizer(on a subset of corpus), TokenDataset of a big corpus and training
I run out of RAM while creating TokenDataset(I got 30gb ram on kaggle). This seems strange, given original corpus size is far less than 30GB
Is there a way to encode corpus into other file(with smaller batches) and then load them lazily for training?
The text was updated successfully, but these errors were encountered: