Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Augmentation is memory hungry #778

Open
sileix opened this issue Sep 12, 2021 · 0 comments
Open

Augmentation is memory hungry #778

sileix opened this issue Sep 12, 2021 · 0 comments
Labels
bug Something isn't working P2 We need to fix it (backlog) training Issues with dataset generation, augmentation, training

Comments

@sileix
Copy link
Member

sileix commented Sep 12, 2021

Currently, the augmentation step will load all parameter values to memory, which could be an issue for some cases like QA over Wikidata. We could either use mmap or do a pre-sampling over the dataset.
@gcampax gcampax added bug Something isn't working training Issues with dataset generation, augmentation, training P2 We need to fix it (backlog) labels Sep 12, 2021
gcampax added a commit that referenced this issue Dec 2, 2021
@gcampax
Make use of the new sqlite-based samplers for augmentation
6cc9dba 
Should help with #778
@gcampax gcampax mentioned this issue Dec 6, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working P2 We need to fix it (backlog) training Issues with dataset generation, augmentation, training
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gcampax @sileix