Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds an option to use
torch.compile
through the appropriate argument in HuggingFace'sTrainingArguments
.In my experiments, with recent versions of PyTorch and Transformers, this resulted in faster training and slightly lower GPU memory usage; more details about my experiments can be found in the discussion in #173. With this change, we can simply add
torch_compile: true
in the config files inchemnlp/experiments/configs
under thetrainer
section to usetorch.compile
, for instance, in the following line:chemnlp/experiments/configs/hugging-face/160M_full.yml
Lines 30 to 31 in 97d6b3f