Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set use_reentrant to False for gradient checkpointing #3882

Merged
merged 4 commits into from
Jan 16, 2024
Merged

Set use_reentrant to False for gradient checkpointing #3882

merged 4 commits into from
Jan 16, 2024

Conversation

arnavgarg1
Copy link
Contributor

@arnavgarg1 arnavgarg1 commented Jan 15, 2024 •
edited
Loading

Fixes the following warnings in newer versions of Torch (>= 2.1.0)

/home/ray/anaconda3/lib/python3.8/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint:
 please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be 
updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that 
you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.

and

`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...

alexsherstinsky
alexsherstinsky approved these changes Jan 15, 2024
View reviewed changes
Copy link
Collaborator

@alexsherstinsky alexsherstinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you so much for the solution for this ever-present warning!

@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 16, 2024 •
edited
Loading

Unit Test Results

       6 files  ±       0         6 suites  ±0   23m 20s ⏱️ + 9m 4s
2 889 tests +2 877  2 852 ✔️ +2 843  12 💤 +9  2 ❌ +2  23 🔥 +23 
2 925 runs  +2 865  2 876 ✔️ +2 834  24 💤 +6  2 ❌ +2  23 🔥 +23 

For more details on these failures and errors, see this check.

Results for commit 0abb640. ± Comparison against base commit b0795e7.

♻️ This comment has been updated with latest results.

@arnavgarg1 arnavgarg1 merged commit 8e8f47b into master Jan 16, 2024
16 of 18 checks passed
@arnavgarg1 arnavgarg1 deleted the reentrant branch January 16, 2024 18:07
vijayi1 pushed a commit to vijayi1/ludwig that referenced this pull request Jan 23, 2024
@arnavgarg1 @vijayi1
Set use_reentrant to False for gradient checkpointing (ludwig-ai#3882)
08dbc1a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexsherstinsky alexsherstinsky alexsherstinsky approved these changes

@w4nderlust w4nderlust Awaiting requested review from w4nderlust w4nderlust is a code owner

@tgaddair tgaddair Awaiting requested review from tgaddair tgaddair is a code owner

@justinxzhao justinxzhao Awaiting requested review from justinxzhao justinxzhao is a code owner

@geoffreyangus geoffreyangus Awaiting requested review from geoffreyangus geoffreyangus is a code owner

@jeffkinnison jeffkinnison Awaiting requested review from jeffkinnison jeffkinnison is a code owner

@Infernaught Infernaught Awaiting requested review from Infernaught Infernaught is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@arnavgarg1 @alexsherstinsky