Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

shift torch dynamo handling to accelerate #23168

Merged
merged 15 commits into from
May 31, 2023
Merged

shift torch dynamo handling to accelerate #23168

merged 15 commits into from
May 31, 2023

Conversation

pacman100
Copy link
Contributor

What does this PR do?

  1. Shits the torch dynamo handling to accelerate
  2. Should be merged after move fsdp handling to accelerate #23158
  3. No user facing change. Now, users can use accelerate launch for torch dynamo, e.g.,
accelerate launch --dynamo_backend=inductor ./examples/pytorch/text-classification/run_glue.py   --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 32   --learning_rate 2e-5   --num_train_epochs 3   --output_dir ~/temp/$TASK_NAME/ --fp16 --overwrite_output_dir --pad_to_max_length --dataloader_drop_last

Current usage like below is unimpacted:

python ./examples/pytorch/text-classification/run_glue.py   --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 32   --learning_rate 2e-5   --num_train_epochs 3   --output_dir ~/temp/$TASK_NAME/ --fp16 --overwrite_output_dir --torch_compile --pad_to_max_length --dataloader_drop_last

@pacman100 pacman100 requested review from sgugger and muellerzr May 5, 2023 12:11
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented May 5, 2023 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

muellerzr
muellerzr approved these changes May 5, 2023
View reviewed changes
Copy link
Contributor

@muellerzr muellerzr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice and clean, great!

sgugger
sgugger approved these changes May 5, 2023
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot!

@pacman100 pacman100 mentioned this pull request May 9, 2023
Merged
@pacman100 pacman100 changed the base branch from smangrul/accelerate-fsdp-integrate to main May 10, 2023 04:46
@pacman100 pacman100 changed the base branch from main to smangrul/accelerate-fsdp-integrate May 10, 2023 04:46
Base automatically changed from smangrul/accelerate-fsdp-integrate to main May 31, 2023 08:40
@pacman100 pacman100 merged commit 03db591 into main May 31, 2023
@pacman100 pacman100 deleted the smangrul/accelerate-dynamo-integrate branch May 31, 2023 09:12
sheonhan pushed a commit to sheonhan/transformers that referenced this pull request Jun 1, 2023
@pacman100 @sheonhan
shift torch dynamo handling to accelerate (huggingface#23168)
0514300 
* mixed precision support via accelerate

* fix issues

* fix for the sharded ddp case

* fix flax and tf failing tests

* `refactor the place to create `Accelerator` object

* move ddp prep to accelerate

* fix 😅

* resolving comments

* move fsdp handling to accelerate

* fixex

* fix saving

* shift torch dynamo handling to accelerate
gojiteji pushed a commit to gojiteji/transformers that referenced this pull request Jun 5, 2023
@pacman100 @gojiteji
shift torch dynamo handling to accelerate (huggingface#23168)
d68c405 
* mixed precision support via accelerate

* fix issues

* fix for the sharded ddp case

* fix flax and tf failing tests

* `refactor the place to create `Accelerator` object

* move ddp prep to accelerate

* fix 😅

* resolving comments

* move fsdp handling to accelerate

* fixex

* fix saving

* shift torch dynamo handling to accelerate
novice03 pushed a commit to novice03/transformers that referenced this pull request Jun 23, 2023
@pacman100 @novice03
shift torch dynamo handling to accelerate (huggingface#23168)
241aa09 
* mixed precision support via accelerate

* fix issues

* fix for the sharded ddp case

* fix flax and tf failing tests

* `refactor the place to create `Accelerator` object

* move ddp prep to accelerate

* fix 😅

* resolving comments

* move fsdp handling to accelerate

* fixex

* fix saving

* shift torch dynamo handling to accelerate
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sgugger sgugger sgugger approved these changes

@muellerzr muellerzr muellerzr approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@pacman100 @HuggingFaceDocBuilderDev @muellerzr @sgugger