Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce TPU Pod launching to accelerate launch #1049

Merged
merged 11 commits into from
Feb 9, 2023
Merged

Conversation

muellerzr
Copy link
Collaborator

Proposed API:

accelerate launch now allows for a configured pod setup through three new params/config items:

  • use_cluster, whether to use a TPU cluster
  • vm, this mimics xla_dist's vm argument and is a list of single compute VM names if you are not using an instance group. (generally not needed)
  • env, this is a list of environment variables to set on each of the compute VM instances

Currently we only support non-Docker, as GCP doesn't support docker yet on the larger pods.

To launch a script on a TPU pod, the API will look like such:

Fully configured:

accelerate launch myscript.py --arg1 --arg2 ...

No configuration:

accelerate launch --tpu --use_cluster myscript.py --arg1 --arg2 ...

Note that if file permissions are given on your GCP cluster, there is a --tpu_use_sudo flag for this, and I recommend also doing sudo accelerate launch ... plus sudo pip install git+https://github.com/huggingface/accelerate. On my specific configuration it was needed, but other users reported they didn't so it's optional.

Closes #501 and closes #471

@muellerzr muellerzr added enhancement New feature or request TPU Bug or feature on TPU platforms labels Feb 8, 2023
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Feb 8, 2023

The documentation is not available anymore as the PR was closed or merged.

Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work on this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request TPU Bug or feature on TPU platforms
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Questions wrt training on TPU Pod Using Accelerate with TPU Pod VM like v3-32
3 participants