Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update TORCH_XLA_USER_GUIDE.md #45

Open
wants to merge 1 commit into
base: llama2-google-next-inference
Choose a base branch
from
Open

Update TORCH_XLA_USER_GUIDE.md #45

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions TORCH_XLA_USER_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,11 +91,11 @@ gcloud compute tpus tpu-vm scp params_70b.json ${TPU_NAME}:params.json --zone ${
gcloud compute tpus tpu-vm ssh ${TPU_NAME} --zone ${ZONE} --project ${PROJECT_ID} --worker=all --command="cd $HOME/llama &&
PJRT_DEVICE=TPU XLA_FLAGS=--xla_dump_to=/tmp/dir_name PROFILE_LOGDIR=/tmp/home/ python3.8 example_text_completion.py --ckpt_dir . --tokenizer_path $HOME/llama/t5_tokenizer/spiece.model --max_seq_len 2048 --max_gen_len 1000 --max_batch_size 2 --mp True --dynamo True"
```
## Commands to Run Llama2 using XLA:GPU (e.g. L4 or H100)
## Commands to Run Llama2 using XLA:GPU (e.g. L4 or H100) without Quantization

`example_text_completion.py` can also be ran on GPUs with XLA:GPU. To do that, you need different wheels than the above such
`example_text_completion.py` can also be ran on GPUs with XLA:GPU without quantization. To do that, you need different wheels than the above such
that you have XLA:GPU support. Please refer to [pytorch/xla](https://github.com/pytorch/xla#wheel) repo to download
a suitable GPU nightly wheel for your environment.
a suitable GPU nightly at 2023/04/22 wheel for your environment.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we using such an old whl @ManfeiBai?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newest nightly wheel has issue of !info.content.empty(), debugging locally now, and 2023/04/22 wheel could pass


After that, you can run the following the command:
```
Expand Down