Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TPU Spawn] Cannot replicate if number of devices (1) is different from 8 Error #7086

Closed
kaushikb11 opened this issue Apr 18, 2021 · 0 comments · Fixed by #7074
Closed

[TPU Spawn] Cannot replicate if number of devices (1) is different from 8 Error #7086

kaushikb11 opened this issue Apr 18, 2021 · 0 comments · Fixed by #7074
Assignees
Labels
bug Something isn't working help wanted Open to be worked on priority: 0 High priority task

Comments

@kaushikb11
Copy link
Contributor

🐛 Bug

TPU available: True, using: 8 TPU cores
Exception in device=TPU:0: Cannot replicate if number of devices (1) is different from 8
Traceback (most recent call last):
  File "/anaconda3/envs/torch-xla-1.8/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 329, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/anaconda3/envs/torch-xla-1.8/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 322, in _start_fn
    _setup_replication()
  File "/anaconda3/envs/torch-xla-1.8/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 315, in _setup_replication

Environment

Note: Bugs with code are solved faster ! Colab Notebook should be made public !

You can get the script and run it with:

wget https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py
# For security purposes, please check the contents of collect_env_details.py before running it.
python collect_env_details.py
  • PyTorch Version (e.g., 1.0):
  • OS (e.g., Linux):
  • How you installed PyTorch (conda, pip, source):
  • Build command you used (if compiling from source):
  • Python version:
  • CUDA/cuDNN version:
  • GPU models and configuration:
  • Any other relevant information:

Additional context

@kaushikb11 kaushikb11 added bug Something isn't working help wanted Open to be worked on priority: 0 High priority task labels Apr 18, 2021
@kaushikb11 kaushikb11 self-assigned this Apr 18, 2021
@kaushikb11 kaushikb11 changed the title Fix Cannot replicate if number of devices (1) is different from 8 TPU Spawn Error [TPU Spawn] Cannot replicate if number of devices (1) is different from 8 Error Apr 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Open to be worked on priority: 0 High priority task
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant