Kaggle and TPUs - Errors popping randomly in and out of existence

[alexander-soare]

This is a general report about what it's like to work with PyTorch Lightining and TPUs on Kaggle. The reason I post it here instead of the XLA repo is that the PyTorch Lightning marketing openly invites us to use Kaggle. The reason I did not follow the bug template, is because I'm reporting something more general than one specific bug.

Over the past few days of trying to use the TPUs I've run into various errors that randomly pop into and out of existence like quantum particles. I can't paste the errors here because of their elusive nature, and I didn't have the hindsight of knowing it would come to this. Here are their descriptions:

• OSError: Undefined Symbol
• Cannot replicate if number of devices (1) is different from 8
• Some "failed to meet rendezvous point" error.
• Execution hangs indefinitely after INIT TPU local core: 0, global rank: 0 with XLA_USE_BF16=None. Also for 8 cores
• For 8 cores only, execution hangs indefinitely when progress bar appears (PyTorch Lightning)

Note that for ALL of these examples I've sometimes been able to do some combination of the following to fix it (without changing my code at all)

• Restart runtime
• Change to standard runtime then change back to TPU
• Refresh the page
• Run the same cell again

Unfortunately, I spend 80% of my time restarting my runtime and reinstalling XLA, and only 20% of my time feeling hopeful that "maybe I've fixed it for good"

By the way, this is how I set up XLA:

!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev

[Borda]

@zcain117 this seems to be directly torch XLA, right?

[alexander-soare]

Perhaps. I may try there too (in fact I have already posted one of the bugs there). Never sure who's going to want to call "mine" on this one. May also belong to Kaggle. The logic of choosing Lighting was that they seem to have a vested interest (amongst others) in making sure Kagglers can use it.

[tchaton]

Dear @alexander-soare,

Mind trying this PR: #6272

Best,
T.C

[alexander-soare]

@tchaton just gave it a shot here https://www.kaggle.com/alexandersoare/pytorch-lightning-tpu-error?scriptVersionId=55790594
The undefined symbol error is one of those ones that I've seen happening sometimes (and sometimes not).

Also tried it on another private notebook and got the issue where execution just hangs after I get confirmation that I'm using a TPU with 8 cores.

[tchaton]

Dear @alexander-soare,

Yes, I had still another error when trying your notebook coming from torchvision

.

I will try to run it with our docker image to see if it is third party bug or Lightning bug.

Best,
T.C

[tchaton]

Dear @alexander-soare,

I have run the script several times and it seems to be working fine inside a TPU VM.

I will try to debug the Kaggle Notebook, but it seems harder to properly reproduce the bug.

Best,
T.C

[edenlightning]

@tchaton any update?

[alexander-soare]

@tchaton it is tough to debug. I didn't even know how to write the report. It's just a bunch of different things, perhaps further complicated by the fact that it's running through Kaggle. Thanks for looking into it.