[CI] [BE] ROCM wheel build CI jobs failing

[danielvegamyhre]

As a test, I created a test PR branched off of main, which just adds a print statement to setup.py: #2545

As you can see, the ROCM wheel builds fail in CI.

Example: https://github.com/pytorch/ao/actions/runs/16298678380/job/46027251126?pr=2545

cc @jerryzh168 @jcaip

[danielvegamyhre]

@petrex I see you set up ROCM wheel build here: 2c901b3 would you mind taking a look at this issue?

[petrex]

looks like an infra issue to me .... error out because no AMD GPU is available.
also do you guys notice some CI jobs have been cancelled starting last Wed...

[danielvegamyhre]

looks like an infra issue to me .... error out because no AMD GPU is available.
also do you guys notice some CI jobs have been cancelled starting last Wed...

@wdvr are you aware of any ROCM infra issues that could affect our CI?

[petrex]

Its both cuda and rocm...might be some infra outage??

[danielvegamyhre]

Its both cuda and rocm...might be some infra outage??

many of our tests that run on cuda sm89, sm100 are passing though

[danielvegamyhre]

Update: noticed pytorch nightly rocm build is used in CI, so I went to the file were the error is thrown and found a PR related to ROCM was submitted 2 days ago, around the time of this failure starting. Looped the authors in here: pytorch/pytorch#156903 (comment)

[danielvegamyhre]

@petrex it seems this line of code added recently is not playing nicely with the runner configuration used for building AMD wheels:
https://github.com/pytorch/pytorch/blob/e4c17d5e1ccd0e730caef484af291243bc1d9cde/torch/distributed/tensor/experimental/_attention.py#L48

Any ideas on what could be wrong? Perhaps torch.cuda.get_device_properties does not have handling for the case where we are compiling/building for AMD/ROCM but have no GPUs attached? I see this function used for CUDA in other places in the code base, and we also build for CUDA without devices attached