find rocm components individually

[afzpatel]

Required for spack since all the rocm components are located in different paths

[i-chaochen]

retest Ubuntu-GPU-multi please
retest Ubuntu-CPU please

[hsharsha]

retest Ubuntu-CPU please
retest Ubuntu-GPU-single please

[afzpatel]

retest Ubuntu-GPU-single please

[hsharsha]

LGTM

[i-chaochen]

retest Ubuntu-GPU-single please

[afzpatel]

retest Ubuntu-GPU-single please

[afzpatel]

retest Ubuntu-GPU-single please

[afzpatel]

I'm not able to reproduce the ci build failure for Ubuntu-GPU-single on my local machine, any ideas what the issue might be?

[afzpatel]

retest Ubuntu-GPU-single please

[hsharsha]

I'm not able to reproduce the ci build failure for Ubuntu-GPU-single on my local machine, any ideas what the issue might be?

I think it is landing on a node where the rocm driver is not stable which causes the tests to fail. If you can add the logs to your local run, here we can disregard the CI failure and merge this change.

[i-chaochen]

retest Ubuntu-GPU-single please

[i-chaochen]

retest Ubuntu-GPU-single please

[i-chaochen]

I'm not able to reproduce the ci build failure for Ubuntu-GPU-single on my local machine, any ideas what the issue might be?

It's because one of nodes has driver issue, we already marked it offline and I'm tring to retest it another node.

[afzpatel]

Here's a log of my test run on mi100. There are only 2 test failures, one of which is that the test failed to allocate enough memory on the device. The errors I was getting on the ci are not present in the local run.
mi100_command.log

[i-chaochen]

retest Ubuntu-GPU-single please

[i-chaochen]

Here's a log of my test run on mi100. There are only 2 test failures, one of which is that the test failed to allocate enough memory on the device. The errors I was getting on the ci are not present in the local run. mi100_command.log

Yes, it wouldn't have any problem because I can see gpu-pycpp is green

[afzpatel]

Now that the pr is merged, this change needs to be backported to the r2.14-rocm-enhanced-spack branch. Also another branch called r2.16-rocm-enhanced-spack should be created with this change and f4f4e86

[hsharsha]

f4f4e86
Instead of creating a new branch w can we just use that as patch and apply it during spack tf installtion? That way we can use the rocm-2.xx-enhanced along with the patch.

[afzpatel]

f4f4e86 Instead of creating a new branch w can we just use that as patch and apply it during spack tf installtion? That way we can use the rocm-2.xx-enhanced along with the patch.

Sounds good, I'll try that out.