[CI][GPU] Add jobs for CUDA 13

[raulcd]

Describe the enhancement requested

The runs-on GPU instances we use are built with CUDA 12. See log from nvidia-smi:

| NVIDIA-SMI 575.57.08              Driver Version: 575.57.08      CUDA Version: 12.9     |

I tried installing cuda-compat-13-0 as suggested here:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08              Driver Version: 575.57.08      CUDA Version: 13.0     |
|-----------------------------------------+------------------------+----------------------+

But the job still failed with:

cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_IMAGE: This indicates that the device kernel image is invalid. This can also indicate an invalid CUDA module.

We might need to, build our own instances with packer installing newer versions of CUDA as suggested here, wait until runs-on provides images with CUDA 13 or try reinstalling the drivers on the instance.

This is a follow up issue to:

• [CI][GPU][C++][Python] Add new jobs for cuda #48582

Component(s)

Continuous Integration, GPU

[raulcd]

We used CUDA 13.0.2 in the old voltron data runners.

[HyukjinKwon]

I don't have much context but taking a look at this for fun at #48796 more to understand it better!

[raulcd]

Thanks for taking a look! We should merge the PR to add the new runner jobs first and then we can modify the CUDA 12 jobs to use CUDA 13. One possibility, in-line with what you were trying is to install CUDA 13 drivers on those runners. The issue and associated PR that should be merged first:

• [CI][GPU][C++][Python] Add new jobs for cuda #48582
• GH-48582: [CI][GPU][C++][Python] Add new CUDA jobs using the new self-hosted runners #48583