[Bug] Cannot provision instances with the A10 GPU on the azure backend

[peterschmidt85]

Steps to reproduce:

1. Run dstack run . -b azure --gpu A10 --spot-auto

• The run hangs for 20-ish minutes and then fails
• The server throws an error
• The instance remains as Running

[r4victor]

The serial log:

main-pink-bobcat-1-0-0-westeurope.edd24e59-a6c4-4fe5-a18f-411dec74b1d4.serialconsole.log

[r4victor]

I believe this is the reason:

The Azure NVads A10 v5 VMs only support GRID 14.1(510.73) or higher driver versions.

Azure requires NVIDIA GRID drivers for NV-series, but dstack only installs CUDA drivers.

https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup

[peterschmidt85]

Steps to reproduce the problem

1. Install the normal CUDA driver (as we do it in our packer script)
2. Reboot the instance
   It hands forever on reboot

• Reproduced

Potential fix

1. Instead of the normal CUDA driver, install the GRID driver (e.g. by following Azure's guide)

• Reproduced

Implementation notes

The Azure's guide is using 535.154.05 version and it seems to be compatible with what we use (535.54.03-1). Where to download the exact 535.54.03-1 Grid driver – I couldn't find it. Because both versions seem to be compatible, we theoretically could go with Azure's guide instructions.

It seems that for A10 for Azure, we need to build a separate AMI.