NVIDIA GPU PCI pass through not working for Talos VM image

[purankhoeval]

Hello! I'm using LXD to spin up a VM and have attached a GPU through device config and able to see the passthrough GPU attached through VFIO-PCI driver. ( I have blacklisted NVIDIA Host drivers)

Further I have installed Talos OS image built with the requisite system extensions for Kata containers, NVIDIA container tool kit and open source GPU. The modules are patched with the patch file described in the Talos docs however in Talos console I see the error as NVIDIA kernel modules are not loaded and NVRM: This PCI I/O region assigned to your NVIDIA device is invalid.

Am I fundamentally missing something here. New in this space, any help is much appreciated!