Add generic support for different GPU hardware.

[harshbafna]

Is your feature request related to a problem? Please describe.

Currently, TorchServe's sanity suite, regression suite, and the recent changes related to logging GPU info in the model description are dependent on nvidia-smi command which is NVIDIA specific.

Also, the default handlers currently support Cuda devices or CPU only.

Describe the solution

Add generic support for different GPU hardware.

[chauhang]

@harshbafna What is the additional work remaining for this issue?

[harshbafna]

@chauhang: We currently only support CUDA GPU hardwares and it is hard-coded in certain sections of code and test suites. This issue has been logged to enhance this to a more generic solution and will be taken up in future releases.

[marcusnchow]

Does pytorch serve support amd rocm gpus now that pytorch officially supports them?

[msaroufim]

@marcusnchow not yet but if you could share a bit more about your interest in ROCM we should restart the discussion with the team

[marcusnchow]

There are various AMD multi-gpu systems, like AMD instinct (similiar to Nvidia's DGX systems) that are used for inference workloads and would benefited by support from pytorch serve. We have an 8 amd gpu system that we use for bench marking inference servers. We already have tensorflow and tensorflow serving running and would like to compare with pytorch for inference performance. What modifications would be needed to support AMD gpu?

[msaroufim]

Out of curiosity what's the performance difference you've observed on a few models when comparing DGX vs Instinct?

It's probably possible to support Instinct just trying to understand the immediate value, I would suspect we need to generalize all the places in the codebase that mention CUDA to support ROCM as well which given that Pytorch just supports ROCM now shouldn't be a huge pain

[marcusnchow]

oh we dont have access to a DGX, i guess i was using it as an example. but we would like to use pytorch to benchmark AMD GPUs for inference performance :)

[ismu]

Hi,
Any update on this topic? Is there a chance for AMD GPU support at this moment when we have ROCm PyTorch?