ensemble multi-GPU

[xiazi-yu]

In ensemble mode, is it possible to set instance_group gpus: [0,1,2,3] in the config.pbtxt so that the pipeline runs in the following manner: preprocessing (GPU0) ---> inference (GPU0), preprocessing (GPU1) ---> inference (GPU1), preprocessing (GPU2) ---> inference (GPU2), preprocessing (GPU3) ---> inference (GPU3), instead of preprocessing (GPU0) ---> inference (GPU1)?

[rmccorm4]

Hi @xiazi-yu, I believe this is not possible to force which GPUs are selected when scheduling between models within an ensemble and multiple GPU choices (multiple model instances) are available.

CC @Tabrizian as I believe you investigated this in the past, in case you have any extra comments.

[xiazi-yu]

My ensemble model consists of over a dozen sub-models. When using multiple GPUs compared to a single GPU, there is a significant performance drop.