Support for multimodal model

[spoonbobo]

Does tensorrtllm_backend supports multimodal LLM like LLaVA like those listed in https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/multimodal?

[symphonylyh]

Hi @spoonbobo , not yet. We're currently working on a general backend for structures like encoder-decoder and multimodal models. Encoder-decoder work is in progress and multimodal follows it. The progress is tracked in NVIDIA/TensorRT-LLM#800.

Meanwhie, if you're referring to a Triton Python backend, do you think it's ok for users to implement a multimodal workflow based on the gpt example

tensorrtllm_backend/all_models/gpt/tensorrt_llm/1/model.py

Line 42 in 2c8c6ae

class TritonPythonModel:

?

[spoonbobo]

hi @symphonylyh. Appreciated the efforts you've put on providing general encoder-decoder support. Haven't tried implement a workflow based on this example, I think definitely worth a try.

[FernandoDorado]

I'm also very interested in these capabilities. Looking forward to try the examples provided. Is there any reference tutorial or guide?