initial tensorrt ep commit

[manickavela29]

• Adds in TensorRT EP
• Observed that Encoder Model is much faster with TRT than with CUDA backend
• Initialization of onnxrt session takes more time as TRT models are generated and converted

ToDo :

• Validate Accuracy of the model inference
• Reducing the starting time

Ref : #40, #41
CC : @csukuangfj

[manickavela29]

I will send a separate PR for handling configs of OnnxRT EP configs separately.
along with TRT config options.

In the middle of something, and it might take sometime

[manickavela29]

I think the build failures are coming in from the python dependency of nvinfer and similar, I will try to add some libraries and check

[csukuangfj]

https://github.com/k2-fsa/sherpa-onnx/actions/runs/9354678305/job/25748084556#step:5:1121

[ 62%] Linking CXX executable ../../bin/sherpa-onnx
/opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: ../../lib/libsherpa-onnx-core.so: undefined reference to `OrtSessionOptionsAppendExecutionProvider_Tensorrt'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/sherpa-onnx] Error 1
make[1]: *** [sherpa-onnx/csrc/CMakeFiles/sherpa-onnx.dir/all] Error 2
make: *** [all] Error 2

I think an extra lib should be linked to for tensorrt.

[manickavela29]

yes, it seems directly having OrtSessionOptionsAppendExecutionProvider_Tensorrt() as in interface is not working out
and AppendExecutionProvider_TensorRT_V2 is actually a wrapper around that function.

so it should be alright now

[csukuangfj]

Please merge the latest master into your branch and the CI should pass or you can just ignore the failed tests.

Could you describe what users need to do to build sherpa-onnx with TensorRT support?

[csukuangfj]

please leave a comment if you think it is ready to merge.

[manickavela29]

Regarding what is required to run TensorRT,

Hardware : Nvidia GPU 😄 ,(not sure how compatible AMD GPUs are)
Software lib : TensorRT requires libnvinfer, libnvinfer-dispatch, libnvinfer-plugin, libnvonnxparsers8 (from my experience)

But seeing that CI/CD's are working fine, I think onnxrt is able to handle these,
as long as someone is having a compatible GPU it should work fine.

Just setting the appropriate provider argument should be enough,
if there are any github pages/doc, this can be updated there

--provider=trt

[manickavela29]

It can be merged,
after ensuring that basic builds are working fine

Thank you for the review 😄

[manickavela29]

Regarding what is required to run TensorRT,

Hardware : Nvidia GPU 😄 ,(not sure how compatible AMD GPUs are) Software lib : TensorRT requires libnvinfer, libnvinfer-dispatch, libnvinfer-plugin, libnvonnxparsers8 (from my experience)

But seeing that CI/CD's are working fine, I think onnxrt is able to handle these, as long as someone is having a compatible GPU it should work fine.

Just setting the appropriate provider argument should be enough, if there are any github pages/doc, this can be updated there

--provider=trt

Update this comment, by mistake I had typed ' don't think onnxrt' but edited it

[manickavela29]

Please merge the latest master into your branch and the CI should pass or you can just ignore the failed tests.

Could you describe what users need to do to build sherpa-onnx with TensorRT support

I think the builds are also good enough.
Shall we merge it @csukuangfj

[csukuangfj]

Thank you for your contribution!