[Question] What is the recommended way to run Triton ?

[MatthieuToulemont]

I am running Triton behind a reverse proxy and flask.
I am mostly curious about how other people run triton in practice and also what is the recommended way from Nvidia ?

For context we process tens of millions of images per day at a sub second latency on traditional vision tasks and diffusion related tasks

[kthui]

cc @GuanLuo @tanmayv25 on "how other people run triton in practice and also what is the recommended way".

[tanmayv25]

@MatthieuTPHR Can you elaborate more on this? Triton has a very wide adoption. Depending upon the use-case, customer use Triton as an inference microservice. For this Triton gRPC and HTTP end-points can be utilized. The users can also write their own service/application and link it to Triton shared library using C API interface.
More info here: https://github.com/triton-inference-server/server/blob/main/docs/customization_guide/inference_protocols.md

If you would like to learn more about the optimization then you can consult:

1. https://github.com/triton-inference-server/server/blob/main/docs/user_guide/optimization.md
2. https://github.com/triton-inference-server/server/blob/main/docs/user_guide/performance_tuning.md

[MatthieuToulemont]

Hello,

Does this happen when there are large number of inflight requests on the server?

Indeed, this happens when there is a surge of requests to the server.

Are you using Triton python HTTP or gRPC clients?

We are indeed using the gRPC client. In our configuration we have 15 flask workers each running an instance of tritonclient.grpc.InferenceServerClient and use the default parameters

• Does each instance of InferenceServerClient reuses Stubs / channels ?
• I am serving multiple models per Triton container, does each instance of InferenceServerClient have a unique stub / channel / connection per model or does it create a new stub / channel / connection per model at each request.

It looks like having parallel flask workers sending requests to Triton is a "good way" to reach the maximum limit of concurrent streams and that a framework like FastAPI might more suited as it has only one worker but I might be wrong.

Overall would you recommend gRPC or HTTP ?

What configuration would you (Nvidia) recommend in the case where we want to serve multiple models per Triton container at a very high number of requests per second ?

[sourabh-burnwal]

@tanmayv25, my use-case involved flask only for admin usage like model state maintenance, and health checks. I was not routing inference requests through flask like @MatthieuTPHR is doing.

[MatthieuToulemont]

@sourabh-burnwal thank you for your help ! If I may ask, how do you usually route requests to Triton ?

[tanmayv25]

Does each instance of InferenceServerClient reuses Stubs / channels ?

For Python gRPC, each instance of InferenceServerClient creates a new channel. It does not reuse the channel.

I am serving multiple models per Triton container, does each instance of InferenceServerClient have a unique stub / channel / connection per model or does it create a new stub / channel / connection per model at each request.

There is a single channel connection per InferenceServerClient. The same channel is used for all requests (all model infer requests and also non-infer requests are pushed through the same channel)

We don't have a specific recommendation to follow. However, our C++ client library is more tuned for performance. The C++ gRPC does re-use the channel.

[MatthieuToulemont]

Thank you very much !

[ken-kuro]

Hi there @MatthieuToulemont, I'm trying to setup an FastAPI/Flask to make inference request to Triton Inference Server, would you mind share some of your experiences on this, especially on when it comes to a surge of requests to the server? Have a nice day