Nvidia Triton Inference Server

[sazzadhrz]

Search before asking

• I have searched the YOLOv8 issues and discussions and found no similar questions.

Question

Could you please share an implementation of yolov8 to run on Nvidia Triton Inference Server?

Additional

No response

[AyushExel]

Triton server inference is not yet supported on ultralytics repo. We'll work on it soon

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

[mltoml]

I'm also facing the same issue, do you have any updates or a timeline for such implementation?
I see some possible starting points*; I tried it but for some reason the output looks strange (tested it with an onnx model deployed on triton and python tritonclient used for inferencing).

*https://github.com/dinhkt/triton-trt-object-detection

[mltoml]

@AyushExel @glenn-jocher

[glenn-jocher]

@AyushExel @mltoml triton server is actually in our supported AutoBackend formats, but this is inherited from YOLOv5 as part of our work with OctoML and not tested yet for YOLOv8.

[mltoml]

yeah @gaziqbal did a very nice work integrating YOLOv5 detect with triton*, hopefully we see that replicated in YOLOv8 soon.
*ultralytics/yolov5#9228

[mltoml]

@AyushExel @glenn-jocher Would it be possible to reopen this issue again?

[glenn-jocher]

@mltoml certainly! We apologize for any confusion - this issue can absolutely be reopened if you have further questions or if an update to the status of Triton support is desired. As Glenn mentioned previously, Triton server inference is supported as part of our work with OctoML and YOLOv5, but it has not been tested yet for YOLOv8. However, we are always looking to expand and improve the capabilities of our software, so we encourage you to stay tuned for future updates! In the meantime, please feel free to check out the Ultralytics Docs and HUB for additional resources, and don't hesitate to ask any further questions you may have.

[marunava21]

@glenn-jocher may i know if the inference is available for yolov8 yet? as i don't see any further comment for this thread.

[glenn-jocher]

Yes we have a PR open for this now , please check PR section and leave any comment in the Triton Server PR.