Slow inference on CPU #5597
Description
System information
- OS version/distro: Windows 10.0 Build 18363
- .NET Version (eg., dotnet --info): 3.1.405 / 5.0.102
Issue
- What did you do?
- I am very new to ML.Net, to be honest, I have never used it until recently. I have a model trained on pytorch which I exported to ONNX. I used a yolov5 network (https://github.com/ultralytics/yolov5) and used this code (https://github.com/BobLd/YOLOv4MLNet/tree/yolo-v5-incl) to run the inference. Please note that I am not running tests in batches, rather one image at a time or batch size = 1. My inferences are all the time ran in the CPU.
- What happened?
- In python/using pytorch the inference time for one image on the CPU would take me around 100~300 ms on average. However, when I ran it using the implementation I found on the second repo I mentioned above the runtime for each image varies from 2000~3700 ms. And these timings are just to run the prediction function, not to setup the model or anything else.
- What did you expect?
- I was expecting to get a runtime similar to the one I go in python if not better. As I am not including the times taken for post-processing or the time for parsing the outputs the timing should stay close if not a little better/worse. But going from about 200/300ms to 2000/4000ms is a big jump.
Source code / logs
Once again these two are the repos
https://github.com/ultralytics/yolov5 (Original python implementation of yolov5)
https://github.com/BobLd/YOLOv4MLNet/tree/yolo-v5-incl (ML.Net code to take the ONNX model and run inference)
Please paste or attach the code or logs or traces that would be helpful to diagnose the issue you are reporting.
Lastly, line number 75 is what is taking the longest amount of time. It takes 80-90% of the total runtime.