tensorrt .engine file validation

[shihanyu]

"add the tensorrt .engine file validate ability by modifying the val.py file and adding the yolov5_trt.py in utils/. (pycuda should be installed by command pip3 install pycuda)
val command:
python path/to/val.py --data aaaaa.yaml --img 640 --engine_library bbbbb.so --engine_path cccc.engine
(The way to generate the .so file and the .engine file can be refered to https://github.com/wangxinyu/tensorrtx/tree/master/yolov5 )
and the file can only be used on the same type GPU as where it was generated .that's why I did not upload my .so and .engine file.
Hope it helps, and if there is any problem,let me know.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Integration of TensorRT inference within YOLOv5 validation script for boosted performance and speed.

📊 Key Changes

• 🔧 Added pycuda to requirements for CUDA operations and TensorRT compatibility.
• 🐍 Added new Python file utils/yolov5_trt.py for handling TensorRT operations.
  • This module wraps the TensorRT API and provides methods for image pre-processing, post-processing, and running inferences.
• 🏎️ Updated val.py to support TensorRT engine validation:
  • Included ability to load TensorRT engine and run inferences directly from the validation script.
  • Validation can now be performed with .engine files by utilizing the new --engine-library and --engine-path arguments.

🎯 Purpose & Impact

• ⚡ Increased Inference Speed: TensorRT's optimized engine accelerates inference, which can be highly beneficial for deployment on edge devices or scenarios where latency is critical.
• 🔄 Flexibility: Users can choose between standard PyTorch validation or high-performance TensorRT validation based on their needs.
• 🌐 Broader Usage: This change appeals to users who prefer or require using TensorRT for inference due to its performance benefits on NVIDIA GPUs.

[github-actions]

👋 Hello @shihanyu, thank you for submitting a 🚀 PR! To allow your work to be integrated as seamlessly as possible, we advise you to:

• ✅ Verify your PR is up-to-date with upstream/master. If your PR is behind upstream/master an automatic GitHub actions rebase may be attempted by including the /rebase command in a comment body, or by running the following code, replacing 'feature' with the name of your local branch:

git remote add upstream https://github.com/ultralytics/yolov5.git
git fetch upstream
git checkout feature  # <----- replace 'feature' with local branch name
git merge upstream/master
git push -u origin -f

• ✅ Verify all Continuous Integration (CI) checks are passing.
• ✅ Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." -Bruce Lee

[glenn-jocher]

@shihanyu thanks for the PR! I think I may have not explained the DetectMultiBackend() module correctly to you. This module is responsible for ALL inference on any backend (i.e. ONNX, PyTorch, TensorFlow, CoreML). This is where ALL TorchScript specific logic must be placed, with the output from DetectMultiBackend() matching across all backends.

yolov5/models/common.py

Lines 276 to 287 in e80a09b

	class DetectMultiBackend(nn.Module):
	# YOLOv5 MultiBackend class for python inference on various backends
	def __init__(self, weights='yolov5s.pt', device=None, dnn=True):
	# Usage:
	# PyTorch: weights = *.pt
	# TorchScript: *.torchscript.pt
	# CoreML: *.mlmodel
	# TensorFlow: *_saved_model
	# TensorFlow: *.pb
	# TensorFlow Lite: *.tflite
	# ONNX Runtime: *.onnx
	# OpenCV DNN: *.onnx with dnn=True

Also please check CI failures: https://github.com/ultralytics/yolov5/runs/4220927192?check_suite_focus=true

[shihanyu]

Thank you for your reply! I misunderstood your requirement. My update can only do the validation for trt .engine file. And the merge did not pass just because pycuda cant be Installed and the trt environment is demanded which cant be installed by simple pip command. Anyway, although my code does not help,still very thank you for your contribution and sharing. I will keep following your projects! 从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送 发件人: Glenn ***@***.***> 发送时间: 2021年11月16日 18:17 收件人: ***@***.***> 抄送: ***@***.***>; ***@***.***> 主题: Re: [ultralytics/yolov5] tensorrt .engine file validation (PR #5626) @shihanyu<https://github.com/shihanyu> thanks for the PR! I think I may have not explained the DetectMultiBackend() module correctly to you. This module is responsible for ALL inference on any backend (i.e. ONNX, PyTorch, TensorFlow, CoreML). This is where all TorchScript specific logic must be placed, with the output from DetectMultiBackend() matching across all backends. Also please check CI failures: https://github.com/ultralytics/yolov5/runs/4220927192?check_suite_focus=true ― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#5626 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE546SSU7QF7WXOX4DDVPT3UMIVUFANCNFSM5H4NGF3A>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[glenn-jocher]

@shihanyu closing this PR to focus on #5699. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐