AutoShape Usage #7128
Comments
|
@sezer-muhammed usage examples are shown right after export. Just follow the TRT export code and then do whatever you want with the exported model: # TensorRT export example
!pip install -U nvidia-tensorrt --index-url https://pypi.ngc.nvidia.com # install
!python export.py --weights yolov5s.pt --include engine --imgsz 640 640 --device 0 # export
!python detect.py --weights yolov5s.engine --imgsz 640 640 --device 0 # inferenceONNX output example:
|
|
Hi, I tried this code:
It works as expected with .pt models. Uploading model with detectmultibackend works. but autoshape gives error when I try to run it. Any solution for this? .engine file can be used with autoshape? |
|
@sezer-muhammed AutoShape and DetectMultiBackend classes are used internally and not externally exposed, therefore their usage is undocumented. |
|
Exactly why does connecting the engine with an autoshape forward cause an error below? File "./yolov5/models/common.py", line 569, in forward
t.append(time_sync())
File "./yolov5/utils/torch_utils.py", line 88, in time_sync
torch.cuda.synchronize()
File "/home/nvidia/miniforge3/envs/nbas/lib/python3.6/site-packages/torch/cuda/__init__.py", line 402, in synchronize
return torch._C._cuda_synchronize()
RuntimeError: CUDA error: an illegal memory access was encountered |
|
@youngjae-avikus yes this is a known YOLOv5 TRT issue with AutoShape. The following code works correctly: But if we use the same model for AutoShape inference we get the above CUDA error you mentioned. I have no idea why, I've looked into it several times and can't find the cause. If you have any ideas or discover a solution please let us know! EDIT: It's extra strange because both methods are essentially the same under the hood. They both use the DetectMultiBackend class here for inference: Lines 279 to 293 in 404b4fe |
|
Ah, I understood the cause, and when I modified it, it temporarily worked on the engine file Lines 533 to 538 in 7043872 So, temporarily changing it to the code below. That's fine t = [time_sync()]
p = next(self.model.parameters()) if self.pt else torch.zeros(1) # for device and type
autocast = self.amp and (p.device.type != 'cpu') # Automatic Mixed Precision (AMP) inference
if isinstance(imgs, torch.Tensor): # torch
with amp.autocast(autocast):
return self.model(imgs.to(p.device).type_as(p), augment, profile) # inference
device = select_device('cuda:0') if torch.cuda.is_available() else 'cpu'
p = p.to(device)or p = next(self.model.parameters()) if self.pt else torch.zeros(1) # for device and type
if self.dmb:
device = self.model.device
p = p.to(device)
autocast = self.amp and (p.device.type != 'cpu') # Automatic Mixed Precision (AMP) inference
if isinstance(imgs, torch.Tensor): # torch
with amp.autocast(autocast):
return self.model(imgs.to(p.device).type_as(p), augment, profile) # inferenceIt's not up to me to decide if I should change this part |
|
@youngjae-avikus ohhh yes I see. We have a device issue. Thanks for the fixes, I'll test them! |
Solution proposed in #7128 to TRT PyTorch Hub CUDA illegal memory errors.
Solution proposed in #7128 to TRT PyTorch Hub CUDA illegal memory errors.
|
@sezer-muhammed @youngjae-avikus good news 😃! Your original issue may now be fixed ✅ in PR #7560 implementing a solution by @youngjae-avikus.
To receive this update:
Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀! |
|
I don't know exactly if it's something that needs to be solved Lines 559 to 562 in 950a85d |
|
@sezer-muhammed in general letterbox() function will pad images according to minimum stride constraints, but this will vary by image so you should be very careful exporting fixed size rectangular models. To get started the most easily I would export a square model and then maybe run some experiments to see what needs to be done to force rectangular inference at the size the model is expecting. |
|
If it's a fixed image size RTSP video stream input, it's better to force it to run faster, right? |
|
@youngjae-avikus you could try to see what shape detect.py turns the stream into and then see if an exported engine model works with detect.py at that shape |
Solution proposed in ultralytics#7128 to TRT PyTorch Hub CUDA illegal memory errors.
Solution proposed in ultralytics#7128 to TRT PyTorch Hub CUDA illegal memory errors.
Search before asking
Question
Hi,
I want to use TRT model in my code.
There are some repos about it but these are complicated, on the other side yolov5s's codes are clear and easy to follow.
So I want to implement trt model just like in detect.py
There are multidetect backend class, and autoshape class.
The torch hub uses autoshape and returns detections object which is very usefull.
As far as I see in the codes I may be able to use trt model with autoshape class and get detections class as return.
But I cannot figure it out how to make it.
The question is how to do this?
Additional
No response
The text was updated successfully, but these errors were encountered: