Hubconf won't process different height and width image size using exported models (detect.py works, pt models work)

[dmccorm2]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Detection, PyTorch Hub

Bug

When exporting a model with different height and width for imgsz, for example, 2560 x 1440 (which becomes 1472):

python export.py --weights yolov5m6.pt --include coreml --device=cpu --imgsz 1440 2560

I can run detect.py without issue: python3 detect.py --imgsz 1472 2560 --weights=./yolov5m6.mlmodel --source=./myvideo.mp4 --classes 0

However if I load the model from Pytorch Hub:

Using v6.2
model = torch.hub.load('ultralytics/yolov5:v6.2', 'custom', '~/yolov5/yolov5m6.mlmodel', force_reload=False)

Then load the video using OpenCV:

cap = cv.VideoCapture('./myvideo.mp4')
while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        print("Can't receive frame (stream end?). Exiting ...")
        break
    scaled_bgr_frame = cv.resize(frame, (2560, 1472), interpolation=cv.INTER_AREA)
    results = model(scaled_bgr_frame)

I get the following error, as it defaults to the 640 size:

RuntimeError: {
    NSLocalizedDescription = "Input image feature image does not match model description";
    NSUnderlyingError = "Error Domain=com.apple.CoreML Code=0 \"Image size 640 x 640 not in allowed set of image sizes\" UserInfo={NSLocalizedDescription=Image size 640 x 640 not in allowed set of image sizes}";
}

However, if I use the provided PT model:

model = torch.hub.load('ultralytics/yolov5:v6.2', 'yolov5m6', pretrained=True)
# ...
 results = model(scaled_bgr_frame)

Works fine.

If I try specifying size as an array, as you can do in detect.py using the --imgsz flag. I get this trace:

    results = model(scaled_bgr_frame, size=[1472, 2560])

File ~/.cache/torch/hub/ultralytics_yolov5_v6.2/models/common.py:621, in AutoShape.forward(self, imgs, size, augment, profile)
    619 s = im.shape[:2]  # HWC
    620 shape0.append(s)  # image shape
--> 621 g = (size / max(s))  # gain
    622 shape1.append([y * g for y in s])
    623 imgs[i] = im if im.data.contiguous else np.ascontiguousarray(im)  # update
TypeError: unsupported operand type(s) for /: 'list' and 'int'

If I specify size as an integer using either 1472 or 2560 in this example, it tries to run the model with a scaled size x size image which fails the exported model check

RuntimeError: {
    NSLocalizedDescription = "Input image feature image does not match model description";
    NSUnderlyingError = "Error Domain=com.apple.CoreML Code=0 \"Image size 1472 x 1472 not in allowed set of image sizes\" UserInfo={NSLocalizedDescription=Image size 1472 x 1472 not in allowed set of image sizes}";

I'm expecting to omit size or provide an array to run a different height and width source on the exported model via pytorch hub successfully as detect.py does without issue.

Environment

• YOLOv5: YOLOv5 🚀 v6.2 (current tag)
• Python: 3.9.13
• OS: macOS 12.4 (M1)

Minimal Reproducible Example

1. Create exported model to use on the zidane jpg sample

python export.py --weights yolov5m6.pt --include coreml --device=cpu --imgsz 720 1280

2. Verify detect.py works (768 to match the max stride calculated from the export)
   python3 detect.py --imgsz 768 1280 --weights=./yolov5m6.mlmodel --source=./data/images/zidane.jpg
image 1/1 ~/yolov5/data/images/zidane.jpg: 768x1280 2 persons, 1 tie, 73.3ms

3. In python shell

import torch
import cv2
im = cv2.imread('./data/images/zidane.jpg')
model = torch.hub.load('ultralytics/yolov5:v6.2', 'custom', './yolov5m6.mlmodel')
results = model(im) # will fail here
results.print()

Expect to see

RuntimeError: {
    NSLocalizedDescription = "Input image feature image does not match model description";
    NSUnderlyingError = "Error Domain=com.apple.CoreML Code=0 \"Image size 640 x 640 not in allowed set of image sizes\" UserInfo={NSLocalizedDescription=Image size 640 x 640 not in allowed set of image sizes}";
}

5. Can then try
   results = model(im, size=[768, 1280])
   Error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/path/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "~/.cache/torch/hub/ultralytics_yolov5_v6.2/models/common.py", line 621, in forward
    g = (size / max(s))  # gain
TypeError: unsupported operand type(s) for /: 'list' and 'int'

6. Can also try results = model(im, size=768)

RuntimeError: {
    NSLocalizedDescription = "Input image feature image does not match model description";
    NSUnderlyingError = "Error Domain=com.apple.CoreML Code=0 \"Image size 768 x 768 not in allowed set of image sizes\" UserInfo={NSLocalizedDescription=Image size 768 x 768 not in allowed set of image sizes}";
}

7. Remove size and revert to the provided model
   model = torch.hub.load('ultralytics/yolov5:v6.2', 'yolov5m6')

8. results = model(im) now works with results.print()

image 1/1: 720x1280 2 persons, 1 tie
Speed: 6.1ms pre-process, 353.2ms inference, 5.7ms NMS per image at shape (1, 3, 384, 640)

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[github-actions]

👋 Hello @dmccorm2, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@dmccorm2 CoreML models are only capable of inference at a fixed size, PyTorch models are capable of dynamic input sizes.

We don't have CI in place for CoreML export and PyTorch Hub inference at various sizes so this feature may not exist.

The fastest and easiest way to incorporate your ideas into the official codebase is to submit a Pull Request (PR) implementing your idea, and if applicable providing before and after profiling/inference/training results to help us understand the improvement your feature provides. This allows us to directly see the changes in the code and to understand how they affect workflows and performance.

Please see our ✅ Contributing Guide to get started.

[glenn-jocher]

@dmccorm2 the preprocess step in the AutoShape forward method would be the place to debug and modify for your use case:

yolov5/models/common.py

Lines 609 to 632 in f258cf8

	# Pre-process
	n, ims = (len(ims), list(ims)) if isinstance(ims, (list, tuple)) else (1, [ims]) # number, list of images
	shape0, shape1, files = [], [], [] # image and inference shapes, filenames
	for i, im in enumerate(ims):
	f = f'image{i}' # filename
	if isinstance(im, (str, Path)): # filename or uri
	im, f = Image.open(requests.get(im, stream=True).raw if str(im).startswith('http') else im), im
	im = np.asarray(exif_transpose(im))
	elif isinstance(im, Image.Image): # PIL Image
	im, f = np.asarray(exif_transpose(im)), getattr(im, 'filename', f) or f
	files.append(Path(f).with_suffix('.jpg').name)
	if im.shape[0] < 5: # image in CHW
	im = im.transpose((1, 2, 0)) # reverse dataloader .transpose(2, 0, 1)
	im = im[..., :3] if im.ndim == 3 else cv2.cvtColor(im, cv2.COLOR_GRAY2BGR) # enforce 3ch input
	s = im.shape[:2] # HWC
	shape0.append(s) # image shape
	g = (size / max(s)) # gain
	shape1.append([y * g for y in s])
	ims[i] = im if im.data.contiguous else np.ascontiguousarray(im) # update
	shape1 = [make_divisible(x, self.stride) if self.pt else size for x in np.array(shape1).max(0)] # inf shape
	x = [letterbox(im, shape1, auto=False)[0] for im in ims] # pad
	x = np.ascontiguousarray(np.array(x).transpose((0, 3, 1, 2))) # stack and BHWC to BCHW
	x = torch.from_numpy(x).to(p.device).type_as(p) / 255 # uint8 to fp16/32

[glenn-jocher]

@dmccorm2 test #9072 to see if this works for your use case:

model = torch.hub.load('ultralytics/yolov5:update/autoshape_hw_size', 'yolov5m6', pretrained=True)

results = model(im, size=(h,w))

[dmccorm2]

@glenn-jocher This worked great for me - am able to leverage Autoshape with a specified different w,h tuple and works on CoreML export (as long as it matches the same model w,h obviously)

[glenn-jocher]

@dmccorm2 great, PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐