Mask not resized when using an Image input with tool="sketch" and a shape argument

[leszekhanusz]

Describe the bug

if we use an Image input with tool="sketch" and a shape argument, then only the image is resized to the shape and not the mask.

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

import gradio as gr                                                                                                     
                                                                                                                        
def run(sketch):                                                                                                        
    image = sketch["image"]                                                                                             
    mask = sketch["mask"]                                                                                               
                                                                                                                        
    return str(image.size), str(mask.size)                                                                              
                                                                                                                        
interface = gr.Interface(                                                                                               
    run,                                                                                                                
    inputs=[                                                                                                            
        gr.Image(interactive=True, type="pil", tool="sketch", shape=(512,512)),                                         
    ],                                                                                                                  
    outputs=[                                                                                                           
        gr.Textbox(label="Image shape"),                                                                                
        gr.Textbox(label="Mask shape"),                                                                                 
    ],                                                                                                                  
)                                                                                                                       
                                                                                                                        
interface.launch()

Screenshot

No response

Logs

No logs.

System Info

Gradio v3.1.6

Severity

annoying

[pngwn]

@leszekhanusz Can you see if this bug is still present in gradio 3.4, we have made a lot of changes to the sketch tool and i think this should be fixed now.

[leszekhanusz]

Nope, this is not fixed. I just tested with gradio 3.4 and the problem is still there.
Btw you can reproduce it very easily by:

• copy-pasting the reproduction code in a file
• run it
• go to http://127.0.0.1:7860
• upload any image with a different resolution than 512x512 px
• Click submit
• If it works then the mask shape should be (512, 512), like the image shape.

[pngwn]

Will take a look, thanks!

[pngwn]

After a little thinking + discussion, I know why this is happening and we should respect the shape argument when it is provided but we need think a little about how we implement it.\

Problem

The reason it isn't being respected is because the shape cropping is handled on the backend (and isn't currently applying to the mask only the image). We can fix that but it still leads to a pretty poor user experience as users of the app (as opposed to authors) don't really have any indication that a shape constraint is being applied. We need some visual indicator that the image (and any additional mask of sketch) will be constrained to a specific shape. Additionally, the current implementation doesn't give much flexibility around how the image is resized and cropped to match the shape argument.

Proposal

I propose adding a crop window when the shape kwarg is provided by the author and a user uploads an image that doesn't match the the dimensions of the shape. The crop bounds will be fixed to the ratio of the provided shape dimensions, allowing users to decide exactly how the image is cropped (zooming/ resizing/ panning but not modifying the crop ratio), when the crop position + selection is accepted the image will then be resized accordingly (we can either do this on the frontend or the backend it doesn't matter much). The backend will validate the dimensions + apply any necessary transforms if they are not consistent with the provided shape dimensions. This final step is to guard against user 'forcing' wrongly sized images which can could cause performance issues.

I think we should also consider augmenting the shape argument. Although some models need specifically sized images, a lot of the time people use the shape argument to 'limit' the size of an image so that it doesn't degrade performance for all users. We can address this issue much more directly by supporting max dimensions. We can either do this via the shape argument or with another option. Suggestions welcome.

[pngwn]

We could also resize images (maintaining dimensions) on the server if images are uploaded that are too large. Depends on the reason for setting max really. If it is because of model performance then it might make sense but if the wrangling of large images itself is a bottleneck, then blocking the upload + simple validation on the server makes more sense. We could also resize in the browser.

[leszekhanusz]

As an example, here is how it is done for the automatic1111 fork of Stable Diffusion:

• The shape argument is not used
• You can define the width and height of the generated picture
• there is a resize_mode input which defines how to fix the aspect ratio with the following options:
  • Just resize which resize the image and potentially change the aspect ratio
  • Crop and resize which will blindly crop the image in the middle to keep the aspect ratio
  • Resize and fill which will fill the picture on the sides with lines to keep the aspect ratio

An integrated cropping tool designed to keep a specific aspect ratio and giving the user the control and visibility about the potential cropping would definitely be a great addition to this library.

What would be absolutely needed in that case is that the shape of the image could be defined by other inputs (the width and height inputs in that case)

[pngwn]

@leszekhanusz Sounds good. In this case the dimensions would be updatable via the gr.Update() method.

[abidlabs]

Just tested this on the latest version of gradio and it's fixed now:

It was probably fixed as a result of PR #3277!

There's a separate issue (visible in the screenshot) of the image being resized incorrectly (but that is an existing issue here: #3093), so I'll go ahead and close this issue. I'll also move your suggestions about cropping to this issue here: #1451