ImageGrab.grab(bbox=) fails on Mac with Retina screen, or on secondary monitor

[resnbl]

What did you do?

Displayed a window using tkinter and attempted to file a screenshot of it.

What did you expect to happen?

Create an image file showing the window contents.

What actually happened?

The generated image shows an area above and to the left of the intended target, and only covers an area 1/4th of the expected size.

What are your OS, Python and Pillow versions?

• OS: Mac OS 12.2.1
• Python: 3.10.1
• Pillow: 9.0.1

import sys
import tkinter as tk
from tkinter import ttk
from PIL import ImageGrab, __version__ as pil_version


def callback():
    img = ImageGrab.grab()
    img.save('screen.png')

    geom = root.winfo_geometry()
    print('geom=', geom)
    left, top = root.winfo_rootx(), root.winfo_rooty()
    width, height = root.winfo_width(), root.winfo_height()
    print(f'{left=} {top=} {width=} {height=}')
    bbox = (left, top, left+width, top+height)
    img = ImageGrab.grab(bbox=bbox)
    img.save('window.png')


print('Python:', sys.version)
print('PIL:', pil_version)
root = tk.Tk()
root.title('ImageGrab Test')
root.geometry('200x300+100+200')
message = tk.Label(root, text='Hello, World!')
message.pack()
save_btn = ttk.Button(root, text='Save Me', command=lambda: callback())
save_btn.pack(ipadx=5, ipady=5, expand=True)
root.mainloop()

Below is a (cropped) version of the 'screen.png' file generated by my script:

Here is the captured image:

ImageGrab.grab() is failing to account for the 144 DPI (not 72 DPI "virtual pixels") image generated by screencapture for my Retina display, so the bbox passed to the crop function needs to be scaled by 2 (all 4 parameters).

Secondary problem: this all fails completely when the app window is on a secondary monitor.

Proposed solution: Apple knows best!
Instead of capturing the entire screen and then cropping it down, make use of the provided -R{x,y,w,h} option for thescreencapture command. This will:

• capture the right portion of the screen (Apple knows to scale "virtual" to "physical" pixels)
• only load the desired portion of the screen (a much smaller image to load into memory)
• works perfectly on any monitor (my 2nd display has left/top of -1920,0; .crop() can't deal with that bbox...)

[PS: for those outside the "reality distortion field", Apple's current lineup contains only Retina displays...]

[nulano]

To be clear, I understand that you are asking for:

1. bbox to be specified relative to the macOS coordinates of the top left pixel (which may be negative with multiple monitors);
2. bbox to be specified in 72 dpi coordinates rather than physical pixels.

I agree 1 sounds like a bug.

As for 2, it is not clear to me whether this is a Pillow bug or a tkinter bug (should tkinter return physical pixel coordinates instead?).

I don't use any Apple devices myself, but I'll provide a Windows perspective:

1. works on Windows, the top left pixel is read from the system when taking a screenshot and the offset is applied to bbox correctly;
2. on Windows, you are expected provide physical coordinates for bbox. I haven't tested whether tkinter applies scaling on Windows (and I don't use scaling myself), but I expect it returns "unscaled" coordinates assuming 96 DPI.

Windows also has display scaling, similar to macOS. Many Windows laptops default to a scale of 150% (144 DPI) or even 200% (192 DPI) rather than 100% (96 DPI). On Windows, it is expected that "old" applications run at 96 DPI (with Windows scaling their windows) while "new" applications that are "DPI aware" use scaled pixel values when communicating with the OS. I would therefore argue that on Windows it is expected to pass "scaled" pixel coordinates to bbox.

As I don't have experience with macOS, I don't know whether applications are expected to use physical or scaled pixel coordinates, so 2 could either be a bug or a feature. Given 1, I'm assuming ImageGrab.grab currently captures all monitors on macOS. Is it possible to have different scaling on different monitors? How does that affect the "physical" pixel coordinates in the output screenshot? Edit: Based on reply below, it doesn't.

[resnbl]

I'm assuming ImageGrab.grab currently captures all monitors on macOS

That is not correct for ImageGrab's usage of the Mac screencapture program. In practice, if you want to capture all monitors, you must provide 1 output file path for each screen, as in:
screencapture main.png second.png third.png ...

I can't speak to different scaling on on different monitors, as both of mine generate 144 DPI .png files when captured. The main monitor .png is 4096x2304 pixels @ 144 DPI, but the co-ordinates I've used were provided by tkinter which reports the screen size as 2048x1152.

In my own working code, I passed those same "virtual" values directly to the screencapture -R... command and got what I expected, so the OS appears to be working in "virtual pixels", as well.

[radarhere]

Command-Shift-4 on macOS brings up a screen capturing tool that shows the width and height of the image that you are about to capture. Those units are in virtual pixels, so I agree with that expectation.

I've created PR #6152 to resolve this.

[nulano]

From https://developer.apple.com/library/archive/documentation/GraphicsAnimation/Conceptual/HighResolutionOSX/APIs/APIs.html#//apple_ref/doc/uid/TP40012302-CH5-SW2:

There are very few, if any, situations for which you need to use device coordinates directly.

So I would agree that virtual pixels are expected (edit: on macOS).