See if its possible to calculate image 'sizes' attribute inside of Gutenberg

[danielbachhuber]

If I were a betting man, I'd bet that the sizes attribute is wrong for most images on WordPress sites. A correct sizes attribute is really important for the browser to know which srcset image to download.

RespImageLint (https://github.com/ausi/respimagelint) is a nifty bookmarklet to lint your sizes and srcset values. It would be neat if we could reuse its logic to calculate sizes attributes inside of Gutenberg.

However, I have no idea if this is technically possible! Before we commit to such a feature, we should evaluate whether it's feasible and what might be involved.

Done is:

• We've evaluated how we might incorporate RespImageLint into Gutenberg to compute sizes attributes for images.

[skorasaurus]

I'm just checking this out hastily but see https://core.trac.wordpress.org/ticket/45407

[danielbachhuber]

@skorasaurus Ah! Yeah, that's exactly in the same space.

@mor10 Any thoughts on ^?

[graylaurenm]

@danielbachhuber last I heard @mor10 wasn't contributing recently. Just an FYI as he was spearheading a lot of the image discussions in 2018/2019.

Can you expand on what you think would be needed to start bringing this proposal to life? Since I also create custom sizes definitions for my clients, I'm pretty invested in how this could be possible with blocks!

[danielbachhuber]

Can you expand on what you think would be needed to start bringing this proposal to life?

@graylaurenm I think we need a reasonably strong JavaScript developer to spend a day hacking on it. I'm not 100% confident how feasible it is — a day of effort would probably give us a sense of "yes, this seems possible" vs. "uh oh, there are problems".

[graylaurenm]

I'm definitely not that developer, but I'm interested in understanding this approach enough to find that person.

For the moment, I'm going to mention #6177 because it–and Trac 45407 above–are the two tickets everything about srcset sizes ultimately comes back to. So I want to link this up.

[danielbachhuber]

@graylaurenm Yep, makes sense!

[mor10]

The challenge here, as it has been from the start, is the only entity with knowledge of the final rendered width of an image is the current active theme and block. The browser cannot know this, which is why the sizes attribute breaks the content/presentation separation.

With themes, the solution is to provide the developer with the means to specify max displayed widths and breakpoints for image areas as I proposed in 45407. The natural extension of this is blocks inheriting and respecting these values dynamically. That way if and when the site owner changes their theme, the max sizes of images in blocks change accordingly.

This in turn requires the sizes attribute to be JIT rendered as a page is called by a visitor based on 1) the intrinsic size of the image, and 2) theme specifications. The sizes values cannot be stored in the <img> element in the post object in the database.

I suggest reaching out to @yoavweiss to get first-hand knowledge of how responsive images are rendered at the browser level. He might also be able to point you towards people who are working on large scale solutions right now.

Also worth looking at is CDN solutions like Cloudinary, though even there the tools require the site providing info about max displayed widths for each image.

[danielbachhuber]

The challenge here, as it has been from the start, is the only entity with knowledge of the final rendered width of an image is the current active theme and block. The browser cannot know this, which is why the sizes attribute breaks the content/presentation separation.

@mor10 I feel like I'm missing something obvious but I don't know what it is 😞

If the block editor can emulate a range of potential viewport widths, why can't it determine what the correct sizes attribute should be for each <img>?

[mor10]

For each image the sizes attribute answers the question "what is the displayed with of this image given the current viewport." This question must be answered before the layout is painted and CSS is applied to the page (which runs counter to how all other painting in the browser happens). It's a layout question asked prior to layout, so it must be coded on an image-by-image basis based on prior knowledge of what the layout will be:

sizes="(min-width: 66em) 33vw,
  (min-width: 44em) 50vw,
  100vw"

Read from the bottom this translates as

• for viewports under 44em, the max displayed width of this image is 100vw (mobile layout)
• for viewports between 44em and 66em, the max displayed with of this image is 50vw
• for viewports wider than 66em, the max displayed width of this image is 33vw.

Based on this info, the browser checks the srcset list for the image width the smallest w value or x value to meet the sizes criteria and loads that image.

https://web.dev/learn/design/responsive-images/#sizes

The intention is to give the developer capabilities to load the smallest possible image for any layout. For example, in a responsive layout with a sidebar, the image width at a narrow viewport may be wider than the image width in a wider viewport because the sidebar reduces the layout space for the image. The sizes attribute can account for this, but because this happens prior to CSS layout paint, knowledge of the final layout is captured in the HTML markup.

This is one of the Very Sticky Problems with the whole RICG spec and why its rollout has been so challenging: it runs counter to how we understand RWD and how the rest of the web platform works. And to top it all off, the spec often ends up counteracting its original purpose - to reduce image sizes on smaller screens - because smaller screens now often have extremely high resolution and therefore pull huge images when the sizes attribute allows for it.

The current best-practice recommendation for sizes markup is to limit the max size to 2x across the board, and to set a firm max-width at the max-width of the main content. That way the worst case scenario is someone on a mobile device with a 6x screen pulling a 2x image when what they really need is a .5x image.

My recommendation in general remains what it was back in the day: Have the theme set a global max width for regular content and default the sizes attribute of all images to that width as the max. Then for special cases like full-bleed images have a special function to deal with them. What WordPress did last time I checked (2019) was setting the sizes to the image width and falling back to 100vw which means any large image will load the largest possible image regardless of screen. This is incorrect for all images except full-bleed images.

No matter which way you turn this particular Gordian knot, the end result will always be to somehow dynamically generate the sizes attribute based on the current theme, block layout, and other factors, at the moment of loading. That's the only way to make it work as the spec intends.

https://web.dev/learn/design/responsive-images/#sizes

[danielbachhuber]

Thanks for the detailed explanation, @mor10. I appreciate you taking the time to weigh in.

This question must be answered before the layout is painted and CSS is applied to the page (which runs counter to how all other painting in the browser happens).

I guess I'm being dense but I still don't understand why some JavaScript can't inspect the actual image size after layout (across a variety of viewports), and then use this data to provide a suggested sizes value. It seems to work pretty well for https://github.com/ausi/respimagelint

The current best-practice recommendation for sizes markup is to limit the max size to 2x across the board, and to set a firm max-width at the max-width of the main content. That way the worst case scenario is someone on a mobile device with a 6x screen pulling a 2x image when what they really need is a .5x image.

My recommendation in general remains what it was back in the day: Have the theme set a global max width for regular content and default the sizes attribute of all images to that width as the max. Then for special cases like full-bleed images have a special function to deal with them.

This seems like it could be a nice win 😊

[mor10]

If I understand you correctly you want to pre-render the page upon save, inspect the layout, then populate the sizes attributes accordingly? If so, then yes, sure, that could work, if the script runs through all possible viewport widths to find the breakpoints etc.

I'm all for practicality, and rather than make some overtly complex system I recommend doing the simplest possible thing which is to 1) identify the max-width of the main layout area for the theme, 2) make that the max-width for all images (with exceptions for full-bleed), and 3) configure a standard set of progressive image sizes (small, medium, large, xl, etc) that are always the same across all themes for consistency. That way you don't have to regenerate custom image sizes every time a new theme is added.

Something like this:

<img
 src="image-large.png"
 alt="A description of the image."
 width="300"
 height="200"
 loading="lazy"
 decoding="async"
 srcset="image-thumb.png 150w,
  image-small.png 300w,
  image-medium.png 800w,
  image-large.png 1200w,
  image-xl.png 1800w"
 sizes="(min-width: [$theme_breakpoint]) [$theme_max_width],  100vw"
>

[danielbachhuber]

Cool, makes sense on both accounts 👍

To help decide "how perfect do we need sizes to be", it would be interesting to do an analysis of some existing WordPress sites, evaluate how much bandwidth is wasted by oversized images, and then produce some estimation of the savings for various levels of correctness.

[mor10]

A straight-forward way of testing this:

1. Set up a site with a default theme
2. Create a post with a bunch of different image sizes and layouts
3. Upload Very Large Images to use in these layouts (width 1800px or higher)
4. Do a benchmark of the load size across different viewports and screen resolutions
5. Swap the theme
6. Repeat 4-6 as required

[roborourke]

On this topic I tried out a small hack project this weekend to take another approach which is to separate the dependency on srcset choice from the viewport width as such. This uses a resize observer on images and sets the sizes attribute to the current pixel width of the image which works nicely.

It's ~500B of JS, a progressive enhancement, using a ResizeObserver per image:

• JS library https://github.com/roborourke/rsp-img
• WP plugin https://github.com/roborourke/wp-rsp-img

Very ugly demo site here https://gigantic-three-sparrow.glitch.me/

If I get some time I'll try and do the maths per @mor10's comment, but it's not my strong suit!