Find ways to support the CDN use case

[chrishtr]

CDNs want to perform content transformations to optimize pages, but can't do so if they might break content. Once #16 is implemented by browsers, the next step is to find a way for CDNs to specify image sizing hints. However, as mentioned in #16, width and height attributes cannot be just set willy-nilly because if there is no CSS setting auto/responsive width & height in addition, the width and height will cause the wrong final layout.

One option could be a new attribute on image elements called, say, sizingistentative (not an awesome name I know). This would explicitly say that the width and height attributes should be overridden by both CSS and the actual bitmap dimensions of the loaded image.

With such an attribute I think CDNs could start setting width and height based on their caches.

[yoavweiss]

/cc @kornelski @nicjansma @triblondon for CDN opinions

[triblondon]

Discussed this with a few colleagues here at Fastly. We are struggling to see the use case here. A CDN might well want to choose an image width based on client hint headers. Are we talking here about also rewriting the width and height properties of the image in the associated HTML resource?

This seems like an unnecessary thing to do to us. We'd prefer that the width and height simply reflect the correct aspect ratio at a default display size. #16 provides a solution for laying out the container responsively. Serving the bytes of the image then only presents a problem if we optimise the image down to a size smaller than the container ends up being rendered at, and the author does not want the image to be upscaled. In practice I would imagine that authors would configure CDN settings to err on the side of serving a slightly larger image.

If I understand correctly the implications of sizingistentative then it's essentially an override that re-introduces a potential reflow on top of an override that is designed to prevent reflows. I appreciate the thought and the attempt to solve the problem but I'm not convinced this is a problem that needs to be solved.

[chrishtr]

Thanks for the response!

Are we talking here about also rewriting the width and height properties of the image in the associated HTML resource?

Not rewrite, but set if they weren't already, plus an attribute to indicate the width and height are only hints. This issue is not about optimizing the image bitmap size, it's only about providing hints to help the browser avoid extra reflow.

The CDN would then recommend to developers not to set the width and height attributes, and instead control width and height with CSS. (If the developer set width or height anyway, then the CDN would be unable to do anything, for fear of breaking the site.)

The CDN would set width and height, plus sizingistentative, to indicate aspect ratio and default layout sizing. Any width/height specified in CSS would override these values. The actual bitmap dimensions and aspect ratio of the image once loaded would override these values.

This way developers don't have to set width and height and keep it in sync with their image assets. The CDN can do it for them. And it can be done in a way that avoids reflows in all the situations that they can be avoided today, and also in responsive design situations that depend on aspect ratio.

If I understand correctly the implications of sizingistentative then it's essentially an override that re-introduces a potential reflow on top of an override that is designed to prevent reflows.

It's the opposite. The proposal is about avoiding reflows. I don't think it introduces any new ones, unless I'm missing something.

[kornelski]

From Cloudflare's perspective:

• We'd be interested only if intrinsicsize was merely a hint, and image's actual intrinsic size took priority over the attribute. If intrinsicsize is an overriding directive that can visually break the page, we can't just add it automatically for our customers.

• sizingistentative + intrinsicsize seems acceptable from robustness perspective. However, since I prefer intrinsicsize to be tentative by default, it seems silly to me to have it with the undesirable behavior and yet another attribute that disables the undesirable behavior.

• We already have a different solution that is entirely safe and doesn't even require markup rewriting: https://blog.cloudflare.com/parallel-streaming-of-progressive-images/

[chrishtr]

• We'd be interested only if intrinsicsize was merely a hint, and image's actual intrinsic size took priority over the attribute. If intrinsicsize is an overriding directive that can visually break the page, we can't just add it automatically for our customers.

• sizingistentative + intrinsicsize seems acceptable from robustness perspective. However, since I prefer intrinsicsize to be tentative by default, it seems silly to me to have it with the undesirable behavior and yet another attribute that disables the undesirable behavior.

The option being discussed in this issue is to not have intrinsicsize at all (at least not for now), and instead width plus height plus sizingistentative would indicate the hinting. e.g.

<img width=100 height=200 sizingistentative>

would reserve layout space of 100px by 200px by default (if there was no other CSS specified), and also use a default aspect-ratio of 0.5 for any layout computations. Once the image resource has loaded, or enough of it to indicate its dimensions & aspect ratio, those would take priority over anything specified in the width or height parameters.

From your comment, I think this would work for cloudflare? Just double-checking. Thanks for the response so far.

[kornelski]

OK, that would work in browsers that implement sizingistentative.

What about backwards compatibility with browsers that don't support sizingistentative?

[yoavweiss]

You can probably differentially apply this optimization only to supporting browsers. Right now, the UA string is your best option. In the future, assuming there's appetite for that kind of optimization, we can consider adding it as a Client Hint.

[kornelski]

Client Hint doesn't seem that useful, given that it couldn't work on the first HTML request.

Is there a reason to make width/heigth dual-purpose instead of either sticking to intrinsicsize=wxh syntax, but without the breaking behavior, or adding a pair of new attributes such as nwidth/nheigth (for naturalWidth/naturalHeight hint) instead?

[chrishtr]

Is there a reason to make width/heigth dual-purpose

One reason: width and height could (should?) transition over time into a best practice of being considered sizing & aspect-ratio hinting attributes only, and then CSS should be used to actually size. Then there is not a dual-purpose.

[triblondon]

The proposal is about avoiding reflows. I don't think it introduces any new ones

I wrapped my head around this now. So, the situation is this:

• If CSS has aspect-ratio defined using attr(width) and attr(height), and the CSS is overriding the HTML sizing attributes, then width and height will serve to get the aspect correct, and sizingistentative is irrelevant.
• If CSS is overriding both the HTML sizing attributes, and does not use aspect-ratio, then width and height attributes are ignored and sizingistentative is irrelevant.
• If CSS is not overriding any HTML sizing information, then the image is laid out according to the HTML attribs and sizingistentative (and aspect-ratio) is irrelevant.
• If CSS is overriding only one HTML sizing attribute, and does not use aspect-ratio, then width and height attributes may interfere with the ability of the CSS to scale one of the dimensions of the image in proportion with the other. This is where sizingistentative would allow the natural aspect of the image to win out.

So the scenario is:

img {
  width: 100%;
  /* height: auto;   <-- not set */
  aspect-ratio: aspect-ratio: attr(width) / attr(height);
}

In this situation, setting width and height is necessary to calculate the aspect ratio, but will also fix the height to an absolute value, causing the width: 100% to stretch the image out of proper aspect.

Speaking for Fastly, we do not transform customer HTML, so this is a slightly moot issue for us, but speaking as a web standards nerd, I'm not especially keen on this being an HTML attribute. It seems like it's so easily solved (by setting the relevant dimension to auto), that I don't see why we need a new thing. If we did need a new thing that specifies the priority for determining image dimensions, perhaps a CSS property would be better:

img {
    width: 100%;
    height: auto;
    aspect-ratio: attr(width) / attr(height);
    sizing-priority: image-data, html, css;  /* also possibly sizing-priority-width, sizing-priority-height */
} 

Client Hint doesn't seem that useful, given that it couldn't work on the first HTML request.

Quite.

[chrishtr]

Here is another argument against the sizingistentative idea (via @eaenet): if this is only useful for middleware like CDNs, that is odd and a sign this is not a necessary API. For the middleware use-case such as CDNs, developers could also hint to the middleware on which images it's ok to set width and height, because the developer promises that will still result in the correct layout. e.g. something like:

<img hey-middleware-you-can-set-width-and-height src="..."/>

And the CDN would add width and height attributes, plus remove the custom one. Or it could be a setting on the HTML resource cached by the CDN.

For the non-middleware case (or to make it easy to be safe for such middleware) it seems easy enough to add a rule at the beginning of the first style sheet like:

img {
  width: auto;
  height: auto;
}

that would be overridden by any other setting of width or height, if desired, but if not desired, then it's safe to set the width and height attributes to anything at all and definitely not break the final layout after image load.

[kornelski]

@chrishtr For some enterprise CDN customers such markup changes are much much harder than it seems. Large corporations usually have more CMS-es and microsites than they can count; set up by different branches, teams, ex-employees, and many many vendors who may be expensive and/or unresponsive. "Add an attribute to all <img> tags" is going to be a multi-year project needing at least a dozen intercontinental flights to meetings.

So there's a monumental difference between "hey, we've made your site faster, automatically!" and "we could make your site faster, but it may break, so compile a list of all your web properties, find people responsible for maintaining them, and tell every one of them to test and review the changes, and update markup accordingly".

[chrishtr]

@kornelski interesting. I hear you about the desire to have a solution that avoids such coordination.