[css-images] Images on the web

[annevk]

I think it's problematic that both CSS and HTML attempt to define images for the web.

We need a central place that is responsible for fetching and decoding a byte sequence/stream (and dealing with EXIF) into an abstract representation of an image, that can then be used for various purposes. If we define this in different places in conflicting ways, the differences can lead to security issues. (Perhaps some of these concerns can be split, but an image being generated from an opaque response is an important property that needs to be part of the model and thought through whenever new features are added.)

See also #4929 and #5165.

cc @domenic

[annevk]

@tabatkins @fantasai @LeaVerou thoughts? whatwg/html#5574 wants to standardize more EXIF data. As pointed out above we need a shared strategy for metadata and opaque responses. It would probably be good to make some progress here on these issues to make the fundamentals a bit clearer to implementers, especially now that it's still somewhat early days for taking EXIF data into account.

cc @noamr

[annevk]

cc @heycam

[lrosenthol]

And as more embedded metadata values will become important to browsers, I would agree that a single place to address the situation would be a good thing.

[faceless2]

to make the fundamentals a bit clearer to implementers,

👋

We've walked right into this one - I thought we'd fully implemented css-images-3 but, having found whatwg/html#5574 and the corresponding testcases I can see that's the wrong thing to do if we want to match browser implementations.

As specified in css-images: the image-resolution (theoretically from the image-resolution property, although it's unimplemented) defaults to 1dppx, but can be set to from-image to take the natural resolution from a bitmap. If the image is loaded from image-set(), the natural resolution is overridden by the resolution from there. The exact definition of natural resolution is undefined in CSS, but given it is defined in JPEG (JFIF) and PNG (pHYs) it seemed clear enough.

What actually happens is the image resolution defaults to the natural resolution of the image, but only if its specified by a particular combination of EXIF tags rather than the tags designed for this purpose in the JPEG and PNG specs (for compat reasons - having read the full PR above I understand why, but this isn't even hinted at in css-images-3 and it's far from obvious). If loaded from image-set() the natural resolution is multiplied by the resolution from there (see #6241)

It's quite different. Particular issues:

1. The rules in the HTML spec apply even to background-image, content, SVG <image> and contexts where I had not expected the HTML image sizing rules to apply.
2. The rules for extracting natural dimensions are very specific and cannot be applied to PNG, despite PNG images with resolution units being used for some related examples, eg https://webkit.org/demos/srcset/. These are false friends - the resolution units are in the files, the example is demonstrating resolution units, but those facts are unrelated: the units from the file are not used. This only became clear when we revisited #6241 and found it meant scaling was now being applied twice.

I write this in the hope an anatomy of our failure can improve things, but the short version is:

I think it's problematic that both CSS and HTML attempt to define images for the web.

[noamr]

to make the fundamentals a bit clearer to implementers,

👋

We've walked right into this one - I thought we'd fully implemented css-images-3 but, having found whatwg/html#5574 and the corresponding testcases I can see that's the wrong thing to do if we want to match browser implementations.

As specified in css-images: the image-resolution (theoretically from the image-resolution property, although it's unimplemented) defaults to 1dppx, but can be set to from-image to take the natural resolution from a bitmap. If the image is loaded from image-set(), the natural resolution is overridden by the resolution from there. The exact definition of natural resolution is undefined in CSS, but given it is defined in JPEG (JFIF) and PNG (pHYs) it seemed clear enough.

What actually happens is the image resolution defaults to the natural resolution of the image, but only if its specified by a particular combination of EXIF tags rather than the tags designed for this purpose in the JPEG and PNG specs (for compat reasons - having read the full PR above I understand why, but this isn't even hinted at in css-images-3 and it's far from obvious). If loaded from image-set() the natural resolution is multiplied by the resolution from there (see #6241)

It's quite different. Particular issues:

1. The rules in the HTML spec apply even to background-image, content, SVG <image> and contexts where I had not expected the HTML image sizing rules to apply.

HTML sizing rules have always applied to CSS - e.g. determining the default tile size in backgrounds. I am surprised though that there is no explicit link from CSS natural sizing definition to HTML natural sizing definition. Not sure if I'll have time to propose a PR for this myself :(

2. The rules for extracting natural dimensions are very specific and cannot be applied to PNG, despite PNG images with resolution units being used for some related examples, eg https://webkit.org/demos/srcset/. These are false friends - the resolution units are in the files, the example is demonstrating resolution units, but those facts are unrelated: the units from the file are not used. This only became clear when we revisited #6241 and found it meant scaling was now being applied twice.

Why can they not apply to PNG? PNG can include EXIF tags embedded in XMP. In fact, the Mac/iOS Safari implementation does this out of the box because it relies on the Core Foundation libraries that parse XMP-EXIF internally.
It's true though that this is not tested properly nor implemented in Chrome/Firefox, but that's more of an implementation gap than something inherently missing in the spec.

[faceless2]

Fair point on EXIF in XMP - good to know that's the intention, thanks for letting me know.

I agree a backlink from css-images natural sizing to HTML natural sizing would help significantly, given that's where the algorithm is specified. SVG2 auto-sizing of images links to css-images so no changes required there.

[svgeesus]

PNG can include EXIF tags embedded in XMP.

True. PNG also has an eXIf chunk

[tabatkins]

I am surprised though that there is no explicit link from CSS natural sizing definition to HTML natural sizing definition.

Those are both the same link.

[noamr]

I am surprised though that there is no explicit link from CSS natural sizing definition to HTML natural sizing definition.

Those are both the same link.

Oops, wrong paste, I meant https://html.spec.whatwg.org/multipage/images.html#density-corrected-intrinsic-width-and-height

But seems like the HTML spec has a reference to setting the CSS natural image dimensions here: https://html.spec.whatwg.org/multipage/images.html#prepare-an-image-for-presentation

Would be good to have a back reference from CSS somehow.

[tabatkins]

Yup, would definitely be good, I'll see what I can do. Reason it's lacking currently is the CSS text was written before the HTML stuff was, and I haven't realized I needed to go back and do more edits.

[tabatkins]

(And this thread, in general, slipped into my "pandemic made me bad at tracking literally anything" backlog pile, sorry.)

[AndresInSpace]

If this is not in same topic of the thread, I apologize, the topic seems to be for discussing issues between both html and css defining images, and those processes / specs.
I believe it's worth mentioning that there is a discussion in HTML wg #4421 regarding an issue where at the browser level there is no optimization happening for the current selection process of the image src from a provided srcset, which I believe translates to image-set in CSS and seems to go against CSS spec and mdn documentation regarding optimization