RFC: New image API

[ascorbic]

⚠️ This is now outdated ⚠️

For the latest on the new image plugin, see #27950

---

Original RFC below

We are currently working on a new image plugin to replace gatsby-image, which greatly improves performance (Lighthouse 💯 again) and adds easy static images (no GraphQL). Part of it is also a new, simpler API for gatsby-transformer-sharp. This RFC is to discuss a proposal for this API. You can take a look at the branch here: #27443

Background

There are two new components that will use this API, GatsbyImage and StaticImage.

GatsbyImage

This component will be used for all of your dynamic images, such as the ones coming from a CMS or Markdown. Just like the current gatsby-image component, you will query the data using GraphQL. However the API for this is a bit different, and should be simpler. There are no more fragments, with the options passed as arguments to the resolver instead, and the data is returned as a JSON object rather than individual fields. You should treat this image data as an opaque object, and pass it directly to the component. The syntax is like this:

query {
  file(relativePath: { eq: "dino.jpg" }) {
    childImageSharp {
      gatsbyImage(layout: FIXED, width: 125, height: 125, fit: INSIDE) {
        imageData
      }
    }
  }
}

...and the data is passed to the new component like this:

import { GatsbyImage, getImage } from "gatsby-plugin-image";

export function Dino({ data }) {
  const imageData = getImage(data.file);
  return <GatsbyImage image={imageData} />;
}

(getImage is a simple helper that takes a file node and returns file?.childImageSharp?.gatsbyImage?.imageData. It is not required, but saves typing)

As shown above, the options are passed as arguments to the resolver, rather than using fragments. Many of these options are likely to match those used by the current fixed() and fluid() resolvers, but there are some new ones.

StaticImage

This is a simpler component for static images, where the src is known at compile time. The API is similar, but the options are passed as props to the component:

import React from "react";
import { StaticImage } from "gatsby-plugin-image";

export const Dino = () => (
  <StaticImage
    src="trex.png"
    placeholder="none"
    layout="fluid"
    grayscale
    maxWidth={200}
    alt="T-Rex"
  />
);

Image API

These are the proposed options, which would either be passed to the GraphQL resolver or as props to the component. Most of these would be optional, and most people wouldn't need to use them. They are slightly different from the ones currently implemented in the alpha. We would welcome feedback on the API.

Option	Description
layout	The layout for the image. FIXED: A static image that does not resize according to the screen width FLUID: The image resizes to fit its container. Pass a "sizes" option if it isn't going to be the full width of the screen. CONSTRAINED: Resizes to fit its container, up to a maximum width, at which point it will remain fixed in size.
placeholder	Format of generated placeholder image. DOMINANT_COLOR: a solid color, calculated from the dominant color of the image. BLURRED: a blurred, low resolution image, encoded as a base64 data URI (equivalent to current "base64" option) TRACED_SVG: a low-resolution traced SVG of the image. NONE: no placeholder. Set "background" to use a fixed background color.
formats	Array of formats to generate. AUTO (output the same format as the source image), JPG, PNG or WEBP. Default is [AUTO, WEBP] meaning it will generate images in the same format as the source, as well as in WebP. Coming soon: AVIF...?!
outputPixelDensities	A list of image pixel densities to generate, for high-resolution (retina) screens. It will never generate images larger than the source, and will always generate a 1x image. Default is [ 1, 2 ] for fixed images, meaning 1x, 2x and [0.25, 0.5, 1, 2] for fluid. In this case, an image with a fluid layout and width = 400 would generate images at 100, 200, 400 and 800px wide
sizes	The "sizes" property, passed to the img tag. This describes the display size of the image. This does not affect the generated images, but is used by the browser to decide which images to download. You can leave this blank for fixed images, or if the responsive image container will be the full width of the screen. In these cases we will generate an appropriate value, where width is the image width or maxWidth: • fixed: "width px" • fluid: "100vw" • constrained: "(min-width: width px) width px, 100vw"
maxWidth	Maximum display width of generated files. The actual largest image resolution will be this value multipled by the largest value in outputPixelDensities. For "constrained" images, this is the maximum displayed width. This option only applies when layout = FLUID or CONSTRAINED. For other layout types, use "width"
maxHeight	If set, the generated image is a maximum of this height, cropping if necessary. If the image layout is "constrained" then the image will be limited to this height. If the aspect ratio of the image is different than the source, then the image will br cropped.
width	The display width of the generated image. The actual largest image resolution will be this value multipled by the largest value in outputPixelDensities. Ignored if layout = FLUID or CONSTRAINED, where you should use "maxWidth" instead.
height	If set, the height of the generated image. If omitted, it is calculated from the supplied width, matching the aspect ratio of the source image.
pngOptions	Options for generated PNG images. quality, compression speed
jpegOptions	Options for generated JPEG images. quality, progressive
webpOptions	Options for generated WebP images.
tracedSVGOptions	If placeholder is TRACED_SVG, these options are passed to potrace
blurredOptions	Options for low-res placeholder. width (size of the low-res preview. Default 20px), format (force the output format, rather than using the same as the source image)
fit	Select the fit strategy for sharp to use when resizing images. Available options are: COVER, CONTAIN, FILL, INSIDE, OUTSIDE. See Sharp’s resize.
cropFocus	Change the cropping focus. Available options: CENTER, NORTH, NORTHEAST, EAST, SOUTHEAST, SOUTH, SOUTHWEST, WEST, NORTHWEST, ENTROPY, ATTENTION. See Sharp’s resize.
background	A hex color code to display as the background for the image wrapper. If you choose DOMINANT_COLOR for the placeholder then it will be chosen for you
rotate	Rotate the image (after cropping).
grayscale	Uses Sharp’s greyscale to convert the source image to 8-bit greyscale, 256 shades of grey
duotone	Applies a "duotone" effect to the source image if given two hex colors shadow and highlight defining start and end color of the duotone gradient

These API options are not decided, and so we welcome your feedback on how you would use them.

[ascorbic]

I'm still in two minds as to whether having formats as an array if a good idea or not. My concern is whether peopel will be confused and end up have it convern all the JPEGs to PNGs or something equally bad. However it is the most flexible way, and most people could leave it as the default ([AUTO, WEBP]). Once we add AVIF support the benefits will be clearer too.

[nkuehn]

+1 for simplifying the GraphQL pattern and providing a <StaticImage>.

Concerning the options I would appreciate if a new API would transparently pass through to the actual sharp options wherever possible from a functional perspective (plugin config, component props, docs, etc). That means to neither redefine, rename, validate or separately document them at all. I spent too much time digging the gatsby repository trying to find out what exactly is going on with an option.

For example, pngOptions could be called sharpPngOptions and its documentation would refer to the options object defined in https://sharp.pixelplumbing.com/api-output#png. Sharp is an actively maintained and well documented project with a clean API. Analogously for pngQuant related options (requires guesswork which option is a pngQuant option and which a sharp option), "cropFocus" (why uppercase the sharp values and repeat the docs?), fit (same) and maybe even more.

Since the option values are not abstracting from the underlying library's interpretation of an option's value anyways I think it would be better practice to be transparent and explicit in passing through.

[ascorbic]

I largely agree, and moving the format-specific options into objects was a step towards that. We can't be totally transparent in passing the options through, because we need to define the GraphQL schema, including the arguments for the resolver. That said, it shouldn't be hard to keep them in sync, because sharp upgrades are always something we undertake with care.

[nkuehn]

Yes, GraphQL and ideally Component types would have an explicit model for the officially supported subset. The naming, structure and documentation could nevertheless be more explicit in referring and delegating to the underlying sharp library.

Concerning the version: Since the sharp / libvips versions were a common source of installation issues we ended up making them an explicit resolution in yarn to get the latest sharp in any case.

[nkuehn]

(Separate comment because it's a likely controversial topic - but introducing a new breaking API is the seldom opportunity to at least reconsider previous decisions)

Placeholders and fade-in animations are an opinionated UX choice that was popularized by medium.com but have to my observation not become a de facto standard UX across the web. There are valid pro and contra arguments and many like the effect but the pros and the "popular vote" are not strong enough to me to want a site generator framework apply end user visible animations to all images as a default, esp. given that it happens at the cost of significant build performance and potential frontend performance. I suggest to consider changing the defaults to NONE (placeholders) and 0ms (animation).

[nkuehn]

Unfortunately I don't know of serious public research yet, that's why used "potential" referring to the fact that animations require some computation and have historically been a notorious performance topic on older user agents and low spec devices.

Thinking about the UX side it a day after, my doubts about blur-up animations are rather related to the specific current implementation of the loading behavior: A blur-up animation that only triggers when the image enters the viewport is making a technical loading process visible to the user even if the rendering engine would have had time to load the images while they are not in the viewport yet. It's forcing the user to see the loading process when it could have been hidden to the user. My hypothesis is that a "perfect" UX would optimize to completely hide any technical loading and optimization from the user, the content would be "just there" and only show placeholders in situations where higher priority resources need the network capacity. For a middle ground, the browser's loading=lazy instruction allows it to do much more informed optimizations than we can do in JS "userland" with the intersection observer which is inherently too late to still hide the technical process from the user.

The half second default animation is questionable to me because it is only adding to the time until the user sees the content. It triggers when the image is already loaded 100% and at that time could immediately be shown to the user. It's an opinionated topic of "looking smooth" vs. "minimal time to see content" that has no right or wrong answer. You'll have someone opposing any default here so you likely better ignore that part of the feedback and decide by personal taste :-).

PS: I quickly checked facebook and instagram since you mentioned it - I am not seeing that they start loading only after the image is in the viewport nor do i see blur-up animations in their web apps. Later posts are loaded incrementally but if possible outside the viewport and if I push scroll speed to limits I see placeholder wireframes but no animations once the content is there. Not scientific research, but at least two data points. (mobile apps may vary, i just did a quick check on the feeds).

PPS sorry for the lengthy response on an RFC in which it is a side topic.

[axe312ger]

My friend @technopagan does/did a some research regarding blur up and frontend performance for SQIP. Maybe this helps:

axe312ger/sqip#116 (comment)

https://github.com/axe312ger/sqip#background--reseach-about-image-placeholder--previews

[pieh]

For a middle ground, the browser's loading=lazy instruction allows it to do much more informed optimizations than we can do in JS "userland" with the intersection observer which is inherently too late to still hide the technical process from the user.

IntersectionObserver support rootMargin config option that allows to expand "test bounding box" (meaning it can trigger callbacks before element is actually in the viewport).

Current implementation in gatsby-image uses hardcoded 200px value (

gatsby/packages/gatsby-image/src/index.js

Line 161 in 5869cc5

{ rootMargin: `200px` }

). The chrome native loading="lazy" use thresholds between 3000 and 8000 on the other hand depending on network condition ( https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/renderer/core/frame/settings.json5;drc=e8f3cf0bbe085fee0d1b468e84395aad3ebb2cad;l=971-1003 ).

So this is quite possible to mimic very similar behaviour which would better "hide" loading/transitions. It can even do pretty much the same thing as native feature because (at least chrome expose connection effective type via navigator.connection.effectiveType meta). Of course bumping those margins have trade-offs - picking threshold value is trying to balance between having images already loaded and visible as user scroll through page versus potentially wasteful downloads for images that user might not even get to see.

[pieh]

Oh and also gatsby-image should use native lazy loading if it's supported by browser instead of IntersectionObserver ( #13217 ) - if it doesn't, there was uncaught regression

[nkuehn]

gatsby-image uses hardcoded 200px value

Good to know, thanks @pieh ! The 200px and the Chrome implementation obviously try to achieve different goals - at least having started before the viewport "hits", but try to avoid any unnecessary network traffic vs. having the image loaded in as many cases as possible while not loading images that are very unlikely to be seen.

potentially wasteful downloads

Also depends on the goal: Do not "waste" bandwidth that could be used for higher priority assets at a given time vs. do not "waste" any bandwidth at all e.g. to save cost. Given that Gatsby Links are prefetching content of linked pages it does not feel consistent to me to nevertheless avoid downloads of images on the same page.

If this were a Christmas wishlist, two "modes" would be beautiful - a User-optimized one following chrome's philosophy of aggressive prefetching once bandwidth is available (basically a polyfill for native lazy) and one that follows the current goal of saving bandwidth in absolute terms.

[ascorbic]

Currently I'm thinking of moving fit, cropfocus, grayscale, rotate and duotone into a transform option object.

[axe312ger]

That be great! Note: Contentful has similar but not the same transformation options. https://www.contentful.com/developers/docs/references/images-api/

[sandren]

Adding objectFit and objectPosition props would be super helpful. It would eliminate the need for wrappers like this.

[axe312ger]

I have to agree. I do the same sometimes 🙃

[ascorbic]

#28520