Tracking: Client-side media processing

[swissspidy]

This is something that was previously mentioned in #55106 and has been heavily explored in Media Experiments already (see for example this blog post for an overview of the latter). I believe the core logic at the heart of Media Experiments is a great foundation to implement such a feature, and Gutenberg is the right place to further develop it and bring it to more people.

---

Overview

Current image processing in WordPress relies on server-side resources and older image libraries, leading to potential performance issues and limited support for modern image formats such as AVIF. This results in a subpar user experience, particularly with resource-intensive tasks like resizing and compressing images. Additionally, the lack of modern compression tools like MozJPEG further hinders optimization efforts.

Client-side media processing offers a solution by leveraging the browser's capabilities to handle tasks like image resizing and compression. This approach not only alleviates the strain on server resources but also enables the use of more advanced image formats and compression techniques, ultimately improving website performance and user experience. By tapping into technologies like WebAssembly, WordPress can provide a more efficient and seamless media handling process for both new and existing content.

Resources

• Add query for LCP image formats and file sizes GoogleChromeLabs/wpp-research#97
• https://github.com/swissspidy/media-experiments

Roadmap

Given the many possibilities client-side media processing unlocks, this project can be split into multiple phases, each with their dedicated goals and success criteria. For example:

Phase 1

Goals

• Client-side sub-size image generation
  • Including PDF thumbnails
• Image compression during upload (for all sizes)
  • Scale down images according to big image size threshold setting
  • Always compress all images, which is different from what WordPress does today
  • Provide opt-out if needed (e.g. for photographers)
• A way in the block editor for compressing an existing image

Nice to have

• Converting HEIF/HEIC images to a web-safe format
• A way in the post editor for bulk optimization of all existing images in a post
  • Not to be confused with regenerating the whole media library

Non-goals

• Converting (transcoding) images to any other format (e.g. JPEG to WebP)
  • Except for HEIC which is a nice-to-have above
• Expand to other media types (e.g. video compression)

Phase 2

Goals

• Converting (transcoding) images to any other format (e.g. JPEG to WebP)
  • Including sub-sizes of course

Nice to have

• Create poster images for videos
• Convert animated GIFs to actual videos
  • Add quick setting to video panel for "GIF-like" playback #19730
• Expand to other media types (e.g. video compression)
• Provide additional client-side image editing capabilities such as cropping, rotation, and scaling
  • Makes it easier for users to make smaller images and thus keep file size down

Non-goals

• UI for regenerating/compressing the whole media library

Phase 3

Goals

• Generate sub-sizes in multiple different formats
  • Useful for <picture> , e.g. for using AVIF with a JPEG fallback
  • Alternatively or additionally, for each sub-size pick the format with the smallest file size
• Expand functionality from block editor to media library, in accordance with media library revamp as part of Gutenberg phase 3

Nice to have

• Expand to other media types (e.g. video compression)

Out of scope

Non-goals at the moment, or something for far out in the future:

• UI for regenerating/compressing the whole media library

Caveats and risks

Technical complexity

WASM-based image optimization requires SharedArrayBuffer support, which in turn requires cross-origin isolation. Implementing that in a robust way without breaking other parts of the editor is challenging. There are currently some known issues in Firefox and Safari due to these browsers not supporting credentialless iframe embeds. Embed previews in the editor currently do not work because of this, until those browsers add support for <iframe credentialless>.

Image compression by itself is rather complex due to the vast amount of different combinations of codecs and encoding options available. While the existing server-side image handling in WordPress has been proven over many years, doing everything client-side is new territory. There will be many edge cases that need to be handled.

As for overall architecture design, the existing Media Experiments project contains a very solid foundation that already solves many challenging problems and could be built upon.

Confidence level

Reception in the community so far has been extremely positive, including from WordPress leadership. There is clearly a desire for this kind of solution.

The biggest risk is about compatibility with other parts of the editor (plugins, embed previews) and certain environments because of the cross-origin isolation required for SharedArrayBuffer. Not breaking existing sites is very important. The risk can be mitigated through extensive testing, additional safeguards, and documentation. Lack of <iframe credentialless> support is not ideal as it degrades editor UX in some cases, but it does not break the authoring experience as a fallback exists.

While image optimization could theoretically be done through other means such as the canvas API, said API is extremely limited across browsers. For example, there is a lack of support for compression options (e.g. lossless or lossy) and mime types (no AVIF everywhere, no WebP in Safari).

Implementation status

• Add experiment for client-side media processing #64650
• Add safeguard to mediaUploadMiddleware #64843
• Media Utils: Add TypeScript support and export more utils #64784
• Add new Media section to preferences modal #64846
• Add new private vips package #64845
• Implement main upload queue

[ramonjd]

Firstly, thanks so much for the detail and all the background work!

Converting HEIC images to a web-safe format

This came up in a recent discussion I was having.

What's your take on the level of effort to integrate this feature? I was browsing the media experiments repo code and was wondering if there were any further considerations, beyond what you've explored already.

It would be a fantastic feature to have - happy to have a shot at it, or try to organize someone to have a shot at it 😄 if you think it's worth it at this stage. I'll take your guidance.

Thank you!

[swissspidy]

I heard of similar discussions recently :)

Always great to hear interest in this kind of topic. And I am glad you already browsed the media experiments code a little bit.

If you are interested solely in HEIC conversion, it by itself could be implemented relatively easily in theory. See #61861 as a demonstration.

However, as with everything else mentioned in this ticket, there are some important aspects to consider. For instance:

• You might want to make this opt-in/opt-out and explain to the user what is happening
  • Make HEIF conversion opt-in swissspidy/media-experiments#347
• You don't want to block the main thread with such operations, so ideally this runs in a separate worker thread
  • Not trivial to add such workers in a WordPress context as you don't have much control over how scripts are served
  • Will probably want to serve the WASM file from a CDN, so it won't work when offline
• You will want more control over such worker threads, for example to cancel conversion operations
• The generated image can be relatively large, because using a canvas to create images is not very efficient
• Thus, you might want to run client-side image compression as well, so you'll discover issues such as cross-origin isolation
  • Cross-origin isolation tracking issue swissspidy/media-experiments#294

And perhaps most importantly: libheif-js and the underlying libheif license is licensed under LGPL 3.0, which is incompatible with GPL v2, but is compatible with GPL v3. Given that WordPress is licensed under GPLv2 or later, it would mean that the final product would need to be distributed under GPL v3 only. IANAL, but I don't think this is gonna fly. So that means an alternative to libheif-js needs to be first evaluated and tested. There are some libraries built on libavif but I haven't had success using those in the past.

(Edit: dynamically linking via loading libheif-js from an externally hosted script might be OK, but again, not a lawyer)

➡️ Aside: Thanks to this POC PR I discovered that the current license check in this repo is broken 😬 I am fixing it in #61868

Bottom line:

I've deliberately split this project into multiple phases as it's more complex than what it might seem on the surface. I strongly suggest tackling this step by step.

Happy to discuss HEIC itself separately elsewhere.

[noisysocks]

Thanks for going into detail Pascal! I like your idea of a "Would you like to convert this to a web safe format?" modal that teaches the user what's going on. I've asked Automattic's legal team for some non-IANAL guidance about libheif.

I think supporting HEIC is the most important outcome here. You encounter these files more and more nowadays and it's very frustrating to users when they don't "just work" in WordPress. I'd push for starting with HEIC support and then expanding on that into other areas (compression, resizing, converting GIFs, etc.)

[ramonjd]

Given that WordPress is licensed under GPLv2 or later, it would mean that the final product would need to be distributed under GPL v3 only. IANAL, but I don't think this is gonna fly.

That's what I'm reading from https://www.gnu.org/licenses/gpl-faq.html#gpl-compat-matrix — if WordPress uses code under LGPLv3, then the "combined" work (WordPress + lib) would then fall under the terms of GPLv3

Bit of a wild suggestion, but is there a chance we could fallback to backend processing via Imagick: https://imagemagick.org/script/formats.php#supported

I say "wild" as I assume HEIC files can be pretty massive, and batching would be pricey in terms of performance.

So that means an alternative to libheif-js needs to be first evaluated and tested. There are some libraries built on libavif but I haven't had success using those in the past.

https://www.npmjs.com/package/heic2any or https://github.com/MaestroError/php-heic-to-jpg (both MIT) might also be worth a looksy 🤷🏻

I've asked Automattic's legal team for some non-IANAL guidance about libheif.

Best answer. 😄

[swissspidy]

Thanks for going into detail Pascal! I like your idea of a "Would you like to convert this to a web safe format?" modal that teaches the user what's going on.

Cool. I'm going to implement it in my plugin soon so we can see how well it works UX-wise.

For WCEU I plan on working on a POC for bringing parts of that plugin to Gutenberg.

I think supporting HEIC is the most important outcome here. You encounter these files more and more nowadays and it's very frustrating to users when they don't "just work" in WordPress. I'd push for starting with HEIC support and then expanding on that into other areas (compression, resizing, converting GIFs, etc.)

Not sure I agree with this one. Performance issues due to too large & heavy images are quite widespread. HEIC is mostly an iPhone thing and not everyone uses those.

As long as there are legal question marks anyway, I'd recommend focusing on server-side HEIC conversion in CORE-53645.

npmjs.com/package/heic2any or MaestroError/php-heic-to-jpg (both MIT) might also be worth a looksy 🤷🏻

heic2any uses the same libheif library under the hood and the usage would be exactly the same, so the MIT license there seems incorrect to me.

php-heic-to-jpg only works server-side and requires an executable file on the server, which is not suitable for WP. For server-side support I recommend focusing on ImageMagick support in CORE-53645.

[noisysocks]

As long as there are legal question marks anyway, I'd recommend focusing on server-side HEIC conversion in CORE-53645.

I agree. Thanks @swissspidy, sorry for derailing your issue here with talk of HEIC 😀

[swissspidy]

At WordCamp Europe I had a great conversation with @youknowriad and @gziolo where I was able to demo Media Experiments and elaborate on its inner workings and some design decisions. Now, I made a ton of (documentation) improvements so people can better understand how it works. The technical overview doc is a great entry point.

I would appreciate if y'all could check it out. All code is 100% TypeScript and well-documented (check out the readmes!), which should help with reviews as well.

cc @ntsekouras who previously showed interest in reviewing the codebase

As the plugin name implies, there are some experimental features in there right now, but the core part, the client-side media processing & compression is what matters most and is what I am proposing to gradually merge into Gutenberg.

That said, if we think this functionality is better tested at scale in its own plugin for now, we could also consider further testing Media Experiments separately as part of the Performance Lab family of plugins before bringing it into Gutenberg.

[youknowriad]

@swissspidy So I've been thinking about this more (specially the store related stuff). So I've come up to the realization that we might be able to actually bundle most of the package/behavior into the generic block-editor package and store (or as a dependency of block-editor package).

In other words, all block-editors using the block-editor package can gain support for:

• Media uploads queue
• Media compression / optimization...

But, to be able to do so, we need to split the "WP specific logic" from the "Media handling logic". For the "upload API" we already have that it's the mediaUpload setting, so nothing is needed there, no new setting or anything.

For the rest, I'm not sure I understand if there's any WP specific logic in the rest of the logic (the client side optimizations, the compression...). At first glance, there's none right?

---

I guess what I'm proposing is the following:

Splitting the MediaExperiment into two parts:

• One package that handles all the logic of the queue, compression... (the big part I guess). Make this package/code part of "block-editor" package or a sub dependency of block-editor.
• The WP specific parts remains settings we pass to BlockEditorProvider like today.

How feasible would that be?

[mtias]

I'd love to start testing some of these enhancements on the plugin, we can gate them with a feature flag if anything feels too experimental yet.

[ntsekouras]

Riad's suggestion for inclusion in block editor seems like a good plan. I can definitely help reviewing and with whatever needed.

[swissspidy]

So I've come up to the realization that we might be able to actually bundle most of the package/behavior into the generic block-editor package and store (or as a dependency of block-editor package).
How feasible would that be?

Interesting 👀 Let me analyze the code base a bit more to provide a good answer.

Just so I understand correctly:

Right now, everything is built as sort of a drop-in replacement for the media-utils package, so everything behaves the same as in Gutenberg:

• block-editor requires a mediaUpload setting, and without it does not support uploads by default. Is platform-agnostic.
• media-utils exposes a mediaUpload function that validates files, adds items to a queue, handles compression, etc. Is WordPress-specific.
• editor package wraps mediaUpload from media-utils to add the post ID and provides the function as a setting to block-editor. Is WordPress-specific.

What you are suggesting:

• block-editor should now gain all the capabilities for the upload queue and compression etc. The final file is passed to the mediaUpload function passed via settings. If the setting is missing, it still doesn't support uploads by default.
• media-utils does not change, it simply takes a file and uploads it to WordPress.
• editor also does not change, it still wraps media-utils to provide the post ID.

Is that correct?

For the rest, I'm not sure I understand if there's any WP specific logic in the rest of the logic (the client side optimizations, the compression...). At first glance, there's none right?

As mentioned, everything was built as a drop-in replacement for media-utils, so there are some WordPress specifics/assumptions in there. The ones that stand out to me right now:

• After uploading a single image, it adds all the thumbnails to the queue and uploads those to the server (using a different API endpoint than wp/v2/media)
  • This would definitely require some refactoring
• Updating post meta for attachments after upload (e.g. after generating a video poster image or compressing an existing image)
• Updating an existing attachment in the database when cropping an existing image (not yet implemented)
  • Note: I see that useSaveImage in block-editor currently incorrectly depends on api-fetch and is clearly WordPress-specific. Given that I plan on adding client-side image editing as well, this needs to be refactored.

[youknowriad]

Is that correct?

Yes, that's correct.

As mentioned, everything was built as a drop-in replacement for media-utils, so there are some WordPress specifics/assumptions in there. The ones that stand out to me right now:

Yes, the problem with that is that media-utils has always been a stateless package and making it stateful means access to its store from other packages which would break existing assumptions: things like multiple editors for instance is one example of that.

After uploading a single image, it adds all the thumbnails to the queue and uploads those to the server (using a different API endpoint than wp/v2/media)

Updating post meta for attachments after upload (e.g. after generating a video poster image or compressing an existing image)

Yeah, I think my answer here would be to try to make these use mediaUpload setting (like adding things to the arguments to handle these cases) or provide alternative settings to the block editor (which also means potentially disable these behaviors if these settings are not present)

Note: I see that useSaveImage in block-editor currently incorrectly depends on api-fetch and is clearly WordPress-specific. Given that I plan on adding client-side image editing as well, this needs to be refactored.

💯 Yes, I discovered that too late, there's an ESlint rule that is ignored there because of this (see top of the file)

[swissspidy]

OK, understood! In that case I will go through my plugin once more and make the necessary changes so it's easier to reason about & eventually port over.

[swissspidy]

Thanks for going into detail Pascal! I like your idea of a "Would you like to convert this to a web safe format?" modal that teaches the user what's going on. I've asked Automattic's legal team for some non-IANAL guidance about libheif.

@noisysocks Curious, have you ever heard back on this?

[noisysocks]

I did. Your understanding was correct, we'd need to relicense WordPress or use a different library 😞

[swissspidy]

Good news, I found some possible libheif-js alternative in the form of ffmpeg. Will share details at swissspidy/media-experiments#483

[spacedmonkey]

I have been thinking about poster generation for a while. Poster generation can be done without the need to FFMPeg. This can be done using canvas tag. @swissspidy and myself worked on something similar in web stories. See this example.

We could add some middleware to the api-fetch. This can defect if a video file is being uploaded and generate the post image, uploaded and set it as the feature image. There is already logic to detect if it is a media upload request, isMediaUploadRequest.

Poster image generate would be a good POC about handling media in the browser.

[swissspidy]

Poster generation like this is already implemented in my media-experiments plugin and can be ported over in a later stage, according to the roadmap.

[spacedmonkey]

Poster generation like this is already implemented in my media-experiments plugin and can be ported over in a later stage, according to the roadmap.

Where is the roadmap?

[swissspidy]

Literally in the issue description.

[spacedmonkey]

Literally in the issue description.

Thought there would a blog post on make with more detail. The above doesn't really give a tonne of detail about poster images, other than you mentioning that may generate poster.

Updating post meta for attachments after upload (e.g. after generating a video poster image or compressing an existing image)

Can you provide more detail how this might be archived, like using a canvas tag or ffmpeg? Will this be done as part of the upload process transparently to users or requires uses to press a button?

Regarding the roadmap, as I am trying to help, I am looking at other part of it, to see what others, like myself could work on. I also trying to understand why things are ordered how they are.

[swissspidy]

Can you provide more detail how this might be archived, like using a canvas tag or ffmpeg? Will this be done as part of the upload process transparently to users or requires uses to press a button?

You can find a working implementation using https://github.com/swissspidy/media-experiments. Right now it uses canvas but there's an ffmpeg version as well. It's all automatically behind the scenes during the upload, but there is also a button for generating a poster for existing older videos.

Regarding the roadmap, as I am trying to help, I am looking at other part of it, to see what others, like myself could work on. I also trying to understand why things are ordered how they are.

The first step is setting the foundation for client-side media processing, with the focus on images, providing a way to compress images and generate all sub-sizes in the browser. After that, other media formats like GIFs and videos can follow.

If you're interested in any of that, I recommend checking out https://github.com/swissspidy/media-experiments.

[spacedmonkey]

I have checked out the repo. I have been watching the repo since you created it. Not to mentioned much of work in the repo is based on work that you and I worked on in web stories. But what is in that repo are called experiments. It is unclear what is and isn't being ported over to Gutenberg.

My point is this, even through I have followed every step of this, I still think the above "roadmap" is unclear and needs more detail. This detail is required for a number of reasons

• To give new comers to these experiments more context
• Layout a order of actions / pieces of work to be done ( and the reasoning why this order )
• understanding of actions already done to date.

Again, just want to understand your reasoning and support here. This work is great and will be great benefit to everyone. Just want to make that benefit clear to everyone who read this ticket. 👍

[carolinan]

I was not ware of the existing limitations, and to be able to add the features that the designers have requested for the playlist block, I need help with adding similar features as those that are listed under the nice to have's for the video block.
I need to be able to edit the title, the artist, the album, and the poster image.

[swissspidy]

@carolinan What limitations are you referring to? Once that new playlist block lands of course it would be possible to use client-side media processing there as well, that's no problem. But we're still a bit far away from landing anything...