[Feature Request] [Sample Implementation Provided] Support other image formats than PNG for the {clipboard-image} handlebars

[kuroahna]

Description

Using the Yomitan {clipboard-image} handlebars for your Anki templates will take the image from your clipboard and save it into Anki using AnkiConnect. However, at least in Windows (not sure about Mac OS/Linux), both Firefox and Chrome will always take the raw bitmap stored in the Windows clipboard as image/png. This means that even if you save an image in your clipboard as JPEG or any other file format, the browser will not be able to recognize it and always save it as PNG. I have confirmed this by using ShareX and changing the settings to save the image as JPEG in the clipboard and pasting it into Anki directly. Anki is able to recognize the JPEG format stored in the clipboard and properly paste it as jpeg. Also, pasting the jpeg stored in the clipboard into an application like Discord will also convert it to PNG (which I believe is because Discord is written using Electron which is basically running Chromium). Yomitan (more specifically chrome/firefox on windows) will always save it as image/png. Note, looking online, I see that on Linux, it also has the same issue of not being able to copy JPEG.

Looking at the w3c Clipboard API spec, they state:

6.4. Mandatory data types
The implementation must recognise the native OS clipboard format description for the following data types, to be able to populate the DataTransferItemList with the correct description for paste events, and set the correct data format on the OS clipboard in response to copy and cut events.

6.4.1. Reading from the clipboard
These data types must be exposed by paste events if a corresponding native type exists on the clipboard:
text/plain
text/html
image/png

So the browser's implementation must be able to at least support text/plain, text/html, and image/png (image/jpeg and other image formats is not listed as a mandatory data type)

And for the clipboard.read() api

3. Let data be a copy of the system clipboard data represented as clipboard items. For the MIME types defined in the mandatory data types list, data MAY be sanitized, but image/png format has unsanitized payload to preserve meta data.

The browser's clipboard implementation may sanitize the clipboard data.

And according to the w3c clipboard-api github issue Getting file/image from clipboard #134

Generally, browsers will sanitize clipboard contents to prevent security issues (ex. remote code execution vulnerabilities in image encoders), so sites can't expect clipboard contents to have be completely preserved/identical after copying/pasting. One unfortunate side effect of this is that exif rotation and other metadata is commonly stripped, and browser-internal formats may sometimes differ from formats used on the web (Chrome happens to prefer bitmaps/PNGs for clipboard image formats).

I suspect Firefox to also prefer bitmaps/PNGs for clipboard image formats.

Proposal

With the above limitations, my current idea is for Yomitan to handle the image processing logic itself. If we assume that the clipboard image source will always be image/png, then we can simply convert it on the fly to a desired image format before saving it to Anki via AnkiConnect.

Sample Implementation

I have created a sample implementation on my GitHub fork as a proof of concept (not production ready yet) written in Rust+WebAssembly which always converts the clipboard image to JPEG and hardcodes the quality level to 80.

Advantages

The advantage of having this feature is so that it saves disk space when saving images to Anki and makes creating cards blazing fast since it doesn't need to send a lot of data. I did some informal benchmarking, and a ~1.8 MB PNG gets shaved down to ~194 KB JPEG (80% quality). Furthermore, sending the ~1.8 MB PNG over HTTP to AnkiConnect takes ~620 milliseconds, whereas the ~194 KB JPEG only takes 35 milliseconds (~17x faster).

Another potential advantage here is that this could work on mobile (Android/iOS) as well. There's some Anki workflows on Android that uses Yomichan via Kiwi Browser and since we do the image compression using Wasm, it should just work

Disadvantages

The disadvantage of this approach is that it adds extra complexity to the project by introducing Rust+Wasm, and hence extra maintenance costs.

Improvements

Other image formats could be supported such as WEBP or AVIF, which provides even better compression than JPEG. However, webp is currently not available in wasm.

Also, for the sample implementation, we should have this configurable via the Settings page in Yomitan rather than always converting the clipboard image to JPEG and hardcoding the quality level to 80. Looking at the Yomitan settings page, there already exists a Screenshot format option, see images below:

The description says

Adjust the format and quality of screenshots created for cards.

I actually was confused with this before, and thought it also affected the {clipboard-image} handlebars, but it didn't. Currently, this setting only affects the {screenshot} handlebars. We can potentially re-use this settings option and make it so that {screenshot} and {clipboard-image} handlebars are both affected. Or, we create a new settings option strictly for the {clipboard-image} handlebars.

Alternative Proposal

Ideally, Yomitan shouldn't be concerned with the image file format stored in the clipboard, but due to the limitations of the browser implementation, it seems like we have to handle this ourselves. However, there also seems to be the option of supporting Web custom formats for the Async Clipboard API.

Reference

1. Clipboard pickling proposal
2. Web custom formats for the Async Clipboard API

This seems like a cleaner option and relieves Yomitan from handling the image compression logic, however, Yomitan needs to look for the web clipboard type when looping through the clipboard items. And, another disadvantage here is that screenshot programs must somehow prepend this web MIME type so that the MIME type is web image/jpeg or web image/webp or web image/avif so that the browser doesn't sanitize the clipboard image. I'm not sure how this can be done, and ShareX does not do this yet