Support for putting images on the clipboard

[Rob--W]

The current specified API lacks the ability to put images on the clipboard. This seems to be motivated by the concern for exploitation of security vulnerabilities in external applications (e.g. as voiced in #41 and https://lists.w3.org/Archives/Public/public-webapps/2015AprJun/0819.html).

Chrome and Firefox currently already have the ability to copy an image via the "Copy Image" context menu option, and choosing the option allows the copied image to be used in other image editing applications. In both cases, the implementation re-encodes the image before the image is exported. This technique can also be used to support putting images on the clipboard without the security concern.

To support images, at the bare minimum one image should be put on the clipboard. Other considerations are:

• Are there requirements on the images that can be copied (e.g. PNG, JPG, GIF)?
• Are there requirements on the actual copy of the image (e.g. copy as-is, convert to PNG, convert to same MIME).
• What is the number of images that can be copied (just one?)
• When does the image conversion occur (sync vs async).
• How does the API look like? (Is the image provided as a Blob/File, as a canvas/img, or maybe a typed array or string?)
• In case of animated images, which frames are copied (all? any one? the first frame?)
• Can the copied image be retrieved again from the DataTransfer instance?
• Is the answer to the previous question different for cut/copy (navigator.clipboard.write) and paste (navigator.clipboard.write)?

I submitted a patch to Firefox that offers the ability to copy images (https://bugzilla.mozilla.org/show_bug.cgi?id=1356543#c14). This is intended to be a temporary solution until we can agree on a common implementation to be shared between browsers.

// In a copy/cut event:
event.preventDefault();
event.clearData();

var blob = new Blob(['image data here'], {type: 'image MIME type here'});
event.clipboardData.mozSetDataAt('application/x-moz-nativeimage', blob, 0);
// Or, using strings (because setData is specced to only contain strings):
event.clipboardData.setData('application/x-moz-nativeimage', 'PNG image data here');

Within the cut/copy event, whatever that is assigned to the DataTransfer instance (=event.clipboardData) can be accessed as-is (via getData or mozGetDataAt). If the result is pasted, event.clipboardData in the paste event (same or other browser) will contain a File of type image/png (=the converted image, the first frame of the original image). When pasted in an image editor, the image appears as expected.

[garykac]

The current specified API lacks the ability to put images on the clipboard.

The intent of the recently-added Async clipboard API is that it supports image data. The old synchronous clipboard API is not going to support image copy, primarily because of security concerns that can only be addressed with an async API.

See the recent async clipboard proposal for background on this. In summary, for image support the clipboard API needs to be async to allow (1) the image data to be santiized, and (2) to allow user agents to query the user for permission.

Also, in general, execCommand() in somewhat deprecated in that it is not going to be updated and no new specs will be written to rely on it.

We discussed adding helper accessors to read/write "text" and "image" data from/to the clipboard, but so far I've only added the "text" helpers in the spec. Adding image helpers would make it more apparent that the API is intended for images as well, so I should go ahead and do that (and add examples).

This is intended to be a temporary solution until we can agree on a common implementation to be shared between browsers.

The current async proposal is the result of the browser vendors discussing and coming to an agreement on how we should address this.

So, rather than adding a browser-specific temporary solution, you should consider adding support for the async API instead. Since there is already cross-browser agreement for this (and Chrome is currently working on an implementation that will ship soon), it will probably be a lot easier to get approval for your changes.

Let me know if you have any questions -- especially if you start working on an implementation since the first two independent implementations are always the most critical.

[Rob--W]

The intent of the recently-added Async clipboard API is that it supports image data.

Please formalize this intent in the draft spec. Write-access to common image types seems to be left out whereas read-only access seems possible.

If the spec'd API is stable, I can look into getting it implemented in Firefox. There are already some open bugs with no assignee:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1351193 Implement the DataTransfer constructor
• https://bugzilla.mozilla.org/show_bug.cgi?id=1221562 Investigate asynchronous DataTransfer API

In the last bug, there is feedback on the API, but I guess that it never got to you since there is no reply: https://bugzilla.mozilla.org/show_bug.cgi?id=1221562#c4. Perhaps you can take a look at it?

And for image copying, I created an experimental prototype to see what issues would arise in implementing the functionality (independent of how the API looks like). The results are summarized in my initial report here, so please review those bullet points too.,

[garykac]

In the last bug, there is feedback on the API, but I guess that it never got to you since there is no reply: https://bugzilla.mozilla.org/show_bug.cgi?id=1221562#c4. Perhaps you can take a look at it?

Yes, there is general agreement that DataTransfer should also get an async API. When (if?) that happens, the async Clipboard API implementations would be trivially updated to use it.

But we don't want the async Clipboard API to be blocked on resolving Drag-n-drop issues (which would certainly be brought up in the context of revisiting DataTransfer).

I believe the plan (although I don't know of anyone actively working on it) is to have an async Drag-n-drop proposal and use that to motivate changes to DataTransfer. I agree that it would have been nice if DataTransfer already had an async Promise-base API, but that's not likely to happen anytime soon.

And for image copying, I created an experimental prototype to see what issues would arise in implementing the functionality (independent of how the API looks like). The results are summarized in my initial report here, so please review those bullet points too.,

Will do. I'm working on Chrome's implementation, so I've bumped into some of them, but I hadn't considered others (like animated images). Thanks for summarizing.

When I've updated the spec with more info on image support, I'll update this issue so that you can take a look at the changes and make comments.

[rniwa]

As far as I can tell, the primary problem we have is that we don't have a mechanism to asynchronously add image data to DataTransferItemList. If we added such a method, we can support putting images into pasteboard/clipboard without having to add navigator.clipboard at least in WebKit since we can sanitize images inside those methods instead of sanitizing the entire DataTransfer object at once.

[Rob--W]

@rniwa

Would the need for sanitizing the image disappear if the image data is sourced from the browser instead of directly set by the user?
For instance, if the add method of DataTransferItemList were to accept views of images, e.g. HTMLImageElement, HTMLCanvasElement, CanvasRenderingContext2D, or (independent of documents, usable in workers too:) ImageBitMap, then web authors can set an image, and browsers have full control over what ends up on the clipboard.

[rniwa]

It would definitely mitigate the security concern if the browser is the one encoding the image.

[alecazam]

Can you please use the original image if it doesn't need sanitizing, rather than using the re-encoded image. Our application needs to get the original user images with metadata out to other design tools.

[alecazam]

The reason that I don't trust sanitizing images is that we render out up to 32k x 32k PNG and JPG. We also pass-through APNG files, and would like to pass through GIF. I can see the sanitizer trying to open these large files to re-render them and failing, or the alternative is doing all the work on the CPU and taking forever for large images. These files are all small enough once compressed but can still be a few hundred MB, and it feels like all attack vectors are in the header or IDAT lengths in the case of PNG.

[garykac]

Proposal: https://docs.google.com/document/d/1sMcajoYAQ_M_VOIn1RLokG1sVB7skjoeviqt9aZOcDk/edit