Proposal: Offscreen Documents for Manifest V3

[dotproto]

Today, Manifest V3 Chrome extensions cannot leverage DOM-related features and APIs in a background context due to service workers not having access to the DOM. This impacts capabilities like headless audio playback, background document scraping, and stream processing. To continue supporting these features, the Chrome team is proposing adding Offscreen Documents support to the Manifest V3 platform.

Allowing Extensions to Open Offscreen Documents in Manifest V3

The full proposal can be found above, but in short offscreen documents provide a temporary, headless page environment that allows an extension to leverage DOM capabilities in the background. Since the contexts are specifically designed for DOM API usage, they will not have access to powerful extension APIs. Offscreen document lifetimes are not tied to the context that spawned them, meaning that an offscreen may outlive the service worker that created it. As an ephemeral context, offscreen documents will be terminated if they are no longer doing work; extension authors should anticipate and prepare to recover from such scenarios.

We are particularly interested in discussing and iterating on the API surface for this capability and identifying reasons extensions may create an offscreen document.

EDIT 2022-03-10: Added a description. Previously we only had a link to the proposal doc.

[tophf]

1. Will it run in the same main thread as the extension's tabs/popups?

   If it's same-thread, it would make sense to expose it via chrome.extension.getViews with its own type so the visible extension tabs/popups can access it. AFAICT, such exposure won't grant it more abilities than it already has, but it will make the migration less painful as many MV2 extensions were relying on getBackgroundPage API.

2. Will an extension page (popup or tab) be able to access this document directly?

   Primarily to get non-serializable content like DOM elements of the scraped web page or a huge ArrayBuffer after muxing/demuxing a captured video file, which would take a long time to transfer even via web API like postMessage with a transferrable object or via URL.createObjectURL.

   Something in the vein of synchronous chrome.extension.getBackgroundPage() and asynchronous chrome.runtime.getBackgroundPage().

3. Direct messaging from visible tabs/popups or a content script?

   The purpose is to avoid broadcasting to the entire extension via chrome.runtime.sendMessage or BroadcastChannel or waking up the service worker just to find the page in its clients. A somewhat awkward workaround would be broadcasting a unique "id", which the offscreen document will use in chrome.runtime.connect({name: id}) to handshake with the correct chrome.runtime.onConnect listener of the calling page.

   Broadcasting may be unwanted as it wakes up the service worker in case it has an onMessage listener.

   It might be nice if the options parameter of chrome.runtime.sendMessage supported targets property taking an array of 'background', 'offscreen-document', 'all'.

4. Will it have its own content_security_policy entry in manifest.json?

5. Will it have a link in chrome://extensions UI to open it in devtools?

[tophf]

The bad thing is that this will increase resource consumption of extensions (roughly twice compared to either SW or the old background page) while the missing features are implemented for the service worker context, which realistically will take more than 5 years, maybe even 10, judging by the two-year lifetime of MV3 during which only a couple of simple features were addressed out of dozens more complex ones. The web specification isn't interested in many/most of these lost features, and even if they were these specifications change extremely slow due to the need to account for a lot of edge cases that matter for the web, but not for extensions.

Coupled with the fact that service workers offer no real benefits over the classic environment for the majority of extensions as both the time to wake up and the memory footprint as well as performance are practically the same, it means that the decision to remove the old DOM-based background pages is premature and should be postponed to ManifestV4. There's just one rare case where a service worker is better: extensions that observe frequent events like network requests while showing an exceptionally heavy UI that can block the extension's main thread e.g. complex extensions for devtools.

For ManifestV3 a much more performant option is to allow both SW and non-persistent background pages, while enticing the developers to migrate by offering preferential treatment of SW-based extensions e.g. shorter reviews and/or some kind of badge in the store and in the browser. Mozilla is doing the right thing here with their support of the limited event pages, which isn't surprising as they have an actual experienced extension developer involved in the design and implementation.

[nir-walkme]

I have one request regarding the "Purpose".
The user agent may display the "Purpose" to the user or may even require user approval for this "Purpose".
While this can make sense for user installed extensions, I think this should not happen for force-installed extensions.
In these use cases, users are not aware of the extensions that are being installed by the admin. Displaying the "Purpose" to the user may confuse them as they are probably not aware of the extension. Moreover, allowing the users to disapprove the functionality may block important extensions that were deemed necessary by the organization's admin.

Therefore I suggest that for force-installed extensions (or any other method available to install an extension by an admin) the "Purpose" will not be presented to the end user nor any prompt to approve it.

Side note: I think that this proposal is too little too late. MV3 has far too many limitations that this proposal does not address.

[greghuc]

It would be good if the OffscreenDocument had an optional "incognito mode", which does not have the user's cookies and other personal state.

In something like dom scraping, you don't want users having a bad/confusing experience of a particular website, if that website generates a personalised experience based on the user's browsing history, and the history has been "polluted" by an extension's OffscreenDocument accessing certain web pages without the user's explicit involvement. An "incognito mode" would fix this.

For something like user automation (with an extension taking certain actions on a website, on the user's behalf), then you don't want "incognito mode", as the extension will need the user's session to access the website and take action.

[dotproto]

1. Will it run in the same main thread as the extension's tabs/popups?

Yes and no. Chrome is planning to implement offscreen documents in the same process and thread as other extensions pages in our initial implementation, but we may want to change this in the future.

extension.getViews() is challenging largely because it exposes direct access to Window instances. While this can be put to good use (for example, the data transfer optimizations you mentioned in #2), it also limits the browser's ability to parallelize different parts of an extension. If we expose offscreen documents in getViews(), we would have to introduce breaking changes to this method in the future in order to move offscreen documents to a separate thread.

While we're not planning to expose offscreen documents in this method, we are interested in making it easier for other contexts to exchange messages with offscreen documents.

2. Will an extension page (popup or tab) be able to access this document directly?

No. Broadly speaking, we'd like to bring cross-context WebExtensions communications capabilities more in line with the web platform. While there are some ways that two documents can synchronously interact with each other (e.g. using window.open() to open another page on the same origin), this is the exception rather than the rule.

In order for two independently created tabs on the same origin to communicate with each other, they must use asynchronous message passing capabilities like the ones you outlined. If these capabilities are insufficient, we believe it's best to address that problem at the web platform level.

3. Direct messaging from visible tabs/popups or a content script?

We are currently planning to expand runtime.sendMessage() to support passing a message to a specific document and frameID. Since offscreen documents will have both a document and frame ID, this will make it possible for other contexts to pass messages directly to an offscreen document. We are also planning to support open web messaging techniques.

4. Will it have its own content_security_policy entry in manifest.json?

Not yet decided, but at the moment I'm hesitant on this. If you (or anyone else) feel this would be valuable, please share how/why you'd like a separate CSP for offscreen documents (especially since, as a full document, these could be included instead within the HTML element with a tag).

5. Will it have a link in chrome://extensions UI to open it in devtools?

Yes.

[tophf]

changes to this method in the future in order to move offscreen documents to a separate thread

Implementing native DOM API in a separate thread might be architecturally impossible in all browsers, because DOM was never designed to be thread-safe, and I doubt it'll change in the foreseeable future (~10 years), so running it in the main thread may be the only realistic solution.

[jasonmayes]

On the TensorFlow.js side we have a number of users wanting to create ZOOM / Google Meet / other online Video Conf solution based chrome extensions that need access to webcam stream to then manipulate it in some shape or form via offscreen canvas or similar to then send that canvas stream to a virtual camera that can be picked up via the original app in question to make that their webcam source. By means of example here is an example of me joining a Google Meet with an ML powered Chrome extension that does just this: https://www.youtube.com/watch?v=-JpCPIx1WGw

Since publishing the above I have seen more companies and individuals try to replicate this for their own domains too.
A great public example of this would be for Virtual Sapiens - a company who uses TensorFlow.js to understand user behaviors during a meeting to provide constructive feedback so that you can become a better presenter: https://www.virtualsapiens.co/ - check the video demo here: https://www.youtube.com/watch?v=cSyAjMhbw30&

I am seeing a growing interest in this sort of area for long living >5 mins situations that need to constantly handle stream data from webcam/microphone, along with machine learning models that may leverage things like WebGL / WASM for acceleration so that the ML can run fast in the browser environment.

[montrosb]

Simeon -

1. Will the offscreen document have access to the same localStorage data that was previously available to a background script?

2. Will the offscreen document have the ability to send the service worker a message? If so, would this message activate the service worker if it was inactive at the time the message was sent?

[tophf]

Since @dotproto didn't answer in the latest minutes why Chromium team insists that this proposal is better in the long term than the old event pages or the new limited event pages idea, I'd like to point out that this offscreen document proposal (on top of being an unnecessary complication) can be completely subsumed by event pages if the API provides a method to prolong the lifetime of the currently running background script with a justification - exactly as proposed here, but better because a single event page would consume much less resources/CPU than having an SW that constantly/randomly restarts in parallel with an offscreen document.

[birtles]

Hi,

Thank you for posting this proposal. I have read it, and while I have quite possibly misunderstood important parts, I am concerned about its potential long-term impact.

Concern about fragmentation

Suppose we add the ability to play audio from ServiceWorkers via AudioWorklet, obviating the need for offscreen documents for that particular use case. There's every chance that Mozilla or Apple may decide not to implement that feature due to privacy or usability or architectural concerns, or simply due to current resource constraints. In such a situation, extension authors would need to prepare two versions of their extension: the offscreen document version and the ServiceWorker version.

This might appear to be the same as any other platform feature that needs to be feature-detected but it is more significant because it involves a different architecture based on the availability of the feature (i.e. it is not progressive enhancement but moving functionality around).

Furthermore, given the amount of work required to implement a new Web platform API, it seems likely such situations will multiply over time making the permutations of architectures authors need to support unwieldy.

Concern about stability

The requirement to provide a series of Purpose values and a justification makes this API feel much less like a promise (such as the Web platform normally provides) and more like a temporary concession. If at any point the powers that be decide my purpose is no longer valid or my justification is insufficient, then the rug can be pulled out from under me.

Despite the amount of anxiety raised by the transition to MV3, this proposal sounds like extension developers can expect to endure a similar experience of having to scramble to rewrite their extensions every few months as Purposes are obsoleted. Furthermore, those building a business around an extension could wake up any day to find the APIs they rely on are no longer deemed valid and they are out of business.

If you will allow me to be a bit frank, in terms of optics, this comes across as a rather condescending API.

There will undoubtedly also be many a use case that does not neatly fit in any of the provided Purpose categories. Are we really confident we can capture every single use case in an enum? If not, doesn't the "other" category (whether explicit or implicit) undermine the utility of the purpose field altogether?

Perhaps this can be addressed by requiring manual reviews for extensions using the "other" category, but doing so introduces a cost to both extension store providers and extension developers. Would it not be preferable to use an API that does not introduce such costs?

Concern about impact on the architecture of the Web platform

Furthermore, are we (and the TAG etc.) really confident that every reasonable use case should—from an architectural point-of-view—be made available to Service Workers in the future anyway, even if only made accessible to extensions?

For example, my add-on would like, from the background page/worker, to load an iframe from an associated cross-origin Web site and query its offline data via postMessage (that is, the data is already on the user's computer, not on a server, hence why an API call is not suitable). Is that really something that a ServiceWorker should do?

Thank you for all your work on this and for publishing your findings. I hope my comments are constructive or that you can explain where I have misunderstood.

[Juraj-Masiar]

I would like to use Offscreen Document to create thumbnails of pages. (in my speed dial)
Currently I use tabs.hide API in Firefox and in Chrome I open new window for that, which steals user focus (super annoying).
I open the page in iframe and then take the screenshot of the tab (and crop it with canvas).

To make this work I need captureVisibleTab to work with Offscreen Document and I need iframe to fully load the content, just like if the tab was focused. I know there are some performance optimizations that applies to "inactive tabs" so it would be nice to make some compromise here. Maybe limit the re-paint to 1Hz instead of 60+. Or maybe draw the content only after calling requestAnimationFrame or add some new API to force full content paint.

[aeharding]

I strongly suggest chrome.tabCapture is also in this context, since one of the main points of the offscreen document api is to allow video capture... 🤦‍♂️

[jccr]

Are there any updates on this API? The tracking bug hasn't had an associated commit since July (#c24). The Known Issues page mentions "Canary support around October, 2022". So I ask, what is the updated plan for this?

I too wonder about an update on the timeline.

There is a new commit that was sent for review yesterday, following the commit in July:

https://chromium-review.googlesource.com/c/chromium/src/+/4000752

[asamuzaK]

Requires new offscreen permission, have to prepare a new HTML document for offscreen which is very similar to current background page but not the same, have to check if the offscreen document already exists, etc., from a compatibility point of view, I can not agree with this proposal as it just adds more work for the developer.

Can someone on the Google side please explain what is wrong with Limited Event Pages for MV3 #134?

[tophf]

Can someone on the Google side please explain what is wrong with Limited Event Pages for MV3

Chromium team has been effectively ignoring such appeals for the past several years to back their claims about [negligible] problems with background pages or event pages and their claims about [non-existent] improvements of using service workers.

We also see that there are no improvements in simplicity of source code of Chromium - on the contrary it will become more and more complicated as it deviates from the standard service worker specification for the Web platform and it has to deviate because 99% extensions don't need a service worker technology per se (they just need an invisible JS context), which means that all the limitations and caveats of service workers need to be patched/disabled for extensions.

We also see thousands of users still report that service worker registration is unreliable and can break randomly on update.

It's time someone who can admit the reality steps up and stops this service worker + offscreen farce.

[dessant]

It also speaks volumes that we have 26 days left to migrate our extensions to Manifest V3, otherwise we will lose our Featured badges, and a significant source of traffic with it, while a good portion of web APIs have been simply made unavailable to extensions. I'm not sure how much of a motivation it will be for anyone to watch as years of their work gets destroyed while they eagerly await for some crumbs to be dropped in the form of the Offscreen Documents API sometime next year.

[StiliyanHoody]

It also speaks volumes that we have 26 days left to migrate our extensions to Manifest V3, otherwise we will lose our Featured badges, and a significant source of traffic with it, while a good portion of web APIs have been simply made unavailable to extensions. I'm not sure how much of a motivation it will be for anyone to watch as years of their work gets destroyed while they eagerly await for some crumbs to be dropped in the form of the Offscreen Documents API sometime next year.

Very well said!

[oliverdunk]

Switched this to implemented: chrome Implemented in Chrome as the feature is now live: https://developer.chrome.com/blog/Offscreen-Documents-in-Manifest-v3/

[hanguokai]

I tried this api yesterday, and write two feedbacks at:
https://groups.google.com/a/chromium.org/g/chromium-extensions/c/sDNbgeVHEEw/m/7KQ4R1ELAgAJ
https://groups.google.com/a/chromium.org/g/chromium-extensions/c/D5Jg2ukyvUc/m/VaSvEfoHAgAJ

[cacosandon]

I tried Offscreen Documents for calling chrome.desktopCapture or getUserMedia and it doesn't work.

Had to do this workaround: https://github.com/wireworks-app/chrome-screen-recording

[rektide]

I'm very confused why this got built for Web Extensions specifically? Offscreen DOM has been asked for time and time again (notably captured in w3c/ServiceWorker#846 which links plenty of other things).

It feels like rather than obey the Extensible Web Manifesto & build general low level capabilities everyone can use, something botique got built just for Web Extensions. How when and where does the rest of the web start to also benefit from this accidental invention on the side of Web Extensions? Why was this the right arena for this work?

To me, name illustrates the weirdness here. This work would be invaluable as "Offscreen Documents". Making it "Offscreen Documents for Manifest V3" is qualifier that makes it niche & special purpose. The overspecialization make it feel like this work won't parlay appropriately into the greater web project. I'd love to know for example what the WinterCG people might work or might be a mismatch about the ideas here.

[oliverdunk]

I tried Offscreen Documents for calling chrome.desktopCapture or getUserMedia and it doesn't work.

Had to do this workaround: https://github.com/wireworks-app/chrome-screen-recording

Just to close the loop, starting in Chrome 116 we support obtaining a stream ID in a service worker (using the chrome.tabCapture API). This can then be passed to an offscreen document :)

There's some more information here: https://developer.chrome.com/docs/extensions/mv3/screen_capture/#audio-and-video-offscreen-doc

I'm very confused why this got built for Web Extensions specifically? Offscreen DOM has been asked for time and time again (notably captured in w3c/ServiceWorker#846 which links plenty of other things).

It feels like rather than obey the Extensible Web Manifesto & build general low level capabilities everyone can use, something botique got built just for Web Extensions. How when and where does the rest of the web start to also benefit from this accidental invention on the side of Web Extensions? Why was this the right arena for this work?

To me, name illustrates the weirdness here. This work would be invaluable as "Offscreen Documents". Making it "Offscreen Documents for Manifest V3" is qualifier that makes it niche & special purpose. The overspecialization make it feel like this work won't parlay appropriately into the greater web project. I'd love to know for example what the WinterCG people might work or might be a mismatch about the ideas here.

I definitely appreciate the concern. As mentioned in the Summary section, the long term goal is that this functionality should be supported in service workers. Offscreen documents are intended as a temporary migration measure and hopefully over time they will be needed less and less. While the proposal was written before I joined Google, I can definitely see why the team might have chosen to build this as an extension specific API - building something for the web would need a lot of thought due to the long term backwards-compatibility implications. On the other hand, while extension APIs still need thought, it's a place where we can definitely move more quickly which was important in unblocking use cases in the migration to Manifest V3. If we realise this concept is useful beyond migration or learn interesting things we can absolutely feed those back to the web.

[anthonylebrun]

First off, thanks to the chrome team for supporting passing a streamId from chrome.tabCapture in release 116, that is a huge leap forward!

I am working on an extension that needs to support full screen recording (not just on a per tab basis) and was wondering if similar support for chrome.desktopCapture.chooseDesktopMedia is being considered.

Both functions return a streamId that could in theory be passed to getUserMedia in the offscreen document, if only calling chrome.desktopCapture.chooseDesktopMedia was supported from the service worker.

Are there any blocking privacy concerns that would prevent this from ever being supported? Or is this something the team would consider / is already on the roadmap?

[oliverdunk]

Both functions return a streamId that could in theory be passed to getUserMedia in the offscreen document, if only calling chrome.desktopCapture.chooseDesktopMedia was supported from the service worker.

Are there any blocking privacy concerns that would prevent this from ever being supported? Or is this something the team would consider / is already on the roadmap?

@anthonylebrun, glad you're excited about the tabCapture change and appreciate the feedback on desktopCapture! Out of interest, is there anything specific desktopCapture gives you that getDisplayMedia doesn't? It would be nice to lean on the web APIs if possible.

[MassimoDeLuca]

Out of interest, is there anything specific desktopCapture gives you that getDisplayMedia doesn't?

@oliverdunk desktopCapture supports cancelChooseDesktopMedia which allows the extension to determine if the "Choose what to share" screen is open, and close it to avoid multiple simultaneous recordings.

Our extension is designed to only capture one source at a time, and although with getDisplayMedia it is possible to detect if the window is open, its not possible to close it, requiring an additional UI to instruct the user to find the pre-existing "Choose what to share" screen.

It would be nice to lean on the web APIs if possible.

Is there any benefits around using getDisplayMedia? chooseDesktopMedia still leverages getUserMedia for the capture of the screen.

[dangkieub13]

Hi @oliverdunk and everyone,

I have an use case where I the extension open a pop-up which list down a few different website. The moment user click on one of the option, we will open a new tab and record that new tab.

However, it seems like chrome.tabCapture API can only be triggered via action (which is not available if we use popup).

I have tried a few ways to retrieve the tab id

Method 1: Get the tabId from chrome.tabs.create API

  const newTab = await chrome.tabs.create({
    url: "https://www.wikipedia.com/",
  });

    const existingContexts = await chrome.runtime.getContexts({});
  let recording = false;

  const offscreenDocument = existingContexts.find(
    (c) => c.contextType === 'OFFSCREEN_DOCUMENT'
  );

  // If an offscreen document is not already open, create one.
  if (!offscreenDocument) {
    // Create an offscreen document.
    await chrome.offscreen.createDocument({
      url: 'offscreen.html',
      reasons: ['USER_MEDIA'],
      justification: 'Recording from chrome.tabCapture API'
    });
  } else {
    recording = offscreenDocument.documentUrl.endsWith('#recording');
  }

  if (recording) {
    chrome.runtime.sendMessage({
      type: 'stop-recording',
      target: 'offscreen'
    });
    chrome.action.setIcon({ path: 'icons/not-recording.png' });
    return;
  }

  // Get a MediaStream for the active tab.
  const streamId = await chrome.tabCapture.getMediaStreamId({
    targetTabId: newTab.id
  });
  

  // Send the stream ID to the offscreen document to start recording.
  chrome.runtime.sendMessage({
    type: 'start-recording',
    target: 'offscreen',
    data: streamId
  });

  chrome.action.setIcon({ path: '/icons/recording.png' });

Method 2: Query

  const currentTab = await chrome.tabs.query({
    currentWindow: true,
    active: true,
  });

  const streamId = await chrome.tabCapture.getMediaStreamId({
    targetTabId: currentTab[0].id,
  });

[dangkieub13]

For both method mentioned above, we hit an error

Uncaught (in promise) Error: Extension has not been invoked for the current page (see activeTab permission). Chrome pages cannot be captured.

I would like to clarify if chrome.tabCapture can only be used on the same tab that is currently Active. And do not work for newly opened tab (even if the opening action is triggered by the extension pop-up)

[oliverdunk]

@oliverdunk desktopCapture supports cancelChooseDesktopMedia which allows the extension to determine if the "Choose what to share" screen is open, and close it to avoid multiple simultaneous recordings.

Thanks, that's good context!

Is there any benefits around using getDisplayMedia? chooseDesktopMedia still leverages getUserMedia for the capture of the screen.

The benefit for this sort of thing is reducing the amount of code in Chromium + making it easier for web developers to learn extensions. Generally the fewer APIs someone needs to be aware of the better.

I would like to clarify if chrome.tabCapture can only be used on the same tab that is currently Active. And do not work for newly opened tab (even if the opening action is triggered by the extension pop-up)

I believe you can do it for any tab, but the extension must have been invoked on that tab (so a page opened by a popup would not work). You could use the normal getDisplayMedia API for this, or open a feature request to have a way of opening a new tab and immediately starting recording. To be clear, I don't think this is a new limitation in MV3.

For replies to any of the above, let's move to Chrome bugs or https://groups.google.com/a/chromium.org/g/chromium-extensions to avoid adding noise to the WECG :)

[fregante]

A use case for this PR turns out to be… alert()

There are two issues because this API is not available in workers and not available at all in Firefox' and Safari's background pages.

• How to inform the user of errors and progress from a Service Worker? #259
• Proposal: browser.alert.create() #272

There isn't a reason specific to this yet, but it looks like calling alert() from such document works great.

If this API becomes cross-browser and this usage becomes accepted, I think we wouldn't necessarily need a dedicated chrome.alert API.

[Nitesh2503]

Can we get the media streams through the following process :

1. Using desktopCapture in service worker to get the particular streamId
2. Pass the extracted streamId to the offscreen
3. Use getUserMedia API to get the streams in the offscreen

Tried the above way but ended with the following errors :

Could not retrieve stream DOMException: Invalid state
Error: DOMException: Invalid state

[oliverdunk]

Can we get the media streams through the following process :

1. Using desktopCapture in service worker to get the particular streamId
2. Pass the extracted streamId to the offscreen
3. Use getUserMedia API to get the streams in the offscreen

This is possible for tabs using the chrome.tabCapture API in Chrome 116: https://developer.chrome.com/docs/extensions/how-to/web-platform/screen-capture#audio-and-video-new-tab

It is not currently possible for chrome.desktopCapture, but we're tracking that here: https://issues.chromium.org/issues/326509126

I'd suggest the chromium-extensions mailing list if you want to chat more just so we keep this thread focused on the proposal :)

[Nitesh2503]

Thanks @oliverdunk,

Basically this is my use case :

• According to particular tabId using the chrome.tabCapture.getMediaStreamId I want to get streamId out of it.
• Then send this streamId in the offscreen doc
• Now using the navigator.mediaDevices.getUserMedia I want to get the media streams

Following the above while using the chrome.tabCapture getting the below error :

Unchecked runtime.lastError: Extension has not been invoked for the current page (see activeTab permission). Chrome pages cannot be captured.

I have tried removing and attaching the activeTab permission but ending up with the same error.
However I have other permissions in-place like tabCapture, tabs

[oliverdunk]

Hi @Nitesh2503, as mentioned could you move to the mailing list? Happy to help but I'd like to avoid adding too much noise here :)