Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Principle: Use [Exposed=*] to make only purely computational APIs available #509

Open
ptomato opened this issue Aug 22, 2024 · 20 comments · May be fixed by #510
Open

New Principle: Use [Exposed=*] to make only purely computational APIs available #509

ptomato opened this issue Aug 22, 2024 · 20 comments · May be fixed by #510

Comments

@ptomato
Copy link

ptomato commented Aug 22, 2024

Problem statement

In webidl#526 a new Exposed annotation, [Exposed=*], was introduced. It denotes a fundamental set of interfaces that are intended to be exposed in Window, all Workers, all Worklets, ShadowRealm, as well as any future global scopes.

We've heard concerns (e.g., tc39/proposal-shadowrealm#401) that it's not clear when an interface should belong to this fundamental set and when it shouldn't. We'd like to issue guidance for this in a design principle.

I'm interested in feedback on the proposed guidance below, and am willing to write up the conclusions in a PR if it is generally positive.

Proposed guidance

[Exposed=*] should be applied only to purely computational interfaces. That is, they do not perform I/O and do not affect the state of the user agent or the user's device.

Anything annotated with [SecureContext] should not be exposed everywhere; not all global scopes are secure contexts.

Anything relying on an event loop should not be exposed everywhere; not all global scopes have an event loop.

The [Exposed=*] annotation should also be applied conservatively. If an interface is not that useful without other interfaces that are not exposed everywhere, default to not exposing that interface as well.

Further reading

Directly related, but predates [Exposed=*]: #35

Discussion resulting in the addition of [Exposed=*]: webidl#468

Other relevant reading: #325, #360, #448, tc39/ecma262#1120, WebAudio/web-audio-api#2499, tc39/proposal-shadowrealm#398

@zcorpan
Copy link

zcorpan commented Sep 6, 2024

cc @smaug---- @mgaudet

@annevk
Copy link
Member

annevk commented Sep 6, 2024

This sounds reasonable. I'm not sure if it needs to be a principle or could just be advice in the Web IDL standard. It might also be helpful to have some examples so people can evaluate whether they understand the rules correctly.

@ptomato
Copy link
Author

ptomato commented Sep 6, 2024

As it so happens, I do have some examples to propose.

The TextEncoder interface converts a string to UTF-8 encoded bytes. This is a purely computational interface, generally useful as a JS language facility, so it should be exposed everywhere.

localStorage is not purely computational because it affects the state of the user agent, so it should not be exposed everywhere.

The timeout method of the AbortSignal interface relies on an event loop and should not be exposed everywhere.

The Blob interface is purely computational, but Blob objects are primarily used for, or obtained as a result of, I/O. By the principle of exposing conservatively, Blob should not be exposed everywhere.

@mgaudet
Copy link

mgaudet commented Sep 6, 2024

I support having guidance for this. I think this is a good piece of guidance.

I think Blob, as described, sits right on the line.

ptomato added a commit to ptomato/w3ctag-design-principles that referenced this issue Sep 11, 2024
@ptomato
Copy link
Author

ptomato commented Sep 12, 2024

Given the positive feedback so far, I've gone ahead and drafted a text for this in #510.

@annevk
Copy link
Member

annevk commented Oct 2, 2024

(Given the File API discussion in WebApps at TPAC Blob should be out for now as some people think it should maybe count towards a website's quota at some point. That would definitely not make it fit for all realms I think.)

@bakkot
Copy link
Contributor

bakkot commented Oct 2, 2024

In the spirit of clarifying edge cases, what about

  • OffScreenCanvas
  • ImageBitmap
  • WebGPU
  • WebCodecs
    ?

I'm guessing no, but I'm not sure I could articulate exactly what makes OffScreenCanvas or WebCodecs not pure computation (at least the subset of the APIs which doesn't do any network/on-screen rendering stuff).

ptomato added a commit to ptomato/w3ctag-design-principles that referenced this issue Oct 4, 2024
@ptomato
Copy link
Author

ptomato commented Oct 4, 2024

I'm less familiar with the above list, but I believe ImageBitmap's primary use is for drawing on a <canvas>, and OffscreenCanvas's primary use is transferring the contents to an ImageBitmap, so I'd say these are out by the principle of exposing conservatively.

WebGPU and WebCodecs I don't know much about, but they also seem like they lose a lot of their usefulness if you don't have canvases and A/V pipelines respectively. I guess WebGPU can also be used for neural networks, but then there's also WebNN.

@bakkot
Copy link
Contributor

bakkot commented Oct 4, 2024

It's certainly true that WebCodecs lose a lot of their usefulness without having somewhere to draw, but on the other hand I use ffmpeg a lot without ever displaying content, and something equivalent could be built on WebCodecs even without the ability to display content (I believe, anyway).

WebNN might be broadly available someday, but there's already a lot of ML built on top of WebGPU now. I would guess a majority of WebGPU consumers are actually ML, since for rendering we already have WebGL.

@ptomato
Copy link
Author

ptomato commented Oct 4, 2024

Assuming we go forward with this guidance, I suppose the maintainers of WebCodecs, WebGPU, and WebNN are now best positioned to decide whether they want to add [Exposed=*].

@padenot
Copy link

padenot commented Oct 7, 2024

Web Codecs isn't "purely computational", it can use lots of system calls, hardware facilities, and generally very little of the operations on Web Codecs have any notion of strictly bounded run time. Besides, it uses the event loop heavily, because almost everything is async in the API. OffscreenCanvas, ImageBitmap, WebGPU, WebNN` can use hardware acceleration underneath, same deal (but sometimes sync). It's perfectly plausible and often desirable to decide to expose them in some context, but they cannot be exposed in all contexts.

When thinking if something could be [Exposed=*], a good test is to ask "can I use it in an AudioWorkletGlobalScope without causing too many problems". The answer is almost always no, and almost always the same as [Exposed=*], this is because it's the most restricted context in the Web Platform as things stand today.

Another way to frame this would be: "is this facility I want to expose only doing pure synchronous computation on the CPU, that has an execution duration that is going to depend solely on its arguments".

@bakkot
Copy link
Contributor

bakkot commented Oct 7, 2024

All computation involves hardware and is not usually bounded-runtime, so I would still regard Web Codecs as purely computational under the normal meaning of those words. But "pure synchronous computation on the CPU, that has an execution duration that is going to depend solely on its arguments" sounds like a reasonable elaboration of @ptomato's proposed rule to me.

Though that also suggests that perhaps ShadowRealm should be less restrictive than AudioWorkletGlobalScope, because the realtime requirements of audio worklets don't cleanly apply to ShadowRealm - there's no reason, for example, that CompressionStream or WebCrypto couldn't be used in a ShadowRealm, even though those APIs are mostly async. (Exposed=* and Exposed=Nonrealtime, maybe?)

@padenot
Copy link

padenot commented Oct 14, 2024

The normal meaning of those words on the web platform is as such: "hardware" means "hardware accelerated". The opposite on the web platform and in general is "in software", meaning, running on the CPU. This is how the words are used for canvas, gl, web codecs, and more. Here we're interested in workloads that are real-time safe-ish, and so we are interested in somewhat of a bounded run time.

As things stand, Web Codecs doesn't tick the boxes for exposure everywhere, simply because it isn't sync and generally allocates (but some folks now would like a real-time safe API, see w3c/webcodecs#19 w3c/media-production-workshop#45 w3c/webrtc-encoded-transform#226).

If an API isn't synchronous, it follows that it can't have a bounded run time (because the control will go back to the event loop or at the very least a microtask checkpoint, and that is unbounded). If an API allocates, it's the same deal: needing to allocate memory can be unbounded (and generally is).

@bakkot
Copy link
Contributor

bakkot commented Oct 14, 2024

Right. My point is that "in hardware" vs "in software" was not originally part of the proposed rule, which was just about whether an API was purely computational, which I understand to mean something like "this API does not require any capabilities other than computation and memory to implement" (by contrast to, say, fetch, which requires network access). As you point out, this rule is not sufficient to establish that something is suitable for use in audio worklets, so something stricter is needed for Exposed=*.

If an API allocates, it's the same deal: needing to allocate memory can be unbounded (and generally is).

Hm. TextEncoder is sync, but it allocates memory (in the form of a Uint8Array) to hold the encoded text. However, the buffer (and time taken) is proportional to the size of the input. Does that mean it's not suitable for Exposed=*?

@padenot
Copy link

padenot commented Oct 14, 2024

TextEncoder has encodeInto, which is exposable everywhere, since you can use that without making an allocation. The other method allocates.

@bakkot
Copy link
Contributor

bakkot commented Oct 14, 2024

OK, cool. If TextEncoder's encode isn't suitable for Exposed=*, that really suggests to me that we need something other than Exposed=*, because I think it would be really unfortunate if you can't use TexEncoder's encode in a ShadowRealm created in a non-worklet context.

Thoughts on something like Exposed=Nonrealtime (name tbd)? And then say that a ShadowRealm gets access to any Exposed=Nonrealtime or Exposed=* API which exists in its creating context, so that a ShadowRealm in a worklet wouldn't get access to APIs which the worklet itself did not have.

@bakkot
Copy link
Contributor

bakkot commented Oct 14, 2024

Alternatively, @annevk in this comment suggests maybe we shouldn't be worrying about trying to keep things out of audio worklets? But I don't know if that's currently the consensus opinion.

@annevk
Copy link
Member

annevk commented Oct 15, 2024

I guess it shows that tc39/ecma262#1120 is still relevant and we haven't actually figured that out in a way that makes sense to everyone and doesn't create confusion. I don't think audio worklets are actively considered today for new language features (e.g., the new dependencies on the event loop) and they are also not actively considered for web platform features. But the latter is less problematic as there is a way to not expose those in audio worklets.

I'm also not sure if the people developing audio worklets have the ability to maintain a strict boundary as it's easy for new things to sneak in. They would have to be ever vigilant across implementers.

@ptomato
Copy link
Author

ptomato commented Nov 21, 2024

I'd like to move this forward. Based on the discussion above and on other things that have come up while looking at rationales for exposing or not exposing APIs, here are some things that might be able to be added to the design principle:

  • things that rely on user interaction should not be exposed everywhere (e.g., Notification)
  • things that rely on being able to share objects between realms should not be exposed everywhere (e.g, MessageChannel)
  • things that perform non-CPU or unbounded computation should not be exposed (WebGPU)

Does anyone feel strongly about adding any of these to the PR?

@ptomato
Copy link
Author

ptomato commented Nov 21, 2024

As for the AudioWorklet question, I don't really have a strong opinion. That said, my goal here is to make the Exposed=* environment, whether that is a ShadowRealm or something else, as uniform as possible, so I don't like the Exposed=Nonrealtime idea; if that's a concern, I'd rather that the web platform make HostInitializeShadowRealm throw if the incubating realm is an AudioWorklet, so you just wouldn't be able to create a ShadowRealm object.

I get that there are conflicting positions ("AudioWorklets must not expose anything that isn't realtime safe" vs. "Well just don't do that then in AudioWorklets") and I think both of them are reasonable. Maybe this is something for the TAG to settle?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants