Can this API be provided as a standard module rather than globals?

[erights]

At tc39/proposal-weakrefs#2
@allenwb suggests that a particular new API be proposed as rooted in (and exported from) a module rather than new globals.

For a tremendous number of the new apis that have been on the table recently, I would desperately love to see them provided via built in standard modules rather than new globals. Many of us would. A particularly vivid example is the SIMD API. Today, proposed as rooted in globals, it adds a tremendous number of new objects to the population of primordials. Since it raises none of the security concerns of weakrefs, let us explore, first, how SIMD may be provided via module rather than new globals.

[ljharb]

Until the loader spec is resolved, I'm not sure how anything could be provided as a module with a string binding identifier, since we'd run the risk of colliding with something in userland. Could the binding identifier for builtin modules perhaps be something other than a string?

[erights]

@dherman , please look at @ljharb 's question immediately above. Thanks.

[ljharb]

Similarly, I think it would be very important to provide a non-syntactic-import method for getting ahold of builtin modules, as well as a method of overriding them (and freezing them of course, for SES), so that polyfills/shims can correct future bugs in implementations.

[allenwb]

I've been hearing this "we have to wait for a loader spec" for too many years and I just don't believe it any more.

TC39 defines the import statements and TC39 if can define special semantics for specific ModuleSpecifier string formats if it chooses to do so. We know the URI/IURL syntax. We know the file system path syntax of all widely used operating systems. What more do we need to know?

As a very off the cuff strawman: ModuleSpecifier strings whose first character is "" are reserved for definition by TC39. Built-in modules specified by TC39 will be identified by a ModuleSpecifier of the form "". Where is a sequence of characters conforming to the IdentifierName production of the lexical grammar. "**" at the beginning of a ModuleSpecifier is essentially a quoted "". It means that this is not a ModuleSpecifier with any TC39 defined meaning and that the initial "" should be ignored when a loader processes the ModuleSpecifier.

Not having a way to identify built-in modules is driving us to make poor long term design decisions. When should just stop doing that.

[ljharb]

That would be just fine, and totally resolve my concerns about name collisions (ie, it would be problematic to forever remove import foo from 'SIMD' from user space, but a special builtin namespace/format would be fine). That still leaves providing a non-syntactic way to retrieve the bindings for a module, a way to provide/override a missing/broken builtin module, and a way to freeze a builtin module.

I think it would be a poor long term design decision to omit these pieces, so it's not just about the loader spec.

[allenwb]

It seems that the issue of a non-syntactic way to access module bindings or a way to over-ride a built-in module should just be part of the standard (TC39 defined) module loader API. WRT to freezing, bindings export by a module already are immutable from outside the module. If you want to freeze an object exposed by such a binding you can just do it.

I suspect, any open design issues related such functionality would move along much after if we first made the commitment to start defining new APIs as built-in modules.

[johnmccutchan]

I don't see how this issue is directly related to SIMD? Please discuss modules / loader specs in their appropriate repositories.

[littledan]

@johnmccutchan Let's not try to resolve this issue by closing it, but by figuring out how we'll come to an agreement on it. This is a concern raised about SIMD.js (if the argument is specifically that this addition should go in a module), so this is the appropriate place.

First, to clarify, I would love it if we could put this in a module! But modules don't seem to be done yet, so I think it'd be best to let modules and this feature be develoepd separately.

We have discussed this particular issue in TC39 in the past multiple times, and IIRC multiple times came to consensus that SIMD, like Intl, would like to use modules but didn't have a particular way to do so. Now we've reached Stage 3, which I've understood to mean that all of the API shape issues from TC39 have been resolved, and what remains is to incorporate the results of feedback from two implementations and then integrate it to the main spec text. Reopening this API shape question seems to be stepping back to pre-stage-3 questions, and going back on what we had consensus on. I really like that we as a committee have a model where multiple things can proceed in parallel and make progress on various issues independently without one blocking on another, as embodied by the stage process.

While this strawman * syntax sounds unambiguous to me, I can definitely see the aesthetic argument that users would be a little confused by import * as simd from '*simd' -- the second * has a completely different meaning from the first one, where the first indicates wildcard and the second indicates builtin. We continue to not have a final Loader spec, and large changes are being proposed in the last few weeks to a loader design, though it is true that no one is proposing using a leading * for anything.

[erights]

Some proposal needs to be the first to root new api in a module rather than globals. Ironically, one of the reasons SIMD seems a good candidate is because it is otherwise so far along and well settled. Because it is also so self-contained, whatever issues are raised by rooting it in a module should be orthogonal to the rest of SIMD. Finally, what stands out about SIMD is the huge inflation of the set of initial primordials it adds, if it remains rooted in globals.

At this point, I think we're all confident that SIMD will happen, essentially in its current state modulo this rooting issue. Adding this burden to SIMD does slow it down, but it does not endanger it. Because of the benefits to all other proposals on the table, working out the module issue here will speed up everything else more than it will slow SIMD down.

[johnmccutchan]

If a module system was completed and available, SIMD would already be a module, but the module system isn't complete and the reasons why have been discussed at length. We are not willing to tie SIMD to the module system.

[allenwb]

@johnmccutchan
See my initial post above. ES2015 defines ES modules quite adequately for implementing built-ins such as the SIMD functions as a built-in module. The only critical missing piece is the naming convention.

But I agree,that this certainly isn't exclusively a SIMD issue. What we should probably do is open a Ecma262 issue saying that we need to define the conventions for built-in modules. Then this SIMD issue can be a dependent issue of the more general one.

ES2016 is locked down, so SIMD and everything else pending will at the earliest be on the 2017 train. I think it is quite reasonable to set the goal that starting with 2017 we use built-in modules instead of new globals.

[sunfishcode]

TC-39 granted SIMD.js stage 3 approval, even with discussion of this very issue. Is the proposal here that the committee revoke its stage 3 approval until this issue can be resolved in a different manner now, or that the meaning of stage 3 does not mean that the committee is done asking for changes?

[allenwb]

@littledan
consider the possibility of:

   import "*SIMD";

where part of the specification of the "*SIMD" module is that as part of its initialization it creates a property of the global object named "SIMD".

or

   import SIMD from "*SIMD";

assuming that the "*SIMD" module is specified to export a default object.

or

   import {Float32x4} from "*SIMD";

Finally note that these approaches aren't mutually exclusive. A built-in module like SIMD could be specified such that all of them will work.

[allenwb]

@sunfishcode
TC39 certainly can ask for or require changes, up until a proposal is declared to have reached stage 4. However, by the time a proposal reaches stage 3, the expectation should be that the volume or extent of such changes will be relatively low.

Note that nothing in this thread has said anything about the core functionality or even the core API of the SIMD proposal. It's only about how that API and functionality is made visible to an ECMAScript programmer. That seems like exactly the sort of think that might evolve at the stage 3 level of the process.

[sunfishcode]

@allenwb TC39 discussed this issue several times at various stages of SIMD.js's progress, and each time the committee agreed that, since the SIMD.js work would complete before the module work did, standardized modules shouldn't block shipping SIMD.js. The consensus was that, like other APIs, organizing the standard libraries into modules could be done orthogonally. I understand you'd like to see more progress on standard modules, but multiple browser vendors are anxious to ship and have followed TC39's process, and we're not interested in holding SIMD.js back if it's just for the purpose of encouraging progress on the library reorganization effort.

[wycats]

I've been hearing this "we have to wait for a loader spec" for too many years and I just don't believe it any more.

We separated the loader spec from ECMA 262 in order to allow it to progress separately from ES6 with more direct input from the web platform about a year ago.

Since then, the web platform has made tremendous progress on the loader spec:

• The loader repository (whatwg/loader), which defines the specification for the loader, has had 77 closed issues and 24 closed pull requests, resulting in 121 commits.
• There is an active polyfill that tracks those semantics.
• <script type="module"> was specified and merged (after 168 comments).
• Multiple high-fidelity userspace implementations of the core module semantics have been implemented, and modules are now used by all major web frameworks in their latest versions.
• There is an active (drawing to a close) discussion in the node project about compatibly bringing modules to node.js.

Both the discussion about <script type="module"> and the node discussion have drawn out important compatibility constraints, that have in turn helped drive the loader specification itself.

There was a time (when the loader was being specified as part of ES6, and for a time after it was split), where progress was slow, but that is no longer the case. Lots of people are working extremely hard day in and day out to bring that process to a close, shipping incrementally along the way.

We're not done yet, but the work is not stagnating.

[dherman]

Remember, we've discussed this multiple times, and it's unfair to the champions to claim this late in the game that a thing we told them wasn't a blocker is a blocker now. That said -- @allenwb, I completely agree that we don't need to block standard modules on a complete loader API (although as @ljharb says we have to coordinate with the standard loaders and ecosystems). In fact, I'd go even farther and say I agree with you that we need to up the priority level of that work since we want to ensure that the conventions we settle on for standard modules are compatible with existing and nascent ecosystems coming out of the loader work. Nevertheless, SIMD is ready to ship, and for better or worse module organization requires more discussion.

[bterlson]

I understand having second thoughts about adding SIMD as a global as we have recently increased the number of built-ins on the radar and also made significant progress on both module specs and implementations. There is light at the end of the built-in module tunnel, and seeing that light, it's tempting to prove the way with SIMD (for the reasons @erights mentions). If it's possible to spec built-in modules first without delaying SIMD significantly, we should still do it IMO, but I am not sure it's possible.

[johnmccutchan]

Multiple browser vendors are getting close to wanting to ship SIMD. We cannot request that they delay that work because TC-39 has chosen to add on a new requirement to wait for the unfinished modules spec. I would recommend that TC-39 finalize built-in module support and once that is implemented and shipping in multiple browsers, approach the SIMD working group again and we will see where we are.

[caridy]

@allenwb

I've been hearing this "we have to wait for a loader spec" for too many years and I just don't believe it any more.

The problem of using modules without a loader, and specifically, allowing built-in modules without a loader is that you completely loose the ability to, in user-land, polyfill SIMD, or any other built-in module API, because you have no way to control the registry, nor the instantiation mechanism. And this is a fundamental issue, departing from our current position which is to allow any global API to be polyfilled.

[johnmccutchan]

Thanks @bterlson. When tc39/ecma262#395 is fixed and shipping in multiple browsers, please re-open this issue and we can discuss making SIMD into a module.

[erights]

I concur that this is a fine reason for closing the SIMD-specific manifestation of this issue. Let's get tc39/ecma262#395 done in time to apply to everything on the ES2017 train, including this one. But we can coordinate this there, not here.

[allenwb]

@johnmccutchan
just keep in mind that we won't have browser implementations until we actual specify built-in standard modules. Somebody is going to have to be the first one. It possible that SIMD will be asked to go first.