Minimum requirements for releasing new experimental features

[GeoffreyBooth]

@mcollina and I were discussing in #45018 a general question that I haven’t seen addressed before: what minimum requirements do we have or should we have before a new feature is made publicly available, even as experimental?

The particulars of #45018 are that:

• It’s designed to be used in CommonJS, and may or may not work equivalently in ESM. (There are no docs or tests for ESM usage currently, but @mcollina has used it in ESM in his projects so it’s not broken there; probably somewhere in between.)
• When released as experimental and working as expected, it has the potential to be extremely popular, as it would solve a common problem users are having.

My concern is that I don’t want this feature being released to the public, even experimentally, and becoming wildly popular before it has full ESM support. If that were to happen, we’ll have another case where we hear more user complaints of “I want to use ESM, but I can’t until this feature supports ESM.” This has been a source of great frustration for years, and has held back ESM adoption. I really don’t want more things trapping people in CommonJS, and I don’t think it’s good for the project generally if we provide a better user experience or more functionality in CommonJS than in ESM. It’s also not fair to the people working on modules to have to try to complete this work to make all features compatible with ESM, when arguably it should be the responsibility of the original authors/teams.

On the other hand, I see the point about merging in something small and iterating; that if the initial requirements for landing anything are too high, new features won’t land and grow. On the thread we discussed approaches to split the difference, such as landing on main only and attaching dont-land flags to prevent the incomplete feature from being released; or gating the feature behind a build flag, or following the pattern of https://github.com/nodejs/ecmascript-modules where a big new feature is developed over many PRs in a forked node repo and merged in when it’s ready. Other ideas for solutions to this problem would be welcome.

So my question for the group is basically, what minimum requirements should we have for landing new experimental features and for releasing new experimental features? Since things can be landed but held back from public availability. Is near-parity in ESM a fair requirement? Are there others? cc @nodejs/tsc @nodejs/loaders @nodejs/modules

[Qard]

I don't like the idea of full ESM being a hard gate for new features to land. Given it itself is still considered experimental, that's a moving target against another moving target. Perhaps we could clarify our stability ratings by marking anything stable in CJS and not in ESM specifically in docs, until they are fixed? That would make stability of those things not necessarily a blocker for ESM anymore while also allowing new development without necessarily needing to support ESM right away. We could also consider having ESM support as entry criteria for being marked as stable.

This would allow us to keep up development velocity while still clearly communicating which parts of the system can be trusted. It also better models how experimental really works given that it's generally incompatibility with specific other subsystems that hold things back. For example, right now async_hooks and ESM are relatively in conflict due to a lack of development effort to solve the particular cases where they overlap.

[aduh95]

ESM is longer experimental since Node.js v12.22.0. I don't think splitting support would help at all, the question here is about landing something that's experimental.

[Qard]

ESM the syntax is stable. ESM Loaders are not.

[GeoffreyBooth]

Loaders are a customization feature. That’s not part of ESM proper. And loaders also apply to CommonJS (even more experimentally).

This is also circular logic. If we don’t support ESM because ESM isn’t ready, and ESM can’t be ready until all features support it, then ESM will never be ready.

[Qard]

I'm not saying ESM in general isn't ready. I'm saying specific components related to ESM are not ready. We often describe ESM with broad strokes though, which we shouldn't do.

I think it's fine for people to use ESM as it is now, but I also think we need to more clearly communicate which features related to it are not quite done yet so users can properly understand the trade-offs to using it. For example loaders being experimental still means they might not get support from APMs or certain other tools that need to hook into the module lifecycle. It's fine that those things don't work yet, and I hope that someone with sufficient motivation will solve those issues eventually, as that's how we keep OSS moving forward. But I don't think we should be blocking on any of these things being usable until they support everything perfectly. Just that we should clearly communicate where those sharp edges are.

[aduh95]

I don't think full ESM support is a reasonable requirement for landing something experimental, in fact I don't think there should be any minimal requirement as long as the end goal is desirable, and attainable (which is left to best judgements of the reviewers).

[GeoffreyBooth]

I don't think full ESM support is a reasonable requirement for landing something experimental, in fact I don't think there should be any minimal requirement as long as the end goal is desirable, and attainable (which is left to best judgements of the reviewers).

Well we always have at least some requirements, like that all features have tests and docs.

[aduh95]

Not really, we do have requirements for landing a PR such as there must be at least one approving review and no one blocking it, a green CI, a commit message that adheres to the guidelines. Now as a collaborator you can request to have docs and/or tests added to the PR, and most reviewers choose to make sure docs are updated in the PR, but that's not an official requirement, and I don't think it needs to be.

[GeoffreyBooth]

Those are the formal requirements, but my point is that we do have informal, unwritten requirements. And the point of this discussion is that we should probably write some of those down, so that there’s clarity on what’s reasonable or unreasonable to block on.

[Trott]

There's a larger issue here that probably is healthy to spell out (especially if it turns out I'm wrong about seeing it this way):

There are people (such as @GeoffreyBooth) who view ESM as something that should be embraced by users over CJS for a variety of reasons. The impression that I get is that Geoffrey and others expect (and want) ESM to become the standard and CJS to become the legacy thing that people mostly don't use anymore and you find only in old code.

And there are people (such as @mcollina) who see CJS as the better option for many use cases and think that CJS should not be constrained by ESM limitations. The implication that I get is that to Matteo and others, CJS is unlikely to be completely replaced by ESM any time soon and that a dominant ESM module ecosystem (with very little in the way of new modules with CJS support being released) might even be undesirable.

It's possible I've misstated the views, but broadly, it seems to me that these two camps have very different and contradictory goals for the medium term at least and it might be a good thing to figure out what the TSC and/or the project as a whole can agree on.

[Qard]

I also count myself among those not convinced by ESM. The fact it is an actual spec is good, but beyond that we don't really gain much from it. It currently has a lot of usability issues compared to CJS, so I generally stick to CJS still. I'm a core contributor and understand those usability issues well yet I don't use it--I can't imagine how frustrating they must be for the uninitiated. ESM is just a whole lot more complicated to use than CJS.

I would definitely like to see it reach the level of maturity of CJS. I'm always a fan of aligning on specs. But it's just not there yet.

It's unfortunate that it has taken this long to catch up with CJS, but I don't see that as a compelling reason to make support for it block experimental feature development. Node.js is driven by volunteer time, and that time is an extremely limited resource. Forcing all our contributors to invest a significant chunk of their time into supporting a feature very few actually use will just drive contributors away. Make the feature compelling and people will want to support it.

[GeoffreyBooth]

I would definitely like to see it reach the level of maturity of CJS. . . . Node.js is driven by volunteer time, and that time is an extremely limited resource. Forcing all our contributors to invest a significant chunk of their time into supporting a feature very few actually use will just drive contributors away. Make the feature compelling and people will want to support it.

There’s an inherent conflict here. Of course we want to encourage contributions, and naturally those will come in the areas of greatest interest to our contributors. But allowing anyone to land anything they want is ultimately allowing the contributors to drive the project in many different and possibly discordant directions, which isn’t in the best interest of the project as a whole. Aside from paying for work, the only way we can influence the direction of contributions is by withholding something contributors want, like releasing their feature, in exchange for doing a little extra (or a lot extra) work. And yes, the latter approach has the risk of driving away contributors or reducing contributions.

So ultimately it’s a question of finding that balance between valuing our own technical goals enough to encourage their enactment, but not so much that we get no progress at all. I don’t think the answer is on one end of the spectrum or the other. I think we need to acknowledge that ESM parity is important and that some effort needs to be devoted to ESM support, and what that is in particular is hazy and may vary by feature, but at least some effort needs to be made.

The vast majority of features have no intersection with the module systems and the only “extra work” required would be writing the docs examples in both CommonJS and ESM. I think that’s a fair minimum. For the features that do behave differently between CommonJS and ESM, I think the deciding factor should be whether the feature’s potential popularity is such that if the feature lacks ESM support, that shortcoming would discourage ESM adoption. If it does, then I think we’re at the other end of the spectrum where ESM support from the get-go needs to be required.

[Qard]

But why is ESM parity important? What is the selling point that compels us to value ESM at the same level as the much more mature CJS? Because I don't see it, nor do many others. Which is not to say it has no value, but because this is a volunteer-led organization there needs to be some sort of compelling case for people to want to work on something.

I think forcing contributors to support it without making a very clear and compelling case why would be a mistake and would drive many contributors away. We already have retention problems just getting people to go through all the steps for properly formatted commit messages, linting, etc. and those are comparatively very small asks.

[mcollina]

I don't have much more to add to this vs what others signalled, minus a quick errata.

And there are people (such as @mcollina) who see CJS as the better option for many use cases and think that CJS should not be constrained by ESM limitations. The implication that I get is that to Matteo and others, CJS is unlikely to be completely replaced by ESM any time soon and that a dominant ESM module ecosystem (with very little in the way of new modules with CJS support being released) might even be undesirable.

Errata: I welcome an ESM dominant registry. Unless there is a massive acceleration, I think it's impractical to think it will happen in a short or medium timeframe. I think we should put our users first.

[ronag]

would drive many contributors away

I second this. But I would replace many with some.

[jasnell]

I think for every experimental new thing added we have three lifecycle phases to be concerned with: (1) Landing in main, (2) Released, (3) Supported/Graduated.

For every experimental thing, it's reasonable to have objective criteria that must be met to progress through any of these transition points. However, I think those criteria should be taken on a case by case basis. Not every experimental feature will be relevant to both cjs and ESM. For instance, would we consider blocking an experimental addition to import.meta simply because it isn't relevant to cjs? That wouldn't make sense.

When a new experimental feature is proposed, it's perfectly fine to say "Don't put this in an actual release until ..." Which gives us a natural deadline every six months. Either the experimental thing has progressed to the point that we are ok with it rolling automatically into the next major or we decide whether to pull it out because it hasn't progressed. Either way we have to make a decision. Which is good.

[legendecas]

I think for every experimental new thing added we have three lifecycle phases to be concerned with: (1) Landing in main, (2) Released, (3) Supported/Graduated.

I find there can be more (optional) phases between (1) and (2) for different new features:

• Build time flags to enable the feature. This would discourage greater adoption as people need to build the Node.js binary themselves to experiment with it.
• Experimental flags to enable the feature. People can not access the feature without explicit command-line opt-in. However, they don't need to build Node.js themselves and can start testing with it easily.
  • This is how new ECMAScript proposals go through V8's implementation process.
  • We've got --experimental-vm-modules too.
• Enabled-by-default experimental flags, with scary experimental warnings.
• Experimental mark in the documentation. This is how most features are released.

Discouraging new features to be landed on the main branch is a frustration for people to bring new features to Node.js too. The above phases indicate different confidence in the maturity of the feature, and clarifying the criteria for a feature to be landed in the main branch and to be promoted towards the "final release" may be a good path for the new-feature author and existing feature-maintainer -- the new-feature can advance without initial full-support with existing features and need the meet more criteria to be promoted to the next stage without burdening other maintainers.

[RaisinTen]

I would say that as long as the feature makes sense in node core and the PR also has docs and tests prepared, it's good to land and release, regardless of ESM support. ESM support can be added later on by anyone who feels strongly about it incrementally.

[RaisinTen]

it abdicates the TSC’s responsibility for setting the direction of the project. We want ESM to be a first-class supported platform.

Is that something the TSC has consensus on?

That can’t happen if the majority of contributors aren’t using it, and then won’t use it because it doesn’t support the latest features—the catch-22 again.

I don't think that's totally true. I never even used Deno or Rust before I sent my first contribution to Deno - denoland/rusty_v8#1049. I thought of working on it because it sounded fun to me.

The few contributors working on modules issues have their hands full with loaders; they can’t also tackle issues in vm, worker_threads and elsewhere (and none of us know C++, which doesn’t help).

It doesn't have to be someone from the loaders team though. I also think that if the loaders-related context is shared with someone who is confident with working on a subsystem other than loaders, it would be possible for them to implement what you're asking for without having the loaders team to directly implement that thing.

I agree with this, pretty much any time any concern is deferred with "oh you can just do it in another PR!" it's incredibly hostile and harmful to the health of the project. However, that's been a consistent refrain with node, in particular as it relates to ESM.

I'd love to see the project make an overarching decision to stop using this "kick the can" approach.

I don't think that's hostile or harmful to the health of the project. The project does make overarching decisions like for example the one we did for the node: prefix. We can definitely try to ask the TSC if they have consensus on blocking PRs that add features without ESM support but I feel that that should be discussed in a separate dedicated issue because this one is is about experimental features in general.

[GeoffreyBooth]

it abdicates the TSC’s responsibility for setting the direction of the project. We want ESM to be a first-class supported platform.

Is that something the TSC has consensus on?

I don’t know if the question has ever been posed so directly to the TSC, but it’s listed prominently in the project’s Technical Priorities document: https://github.com/nodejs/node/blob/main/doc/contributing/technical-priorities.md#esm and it’s one of the TSC’s initiatives: https://github.com/nodejs/node/blob/HEAD/doc/contributing/strategic-initiatives.md#completed-initiatives (as “modules”).

[RaisinTen]

I think being on the technical priorities document means that the project thinks that it would be more successful if folks worked on the tasks listed there and the TSC initiatives list is for a list of substantial tasks that are currently being worked on and it is the responsibility of the TSC to make sure that those are active and has the support that it needs.

None of these suggest that other PRs are not supposed to land if those don't implement support for any of the tasks listed here if applicable. I think it means that we are likely to reject a PR that detrimentally affects any of the mentioned tasks.

[GeoffreyBooth]

I think it means that we are likely to reject a PR that detrimentally affects any of the mentioned tasks.

I think this is another way of saying what I was referring to earlier as avoiding improvements that widen the gap between CommonJS and ESM, slowing ESM adoption or encouraging migration back to CommonJS. The ideal state is that projects could choose either module system based only on the particulars of that system, not on unrelated features like async_hooks or vm working significantly better in one system or the other. A PR that widens the gap would be the type of PR that I think should get the highest scrutiny; but fortunately it should happen very rarely, and even #45018 is not likely to be such a case as I understand it so far.

[RaisinTen]

A PR that widens the gap

I guess the PR you're referring to here should be the one that's moving the feature from experimental to stable because that's really what's supposed to promote the usage of that feature in the ecosystem. It shouldn't be the PR that's just adding maybe a first iteration of the feature though.

avoiding improvements that widen the gap between CommonJS and ESM

I don't think anyone is opposed to adding such improvements before marking the feature as stable.

[gireeshpunathil]

IMO, these can form a minimum requirements for experimental:

• usual caveats for adding anything in core
• bug free
• documented
• ability to opt-out with a flag

It is perfectly alright to ask for specific enhancements, and it should be perfectly alright to propose it to be enhanced later; those may be subjected to the willingness of the contributor. This approach aligns with the philosophy of experimental, as we expect usage feedback to cause amendments / enhancements to the feature anyways, before graduating to stable.

[Flarna]

• bug free

In that case node itself wouldn't be ready for experimental as I know there is at least one bug left somewhere :o).
No know bugs would work or maybe test must exist and don't fail.

[gireeshpunathil]

agree, I meant no known bugs.

[mcollina]

Part of this problems arises from the async_hooks experimental status. We have been abusing the experimental status significantly for this and it has lost its meaning.

async_hooks:

1. is old
2. work as expected
3. is used in production by almost everybody
4. is used inside testing frameworks
5. ...

In all intent and purposes async_hooks a stable and proven API. We cannot ship a Node runtime without it and avoide a massive uproar from our production users.

Unfortunately, async_hooks is a complex API that we do not like.
As things are, we have been kicking the can down the road, hoping for somebody to have time to improve the async_hooks API.
This is unlikely to happen, so we should just call async_hooks stable due to major popular usage.

[GeoffreyBooth]

This is the complete opposite point of view from the top.

Well we all agree it’s an API with major issues; the disagreement is around what to do about it. There are other options besides “do nothing” and the extremes of “make it stable” or “make it internal.” We could mark it as deprecated, even though that’s not something we really do for experimental features. This would be a gentler nudge to vendors to migrate off of it. We would follow through on the deprecation removal (in this case, making it internal) once enough people had migrated. We could also have it start emitting loud experimental warnings, or deprecation warnings, or both.

Edit: I wrote the above before @Qard’s replies appeared for me. I agree with the sentiment, that some kind of marker like legacy or deprecated to prod vendors to prioritize migration (and PRs to enable migration) is something we should look into. Also if we can backport AsyncLocalStorage becoming stable to 14.x, that should also help.

[jasnell]

This particular thread is straying pretty far from the topic of discussion. Perhaps it would make sense to create a separate top level discussion that focuses on async_hooks?

[Qard]

Well, I think the point that "experimental" is a very murky definition due to how we've handled async_hooks is a relevant point to the discussion. I think part of defining requirements for how experimental features interact with each other necessitates better clarification of what "experimental" actually means. I do agree that specifics of what we should do about async_hooks outside of its current experimental status is out-of-scope though.

[Flarna]

Besides the legacy term we could use deprecated like in domains. But as long as there is no replacement we should not issue any sort of warning message or similar. So in the end it's just about a single word in the docs.

[Qard]

Well, my plan was to get the core users (context management, APM, etc) off of it and then turn on a deprecation warning at that point to encourage any remaining users to talk to us about what they're doing so we can make a better API for their use cases.

[mcollina]

I would hardly constraints anything landing as experimental. The most riskier features could be behind a flag. Others emit a warning, or even nothing. It really depends on the perceived risk.

I would rather abstract the discussion from ESM. In broad terms, for any feature to graduate from experimental should not have major bugs and design flaws that prevent it to work with other features of Node.js. In case of features with broad ecosystem reach, some real-world usage is recommended too.

[jasnell]

In the case of SynchronousWorker, just as an example, there was no "kicking the can down the road" for ESM support. I wanted to do it incrementally over multiple PRs to make it easier to review and easier to iterate on. I wanted the first PR to be as close as possible to the thing being vendored in so that it would be easier to review and reason about. I was already working on the necessary improvements for ESM support as a separate PR, and was also working on a couple other fixes. The intent was not to ignore esm at all, but to make it easier to review the set of changes and the evolution of the impl over multiple steps.

Any discussion of whether esm is preferred, or mature enough, or whether we like it or not, etc is immaterial in my opinion. It's a supported feature of Node.js and must be taken into consideration. I just don't think it should generally be a gating factor for landing and iterating on new experimental stuff.

[GeoffreyBooth]

I was already working on the necessary improvements for ESM support as a separate PR, and was also working on a couple other fixes.

Yes, and thank you! Sorry if this thread came across as even implicit criticism. I felt like I needed to explain the context for where this question came from, but the point of this discussion was to try to arrive at some guidelines that could apply generally, beyond that one PR.

[devsnek]

I am uncomfortable with the way this has been brought up.

My concern is that I don’t want this feature being released to the public, even experimentally, and becoming wildly popular before ...

If you are concerned that an experimental feature will gain popularity, perhaps it would be more constructive to open a discussion about better ways of gating experimental features, rather than blocking individual prs. Especially since the authors of the PR in question were willing to iterate on the functionality you were requesting, which is a fantastic use case for experimental gates, and one of the reasons we have them in the first place.

[GeoffreyBooth]

perhaps it would be more constructive to open a discussion about better ways of gating experimental features

That was part of the top post:

approaches to split the difference, such as landing on main only and attaching dont-land flags to prevent the incomplete feature from being released; or gating the feature behind a build flag, or following the pattern of nodejs/ecmascript-modules where a big new feature is developed over many PRs in a forked node repo and merged in when it’s ready. Other ideas for solutions to this problem would be welcome.

The specific concerns around the PR that spawned this discussion have been resolved. I opened this discussion because of talking through those issues with @mcollina. This discussion is meant to see if we have consensus around these topics more generally.