a more deliberate release process

[othiym23]

<whining>I'm going to be straight-up emotional for just a second here: this is the second Sunday in the last month where I've woken up to a brewing crisis involving io.js and npm (#1850, #1591). It is incredibly annoying, as well as demotivating, to have to clean up these situations on what should be one of my days off. I try very hard to be a conscientious project lead / OSS maintainer / io.js collaborator, but for human psychological and physiological reasons I need some downtime, and the weekends are that time for me. The suggestions below are offered in the spirit of making it so that I don't have to choose between being responsible to the io.js and npm user communities, and being responsible to myself. Thank you for reading this with that in mind.</whining>

Also, for all of the following, I frame things in terms of npm, but this is just as true of V8 and changes made to Node's own APIs.

About a year ago this time, npm introduced a new release process that included as one of its key features the notion of publishing every new release with a dist-tag of next, and only promoting it to latest after at least a few days (generally a week) so that interested members of the community could use / test it and identify any latent issues with the release. This was done as a reaction to what used to be an all-too-common pattern with Node.js releases, where a new release of Node.js would be followed almost immediately by another due to something messed up inside the version of npm bundled with Node, which was embarrassing and wasteful. This basically hasn't happened since npm's new process was put into place, which I think everybody agrees is a big improvement.
#1850 shows that this only deals with one side of the problem, though. If there's something in a new Node release that makes it incompatible with npm, we can end up putting out versions of Node that render npm unusable, which cripples the release. There's a lot of room for improvement in how npm is tested and integrated with io.js's CI, but even with that situation substantially improved (which a number of us are working on), there's no substitute for real use of the product to give it some real-world smoke-testing.

io.js has a much quicker release cadence than Node.js has had for a while, and this is one of the primary drivers of io.js's quick progress (and is great). New versions come out frequently, and when things break, a new release that fixes them is generally pretty quick to appear. However, because things like Travis pin to the latest version, and many many projects take advantage of this, official io.js releases can magnify issues into an avalanche of issue-tracker and support traffic.

Therefore, I have a few suggestions I'd like to make about how io.js is released:

1. when upgrading dependencies within Node, they should be included in at least 1 (preferably more) nightly release before an official release is cut (I don't know how widely-used the nightlies are, but this is better than nothing)
2. to get potential releases in front of more people, it would be great to have some notion of release candidates for "larger" releases (semver-major definitely, semver-minor probably, semver-patch it would be nice but also probably a gratuitous amount of work [ETA: finished incomplete thought])
3. io.js releases should only happen during the week. I completely understand that for many io.js contributors, the weekend is the only time they have available for OSS / io.js-specific work, which is why I haven't raised this point before. At the same time, many of us are already spending pretty full weeks working on things related to Node – dealing with this is our job, and as such we just need some time off.

Note that these suggestions are orthogonal to adding more / better tests of dependencies, better integration of the tests we do have into CI, acceptance tests, and other prophylaxes intended to improve release quality. All of those things would be great, as would some kind of defined QA process for the project. I still think, however, that Node and npm are both complex and widely-used enough that there's no real substitute for actual, hands-on use to help ensure quality.

[vitaly-t]

This is the price for being the front-runner, you manage to break things on the way.

NodeJS, in contrast, didn't rush things like this, faced no such predicament, and lagged behind happily for years to come.

I believe the truth is in the middle. Therefore, looking forward to the merge of the two platforms.

[meandmycode]

I think quite a lot of issues brought up in io.js focus far too inside the box, the whole ongoing thing with V8 4.3 onward has focused heavily on how will io.js / nan solve this, I think there needs to be a bigger effort to external thought, and the ecosystem as a whole.

Quite often a lot of these small changes to clean the code base are based on quick and dirty searches on google, github and npm, and the assumption this is good enough to put in a deprecation notice (which, are mostly useless in my experience, as the people who care enough to do something are generally those who are eager to maintain their software anyway).

It would be great if some external tooling were to start to happen, for example; the chrome guys have metrics that they use to inform changes, it would be great if npmjs could do some static analysis on packages to establish some real connectivity of code use and its flow between packages. This would be really interesting outside of this issue as well, and could be used to establish quality of packages over time, and perhaps even some day some automation around automatically updating dependencies.

I'm not saying the above is the answer, or even a good one- but its trying to use aspects of the ecosystem that aren't just code. Aside to this, there's so much worry about upgrading V8 too often and how code can solve that, but what about outside of that? can npm help? can community outreach help?

More on topic, and I'm really out of my depth here not having any experience bar following along reading issues most days, the node/iojs build + test does seem woefully inadequate for how important this platform is.

[dashed]

First I'd like to say thank you to everyone and anyone who took the time to spend the Sunday to try and resolve this particular issue.

I think #2, as @othiym23 has suggested, may be a good compromise. If at all possible, I'll be willing to add something like iojs-latest-rc to .travis.yml (or some CI platform), and maybe a lot of projects would as well; and this would the cheapest way to test the waters that a new release is stable enough for the rest of the ecosystem.

[mscdex]

@meandmycode Static analysis is already in the works. See @chrisdickinson's estoc module.

[isaacs]

@meandmycode +1, you hit the nail on the head.

The focus of io.js (and for that matter, Node as well, more often than not) has been on polishing and improving io.js.

There are still features of the platform that are not documented, not tested, and perhaps maintainers of io.js may assume are "internal".

However, we need to change our thinking on that.

If it's exposed, it's public API. Period.

Imo, changes to the public API require, at minimum:

1. A very damn good reason. ("It's ugly" or "it should be internal" is not satisfactory. Too late, sorry.)
2. Explicit deprecation in the docs for a major release.
3. Deprecation via util.deprecate for another major release.
4. Assuming that no riots form when we do (3), remove it in the next major release.

We can add new stuff all day long, or provide better APIs in the meantime. But we can't break stuff without breaking users.

Related: #1807 (comment)

Sorry, still haven't gotten around to writing up the issue for a stable release checklist. I think we're seeing from this fiasco that it's not just a matter of keeping up with breaking V8 changes, but also pure-js changes as well.

[ChALkeR]

@isaacs What about _-prefixed properties?

[ChALkeR]

Btw. When I asked in IRC why was .client deprecated in a minor version (after the 2.2.0 got released), I was told that deprecating in a minor is ok, along with actual removing in a major. If that is not so, you should probably form a policy for such things and put it somewhere.

[silverwind]

@ChALkeR I was refering to this part of semver:

Minor version Y (x.Y.z | x > 0) MUST be incremented if new, backwards compatible functionality is introduced to the public API. It MUST be incremented if any public API functionality is marked as deprecated.

The .client deprecation itself was improperly handled which lead to #1850.

I don't expect people to run test-npm for every commit, but it should be a requirement for every release at least, as it's been proven again that npm itself is a good test for the ecosystem.

[isaacs]

@silverwind Running make test in npm will be less of a good test of the ecosystem once we finish making all of the npm tests run network-free. A much better test of the ecosystem is selecting the top 100 modules, and making sure that you can do npm install && npm test on each of those. (npm itself would presumably be in that set.) In the case of 2.2.0, this would have been 0/100 passing, since npm install is broken.

@ChALkeR Everything that is exposed is exposed. We've had a firm "this is internal" policy regarding process.binding(...) stuff, but even then, I think we should verify that breaking changes won't break user code.

In the case of 1eec5f0 specifically, even though the client property was not used internally, because it was exposed, it was a part of the public API, documented or not. If client is to be deprecated in favor of socket, then that ought to be changed in 3.0.0 by adding a note to the docs, then in 4.0.0 by using util.deprecate, and then in 5.0.0 by removing it entirely. Moving it to an underscore-prefixed property is inappropriate in this case.

Of course, mistakes happen, and slip past review. That's why we need an explicit objective release gating process, so that we can get away from the (currently valid!) stigma that io.js is unstable, prone to break in minor releases, and not trustworthy.

[mscdex]

I thought that when the topic of nightlies/RCs was originally discussed months ago, the conclusion was that nobody would likely test/use them (especially with a -nightly-xxxxx or similar version number it would not work with npm)? Also it looks like the last nightly was built on 5/14/15.

[silverwind]

Nightly usage has been pretty minimal. That might change once we converge, but for now, they're not getting the exposure they need.

[ChALkeR]

@isaacs

Everything that is exposed is exposed. We've had a firm "this is internal" policy regarding process.binding(...) stuff, but even then, I think we should verify that breaking changes won't break user code.

I always thought that _-prefixed means that there are no guarantees and that whoever uses such properties does it on their own risk.

Technically, Symbol-based properties are also accessible. Will you treat those as «exposed»? Technically, in c++ private members of a class are also accessible. Are those «exposed»?

The question is how far does the user code go in breaking the conventions and to what extent do want to support that.

[silverwind]

To get more nightly usage, we probably need some kind of auto-update mechanism. That would of course require installing into a user-writeable directory so we can update without needing root.

Another unresolved issue with nightly builds is that their version strings don't satisfy the engine requirement of packages, which results in npm printing a warning for every single dependency installed, but I guess there could be made an exception in npm.

[ljharb]

If it would help, I would be pleased to extend nvm to support installing io.js nightlies, provided we can agree on formats/structure/URLs/etc, and provided it will remain the same with node post-merge - that would allow easy testing on travis-ci, as well as easy testing on devs' local machines.

[othiym23]

I always thought that _-prefixed means that there are no guarantees and that whoever uses such properties does it on their own risk.

• If you're an application developer and have conventions for what counts as public vs private state within your development team, this is a fair assumption.
• If you're developing a tool like npm, where downstream consumers need to be able to rely on the behavior of the tool only changing in line with semver, you can get away with marking parts of the tool as off-limits, although realistically you can't change the behavior (documented or otherwise) without lots of advance warning. We've inadvertently caused pain for npm users more times than I care to remember by changing npm's undocumented behaviors, even when we're doing it to fix serious bugs.
• On the other hand, if you're maintaining an open platform, all bets are off. It's nearly impossible to foresee all the things developers are going to want to know about your system, or how they're going to want to hook into its behavior. Your platform will be more useful to developers if you as maintainers keep that in mind, especially in a language like JavaScript that has relatively weak encapsulation and abstraction mechanisms.

Ideally _prefixed should mean "stay out," but when that's the only means available to get a thing done, people are going to use it. See _headers and many of the _methods on process.

I think a good rule of thumb is that if something can be accessed using typical JavaScript means – either exported or exposed from a system object – people are just inevitably going to end up relying on it. There is a lot of Node core that is hidden away in closures and just can't be accessed from third-party modules (sometimes too much). That's a useful strategy for keeping people from messing with the state of the runtime too much.

To be clear, it's unreasonable to expect that every implementation detail of Node's core objects must be treated as a frozen API as soon as it's shipped, and I think @isaacs's guidelines are pretty good.

Technically, Symbol-based properties are also accessible. Will you treat those as «exposed»? Technically, in c++ private members of a class are also accessible. Are those «exposed»?

If a Symbol is only created and used from within a closure / function scope, it's effectively inaccessible to outside users. A C++ private member is only accessible by violating encapsulation, and is definitely not the kind of thing we're talking about here.

[ChALkeR]

@othiym23

If a Symbol is only created and used from within a closure / function scope, it's effectively inaccessible to outside users.

https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object/getOwnPropertySymbols

@isaacs guidelines actually mean that all changes even to _prefixed
properties (like removing/rewriting) should always go through three
(3) major releases. Did I read that correct? Given that during the
whole iojs history we had only two.

I'm not advocating for breaking everything immediately, or for
breaking _prefixed without checking how it's used, but the above
looks to me like going a bit too far.

[ChALkeR]

A C++ private member is only accessible by violating encapsulation, and is definitely not the kind of thing we're talking about here.

Both of those are breaking the convention.

[chrisdickinson]

@ChALkeR To stomp on another programmer's symbol-accessible methods, one would have to getOwnPropertySymbols, pick an index out of the list returned, and then purposefully use it. It is extremely unlikely that folks will do that. There is, however, a lot of evidence to support the notion that users will use _-prefixed methods if they are the only way to get to behavior / information that they need.

The argument is not that we should never be able to change the implementation of _-prefixed properties – it's that we have to be as careful in changing them as we are with other, non-prefixed parts of the API. We came very close to this weekend's incident not happening: if the property had been defined on the instance instead of the prototype, nothing would have broken. We can move forward under these requirements, we just have to do so carefully. I totally whiffed the review of the breaking change that came through (sorry @mscdex, @othiym23! the possibility of breaking a hasOwnProperty check didn't occur to me!) We still have latitude to change implementation – getters and setters afford us leeway – but we have to make sure we employ those tools correctly.

[targos]

Maybe to avoid this kind of issues in the future, Symbols should be used systematically for internal properties when new features are added.
That way when people in userland need to access something that was not initially in the API they would come here and ask for it instead of hacking with _values without telling anyone.

[benjamingr]

to get potential releases in front of more people, it would be great to have some notion of release candidates for "larger" releases (semver-major definitely, semver-minor probably, semver-patch it would be nice but also probably a gratuitous amount of work [ETA: finished incomplete thought])

I think this is a good idea - putting beta versions in front of developers in development (but not in production) so that they use new features and by the time they become stable the developer is finished and is now working in production is good.

This way a lot more people get to test new versions but it doesn't break anyone's production. This is what happens in projects like Angular.

[targos]

@ljharb @benjamingr 👍
If we have realease candidates with nvm I would use them on my dev machine

[benjamingr]

@targos we have release candidates - the problem is that at the moment is they're available for a short time before release. I've installed them multiple times in the past to test new releases but they're not available for long enough in order to test them in actual development.

I think a limited timespan dev channel would be really nice - nothing like the 0.odd 0.even thing node has - but time constrained to relatively short durations (let's say - at least a week and one semver version behind but as soon as we reach a week (or two) we can release).

[targos]

@benjamingr I know we have release candidates but AFAIK they have to be downloaded and installed manually, so it is not that easy to try them.
We also have the master and next branches that could be interesting to test. If it was just a command like nvm use iojs-next I'm sure more people would give it a try.

[benjamingr]

@targos I definitely agree 👍

[rvagg]

Regarding the lack of nightlies and next-nightlies, they should have been more regular but they stopped when we moved the repo ... so I've fixed that and they should continue every day (when there are changes) now. Sorry for this momentary break in transmission.

[YurySolovyov]

With 3.0 release coming, are we going to actually remove any features that were deprecated in 2.x ?

[ChALkeR]

@chrisdickinson

@ChALkeR To stomp on another programmer's symbol-accessible methods, one would have to getOwnPropertySymbols, pick an index out of the list returned, and then purposefully use it. It is extremely unlikely that folks will do that. There is, however, a lot of evidence to support the notion that users will use _-prefixed methods if they are the only way to get to behavior / information that they need.

I am not advocating for keeping Symbols stable. I just want there to be some proper rules defining what is «exposed» and how should deprecation work, not just «Everything that is exposed is exposed». Because technically, Symbols are exposed.

And I would be more glad if ._prefixed deprecation rules were at least a bit different from .unprefixed deprecation rules, but that's not up to me.

[chrisdickinson]

@ChALkeR The proper rule about "what is exposed" is "Everything that is a property is an implicit public API, _-prefixed or not. Symbol-named properties are also exposed if the Symbol itself is exposed to the user." Any changes to an implicit public API have to go through a process, ideally to include:

1. Check the extent of usage using estoc + comparing weekly download counts.
2. If the download counts are reasonably small, begin deprecation process.
   1. Deprecate in the docs in the current major line.
   2. As of next major version, replace with getter/setter that issues deprecation warning.
   3. If the goal of the deprecation is removal (not all deprecation ends in removal), set a target of
      weekly downloads to have fixed before removal.
   4. Once that target has been reached, remove as of the next major version.

The other issue is that we may want to change internal representations, as in _events – in that case we can create a getter/setter immediately that preserves the old behavior, separately from deprecating the old accessor.

[ChALkeR]

@chrisdickinson This looks much nicer, thanks ☺. Could that be put somewhere in the repo so it won't be lost?

[sam-github]

Nightlies don't get much use because they are too bleeding edge. Also, anyone who just wants to run tests off the head of master can build node themselves, they don't need tarballs, though that might be a convenience.

Release candidates might get more traction - there would be less of them, they could get tweeted/announced, they'd be linked to from the PR that proposes a release, etc. They'd be left up for a few days, I hope. Seems worth a try.

[benjamingr]

Release candidates might get more traction - there would be less of them, they could get tweeted/announced, they'd be linked to from the PR that proposes a release, etc. They'd be left up for a few days, I hope. Seems worth a try.

Yes, this is what more in the lines of what I had in mind. I think RCs for a week or two could really help.

[othiym23]

I agree with everyone arguing in favor of RCs over nightlies. When I said "better than nothing", that wasn't exactly an endorsement. Having a blessed release channel / distribution mechanism for them, and getting them integrated into tools like nvm would go a long way to getting them used more.

[Qard]

👍 For RC, similar to npm@next. After a week (or something like that), if nothing is on fire, upgrade it to a release. Otherwise, roll a new RC with fixes.

If there was an easy-to-follow RC channel that I could install via nvm, I'd be updating all my projects to test against RC. :)

[silverwind]

I'd still prefer some sort of auto-update mechanism for maximum exposure. A lot of software is evergreen today. I don't see why node, at least for nightlies (for now), couldn't be.

[sam-github]

integrated into tools like nvm would go a long way to getting them used more.

That would give great exposure, and be fairly safe as long as they -rc's didn't automatically get installed as the latest.

I'd still prefer some sort of auto-update mechanism for maximum exposure.

A debian ppm for -rc packages might work for deb-based systems. All the OS X auto-updates I've seen seem to require a UI to be running, and a user to interact with.

How would you see this working for node?

[silverwind]

How would you see this working for node?

I could see some sort of startup hook build into nightly binaries that does a single HTTP request per day to check for an update. If there's one available, download it and put it into ~/.node/bin with PATH pointing there, pretty similar what nvm does.

This would primarily target people that use node as a tool and not servers. Also, there must be an opt-out and the timestamp of the last update check must be saved to disk. Thinking about this, maybe it's better to keep such things to nvm though, these things can get messy.

[domenic]

Last time we discussed putting out RCs we looked at the nightly builds for next and said "hey, we're already doing everything we would do to put out an RC."

What do you people advocating for RCs over nightlies see happening here? A marketing change perhaps---better links from the homepage?

Nobody really wants to do the extra infrastructure work to change the name from nightly.yyyymmdd to rc.increasingnumericsequence, so hopefully that's not what you're looking for.

[domenic]

Wait, @rvagg, do we even have nightly next releases? If we don't, I guess that's a pretty big missing thing.

[aredridel]

Mostly branding: That the exact build that will become 'latest' bakes for a week as 'next' first. Then promoted, rather than a rebuild or similar -- it makes the difference between 'testing' and 'safe to use' very very small, just earlier -- rather than nightlies that are thrown away, and most never become a 'release'.

[rvagg]

@domenic https://iojs.org/download/next-nightly/ - they only build when there are changes which is why the list is short (and because of the accidental downtime last month)

[domenic]

@rvagg ah awesome. I'll open an issue on the website to link to those.

[rvagg]

I have some unrelated things to do today but if I get time I'm going to set up yet another release process in Jenkins for "rc" builds with special "rc" labels so they don't look so scary - perhaps we can get nvm to pick those up too? 3.0.0-rc1 @ https://iojs.org/download/rc/3.0.0-rc1

[ljharb]

@rvagg if the listing in https://iojs.org/download/rc is an identical format to that in https://iojs.org/dist, then I should be able to get something out this or next weekend, and merged to travis-ci. When they deploy that, I can't tell you, but it'd be included in their next VM image update.

[rvagg]

neato, all 3 of the current directories are using the same format, https://iojs.org/download/next https://iojs.org/download/next-nightly and https://iojs.org/download/release/ which is aliased at https://iojs.org/dist/, they all use index.tab and index.json in the same way

[ljharb]

Great! I'll build it optimistically hoping the future merged node releases follow the same patterns :-)

[rvagg]

I think you'll find that optimisim well founded

[ljharb]

Please let me know when a release candidate exists in https://iojs.org/download/rc so I can test with it :-)

[Fishrock123]

Further discussion at #1997