RFC: Wasmtime 1.0 #14
Conversation
accepted/wasmtime-one-dot-oh.md
Outdated
|
|
||
| Each release of Wasmtime will bump the major version number of Wasmtime itself. | ||
| This means, for example, that Wasmtime 2.0 will be released one month after | ||
| Wasmtime 1.0. After one year Wasmtime will be at 12.0. At this time it is not |
There was a problem hiding this comment.
I would not expect this as a user: I would expect the project to follow semantic versioning and decide every four weeks if the release should be major or minor. Users then get some type of clue about the changes coming to them ("oh, this is a major change, some API must have changed--let me go look at the release notes!"). It seems that the "match the Wasmtime version with the bindings version" can be solved some other way (e.g., wasmtime-py [Wasmtime version]-[binding version])
There was a problem hiding this comment.
That may have been a usual expectation at some point in the past, but many projects have changed their versioning policies in recent years to this sort of release train, for example the LLVM project switched to something like this in 2016 http://blog.llvm.org/2016/12/llvms-new-versioning-scheme.html
There was a problem hiding this comment.
Most of these projects do have breaking changes every release now. Wasmtime probably won't have breaking changws every time when releasing at the suggested interval.
There was a problem hiding this comment.
I started out with this view, too, but talking through it with Alex and others made me pretty convinced that proper fine-grained semver handling would be very, perhaps prohibitively, hard.
As described in the explainer, the language embeddings are a major consideration here: I think it's very clearly desirable to keep their version numbers aligned with Wasmtime's (because you shouldn't have to maintain a mapping table from language embedding version to Wasmtime version in your head). I really don't see a way to achieve this without regularly scheduled version bumps, in particular if we want to enable third-party embeddings to stay version aligned, too.
There was a problem hiding this comment.
If we bump the major version of all crates part of wasmtime to the same version even if only some crates have breaking changes, I guess it wouldn't be much of a stretch to bump the major version every time any language embedding has a breaking change.
There was a problem hiding this comment.
That's true, yes. But I think it'd in reality have the effect of making it harder for embeddings to improve in ways that require these kinds of changes. In particular for lower-tier ones.
As Pat mentioned above, we're not proposing to tread particularly new ground here: LLVM does pretty much the same, and I'd assume for much the same reasons.
There was a problem hiding this comment.
Personally my experience is that maintaining a strategy where you figure out whether breaking changes or not have happened is onerous and difficult to maintain. From the perspective of authoring Wasmtime it means that we'll quickly fall into a trap of "oh when's the right time to have a breaking change?" and try to batch things up, in my opinion creating unnecessary tension for landing improvements and updates to the engine.
From a user's perspective I would consider Wasmtime as a "weighty" enough dependency that even if you want to update patch versions there's likely to be some sort of gotcha depending on how deeply integrated it is into your system. This means that we, as maintainers, will likely spend a lot of time trying to chase down unintended consequences of changes in embeddings we have limited access to.
I feel that the best compromise between these worlds is regularly expected breaking changes (mild breaking) and users having the expectation that they'll have to bump the version. This creates a number of motivations towards what I feel is "the right thing to do":
- As authors we don't feel hindered landing breaking changes (minor ones)
- As authors we are vigilant to clearly document any possible breakage in release notes
- Users should expect that upgrades are hassle-free, not literally free
- Users should expect to read the release notes and determine risks to their application, if any
I don't think it's viable to have a situation where we, as the Wasmtime project, take on 100% of the work of keeping everyone else's applications running. Our job, in my opinion, is providing a solid WebAssembly runtime that users can rely on. Requiring a small amount of end-user assistance, though, greatly helps actually releasing new features and not feeling hindered in development.
| with trying to do this, though, including: | ||
|
|
||
| * Wasmtime developers now have to keep track of breaking changes and determine | ||
| whether any landed in the 4-week release window. |
There was a problem hiding this comment.
Yes! I feel the project should keep track of breaking changes; the user will have a lot harder time figuring that out (see my semver comment above).
There was a problem hiding this comment.
This might be too strong a statement as written: as discussed elsewhere in the explainer, the idea is very much to be mindful of API stability and not make updating harder than it needs to be. So actually breaking changes should be documented as such, which means that this sentence should still be true as per this plan.
There was a problem hiding this comment.
Yeah I'm not getting across what I was thinking with this bullet. It's moreso that having an unpredictable versioning scheme makes it harder for releases, automation, and such. Having to programmatically detect a breaking change somehow does not seem super viable and could complicate what's already bound to be a pretty complicated set of automation for these releases.
I consider it a bare minimum that at all times we're keeping track of breaking changes and logging them in a changelog of some sort.
There was a problem hiding this comment.
Great, agreed with the general feeling that wasmtime is already production-ready!
Would it be useful to indicate that most people can use tooling (e.g. dependabot for Rust on github/gitlab) for upgrading their dependencies, and that the relatively fast train schedule can be accommodated by such tooling?
accepted/wasmtime-one-dot-oh.md
Outdated
| LTS versions will be maintained in separate branches of the Wasmtime repository. | ||
| At this time it's expected that LTS versions will not needed separate branches | ||
| in separate embedding repos and tags will suffice. The branch names for the | ||
| Wasmtime repository will be `lts-latest` and `lts-oldest` for the most recent |
There was a problem hiding this comment.
(a bit nit-picky, sorry) Instead of lts-latest / lts-oldest, how about lts / lts-previous? lts-oldest not technically the oldest one, and previous doesn't leave room to any possible doubt imo.
There was a problem hiding this comment.
How about just many lts-$VERSION branches, the two most recent being the supported ones? No jumping lts-latest from 10.0 to 15.0 etc.
There was a problem hiding this comment.
Yeah in retrospect I don't think latest and oldest are good names, but I like @fitzgen's idea of just having a branch for all of them. It means they become irrelevant and ossified 10+ months after their creation but that's not really an issue with git
There was a problem hiding this comment.
whynotboth.jpg: Could have a moving tag for current-lts and prev-lts + fixed tags with version numbers. But I'll stop with the bikeshedding here, having tags with the version number would be already quite good too!
| project. For example Cranelift is not an API-stable component that will be | ||
| released with Wasmtime. For now Cranelift will likely have the same release | ||
| cadence of Wasmtime for ease of infrastructure, but this may change in the | ||
| future as Cranelift develops its own API and release process. Additionally, |
There was a problem hiding this comment.
It might be an open question at the moment, but still curious: until there is such a separate Cranelift release process, does it mean that the Cranelift shipped with Wasmtime LTS is actually in LTS mode itself? It might have different requirements in terms of backports for bugfixes and sec fixes (e.g. some of these might involve only Cranelift and nothing in Wasmtime, and conversely), so this really calls for a different release process as you suggest. Still, it would be interesting to get a clear stand on where Cranelift is at the moment in terms of support and will be once this RFC gets merged.
There was a problem hiding this comment.
In my view at least, it makes sense for us to uphold the same LTS promises with Cranelift as with Wasmtime with regard to bug-fixes, because a security bug in Cranelift is a security bug in Wasmtime from the point of view of the consumer. So, yes, I would expect that we backport any correctness or security fix to codegen.
(Also FWIW, it's hard for me to imagine at the moment what would cause Cranelift to diverge to another release process, as riding along with Wasmtime gives all of the needed framework for managing stable branches, security patches, etc. -- at most we might introduce a stronger API-stability guarantee on cranelift-codegen and friends -- but for completeness, if Cranelift does ever diverge, I would expect it to keep a very similar LTS and backports policy.)
There was a problem hiding this comment.
My view on this is that as part of Wasmtime's 1.0 stability promise we're promising quality in our transitive dependencies, including cranelift, so any cranelift bugs affecting Wasmtime is something we have to take care of in Wasmtime (patch releases, etc).
If there's a bug in Cranelift, though (security or not), which we probably know cannot affect Wasmtime (e.g. in an architecture not supported or something like that) I think the process is a bit more ambiguous. Most of this RFC is about LTS/point releases for Wasmtime, and I don't think it would apply in the case that the bug from Cranelift isn't exposed through Wasmtime's usage.
Not to say that point releases won't happen, just that I don't think it would always necessitate a Wasmtime point release as well. Ideally the release process for these two would be diverged so that Cranelift could be released without the other (and in theory Wasmtime as well if Cranelift didn't have changes, but that seems unlikely), but that may take some more infrastructure.
There was a problem hiding this comment.
If there's a bug in Cranelift, though (security or not), which we probably know cannot affect Wasmtime (e.g. in an architecture not supported or something like that) I think the process is a bit more ambiguous. Most of this RFC is about LTS/point releases for Wasmtime, and I don't think it would apply in the case that the bug from Cranelift isn't exposed through Wasmtime's usage.
I think this is the key point. The way Cranelift is tested right now in particular doesn't really allow us to make strong claims about its stability and quality properties when used to compile non-Wasm content, and perhaps even when used in other embeddings in general.
I think that will change, but this RFC is about Wasmtime, and in that context it makes sense to treat Cranelift purely as a dependency to Wasmtime, and look at its stability and quality properties through that lens.
it's hard for me to imagine at the moment what would cause Cranelift to diverge to another release process
I also agree with this, though! When Cranelift gains a stability story of its own, that explicitly encompasses other uses, I don't think that'd be a reason for it to also gain a completely separate release process. It seems much better to broaden the scope of guarantees we give for it by introducing the required testing strategies, etc, but still let it benefit from being included in the same release train schedule, etc.
Motion to finalize with a disposition to mergeThis hasn't seen some discussion in some time and has been open for awhile to peruse, so I'm gonna try to hit the button and propose that this RFC is merged. Stakeholders sign-offThis model for a 1.0 for Wasmtime will affect just about everyone using Wasmtime so I've been generous with the tags here for sign-off. ArmDFINITYEmbark StudiosFastly
Google/EnvoyIntelMicrosoftMozillaIBMwasmCloudUnaffiliated |
|
✔️ this sounds quite fine to me, thank you for putting this together! |
|
So I'm not sure I can agree that the code base is 'production ready'... People are writing hand written tests which make the backends fall over on, or produce incorrect code, on a weekly basis. Full SIMD support has only been in, for Intel and Arm, for a month or so and I think we can now all agree that the Wasm spec tests haven't been good enough at catching bugs. So, I imagine there's still quite a few lingering around and I wouldn't want to be the one backporting the fixes for an LTS release. At least, I think we'd need a caveat in the notes about SIMD support. And generally, the lack of aarch64 fuzz testing continues to make me nervous and without it I would struggle to have faith, at least, in the aarch64 backend in a production environment. |
This is true for Cranelift. However, its worth point out that the issues we are opening are for instructions / type combos that are pretty much never emitted from the wasm layer (at least for the ones I checked). We are probably finding about them now exactly because the wasm layer never emits those. So I think this would be similar to the situation @alexcrichton was mentioning where we have bugs in cranelift which are never exposed to wasmtime. I haven't used wasmtime too much directly, so I'm not really the best person to comment on its stability as a whole. |
|
Yes, sorry, I was forgetting that this is for wasmtime as a whole. My concerns over SIMD still stand though generally, it seems most development has been done relying on the wasm spec tests (which has had bugs it in) and for which we haven't had the time or infrastructure to really test it. |
I think these are very valid concerns for SIMD, and we should make sure that they're addressed before enabling SIMD by default. The RFC has a section discussing feature stability, which I think applies here. It seems like right now SIMD doesn't pass the test for items 3 and 4 of the list in that section. Do you think that that view makes sense, and does that alleviate your concerns? |
Ah, thanks. Yes, this seems fair enough to me. |
|
|
||
| Wasmtime intends to follow in the footsteps of many other projects on the matter | ||
| of release cadence with a frequent and predictable release cycle. **This RFC | ||
| proposes releasing Wasmtime on Tuesday every 4 weeks**. The precise date of |
There was a problem hiding this comment.
Monthly releases (e.g. releasing Wasmtime on the first Tuesday of every month) might be easier to reason about, seeing as even this document uses "once a month" and "every 4 weeks" interchangebly.
There was a problem hiding this comment.
My main goal is to make sure that the releases are as automate-able as possible to make the work of a release on us in theory "make sure the changelog is up-to-date and merge a PR". In that sense I don't think that the precise cadence really matters too too much. I'm expecting users will have some sort of regular process to update wasmtime and/or notifications through some bot like dependabot.
All that to say I think the precise number doesn't really matter too too much and I think that whatever is the easiest and most reliable to automate is probably the way to go. I chose a weekly-based cadence since that's what rust-lang/rust does (although 6 weeks instead of 4) and I know there's a fair bit of tooling for things like "when's the next release?" which I believe is easier with a machine-predictable cadence rather than a human-predictable cadence. I don't think it's really onerous one way or another though.
accepted/wasmtime-one-dot-oh.md
Outdated
|
|
||
| Wasmtime will support some releases for an extended period of time relative to | ||
| other releases. These releases will be known as "long term support" releases or | ||
| LTS releases. **Wasmtime will support two active LTS versions at any point in |
There was a problem hiding this comment.
Maintaining LTS releases is a costly process, so starting with two when it's not clear that even one is necessary seems like an overkill. I'd suggest starting with one, perhaps with a slightly longer cycle (~6 months), and adding another one only if there is a clear demand for it.
There was a problem hiding this comment.
The worry we have about only having a single LTS release is that it basically means LTS users have a zero-length upgrade window when a new LTS release is published. That seems counter to what someone in need of an LTS release would want :)
Plus, our hope is that we'll be able to automate our processes around releases to a very high degree, and to make use of that same automation for LTS releases as well—which ideally will address some of the cost concerns.
At the very least, I'd hope that maintaining two LTS releases wouldn't be that much more burdensome than maintaining a single one, in particular given the tightly circumscribed scope of changes we intend to make available to LTS releases.
I think it's worth at least thinking through what it'd mean to start without an LTS release at all: you're right that the need for one is more derived from precedent in other projects than from immediate requests from Wasmtime users.
An option could be to split the LTS part out into a separate proposal that could simmer for a bit longer. The one worry I'd have about that is that it's probably easier to establish the required processes for all of this in one go, so getting up the energy to do the LTS stuff later might simply be harder. But @alexcrichton would much better be able to comment on that part than me :)
There was a problem hiding this comment.
I meant the cost of backporting fixes, which often means rewriting the change from scratch, since trivial cherry-picks to a branch from a few months ago rarely work in an actively developed codebase... Having said that, I'm not very familiar with the codebase or the development pace of Wasmtime, so perhaps that won't be an issue.
There was a problem hiding this comment.
AFAIK most of us working on Wasmtime haven't dealt much historically with maintaining LTS releases of software, so in that sense that may be why we're being a bit overly ambitiuos here. Given how different the LTS process is I don't forsee it being easier to add now vs later if we decide to do it, so in that sense I don't think there's urgency to add it earlier while the release process is all in someone's head.
An alternative model could be one where the LTS windows overlap for 1 release instead of 5 releases. For example 5.0 would be an LTS release for the 5.0-11.0 releases, but when 12.0 is released only 10.0 is a supported LTS release. Something like that means that our window for supporting multiple LTS releases is much smaller and because 11.0 is probably pretty similar to 10.0 any work for backports would primarily be from 11.0 to 5.0 rather than 11.0 to 10.0 (which is likely cherry-pick-able). This would still have the property where there's a window where users can upgrade but still get bug fixes if necessary.
Naturally though I'd expect that anything here can change at any time with sufficient consensus. It's probably a lot easier to add an LTS process than it is to remove one as well. Given that we don't have anyone specifically asking for LTS it may not be a bad idea to not support it initially. We could have some initial releases and see if users find the update-every-4-week cadence to be too onerous. We'd probably, in that case, update the bugfix policy to backport to the current release at the time of the bugfix and perhaps the last 1-2 releases prior to that as well.
Overall I don't personally know the best thing for Wasmtime at this time. We made an initial guess with 2 LTS releases at this cadence, but it's mostly a guess that it'll work out.
There was a problem hiding this comment.
If customers aren't requesting an LTS, it might be worth postponing until the release process has had some time to mature. I completely agree that backporting is very costly, even in relatively mature code bases, so it could be too much work for little gain.
And sorry if I missed it, but is there a plan for stabilization and testing before a release drops, or are we going for an "always releasable trunk"? Particularly if we go for LTS releases, it would be really useful to give developers and users a reasonable amount of time to test before hand, otherwise we'd probably just get a bunch of issues a couple of weeks after the release and it wouldn't be great for trust to push out a buggy LTS.
There was a problem hiding this comment.
I posted a comment at the bottom about this, but personally I agree with y'all's warnings and conclusions that maybe we don't need LTS at all.
For releases yeah the plan is that main is always green with CI and always release-able.
| * `wasmtime-go` - Go bindings for Wasmtime | ||
| * `wasmtime-dotnet` - .NET bindings for Wasmtime | ||
| * `wasmtime-cpp` - C++ bindings for Wasmtime | ||
| * `witx-bindgen` - Canonical API bindings generator for Interface Types |
There was a problem hiding this comment.
Since you've included witx-bindgen here, is this expected to be used by all Bytecode Alliance projects or only by Wasmtime (with sub-crates) and its language bindings?
There was a problem hiding this comment.
Great question! We do expect witx-bindgen to be used by other projects (both in and outside the BA), so in some sense it wouldn't need to have a release schedule aligned with Wasmtime's, and there's nothing tying witx-bindgen to Wasmtime in some fundamental way.
At the same time, I think there are practical reasons for doing so, at least for the time being. As of now, simply as a matter of fact we're making heavy use of Wasmtime in the development process of witx-bindgen. And increasingly that'll be the case the other way around as well. Aligning the release processes means we can rely on available feature sets in these development processes.
Should over time other uses of witx-bindgen become dominant, it'd make sense to reevaluate this alignment.
There was a problem hiding this comment.
Right, at the same time, for someone using witx-bindgen only to generate guest bindings, seeing "random" major version bumps without any significant changes in witx-bindgen might be a weird experience.
There was a problem hiding this comment.
That's true, yes, and to some degree more so than for the Wasmtime embeddings.
Our processes in developing witx-bindgen itself would become increasingly more complicated if we didn't align version numbers like this that I think the benefits outweigh these costs, though. Which seems even more true given that, when using it as a guest bindings generator, one should only need to update witx-bindgen to pull in significant changes. And even then, the major version upgrade should be trivial—except if witx-bindgen has significant API changes which would necessitate a major version bump anyway.
There was a problem hiding this comment.
Currently witx-bindgen has a lot of various components which I suspect is going to end up like the wasm-tools repo where each component is separately managed and versioned. I expect anything wasmtime-related to be tied to wasmtime's own version, but things like the JS code generator or Rust generator for compiled-to-wasm-code probably don't need to be tied to weasmtime's own release process.
|
I'm OK with this proposal. From the perspective of the s390x architecture, I would really like to see the public CI enabled before we call wasmtime "production ready". As of right now, qemu 6.1 contains all changes needed to run CI in emulation. With PR bytecodealliance/wasmtime#3330 and the dependencies mentioned there, the tree builds and passes CI testing on s390x again, so it should now be possible to enable CI once everything is merged. |
|
I think the discussions about LTS have good points, and I agree at this time with taking the suggestion of just dropping LTS entirely from this proposal. That would mean that there would be no LTS release channel and bug/security fixes would only go to the most-recent-released Wasmtime version. Does anyone forsee this being an issue? Do users today have a good motivating use case for the LTS releases? (either 1 or 2 or with different schedules) One of the main reasons for dropping them is the lack of concrete motivation for them at this time. We can always add LTS later but removing it is probably going to be much harder. |
|
Ok I've updated the wording here to remove the LTS channels, so we'll just have one version every 4 weeks and the current version receives bugfixes and backports. Given that we also have additionally sign-offs at this time I'm going to go ahead and... Entering Final Call Period
The FCP will end on Monday September 27. |
|
Ok with no major objections having been raised, I'm going to merge this. Thanks all! |
This commit removes the Lightbeam backend from Wasmtime as per [RFC 14]. This backend hasn't received maintenance in quite some time, and as [RFC 14] indicates this doesn't meet the threshold for keeping the code in-tree, so this commit removes it. A fast "baseline" compiler may still be added in the future. The addition of such a backend should be in line with [RFC 14], though, with the principles we now have for stable releases of Wasmtime. I'll close out Lightbeam-related issues once this is merged. [RFC 14]: bytecodealliance/rfcs#14
This commit removes the Lightbeam backend from Wasmtime as per [RFC 14]. This backend hasn't received maintenance in quite some time, and as [RFC 14] indicates this doesn't meet the threshold for keeping the code in-tree, so this commit removes it. A fast "baseline" compiler may still be added in the future. The addition of such a backend should be in line with [RFC 14], though, with the principles we now have for stable releases of Wasmtime. I'll close out Lightbeam-related issues once this is merged. [RFC 14]: bytecodealliance/rfcs#14
|
Thanks for driving this @alexcrichton. The final version of the RFC looks great! I have only one comment (sorry for missing the main discussion and posting it after the merge): would it be possible to always backport security fixes to the previous version in addition to the current version? At DFINITY we need at least two weeks to incrementally roll out a new version to production. If there are breaking API changes, then we may need even more time. Guaranteed backporting of security fixes to the previous version would give us more predictable upgrade schedule. Otherwise, in case of a security issue we would need to quickly upgrade to the current version sacrificing testing/canary coverage, which may be problematic if the current version has some regression. |
|
We talked about that in today's wasmtime meeting and yeah @ulan we all felt it was reasonable to backport fixes to the latest 2 releases instead of only the latest release. When I get around to implementing all the automation for this I'll write up our release process and such in the wasmtime repo, and I'll be sure to update this when I write it down. |
|
Thank a lot @alexcrichton! |
This RFC is a proposal to release Wasmtime with a 1.0 version. In doing so this
proposal also defines what it means to be 1.0 and details areas such as:
Rendered