-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(rfcs): add expressing npm deps rfc #4
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,129 @@ | ||
# RFC: npm Dependency Expression | ||
|
||
## Summary | ||
|
||
Allow for the expression of npm dependencies in Rust-generated WebAssembly projects that use the `wasm-pack` workflow. | ||
|
||
## Motivation | ||
|
||
In keeping with the team’s goal to allow the surgical replacement of JS codepaths with Rust-generated WebAssembly, developers using the `wasm-pack` workflow should be able to express dependency on packages from the npm registry, other registry, or version-control repository. | ||
|
||
## Guiding Values | ||
- Development on Rust-generated WebAssembly projects should allow developers to use the development environment they are most comfortable with. Developers writing Rust should get to use Rust, and developers using JavaScript should get to use a JS based runtime environment (Node.js, Chakra, etc). | ||
- JavaScript tooling and workflows should be usable with Rust-generated WebAssembly projects. For example, bundlers like WebPack and Parcel, or dependency management tools such as `npm audit` and GreenKeeper. | ||
- When possible, decisions should be made that allow the solution to be available to developers of not just Rust, but also C, and C++. | ||
- Decisions should be focused on creating workflows that allow developers an easy learning curve and productive development experience. | ||
|
||
|
||
|
||
## Solutions | ||
|
||
Any solution to a problem like this involves 2 steps: | ||
|
||
1. How to index the third-party dependencies (in this case: npm packages), and | ||
2. How to "require" or "import" the packages into code. | ||
|
||
The second of these is simpler than the first so let's start with that: | ||
|
||
### Requiring an npm package | ||
|
||
To require an npm package in your Rust code, you will use the `wasm-bindgen` | ||
attribute, passing in `module = "name-of-package"`. | ||
|
||
```rs | ||
// src/foo.rs | ||
|
||
#[wasm_bindgen(module = "moment")] | ||
extern { | ||
alexcrichton marked this conversation as resolved.
Show resolved
Hide resolved
|
||
// imported from moment.js | ||
} | ||
``` | ||
|
||
This syntax is already supported by `wasm-bindgen` for other types of JavaScript imports. | ||
|
||
## Indexing the npm packages | ||
|
||
This question of how to index, or even if, to index, the npm packages is a large one with | ||
several considerations. The below options were all considered. We believe that the | ||
`package.json` solution is the best, at the moment. | ||
|
||
### `package.json` | ||
|
||
*This is likely the best choice. Although it requires that Rust developers use a* `*package.json*` *file, it allows the best interoperability with existing JavaScript tooling and is agnostic to source language (Rust, C, C++).* | ||
|
||
Create a file called `package.json` in the root of your Rust library. Fill out dependencies as per specification: https://docs.npmjs.com/files/package.json#dependencies. You can use `npm install` to add dependencies: Although npm will warn that your `package.json` is missing metadata, it will add the dependency entry. | ||
|
||
Note: All meta-data in this file override any duplicate fields that may be expressed in the `Cargo.toml` during the `wasm-pack build` step. This allows the library author the flexibility to change the value of fields that may be present in the metadata in the `Cargo.toml`. For example, this would allow the user to provide a different name for the npm package (since the naming rules are slightly different). The confusion that may arise from the interaction of the potential duplication of metadata is a downside to this solution. | ||
|
||
Note: semver expression in `package.json` are based on npm rules. This is counter to the implicit `^` in a `Cargo.toml`. This confusion is also a downside to this solution, but is difficult to avoid in any potential solution to this problem. | ||
|
||
Example: | ||
|
||
```json | ||
|
||
{ | ||
"dependencies": { | ||
"foo" : "1.0.0 - 2.9999.9999", | ||
"bar" : ">=1.0.2 <2.1.2", | ||
"baz" : ">1.0.2 <=2.3.4", | ||
"boo" : "2.0.1", | ||
"qux" : "<1.0.0 || >=2.3.1 <2.4.5 || >=2.5.2 <3.0.0", | ||
"asd" : "http://asdf.com/asdf.tar.gz" | ||
}, | ||
"devDependencies": { | ||
"til" : "~1.2", | ||
"elf" : "~1.2.3", | ||
"two" : "2.x", | ||
"thr" : "3.3.x", | ||
"lat" : "latest", | ||
"dyl" : "file:../dyl" | ||
}, | ||
"optionalDependencies": { | ||
"express": "expressjs/express", | ||
"mocha": "mochajs/mocha#4727d357ea", | ||
"module": "user/repo#feature\/branch" | ||
} | ||
} | ||
``` | ||
|
||
### `Cargo.toml` | ||
|
||
*Ultimately this is not a good choice because it lacks interoperability with existing JavaScript tooling, but could be considered if we anticipate that we can get tooling to use this format.* | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This doesn't actually require new tooling support, since we can use the [package.metadata.npm]
moment = "~2.22"
#sugar for moment = { version = "~2.22", type = "prod" }
mocha = { version = "mochajs/mocha#4727d357ea", type = "dev" }
chai = { version = "^4", type = "dev" }
optional = { version = "6.6.6", type = "optional" }
git = { version = "http://asdf.com/asdf.tar.gz" } There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i genuinely think trying to express npm deps in Cargo.toml is not the right idea- it's complicated on several levels that just using a package.json doesn't cause. is there a reason you continue to return to it? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @ashleygwilliams What complications are you referring to? This RFC does not explain them, and it has not really been explained elsewhere either. There are some benefits to the
I believe that RFCs should fairly list the benefits and drawbacks (in thorough detail) of the different approaches, so that way the best decision can be made. That is what I have tried to do with my own RFC. I don't have a particular preference (I was the one who originally suggested There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There's some community experience you might want to consider. Before npm has become the leader in Node and later all web package management, there was Bower with its own registry (like crates) and its own metadata file (bower.json, like Cargo.toml). Package maintainers who wanted to publish into both registries had to maintain two metadata files. The packages could have different names and different sets of dependencies because not everybody published to both registries, and there was (and is) no centralized registrar for package names like there are for DNS domain names. It was a mess to maintain, so Bower including the registry was quickly deprecated. Do you want to repeat the history in the Rust community? There are more and more tools emerge on top of package.json that aren't directly linked to Node packages, like dependency tree analysis on GitHub, or dependency security audit. Npm and Node as organizations for sure can do better in standardizing package.json and ways to extend it beyond Node, but it would be improvement on top of a de-facto standard in a large community of Node/web, rather than something absolutely new, quite opinionated and less portable (toml vs json) in a smaller community of Rust. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @sompylasar To be clear, the situation with Rust is not the same as npm vs bower (which I am familiar with). In particular, Cargo will be used for Rust code (which has dependencies on other Rust packages). Then the Rust code will be compiled down to WebAssembly, and the WebAssembly will be published to npm. So in this case we do need both cargo and npm, I don't see any way to unify them into a single packaging system. Of course if you have any great ideas, we'd be glad to hear them, maybe we missed something! P.S. Also, cargo is not some "absolutely new" system, it is the standard for Rust, and has been for years. There are many thousands of Rust packages already published to crates.io, and it's extremely unlikely that they will all switch from cargo to npm (even if it were technically possible to do so). Switching to npm would actually create the exact same problem as bower: package fragmentation and duplication. It causes less fragmentation if the Rust code is published to Cargo and the WebAssembly + JS code is published to npm. So it's not like comparing bower to npm; it's like comparing Java's Maven to npm, or Go's packaging system to npm, or C#'s nuget to npm, etc. I've contributed to the Fable project, and they tried to consistently use npm for everything, but they ran into various problems. The unfortunate fact is that npm was designed for JavaScript, and it just doesn't work as well for non-JavaScript code. P.P.S. What we are discussing here is not " There is no scenario where we do not have a And no matter what is decided, the end result is that There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Loving the amount of non-aggressive opinion sharing going on in this sub-thread :) However, I feel it is getting steered a bit off course and onto tangents, and I'd like to re-focus the discussion a bit. Let's revisit guiding values:
This text is taken from this very RFC, but it might as well be inside some manifesto for our working group. We want to reach out to users and work with them, however they are already working. We do not want to force them to change their workflow just to do Rust and WebAssembly hacking. It follows that
These decisions are pretty much forced by the guiding values. What is nice is that We are not creating an unholy amalgamation of multiple dependency configuration files and systems here. Everything is cleanly separated. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Sounds good, but then does it mean there should be no initial There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Crates.io has existed for about 4 years, and in the link I posted earlier it says 18,467 crates with 528,272,511 total downloads. I think you are severely underestimating both the age and size of the Rust community. Rust already has extremely well established de-facto (and de-jure!) standards, and it is growing at an incredibly fast rate. Given the size of the community, making any sort of change is not trivial.
As I said, it should be much easier for tools to add support for You are right that JSON is probably the most well supported data format in the world, but TOML also has widespread support. And there are plenty of projects already using it. In addition, it is a very stable standard with a spec. Personally, I would say TOML is about as well supported as YAML. Besides, parsing the data format is usually the easy part, the harder part is actually adding in code that does the dependency analysis (and this code would be the same regardless of whether it's using a new
There is no reuse, because the So the amount of work is at least as much as adding in support for
This is an incredibly hard thing to do. I think you are overestimating just how generic npm and The best attempt so far to create a "one package manager to rule them all" is Nix. It works extremely well, however, it works by reusing existing package managers as much as possible (e.g. it uses npm for JS packages, Cargo for Rust packages, Cabal for Haskell, etc.) This is the only real way to unify the very diverse approaches to package management. Another attempt at a unified package manager is 0install, but in my opinion it is significantly inferior to Nix. Like many areas in programming, package management is something that seems easy at first, but becomes incredibly hard when you start to dig into the details. The lack of a unified package manager isn't just because of people, it's also because of technical difficulties. The reasons for the diversity in package managers are similar to the reasons for the diversity in programming languages (many attempts at a "one programming language to rule them all" have been tried, and all have failed, and not due to lack of effort).
I don't mean to discourage you, but frankly, yes, you do sound inexperienced in this area. The situation with bower vs npm is not the same as Cargo vs Rust: bower and npm had similar features, were targetting a similar community, and were targetting the same language. There was a lot of overlap between them (and thus competition), which prompted developers to publish to both registries (which is not really maintainable). In addition bower was not well maintained, so npm eventually ended up winning. But history could easily have gone the other way, it was not inevitable that npm would win. On the other hand, Cargo is fulfilling a need which is not fulfilled by any other package manager (including npm), it targets a completely different language and a different community, and it has many important features which npm lacks. Cargo is very well maintained (and constantly improving). So there is almost no overlap at all between Cargo and npm, and thus no need to publish to both Cargo and npm (as @fitzgen said, there is a very clear natural division: Rust code goes to Cargo, WebAssembly+JS code goes to npm). Therefore, I do not predict the same bower vs npm style of competition.
Yes, that is one of the potential options, and it is what I was advocating for in this specific sub-thread. I would still like to hear from @ashleygwilliams what they think the downsides of using There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for the long and great write-up, appreciate it! I internally understand all that you're saying from an engineer's perspective, and that's why I originally mentioned my wishes of united communities and fewer tools that do similar things won't ever happen. I observe so much work, which from the features viewpoint is really close, being repeated in yet another language (a multitude of languages from the past and present, now it's JavaScript, next is Rust), and this goes over and over, and this is depressing. A good example from today: I searched a dependency tree graphing tool for ES6 JS, and found a good one written in Node; why isn't there a tool for just dependency graphing (e.g. I need one for JS components and C++ components and HTTP APIs), well, because JS community won't write a generic tool, and other community won't likely write a generic tool that also supports JS. Same with package managers. Rust community made a package manager for Rust. Node community makes one for themselves. All right, wrapping up, thanks for listening. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I agree, but I don't think the solution is npm / In any case, it's not the Rust Wasm working group's job to push for Rust users to move to a different package manager (we are only responsible for WebAssembly, not Cargo or the overall Rust community), so the solutions presented in this RFC are the only viable ones for us.
Thanks for sharing! I think it's good to get multiple perspectives on this issue, including from people coming from a non-Rust background. |
||
|
||
To express npm dependencies, add a table to your `Cargo.toml` called `npm`. This table will have a key-value store of the dependencies you would like to use. The key is the name of the dependency followed by a value that is consistent with dependency values as specc’d in [this document](https://docs.npmjs.com/files/package.json#dependencies), many of which are demonstrated in the below example. | ||
|
||
```toml | ||
# Cargo.toml | ||
|
||
[package] | ||
#... | ||
|
||
[dependencies] | ||
#... | ||
|
||
[npm] | ||
moment = "~2.22" | ||
#sugar for moment = { version: "~2.22", type: prod } | ||
mocha = { version: "mochajs/mocha#4727d357ea", type: dev } | ||
chai = { version: "^4", type: dev } | ||
optional = { version: "6.6.6", type: optional } | ||
git = { version: "http://asdf.com/asdf.tar.gz" } | ||
``` | ||
|
||
### New Manifest File Format | ||
*We rejected this outright based on inherent complexity, community exhaustion, and its lack of interoperability with JavaScript tooling.* | ||
|
||
### Inline Annotations | ||
*This was the original solution that was implemented. It was good because it worked equally well with Rust and other languages such as C or C++. It was not good because it added high management complexity and lacked operability with JavaScript tooling.* | ||
|
||
This would look like this, and have no external manifest file: | ||
|
||
```rust | ||
#[wasm_bindgen(module = "moment", version = "2.0.0")] | ||
extern { | ||
type Moment; | ||
fn moment() -> Moment; | ||
#[wasm_bindgen(method)] | ||
fn format(this: &Moment) -> String; | ||
} | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for writing down these values! They've been vaguely floating around for a while, but it is great to see them crystallized so clearly in one place.