Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(rfcs): add expressing npm deps rfc #4

Closed
wants to merge 3 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions text/003-expressing-npm-deps.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# RFC: npm Dependency Expression

## Summary

Allow for the expression of npm dependencies in Rust-generated WebAssembly projects that use the `wasm-pack` workflow.

## Motivation

In keeping with the team’s goal to allow the surgical replacement of JS codepaths with Rust-generated WebAssembly, developers using the `wasm-pack` workflow should be able to express dependency on packages from the npm registry, other registry, or version-control repository.

## Guiding Values
- Development on Rust-generated WebAssembly projects should allow developers to use the development environment they are most comfortable with. Developers writing Rust should get to use Rust, and developers using JavaScript should get to use a JS based runtime environment (Node.js, Chakra, etc).
- JavaScript tooling and workflows should be usable with Rust-generated WebAssembly projects. For example, bundlers like WebPack and Parcel, or dependency management tools such as `npm audit` and GreenKeeper.
- When possible, decisions should be made that allow the solution to be available to developers of not just Rust, but also C, and C++.
- Decisions should be focused on creating workflows that allow developers an easy learning curve and productive development experience.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for writing down these values! They've been vaguely floating around for a while, but it is great to see them crystallized so clearly in one place.




## Solutions

Any solution to a problem like this involves 2 steps:

1. How to index the third-party dependencies (in this case: npm packages), and
2. How to "require" or "import" the packages into code.

The second of these is simpler than the first so let's start with that:

### Requiring an npm package

To require an npm package in your Rust code, you will use the `wasm-bindgen`
attribute, passing in `module = "name-of-package"`.

```rs
// src/foo.rs

#[wasm_bindgen(module = "moment")]
extern {
alexcrichton marked this conversation as resolved.
Show resolved Hide resolved
// imported from moment.js
}
```

This syntax is already supported by `wasm-bindgen` for other types of JavaScript imports.

## Indexing the npm packages

This question of how to index, or even if, to index, the npm packages is a large one with
several considerations. The below options were all considered. We believe that the
`package.json` solution is the best, at the moment.

### `package.json`

*This is likely the best choice. Although it requires that Rust developers use a* `*package.json*` *file, it allows the best interoperability with existing JavaScript tooling and is agnostic to source language (Rust, C, C++).*

Create a file called `package.json` in the root of your Rust library. Fill out dependencies as per specification: https://docs.npmjs.com/files/package.json#dependencies. You can use `npm install` to add dependencies: Although npm will warn that your `package.json` is missing metadata, it will add the dependency entry.

Note: All meta-data in this file override any duplicate fields that may be expressed in the `Cargo.toml` during the `wasm-pack build` step. This allows the library author the flexibility to change the value of fields that may be present in the metadata in the `Cargo.toml`. For example, this would allow the user to provide a different name for the npm package (since the naming rules are slightly different). The confusion that may arise from the interaction of the potential duplication of metadata is a downside to this solution.

Note: semver expression in `package.json` are based on npm rules. This is counter to the implicit `^` in a `Cargo.toml`. This confusion is also a downside to this solution, but is difficult to avoid in any potential solution to this problem.

Example:

```json

{
"dependencies": {
"foo" : "1.0.0 - 2.9999.9999",
"bar" : ">=1.0.2 <2.1.2",
"baz" : ">1.0.2 <=2.3.4",
"boo" : "2.0.1",
"qux" : "<1.0.0 || >=2.3.1 <2.4.5 || >=2.5.2 <3.0.0",
"asd" : "http://asdf.com/asdf.tar.gz"
},
"devDependencies": {
"til" : "~1.2",
"elf" : "~1.2.3",
"two" : "2.x",
"thr" : "3.3.x",
"lat" : "latest",
"dyl" : "file:../dyl"
},
"optionalDependencies": {
"express": "expressjs/express",
"mocha": "mochajs/mocha#4727d357ea",
"module": "user/repo#feature\/branch"
}
}
```

### `Cargo.toml`

*Ultimately this is not a good choice because it lacks interoperability with existing JavaScript tooling, but could be considered if we anticipate that we can get tooling to use this format.*
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't actually require new tooling support, since we can use the package.metadata field in Cargo.toml:

[package.metadata.npm]
moment = "~2.22" 
#sugar for moment = { version = "~2.22", type = "prod" }
mocha = { version = "mochajs/mocha#4727d357ea", type = "dev" }
chai = { version = "^4", type = "dev" }
optional = { version = "6.6.6", type = "optional" }
git = { version = "http://asdf.com/asdf.tar.gz" }

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i genuinely think trying to express npm deps in Cargo.toml is not the right idea- it's complicated on several levels that just using a package.json doesn't cause. is there a reason you continue to return to it?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashleygwilliams What complications are you referring to? This RFC does not explain them, and it has not really been explained elsewhere either.

There are some benefits to the Cargo.toml approach:

  1. It doesn't require Rust users to learn a new format.

  2. All of the metadata for the package is in one location.

  3. The semver versions can be the same as Rust, further reducing confusion.

  4. It makes it clearer that the npm dependencies for the Rust packages will be merged (unlike package.json which implies that it is the final source of truth).

  5. It makes the workflow very clear: wasm-pack generates a package.json from Cargo.toml.

    As opposed to the package.json idea: wasm-pack merges incomplete package.jsons together and also adds in some metadata from Cargo.toml. So it's a lot more complicated from the user's perspective.

I believe that RFCs should fairly list the benefits and drawbacks (in thorough detail) of the different approaches, so that way the best decision can be made. That is what I have tried to do with my own RFC.

I don't have a particular preference (I was the one who originally suggested package.json!), I just want the best decision to be made. And I think the RFC is missing important details about the various approaches, and their respective trade-offs.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's some community experience you might want to consider. Before npm has become the leader in Node and later all web package management, there was Bower with its own registry (like crates) and its own metadata file (bower.json, like Cargo.toml). Package maintainers who wanted to publish into both registries had to maintain two metadata files. The packages could have different names and different sets of dependencies because not everybody published to both registries, and there was (and is) no centralized registrar for package names like there are for DNS domain names. It was a mess to maintain, so Bower including the registry was quickly deprecated. Do you want to repeat the history in the Rust community? There are more and more tools emerge on top of package.json that aren't directly linked to Node packages, like dependency tree analysis on GitHub, or dependency security audit. Npm and Node as organizations for sure can do better in standardizing package.json and ways to extend it beyond Node, but it would be improvement on top of a de-facto standard in a large community of Node/web, rather than something absolutely new, quite opinionated and less portable (toml vs json) in a smaller community of Rust.

Copy link

@Pauan Pauan Sep 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sompylasar To be clear, the situation with Rust is not the same as npm vs bower (which I am familiar with).

In particular, Cargo will be used for Rust code (which has dependencies on other Rust packages). Then the Rust code will be compiled down to WebAssembly, and the WebAssembly will be published to npm.

So in this case we do need both cargo and npm, I don't see any way to unify them into a single packaging system.

Of course if you have any great ideas, we'd be glad to hear them, maybe we missed something!

P.S. Also, cargo is not some "absolutely new" system, it is the standard for Rust, and has been for years. There are many thousands of Rust packages already published to crates.io, and it's extremely unlikely that they will all switch from cargo to npm (even if it were technically possible to do so).

Switching to npm would actually create the exact same problem as bower: package fragmentation and duplication. It causes less fragmentation if the Rust code is published to Cargo and the WebAssembly + JS code is published to npm.

So it's not like comparing bower to npm; it's like comparing Java's Maven to npm, or Go's packaging system to npm, or C#'s nuget to npm, etc.

I've contributed to the Fable project, and they tried to consistently use npm for everything, but they ran into various problems. The unfortunate fact is that npm was designed for JavaScript, and it just doesn't work as well for non-JavaScript code.

P.P.S. What we are discussing here is not "Cargo.toml vs package.json", instead what we are discussing is "Cargo.toml by itself (and wasm-pack then creates a package.json file)" vs "Cargo.toml and package.json".

There is no scenario where we do not have a Cargo.toml file. So the Cargo.toml approach actually means one metadata file, whereas the package.json approach means two metadata files.

And no matter what is decided, the end result is that wasm-pack always generates a package.json file which is published to npm. So the npm experience will always be identical, we are only discussing the Rust user's experience.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loving the amount of non-aggressive opinion sharing going on in this sub-thread :)

However, I feel it is getting steered a bit off course and onto tangents, and I'd like to re-focus the discussion a bit.

Let's revisit guiding values:

Development on Rust-generated WebAssembly projects should allow developers to use the development environment they are most comfortable with. Developers writing Rust should get to use Rust, and developers using JavaScript should get to use a JS based runtime environment (Node.js, Chakra, etc).

This text is taken from this very RFC, but it might as well be inside some manifesto for our working group. We want to reach out to users and work with them, however they are already working. We do not want to force them to change their workflow just to do Rust and WebAssembly hacking.

It follows that

  • Compiling a Rust crate will continue to use cargo and Cargo.toml. Trying to change that is completely out of scope, even if the Rust crate is being compiled to wasm.

  • NPM dependencies will use package.json because that's how NPM dependencies are already described in NPM-land.

These decisions are pretty much forced by the guiding values.

What is nice is that Cargo.toml dependencies are for compiling Rust to wasm, and package.json dependencies are for running compiled wasm. Note that these are two completely distinct phases, that have completely distinct sets of dependencies. We aren't in a "now we're duplicating dependencies in two places" situation. Additionally, often it isn't even the same developer(s) driving each phase: once Rust-generated wasm is published in an NPM package, then any other NPM packages that depend on it only care about the package.json dependencies. The first Rust->wasm phase might as well not exist or never have happened. And similarly, compiling Rust to wasm doesn't need to know anything about the package.json or NPM.

We are not creating an unholy amalgamation of multiple dependency configuration files and systems here. Everything is cleanly separated.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is nice is that Cargo.toml dependencies are for compiling Rust to wasm, and package.json dependencies are for running compiled wasm.

Note that these are two completely distinct phases, that have completely distinct sets of dependencies. We aren't in a "now we're duplicating dependencies in two places" situation. Additionally, often it isn't even the same developer(s) driving each phase: once Rust-generated wasm is published in an NPM package, then any other NPM packages that depend on it only care about the package.json dependencies. The first Rust->wasm phase might as well not exist or never have happened. And similarly, compiling Rust to wasm doesn't need to know anything about the package.json or NPM.

Sounds good, but then does it mean there should be no initial package.json and everything to generate an NPM package should be in Rust ecosystem's Cargo.toml or in a separate rust-wasm-pack.toml or .json that describes this rust-to-node-wasm transformation? Let's make package.json a compilation target, and not use it as an input that, being in a repo, would confuse readers that this is a Node package while it's a Rust package that can also be built with Node and NPM as the target platform.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder how old is Rust community and how many packages are we talking about here. If there's a standard de-facto, maintainers move naturally.

Crates.io has existed for about 4 years, and in the link I posted earlier it says 18,467 crates with 528,272,511 total downloads.

I think you are severely underestimating both the age and size of the Rust community. Rust already has extremely well established de-facto (and de-jure!) standards, and it is growing at an incredibly fast rate. Given the size of the community, making any sort of change is not trivial.

Many modern tools in the Node/web world support more than one format for their metadata (e.g YAML or JavaScript for dynamic configs), but primary is JSON as it has first-class support in a lot of existing tools and protocols. TOML to me looks like a less widespread format.

As I said, it should be much easier for tools to add support for Cargo.toml (which is an established standard), rather than trying to invent a completely new standard.

You are right that JSON is probably the most well supported data format in the world, but TOML also has widespread support. And there are plenty of projects already using it. In addition, it is a very stable standard with a spec. Personally, I would say TOML is about as well supported as YAML.

Besides, parsing the data format is usually the easy part, the harder part is actually adding in code that does the dependency analysis (and this code would be the same regardless of whether it's using a new package.json metadata or Cargo.toml).

I perceive advantage in unification, update, and reuse of existing software instead of adding new software and integrating it with existing.

There is no reuse, because the package.json metadata you are proposing is a new standard, which requires new code to be written, and changes to existing packages and tools.

So the amount of work is at least as much as adding in support for Cargo.toml (and it's actually probably much more work to create a new package.json standard).

I believe teams should unite to build the package manager (not necessarily single one, but fewer is better, like Bash is de-facto standard for a shell despite there are sh, zsh, and fish).

This is an incredibly hard thing to do. I think you are overestimating just how generic npm and package.json are. There are many important things which they don't do well, such as pre-compilation, cross-compilation, optional dependencies, conditional compilation, guaranteeing one version per package, patching/overriding dependencies, pervasive global caching, etc.

The best attempt so far to create a "one package manager to rule them all" is Nix. It works extremely well, however, it works by reusing existing package managers as much as possible (e.g. it uses npm for JS packages, Cargo for Rust packages, Cabal for Haskell, etc.) This is the only real way to unify the very diverse approaches to package management.

Another attempt at a unified package manager is 0install, but in my opinion it is significantly inferior to Nix.

Like many areas in programming, package management is something that seems easy at first, but becomes incredibly hard when you start to dig into the details. The lack of a unified package manager isn't just because of people, it's also because of technical difficulties.

The reasons for the diversity in package managers are similar to the reasons for the diversity in programming languages (many attempts at a "one programming language to rule them all" have been tried, and all have failed, and not due to lack of effort).

If I sound like a junior idealist, please tell me explicitly, as despite having a decade of experience and understanding technical downsides of excessive generalization, I still believe there's a way to unite communities of different programming languages and tools to produce a better outcome overall.

I don't mean to discourage you, but frankly, yes, you do sound inexperienced in this area.

The situation with bower vs npm is not the same as Cargo vs Rust: bower and npm had similar features, were targetting a similar community, and were targetting the same language. There was a lot of overlap between them (and thus competition), which prompted developers to publish to both registries (which is not really maintainable). In addition bower was not well maintained, so npm eventually ended up winning. But history could easily have gone the other way, it was not inevitable that npm would win.

On the other hand, Cargo is fulfilling a need which is not fulfilled by any other package manager (including npm), it targets a completely different language and a different community, and it has many important features which npm lacks. Cargo is very well maintained (and constantly improving). So there is almost no overlap at all between Cargo and npm, and thus no need to publish to both Cargo and npm (as @fitzgen said, there is a very clear natural division: Rust code goes to Cargo, WebAssembly+JS code goes to npm). Therefore, I do not predict the same bower vs npm style of competition.

Sounds good, but then does it mean there should be no initial package.json and everything to generate an NPM package should be in Rust ecosystem's Cargo.toml

Yes, that is one of the potential options, and it is what I was advocating for in this specific sub-thread. I would still like to hear from @ashleygwilliams what they think the downsides of using Cargo.toml are (and why package.json is better).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the long and great write-up, appreciate it! I internally understand all that you're saying from an engineer's perspective, and that's why I originally mentioned my wishes of united communities and fewer tools that do similar things won't ever happen. I observe so much work, which from the features viewpoint is really close, being repeated in yet another language (a multitude of languages from the past and present, now it's JavaScript, next is Rust), and this goes over and over, and this is depressing. A good example from today: I searched a dependency tree graphing tool for ES6 JS, and found a good one written in Node; why isn't there a tool for just dependency graphing (e.g. I need one for JS components and C++ components and HTTP APIs), well, because JS community won't write a generic tool, and other community won't likely write a generic tool that also supports JS. Same with package managers. Rust community made a package manager for Rust. Node community makes one for themselves. All right, wrapping up, thanks for listening.

Copy link

@Pauan Pauan Sep 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I observe so much work, which from the features viewpoint is really close, being repeated in yet another language (a multitude of languages from the past and present, now it's JavaScript, next is Rust), and this goes over and over, and this is depressing.

I agree, but I don't think the solution is npm / package.json, instead it's something like Nix or 0install. Nix already has support for both npm and Cargo (and you can mix and match them in the same package).

In any case, it's not the Rust Wasm working group's job to push for Rust users to move to a different package manager (we are only responsible for WebAssembly, not Cargo or the overall Rust community), so the solutions presented in this RFC are the only viable ones for us.

All right, wrapping up, thanks for listening.

Thanks for sharing! I think it's good to get multiple perspectives on this issue, including from people coming from a non-Rust background.


To express npm dependencies, add a table to your `Cargo.toml` called `npm`. This table will have a key-value store of the dependencies you would like to use. The key is the name of the dependency followed by a value that is consistent with dependency values as specc’d in [this document](https://docs.npmjs.com/files/package.json#dependencies), many of which are demonstrated in the below example.

```toml
# Cargo.toml

[package]
#...

[dependencies]
#...

[npm]
moment = "~2.22"
#sugar for moment = { version: "~2.22", type: prod }
mocha = { version: "mochajs/mocha#4727d357ea", type: dev }
chai = { version: "^4", type: dev }
optional = { version: "6.6.6", type: optional }
git = { version: "http://asdf.com/asdf.tar.gz" }
```

### New Manifest File Format
*We rejected this outright based on inherent complexity, community exhaustion, and its lack of interoperability with JavaScript tooling.*

### Inline Annotations
*This was the original solution that was implemented. It was good because it worked equally well with Rust and other languages such as C or C++. It was not good because it added high management complexity and lacked operability with JavaScript tooling.*

This would look like this, and have no external manifest file:

```rust
#[wasm_bindgen(module = "moment", version = "2.0.0")]
extern {
type Moment;
fn moment() -> Moment;
#[wasm_bindgen(method)]
fn format(this: &Moment) -> String;
}
```