Thoughts about wing’s library ecosystem

[eladb]

(working backwards)

Wing Libraries

A wing library is a collection of types that can be used by other wing libraries or apps.

Open-source supply-chain attacks are becoming one of the most common and dangerous attack vectors in the industry. To increase the trust, safety and quality of the wing library ecosystem, wing takes a unique approach to package publishing.

The basic idea is that "trusted" winglang libraries are normal npm packages that are published under the @winglang scope, and their code is hosted under the winglang github org. These libraries can be authored by anyone but they are always published by the winglang project and not their authors.

This serves two purposes:

1. It simplifies publishing. The wing publishing system takes care of release management, versioning, backwards compatibility checks, changelogs, etc.
2. It increases trust and quality of the ecosystem using a low friction peer-review system.

We need to decide on the right name that sets the right expectation: "trusted", "community", "approved", "peer-reviewed", something like that. For the sake of this document, I'll call them "trusted".

Let's walk through the user experience of installing and publishing trusted and untrusted wing libraries.

Installing wing packages

To install a trusted wing package, use:

wing install redis
# or
wing i redis
# or
wing i redis@^2

Use it like so:

bring redis;
bring cloud;

let db = new redis.Database();

new cloud.Function(() ~> {
  db.hset("my_hash", "my_key", "my_value");
});

Under the hood, wing uses npm to install the package @winglang/redis into your package.json.

Publishing wing packages

To publish a new wing library, all you have to do is simply submit a pull request to the https://github.com/winglang/libs repository.

This repository includes a directory for each library, and is already set up to take care of builds, tests and releases. Just put your code in the right place and we'll take care of the rest.

Your pull request will be go through a quick review by one of our community members. The purpose of this review is to help you make your library awesome and that it meets the wing standards. Once reviewed, it will be merged and immediately released.

In the future, we will add support for self-service creation of new repository in the winglang github organization, and each library will have its own repository, but for now a single mono repo will make things much easier for everyone and will increase the cohesiveness of the ecosystem.

Untrusted libraries

Again, name pending.

As mentioned above, winglang libraries are simply npm (JSII) packages, so anyone can publish anything to npm.

You can then use npm to install this library:

npm i my-wing-lib

And use bring like normal:

bring "my-wing-lib" as mwl;

mwl.print_hello();

But:

$ wing compile hello.w
ERROR: trying to bring untrusted library "my-wing-lib". Use --allow-untrusted=my-wing-lib to allow.

So you'll have to explicitly opt-in to allow this library:

$ wing compile hello.w --allow-untrusted=my-wing-lib

[3p3r]

Looks good to me. One note: this does not really contribute to supply chain security because these wing modules may be bringing third party other modules under them and they're all hosted on NPM (currently) and we don't have control over npm (and it's not audited by us).

So in case of an actual supply chain attack, if a package is compromised, we can't do anything about it.

It would contribute to supply chain security if we came up with a process to freeze every single package and all its dependencies on release, that would be a valuable addition.

Thoughts on naming: maybe we call it "side loading". So everything under @winglang is verified and you can side load your non verified packages.

[Chriscbr]

One note: this does not really contribute to supply chain security because these wing modules may be bringing third party other modules under them and they're all hosted on NPM (currently) and we don't have control over npm (and it's not audited by us).

I think if we impose that libraries in @winglang/ must bundle all of their dependencies inside, and the maintainers of winglang/libs ensure no malicious package versions are added during routine dependency updates, then I do think this mechanism would protect consumers from supply chain attacks.

[3p3r]

I think if we impose that libraries in @winglang/ must bundle all of their dependencies inside

Yep, that's pretty much what I'm saying. the process needs to be explicit about that. Because if something slips in with a dependency on anywhere else other than @winglang hosted packages, it defeats the purpose.

[MarkMcCulloh]

This feels like security theater because at scale it's going to be all rubber stamps. So if we want this from a marketing perspective then it makes sense but I'm not convinced of the technical merits. My last interaction with something like this was the Jenkins ecosystem, but they at least controlled all dimensions (GitHub org and the actual distribution via a global Artifactory setup)

The more rules/strictness we add the safer it would get of course, but at the end of the day there's only two targets:

1. For those who care about this security: They wouldn't fully trust us anyways and will have their own supply chain mitigations (probably even for the wing toolchain itself)
2. For those who don't care: They would only see the friction

[MarkMcCulloh]

It simplifies publishing. The wing publishing system takes care of release management, versioning, backwards compatibility checks, changelogs, etc.

This part I would normally be on board with, but I really appreciate what projen was able to do with this and I think it would be really cool to continue leaning into that (but now with wing itself as the new projen :) )

[Chriscbr]

This part I would normally be on board with, but I really appreciate what projen was able to do with this and I think it would be really cool to continue leaning into that (but now with wing itself as the new projen :) )

Agreed, in the long term it makes more sense to me to have release management, versioning, backwards compatibility checks, changelogs, etc. as part of the Wing toolchain, not part of a centralized GitHub org. But I do see the merit of having our entire library ecosystem in a monorepo for a v0 "Wing library ecosystem" to help us iterate quicker, and then extract parts out as they stabilize.

[eladb]

I also see this is a healthy way to kick off a healthy ecosystem, and then we can decide on the best way to scale.

It is always possible to publish independently (and we can decide on the level of friction for that), but I have a sense people will appreciate the reduced barrier to publishing.

Projen reduces a lot of the cruft but there are still aspects like token management and rotation that's out of scope for projen, especially if we want to publish to multiple languages.

Another benefit of centralized publishing is that we can control the mechanics and experiment with ideas such as bundling/freezing dependencies across the entire ecosystem.

I agree that this won't hold water for thousands of packages but it could be a great way to start with a higher bar for our ecosystem.

[ekeren]

I would separate the need to allow people a very simple way to publish (versioning, backwards compatibility checks, changelogs, etc) vs promising/validating trust.

The people that will create "untrusted" libraries still needs all of the above.

I would considering separating this into two different discussions:

• Should/How can we ensure trust in specific libraries
• How to ensure good hygiene when publishing wing libraries

It might end up as the same mechanism

[laSinteZ]

Have you considered Deno's module system?
They have:

1. std library (which is a set of modules reviewed by Core team w/o external dependencies)
2. third-party modules (written by community, no guarantees)
3. NPM

I guess we can mark packages which depend only on "standard library" (which is reviewed) as "semi-trusted" packages (or packages w/o "External" dependencies). And well, packages which rely on "semi-trusted" packages as "semi-semi-trusted (lol)". And well, NPM is your wild west, proceed with caution.

I am also not sure if it will be realistic to review all the packages any deeper than

Well, it doesn't say import bitcoin_miner from "bitcoin_miner", lgtm"

anyway, which will provide a false feeling of security to end users – do we want that?

[3p3r]

Here is what I think about this in terms of future plans: Assuming that we'll have a WebAssembly runtime that only implements node core modules that WingSDK needs (crypto, buffer, fs, path, process, and console), we can have Wing modules execute inside an isolated sandboxed runtime in WebAssembly, to make sure it cannot breach out to its environment, even if there is an import bitcoin_miner from "bitcoin_miner" somewhere.

Controlling this sort of thing is really easy going through something like WASI. I like the Cap based module system of Deno, and I like to apply it at runtime level to actually enforce module resolution and not just do some static analysis and say, "well I could not find any import bitcoin_miner from "bitcoin_miner" so it must be good and secure"

What Elad suggests, lays the foundation towards that goal. If we had some sort of pipeline that all our libraries went through, then making Wing libraries compatible with Wing runtime should be a matter of changing build scripts in a single place. So library authors just write normal JS libraries, and our pipeline manages building into JSII or webpackify'ing it so it can go into the WASM runtime in the future.

[AdamTylerLynch]

Requiring the team who writes the code to modify the CLI command parameters for compilation feels a bit out of bounds. I have worked in many environments where the folks who write code can not mutate the build chain in any meaningful way.

I would suggest an additional ability to have a package.conf or another type of package configuration file where one can explicitly allow untrusted packages/libs.

Potentially adding a feature allowing for compile overrides in the form of a chain (env_vars, paclage_conf, CLI params).

[eladb]

Related: https://twitter.com/rybickic/status/1603622787087994880