discussion: Make import maps stable

[kitsonk]

Based on a discussion in Discord, @shian15810 suggests that marking import maps unstable is going to be problematic (as well as brings it up in #4933 but it should have its own issue).

I am torn, the spec only has one implementor (Chromium 74+) and Firefox and Safari as well as other runtimes have no public signals to support it. We implemented it because it solved a specific problem that we didn't want to solve in a non-standard way. The spec repo though indicates there are a few critical risks/issues with the current spec: https://github.com/WICG/import-maps#further-work. None of those really are worrisome, except that support for how multiple maps are dealt with. At this point we don't support that feature (because the spec doesn't) and the semantics are likely to be breaking to the current single file support, in my opinion.

Given it sits unflagged in Chromium is an indication that they do not expect there to be breaking changes with it as the standard evolves (though they have been wrong about that before).

As @shain15810 points out, it is effectively opt-in at the moment anyways, so having two flags to be able to enable a single flag does seem non-sensical. If anything, we could warn when using the flag, that the spec isn't finalised and semantics may be broken in the future.

[shian15810]

Thanks @kitsonk for opening this issue!

There are some major discussions around dependency management, and the general consensus is that this should be done in userland rather than built into deno.

In general, my point is that flagging this as unstable will be a major blocker for future userland innovations in dependency management solutions.

As @shain15810 points out, it is effectively opt-in at the moment anyways, so having two flags to be able to enable a single flag does seem non-sensical. If anything, we could warn when using the flag, that the spec isn't finalised and semantics may be broken in the future.

I think this is the right way to go.

[lucacasonato]

Given it sits unflagged in Chromium is an indication that they do not expect there to be breaking changes with it as the standard evolves (though they have been wrong about that before).

@kitsonk It is not unflagged in Chromium. It is only enabled with the experimental-web-plaform-featues flag. Currently there is no way to use it except for by manually enabling experimental-web-plaform-featues.

As @shain15810 points out, it is effectively opt-in at the moment anyways, so having two flags to be able to enable a single flag does seem non-sensical. If anything, we could warn when using the flag, that the spec isn't finalised and semantics may be broken in the future.

I think requiring two flags makes it very clear that this is a experimental and unstable feature. Everything that is not definitely stable should be behind the unstable flag.

[shian15810]

Double flagging --importmap with --unstable effectively means that those opt in to import maps is forced to be greeted with all unstable APIs. This is unfair to those that intent to opt in only import maps but not other unstable APIs.

[domenic]

I want to second @lucacasonato's observation that this is not unflagged in Chromium; it is behind the experimental web platform features flag. That said, I believe the current explainer / spec / reference implementation reflect something that we are willing to ship, when we are able to prioritize it. I will defer to @hiroshige-g for more information on that.

None of those really are worrisome, except that support for how multiple maps are dealt with. At this point we don't support that feature (because the spec doesn't) and the semantics are likely to be breaking to the current single file support, in my opinion.

I don't see how they would be breaking, especially if (unlike the web) your environment has no current way to feed in two import maps. Feeding in one import map would not change behavior.

The other thing I want to point out is that some features that were in the initial spec were removed, such as fallback support and built-in module support. I'm not sure Deno has kept pace with that.

[bartlomieju]

The other thing I want to point out is that some features that were in the initial spec were removed, such as fallback support and built-in module support. I'm not sure Deno has kept pace with that.

Thanks for pointing that out @domenic, we should revisit our implementation. Built-in modules are not supported, but I think we got fallback working.

[lucacasonato]

Double flagging --importmap with --unstable effectively means that those opt in to import maps is forced to be greeted with all unstable APIs. This is unfair to those that intent to opt in only import maps but not other unstable APIs.

@shian15810 Well Deno currently does not have a multi flag experiment system, only the single unstable flag. So for now everything that is unstable is under one flag, to keep the experience uniform. I don't think we should make an exception for importmap, just because it is more fleshed out than other things. Currently the focus is on getting 1.0 out the door. That means locking everything that is not set in stone behind a flag. Later --unstable could turn into multiple individual flags.

[shian15810]

Currently, since deno doesn't have a dependency management solution built in, there are virtually tons of possible ways that I can think of to make dependency management possible in userland, be it a complete custom solution or a solution that adhere as closely as possible to the available specs. At least for me, I think using import maps is the most spec-adhering and elegant way to make this feasible.

If dependency management is destined to be done in the userland, after 1.0 is released, I can see that window of time will be the perfect breeding ground for lots of innovations on dependency management solutions to pop out. However, with --importmap behind the --unstable flag, library authors could be forced to invent some custom solutions. What concerns me is, there is a possible risk that a solution that doesn't adhere or adhere loosely to specs might gain a lot of traction, and then becoming de facto standard for the community.

Well, you could argue that deps.ts is just a source file and it just have to be a valid TypeScript, and all dependencies can be listed in it. Well, if a project is big enough with tons of dependencies, flattening every dependencies into a single module and then exporting everything is going to make this file a horror. What about dead code elimination? What about tree shaking? What about naming conventions for exported dependencies? What if every dependencies has a lot of files to be exported? What about a dependency manager that bundle source code and dependencies independent of deno? What about a dependency manager that does a lot of magic with deps.ts? There can be a lot of conventions to choose from.

You could argue about anything I have said about deps.ts, maybe what I said are just nonsense, and I could also argue about anything you have to say about deps.ts and how you are going to do with it or use it. Also why don't we just stop there?

Import maps has given us a spec/convention to solve problems like this, and I think we should embrace it rather than to create more problems like this in the future.

Anyway, I could be wrong, only time will tell.

My general point is still the same, please don't make this a blocker behind --unstable. --importmap has already implied an intention to opt in. In any time should import maps be deprecated as specs evolve, just remove it alongside major version bump. This is understandable as deno philosophy of adhering to specs. Deno has no obligation to support it forever if it dropped out of specs.

[dobbs]

I don't have an answer for your conversation, but I do have a cautionary tale. Every programming ecosystem I've worked in has built their own dependency management system and had it grow into something mostly broken after five to ten years. Language after language has set out with the idea that they could improve on the problems from the previous language, and every one of them has only managed to create a slightly new kind of mess. Please do not underestimate the essential complexity of dependencies. A simple map of names to versions does not work in the long run.

When a project is young, everything is small and easier to think and reason about. In a few years it'll all be a tangled mess. The way people solve dependency management when things are small make perfect sense for the small, and then fail as things become big and sprawling.

All languages that came after perl had to invent something that was "better than CPAN". I admit that python's example supports @shian15810's worry: the absence of some package management thing will lead to a proliferation of options.

Even so, I really liked coming to deno and finding a community just using the features of the language itself.

Tangent: Contrast this with java where in the early days, the language itself was so limited on its own it spawned dozens of xml-based programming languages to make up the slack (jsp, ant, velocity, jelly, eventually spring configs). Since I know you don't know me at all, I was in the corner of the jakarta project where maven was born. I'm not just talking theory here. The guys who built maven were really smart and really capable and maven nevertheless became a mess that people raged against—might even have inspired the exodus of developers from java to ruby.

@ry made such a great case in his reflections on mistakes in node for including file suffix in the import and for just using the browser standard of importing from a url instead of having a separate packages.json.

import_maps are the exact same shape as dependencies in packages.json. True, node has proved you can go a long way with that. But also shown the mess that's become of our software supply chain.

For future projects with sprawling code bases I imagine starting with deps.js (thinking .js instead of .ts for the parts that need browser compatability) and growing into something like this (borrowing the example from https://github.com/WICG/import-maps#the-basic-idea):

import moment from "./deps/moment.js";
import { partition } from "./deps/lodash.js";
...

# where moment.js looks like this
export * from "https://cdn.jsdelivr.net/npm/moment@2.24.0/moment.min.js";

The specific organization and grouping of dependencies within ./deps would be project specific. For especially long-lived projects it is a valuable practice to wrap up all the dependencies in project-specific modules anyway. That way the team can control the blast radius in their code base in the never ending upgrade of dependencies.

[shian15810]

@dobbs Thanks for such a detailed feedback!!! I don't mean to sound like my perspective is future proof. I'm just trying to point out what problems have import maps solved.

The declarative way of listing dependencies and their semver constraints in package.json is also what other languages did, like Cargo.toml in rust or Pipfile in python.

The problem of package.json does not lies in package.json itself, but its resolving strategy that creates the renowned dependency hell. When a package.json has some declared dependencies, it must depend on all its dependencies' package.json, and when these dependencies' package.json has some other declared dependencies too, this cycle must go all the way down until all package.jsons are resolved.

However, import maps work on application level only, which means an application-level import_map.json doesn't depends on the import_map.json of its dependencies. To quote https://github.com/WICG/import-maps:

Import maps are an application-level thing, somewhat like service workers. (More formally, they would be per-module map, and thus per-realm.) They are not meant to be composed, but instead produced by a human or tool with a holistic view of your web application. For example, it would not make sense for a library to include an import map; libraries can simply reference modules by specifier, and let the application decide what URLs those specifiers map to.

Expanding on your suggestion on ./deps/ directory, do you mean mirroring the same directory structure of remote dependencies in local ./deps/ directory, with every corresponding file exporting its own module? If that is the case, lodash and moment also expose a lot of other files that can be imported, as can be seen in https://cdn.jsdelivr.net/npm/lodash@4.17.15/ and https://cdn.jsdelivr.net/npm/moment@2.24.0/. From a dependency manager point of view, any valid js and ts files in there should in theory be made importable in deno, since each of them can be linked by a unique and valid url. To make them importable, all of these files will have to be mirrored locally. This would be overkill for just a single dependency IMO.

Import maps provide quite an elegant and declarative way to solve this, for example:

{
  "imports": {
    "lodash": "https://cdn.jsdelivr.net/npm/lodash@4.17.15",
    "lodash/": "https://cdn.jsdelivr.net/npm/lodash@4.17.15/",
    "moment": "https://cdn.jsdelivr.net/npm/moment@2.24.0",
    "moment/": "https://cdn.jsdelivr.net/npm/moment@2.24.0/"
  }
}

import _ from "lodash" and import { foo } from "lodash/bar.js" will both be valid imports. Any files under lodash/ can be imported with just a trailing slash.

[dobbs]

I think rather than try to expand my own gesture at a solution, I'll point to this long article that captures the sociotechnical complexity of dependency management:
https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527

I'm undecided about the specific recommendations the author makes for go (for example his call for a central repository), but his description of the problem is pretty thorough.

[shian15810]

@dobbs Thanks for the article, it is a long but breathtaking read nonetheless!

I can understand why the author recommend a central repository for golang. To quote:

Well, I’m skipping how to build a registry, but not their functional role. By acting as package metadata caches, they can offer significant performance benefits. Since package metadata is generally held in the manifest, and the manifest is necessarily in the source repository, inspecting the metadata would ordinarily require cloning a repository. A registry, however, can extract and cache that metadata, and make it available via simple HTTP API calls. Much, much faster.

As a premise, under current deno module resolution algorithm, any valid URL that can link to a valid JS/TS esmodule is considered importable and can be consumed by deno.

One of the early problems that I found when developing a dependency manager for deno is that when pulling metadata and manifests from a repository, i.e. github, is very time consuming, with a lot of back-and-forth APIs fetching, just to gather enough information to do semver matching, with lots of sanitization rules involved. Also there is no forced standard on a repository to support any language/runtime, and some repository under https://deno.land/x/ don't even has a release or some even don't support semver. So a registry would definitely help a lot in that, especially to keep consuming and publishing of packages in sync. With just a single GET from registry API, no sanitization is needed, and all informations required for installation is provided, how convenient is that. Also a registry could prohibit deleting any version a package has published before. However, in a repository like github, deleting a release is only some clicks away. What to do when a semver range could only match a version and that version got deleted? Or even worse when the whole repository is deleted? So in the future, I could imagine a possible situation where a popular deno package consumed by a lot of people hosted under github got deleted and possibly breaks a lot of automation like CI/CD. IMO, a central registry provides lots of advantage except that who has the authority over the registry.

I would call it a dependency manager rather than a package manager as it currently only concerns about application-level dependencies only, not library publishing. To support library publishing, that would require a central registry to enforce some rules, rather than conventions that spread out over github repos. A convention could be broken by any party, and that will break a lot of things, no one will be happy about it.

However, I am still very optimistic about the current situation without a central registry. Any valid URL that can link to a valid JS/TS esmodule is considered importable and can be consumed by deno, this is elegant enough, just as in the browser. Your website will break if the url of your scripts return 4xx/5xx anyway. Without a central registry, the responsibility of a missing package or version is outsourced to the package maintainer itself.

In my imagination, without a central registry, a deno dependency manager would ultimately focus on application-level dependencies only. Library publishing is as simple as putting your source code online with a reachable URL.

Back to the article, to quote:

For ahead-of-time compiled languages, PDMs are sort of a pre-preprocessor: their aggregate result is the main project code, plus dependency code, arranged in such a way that when the preprocessor (or equivalent) sees “include code X” in the source, X will resolve to what the project’s author intended.

The ‘phase zero’ idea still holds in a dynamic or JIT-compiled language, though the mechanics are a bit different. In addition to laying out the code on disk, the PDM typically needs to override or intercept the interpreter’s code loading mechanism in order to resolve includes correctly. In this sense, the PDM is producing a filesystem layout for itself to consume, which a nitpicker could argue means that arrow becomes self-referential. What really matters, though, is that it’s expected for the PDM to lay out the filesystem before starting the interpreter. So, still ‘phase zero.’

For most languages, there is a path in file system where the language will look for dependencies/modules/packages. So a package manager can take advantage on that and prepare everything in there for the language compiler/interpreter to consume. This is not the case for deno though.

Currently the only way to tap into deno module resolution algorithm is import maps. Without import map, there is no way to tell deno that I want to import a module named deno_std:fs, don't bother about where it comes from or its version, just import it for me, I will manage its url and version independent of you.

We can all agree that a central manifest file is essential to manage a project in the long run. So the responsibility of a deno dependency manager would be to generate deps.ts or import_map.json from a manifest file. Generating an import map is easy, there is a spec to be followed. Generating a deps.ts though is going to be troublesome as stated in the previous comment. The biggest issue with this is all dependencies are flattened into a single module. The suggestion of ./deps/ directory though would essentially mirror the whole remote directory structure locally.

Since there is no way to tap into deno module resolution algorithm, another approach would be for the dependency management tool to do some transpilation of converting dependency names to url before feeding the source code to deno. IMO this is a very ugly way to do it, but would be logical if import maps is not available. Somewhat like https://yarnpkg.com/cli/node.

[dobbs]

I'm afraid I've lured us away from the original topic of discussion.

The question remains whether import_maps should remain behind the unstable flag or given its own flag or returned to be marked stable.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kitsonk]

Chromium will be shipping this unflagged: https://groups.google.com/a/chromium.org/g/blink-dev/c/rVX_dJAJ-eI/m/17r-6-eiAgAJ

We should consider stabilising it now.