[RFC] Import query params

[padmaia]

Problem Statement

Currently in Parcel v2, dependencies are resolved to files and the resulting file paths are matched against the configured globs in the .parcelrc file to see which pipeline the file should be transformed by. This works for most cases, but sometimes you might want to force a dependency down a specific pipeline. For instance, let's say you are building documentation for a React component. By default, importing the React component will bundle the component as a JavaScript module, but what if you additionaly want the component's source as plain text? Maybe you also want to run some transformer over the component that extracts what prop types it expects and bundle that as well.

A very naive solution that would work for any and all tooling would be to just create files for the source text and prop types in a prebuild step and then you can import three individual files. The downside to this approach is that you would then have two build steps, which means you have two build commands that you have to string together and two separate watchers if you want to build in watch mode. Imagine in this scenario you updated MyComponent.js. Both the prebuild and the bundle watchers would pick up the change and start rebuilding. Then when the prebuild step finishes, the derived files are updated, which causes the bundler to rebuild yet again. It makes for a much better developer experience if this can all be handled in one process.

With webpack, this is usually solved by using inline loaders.

import MyComponent from './MyComponent';
import MyComponentSrc from '!!raw-loader!./MyComponent';
import MyComponentPropTypes from `!!extract-react-types-loader!./MyComponent`;

One downside to this approach is that the detail that you are using webpack leaks into your source code. Ideally this could be done in a more bundler agnostic way.

Proposed Solution: Import Query Params

The ECMAScript modules standard (ESM for short) is now supported across all modern browsers and will soon be supported by Node.js as well. One neat thing about ESM is that it essentially treats all module specifiers as URLs, which means adding query parameters is perfectly valid syntax and not something bespoke to a specific bundler. This seems like a great way to specify how you would like a dependency loaded:

import MyComponent from './MyComponent.js';
import MyComponentSrc from './MyComponent.js?as=raw-text';
import MyComponentPropTypes from './MyComponent.js?as=react-types';

So what would it take for Parcel to support this? The most straightforward approach requires no changes to Parcel's configuration so it's got that going for it. All you would have to do to force a file to be transformed by a different pipeline is add a query param to the module specifier and configure a pipeline in your .parcelrc with a glob that matches it.

{
  "transforms": {
    "**/*.js\?as=react-types": ["parcel-transformer-extract-react-types"]
  }
}

The only change we would need to make in Parcel would be to slice off the query params from the module specifiers to resolve the file and then tack them back on to resolve the transformation pipeline. Perhaps it would be nice to have a special glob matcher that makes it easier to write globs of this type. Query params could also be passed on to transformers as well.

import profilePic from './profile.png?width=300&height=300';

Drawbacks

• Since this solution is based on ESM it makes it pretty JavaScript centric. Some other languages and module formats support module specifiers as URLs, but what about the ones that don't? Parcel will either end up encouraging developers to write non standard code that only works because it's run with Parcel or this behavior may not end up being supported in all types of files.
• Text matching can get hairy. It seems possible that this could cause Parcel config files to balloon especially considering there could be multiple query parameters that might influence what pipeline a file should be transformed by.
• This is more of a drawback for the transformation configuration model in general, but one thing it doesn't accommodate is influencing what pipeline to use based on the file that defined the dependency. For instance you might want something different when importing as=url from a JavaScript file vs a CSS file. This could be solved by using a more specific parameter like as=js-url, but the problem still remains for imports without query parameters. If we decided to change the configuration to allow for this type of thing then it could potentially lead to a cleaner/smaller/simpler config.

Alternatives

One alternative would be to make a change to Parcel's configuration. I do like how simple Parcel's configuration is at the moment, but I could imagine this not working perfectly for all cases. I'm not aware of any of these cases at the moment, but that's what the RFC is for. If no one brings up a compelling case that proves that a change to configuration would either be necessary or would be a much cleaner solution, then this seems like a reasonable place to start.

Another alternative could be to use a special extension like extensions leading with a certain number of underscores.

import MyComponent from './MyComponent.js';
import MyComponentSrc from './MyComponent.js.__raw';
import MyComponentPropTypes from './MyComponent.js.__react-types';

Parcel could split the module specifier like it would have to with query parameters and use the parts before the special extension to resolve the file and tack the special extension back on to resolve the pipeline. You could also have a prebuild step that actually creates these files which means the imports would still work with other tooling. One potential issue is that the special extension might conflict with legitimate extensions.

Open Questions

• ESM treats module specifiers as URLs but CommonJS does not. Should Parcel only support this feature on module specifiers that are known to be URLs or should it be less strict?
• Should this be supported in node_modules or only source code?
• What happens when the configured transformer changes the type of the asset? Do we use the query parameters to match the next pipeline or do they only apply to the first pipeline? (Seems like you would probably have to use them to match if you wanted any control over whether to jump to a new pipeline or not)
• Any examples where this type of configuration doesn't cut it?

cc @devongovett @wbinnssmith @jamiebuilds

[mischnic]

Should this be supported in node_modules

I think we shouldn't encourage publishing packages with this (currently) non standard code, at least in the beginning.

[devongovett]

ESM treats module specifiers as URLs but CommonJS does not. Should Parcel only support this feature on module specifiers that are known to be URLs or should it be less strict?

I think we should only support query params for specifiers that are URLs, so for JS, only ESM. CSS and HTML are all URLs, but langs like SASS and LESS may not be. Some research required. CJS shouldn't be supported - ? is a valid character in a filename.

Should this be supported in node_modules

Agreed with @mischnic.

What happens when the configured transformer changes the type of the asset?

I think it should only apply to the first pipeline, and should be stripped off afterward.

[devongovett]

I think in order to make the config easiest to use, we shouldn't require escaping the query params. We can build a custom matcher, that strips off the query param first and matches the glob on the left side, and then checks the query params separately if it matches.

[devongovett]

It would also be pretty nice if config like this worked:

{
  "transforms": {
    "*.png": ["parcel-transformer-png"],
    "**?as=url": ["parcel-transformer-js-url"],
    "**?as=base64": ["parcel-transformer-js-base64"]
  }
}

Currently, that wouldn't work - for a PNG file, only the .png pipeline would run, and not the url or base64 pipeline. It could be made to work if you configured the order differently:

{
  "transforms": {
    "**?as=url": ["...", "parcel-transformer-js-url"],
    "**?as=base64": ["...", "parcel-transformer-js-base64"],
    "*.png": ["parcel-transformer-png"]
  }
}

The ... would cause the next matching pipeline (*.png) to run first, and then continue with url or base64. But that's kinda confusing vs the first.

Any ideas on how we could make this work?

[rtsao]

It makes for a much better developer experience if this can all be handled in one process.

One alternative is to expose a Parcel API for conveniently allowing transformations to write files to disk. This way it's still a single process with a single command and single file watcher, but there's zero "magic" in the JS module bundling phase as only JS files are importing other JS files (as opposed to arbitrary file types). This seems the most bundler agnostic solution, as any other bundler could pick up these plain JS files.

One huge advantage to on-disk code generation is generated code can be statically typed. Type checkers are not going to be able to understand query parameters. However, the transformer could output a file on disk that any type checker will be able to understand.

The main potential drawback I can see is passing configuration at the import site might not be possible (as the files would have to exist already) -- so the config would have to be centralized. This might not necessarily be bad, though. The other drawback is potentially extra work compiling files before they are used. However, this might be avoided using conventions around filenames where transformed files are only generated on-demand.

[jakearchibald]

My opinions here may not fit with what Parcel is trying to achieve, so feel free to ignore.

I don't think build systems should hide. If a file is to be transformed in an uncommon way, it should be obvious from the import that something special is happening.

import text from './whatever.js?as=text';

The above doesn't really look special. It's just a regular URL.

If the above is passed to a system that doesn't recognise the 'as' query parameter, it'll interpret the module as JavaScript, which feels like a bad fallback. If the transform doesn't have an 'acceptable' fallback (if a file fails to minify, the result still works), the import should fail as soon as possible.

Other ideas:

import text from 'as-text:./whatever.js';

The as-text scheme is invalid in module identifiers, so it's obvious something special is happening. Without the transform, this will fail at the parse stage.

import text from './whatever.js/text';

This treats whatever.js as a directory, when it isn't. Transforms would define what a file could 'contain'. Without the transform, this will fail at the fetch stage. That might be too late, and the special nature of the import may not be obvious enough.

• Should this be supported in node_modules or only source code?

I'm not sure if files within node_modules should support this syntax. However, I think files outside of node_modules should be able to use it on files within node_modules.

Eg:

import text from 'whatever-package?as=text';

[devongovett]

That's interesting feedback @jakearchibald! One reason we were thinking about query parameters was to avoid making as "special". It's just an option that is passed to transformers, like other query params (e.g. image width/height). It could be called type or banana for all Parcel core cares.

If we went with something more specific like a URL protocol, it would also affect resolution in addition to transformation and would need to become a bit more integral to Parcel core. But I can see your point about fallback in case a build system isn't configured to use as vs failing early. We can think about that further.

As for the specific suggestions, protocols seem like an interesting idea, but feel a bit like a "source" rather than a transform. e.g. normal protocols like http:, file:, and even std: modules are all sources - you cannot request the same file from different protocols generally. The directory idea is hard as well because it requires cooperation from the resolver plugin as well as the transformer.

[jamiebuilds]

There's a good point made that "falling back" to JavaScript when ?as=text is not supported, is actually worse than not resolving to anything. If you're going to fail, fail early.

A couple constraints:

• It needs to be specified as part of a "URL" (or rather a string-based module specifier), we can't rely on additional syntax being available in the language, because it very well might not exist.
• We need to stick as close as possible to what is considered "normal" in a language, even if that means not supporting every feature everywhere.
  • Ex: Don't try to make Rust support importing JavaScript
  • Ex: CSS has a way to refer to images with url(...) so push people that direction.
• We can't use protocols:// because we allow importing URLs in many forms
• I think there needs to be an abstraction between module specifiers and transformer code. i.e. !raw-loader! is a bad solution because it specifies the transformer directly.

I was in favor of query parameters before, but @jakearchibald made a good point that unknown query parameters rarely change how a route is resolved by a server.

Right now I'm leaning more towards the additional extension:

import "./file.js.txt"

This has the benefit of reusing existing configuration:

{
  "*.{txt,text}": ["@parcel/transformer-raw"] 
}

The problem comes in trying to resolve it. How do you decide ./file.js.txt should resolve to ./file.js?

The answer could be stat-ing basename(name) recursively until you resolve to something, but I haven't seen anything that behaves like that before.

[jakearchibald]

@devongovett

As for the specific suggestions, protocols seem like an interesting idea, but feel a bit like a "source" rather than a transform. e.g. normal protocols like http:, file:, and even std: modules are all sources - you cannot request the same file from different protocols generally.

The only exception I can think of is view-source:http://example.com/ which a few browsers do.

@jamiebuilds

Right now I'm leaning more towards the additional extension:

Yeah, that could work. Would it be chainable? Say I had two transforms:

• Optimise image data
• Data metadata for image (width, height, file size etc)

…would I be able to do:

import data from "./file.png.img-opt.img-data";

[jamiebuilds]

For that specific case I don’t think we’d encourage the use of optimizers via file extension vs Parcel’s config format:

{
  "transforms": {...},
  "optimizers": {
    "*.png": ["@parcel/optimizer-pngmin"]
  }
}

However for the generic case of chaining I’m a little bit torn.

There is some precedence for this in build systems as well, Ruby on Rails’ asset pipeline uses extensions to turn on different file transforms (although that’s at the file system level, not at import-time).

We’d have to consider how the Parcel pipeline interacts with this:

import "./img.png.img-data"

If the config were to look something like this:

{
  "transforms": {
    "*.png": ["@parcel/transform-png"],
    "*.img-data": ["@parcel/transform-img-data"],
  },
  "optimizers": {
    "*.png": ["@parcel/optimizer-pngmin"]
  }
}

There’s a lot of questions about which plugins run and when they run:

• Does the png optimizer run before you get the image data?
• Should the *.png match .png.img-data?
• Should png transforms/optimizers be run at all? Maybe it makes sense in this case but if you were importing .js.raw-source would you want Babel to run first? (Probably not)

I’m not sure we can come up with a pipeline algorithm that will work for all cases. In which case I’d prefer forcing users to do a little more work to be explicit than to have them be frustrated with a bad abstraction.

[jamiebuilds]

Vague late night thoughts:

• I think we need a new plugin type for "viewers" (or something like that).
• This new plugin type would apply only at the import level.
• You would only be able to turn on one viewer per import.
• The viewer plugin would have (declarative) control over the transforms/optimizers to be run (level of control needed should be explored)
• I've flipped on file extensions, I was pulling it from RoR before, but thinking more about the requirements, I don't think it works that well.
• I vaguely like view-source:http://... but I think it implies it's a standard of some sort
• I actually think that Webpack's loader!./file syntax would be nice here, but I explicitly want to use a different syntax than them to avoid feature confusion.

Some ideas on syntaxes other than loader! that don't conflict with the URL standard or what I've seen in any programming language import specifiers, and which will fail to load if you try loading them outside Parcel:

import "viewer~specifier"
import "viewer^specifier"
import "viewer|specifier"

Example

import "image-data~./file.png" // runs optimizer-png, but not transform-png
import "raw-source~http://example.com/script.js" // doesn't run any optimizers or transforms

{
  "transforms": {
    "*.js": ["@parcel/transform-babel"],
    "*.png": ["@parcel/transform-png"]
  },
  "optimizers": {
    "*.js": ["@parcel/transform-uglify"],
    "*.png": ["@parcel/optimizer-pngmin"]
  },
  "viewers": {
    "image-data": ["@parcel/viewer-image-data"],
    "raw-source": ["@parcel/viewer-raw-source"]
  }
}

[Morgul]

Just a random thought.

Personally, I dislike having to change my import statements because of a bundler. Even the query parameter is annoying, because it feels like I'm mixing the concern of loading modules with the concern of bundling code. They're related, but should be more explicitly separate, imho.

Combined with the concern that no all languages support urls for their module imports, the solution seems pretty clear:

• Put the bundler configuration in a comment.

Here's the same examples from the OP:

import MyComponent from './MyComponent.js';

// bundler: { as: 'raw-text' }
import MyComponentSrc from './MyComponent.js';

// bundler: { as: 'react-types' }
import MyComponentPropTypes from './MyComponent.js';

Simple, easily understood, not bundler specific, and supported in all languages with comments. (Syntax obviously up for debate, that was with about 10 seconds of consideration.)

[devongovett]

I still think query params are by far the best option here, even with the concerns raised.

1. It's the easiest to implement and configure. No new plugin type or config. Only transformers need to be involved, not resolvers or other plugins.
2. It's not a special Parcel thing. Query params are valid in all tools that support URLs. Import specifiers need to be understood by all tools that you run over your code, not just your bundler. Type systems, linters, test frameworks, etc. all process import statements in your code and if we introduce new syntax, they will not understand them and will break. Query params should be (if not now, then eventually) supported everywhere since they are part of the URL standard.
3. It's not hard coded into Parcel core at all. Query params are just options that are provided to transformers. You could have a giraffe query param for all Parcel core cares - it's up to transformer plugins to support options, and the user to configure their pipelines in .parcelrc if necessary.

The fallback issue when switching tools is valid, but honestly not that bad I think. You'll already have other problems in anything complex. There are many cases where behavior of imports relies on specific configuration. import './foo.png' in a tool that isn't configured to load PNGs, and it'll either try to process it as JS directly, or maybe give you back a URL or base64 string. Depends on how the tool is configured. You need to configure your whole toolchain to support processing images in order to fix it. Query params are the same thing.

I think we should try to keep this as simple as possible and not try to make Parcel even more complicated to support this one feature. Query params can be widely used for many purposes, including changing output type or passing other options (e.g. image resizing). Supporting them is simple, and opens up the doors to many features without much added complexity.

[jamiebuilds]

I still think query params are by far the best option here, even with the concerns raised.

I strongly disagree with that.

We already should be supporting query parameters for other purposes, if someone specifies a file https://example.com/image.png?width=100&height=100 then we should be respecting that and fetching the file as written.

By assigning a query parameter, we're changing the meaning of query parameters. What if someone actually means to specify a ?as= query param? We would have to strip it out because we will inherently have a different meaning than they give it in their API.

Not to mention that using an existing standard would be harder to unwind than something non-standard. Failing early is far better than failing softly or silently. Making it easier for users to debug ahead of time.

It costs us nothing to have the syntax be something other than query parameters. Having it be query parameters does come with a cost.

The fallback issue when switching tools is valid, but honestly not that bad I think. You'll already have other problems in anything complex. There are many cases where behavior of imports relies on specific configuration. import './foo.png' in a tool that isn't configured to load PNGs, and it'll either try to process it as JS directly, or maybe give you back a URL or base64 string.

It wouldn't try to do any of those things, unless you're talking about Parcel in which case it also should not do any of those things, it should fail the same way it's meant to fail:

import "./image.png"
// TypeError: 'image/png' is not a valid JavaScript MIME type.

import "./file.js?as=raw-source"
// No Error

Module loading is incredibly complicated today because of a whole bunch of early decisions that were not all that well thought out. Having been one of the people (on Babel) that helped make decisions that directly led to the ecosystem having a ton of "problem code" today, I really wish we had done things differently.

There are a lot of things Parcel can make short term decisions with, but one of them should not be module loading or user code, because it sticks around for a very long time.

I think we should try to keep this as simple as possible and not try to make Parcel even more complicated to support this one feature. Query params can be widely used for many purposes, including changing output type or passing other options (e.g. image resizing). Supporting them is simple, and opens up the doors to many features without much added complexity.

There is a bigger problem here than parameters to transformers though. Something like ?as=raw-source requires a completely different pipeline than whatever it's original extension was. And that pipeline looks completely different from what ?as=image-data should be.

There are two problems:

A. Passing parameters to transformers.
B. Modifying the pipeline that a particular import goes through.

I agree that both need to be solved, but:

• You cannot solve B by solving A. Passing query params is not enough to modify the pipeline.
• You can solve A by solving B. Modifying the pipeline can modify the query params as it goes.

If you solve A before you solve B, you will have to support two things. But if you solve B and A at the same time, you can have a unified solution.

Not to mention that query parameters decentralize your pipeline from out of your config file and plugins and into your code.

[devongovett]

By assigning a query parameter, we're changing the meaning of query parameters. What if someone actually means to specify a ?as= query param? We would have to strip it out because we will inherently have a different meaning than they give it in their API.

I don't think it's changing the meaning at all. as wouldn't be a special parameter in Parcel core, just what a plugin has implemented. Nothing is stopping transformers from implementing query parameters that return different types of assets depending on the value. For example, the raw transformer could implement support for as=base64 and as=text in the same plugin based on the query param. This would work even without modifying the pipeline config at all.

So, if we add another syntax for this, we'd effectively be supporting two ways to do the same thing.

[jamiebuilds]

I don't think it's changing the meaning at all. as wouldn't be a special parameter in Parcel core, just what a plugin has implemented.

But that is changing the meaning. Query parameters, and module specifiers in general only affect the way a module is resolved. There is currently no standard for transformations and if one existed it would not be based on query params because it would be a backwards-incompatible change.

The only question you need to ask is “If I removed Parcel from the equation would this fail immediately or potentially quietly/silently.”

For example, the raw transformer could implement support for as=base64 and as=text in the same plugin based on the query param. This would work even without modifying the pipeline config at all.

That’s not how the pipeline config works though. If I import file.js?as=raw-source it’s gonna match *.js unless the config file also specifies *?as=raw-source in which case it does require a modification to the pipeline config and it does not give the raw transformer control over the query param.

The only alternative I can come up with is specifying raw transformer to match ** and fall through to other transformers if it doesn’t have the query param, but now you’re breaking the design contract of .parcelrc and you’re loading in a transformer on every file.

Nothing is stopping transformers from implementing query parameters that return different types of assets depending on the value. [...] So, if we add another syntax for this, we'd effectively be supporting two ways to do the same thing.

People are always going to find ways to use Parcel in ways we did not intend or want them to. There are dozens of Babel plugins that I wish didn’t exist.

But that’s an argument for being more restrictive and provide more guidelines. Letting people hack away in whatever way they please is only going to make it harder to make changes to Parcel in the future.

[jakearchibald]

Something came up at TPAC that feels relevant, but sorry if this is just noise:

import obj from 'https://example.com/whatever.json';

There's a proposal to make this work in browsers, where whatever.json is parsed as JSON and the resulting JS object is imported. But, this creates a situation where example.com can change the content type of whatever.jsonto text/javascript, and now they're executing JS.

It wasn't resolved at TPAC, but to me it feels like the importer should have primary control over how the resource is processed, not the server. Something in the import statement needs to say "treat this as JSON", and it can't be a syntax that, if JSON imports are unsupported, falls back to "treat this as JS".

@devongovett

It's not a special Parcel thing. Query params are valid in all tools that support URLs. Import specifiers need to be understood by all tools that you run over your code, not just your bundler. Type systems, linters, test frameworks, etc. all process import statements in your code and if we introduce new syntax, they will not understand them and will break

It comes down to whether build tools should behave like JS or CSS. In JS, if you call an unrecognised function, it throws. In CSS, if you set an unrecognised property, it's a no-op.

[devongovett]

@jamiebuilds

Query parameters, and module specifiers in general only affect the way a module is resolved.

img.png?width=100&height=100 doesn't affect resolution, but transformation. In this case it just doesn't change the type of asset being returned.

Servers can implement an as parameter today and return different things if they want. Not sure why parcel is any different. Just pretend transformer plugins are servers. If the server changes what the query params it supports mean, then it'll break.

[jamiebuilds]

img.png?width=100&height=100 doesn't affect resolution, but transformation. In this case it just doesn't change the type of asset being returned.

If you fetch that URL it does resolve to an asset, the server might be behind the scenes “transforming” that, but from the client you are only aware of resolution — and that resolution is important for the client for caching.

Servers can implement an as parameter today and return different things if they want. Not sure why parcel is any different. Just pretend transformer plugins are servers. If the server changes what the query params it supports mean, then it'll break.

Except Parcel and the server can conflict.

If my server wants ?as=foo|bar and parcel only allows ?as=baz|bat then we now need to do some parcel magic to get it all working together.

If the server changes what the query params it supports mean, then it'll break.

Exactly, and it will probably break quietly or silently which is why well designed APIs will intentionally make it loudly fail when someone passes a deprecated/removed query param.

We don’t have the ability to do that, if someone uses a module relying on Parcel-specific query params outside of Parcel it’s going to fail quietly or silently. If they use something like raw-source~file.js it will fail immediately every time.

[devongovett]

If they use something like raw-source~file.js it will fail immediately every time.

Yup, and they'll likely have no recourse but to change their code, thus making it incompatible between tools. At least with query params there is some chance of other tools implementing them. I really don't want to be in the business of inventing syntax with Parcel.

[devongovett]

FWIW, there was a discussion in create-react-app a while back about this: facebook/create-react-app#3722. @gaearon made a proposal to use named imports for this. Not sure where it ended up, but appears that query params were also proposed as an option in the thread. Would be useful to get some buy in from other tools for whatever we move forward with.

[jamiebuilds]

Other tools could also implement a viewer~specifier syntax. That is not limiting. But we need to design something that would get more consensus, which query params will not.

There’s still always the possibility that a standards body will want to get involved here, there’s been talks about having “loaders” for years now.

By designing something more robust we can have more influence on the design from standards bodies. But if we design something with obvious flaws, then it will be discarded.

[kwelch]

I am sort of intrigued by the idea of protocol again with the combination of viewers. Using a protocol like view-*:, could sort of blend the line between standard and new.

view-source:./foo.js
view-data:./butterfly.png

It would be nice to also be able to select if optimizers are ran or not.

[devongovett]

FYI I implemented "named pipelines" in #3613 using a protocol-like syntax. It's not implemented in import specifiers yet, only in .parcelrc. For now, it's used to run a different pipeline based on the target (e.g. main, module, types).

[devongovett]

Named pipelines are implemented in import specifiers now. Query params for options are not yet implemented.

[svr93]

I think it will be good to use a single syntax for the whole JS ecosystem so I suggest combining different solutions into one - tc39/proposal-import-attributes#22