Skip to content

Commit

Permalink
Feature(trace-deps, trace-pkg): User conditions (#12)
Browse files Browse the repository at this point in the history 
Adds user `conditions` flag. We're following the https://github.com/vercel/nft#exports--imports model of **always** including some built-in Node.js conditions because the Node.js runtime actually does this (e.g., if I'm running `node -C my-bespoke ./file.js` and there's an `import` condition before `my-bespoke` the Node.js built-in matching means `import` wins).

- Supersedes FormidableLabs/trace-deps#73 (Thanks so much for your work @yankovalera -- this is mostly just a retweaking of your original work here).
- Fixes FormidableLabs/trace-deps#56
- Major release: It's a very "minor Major" in that we're removing `production` and `development` user conditions by default, but practically no one has code that is likely impacted....
  • Loading branch information
@ryan-roemer
ryan-roemer authored Sep 10, 2022
1 parent 0370bab commit 43552d1
Show file tree
Hide file tree
Showing 8 changed files with 1,371 additions and 1,075 deletions.
8 changes: 8 additions & 0 deletions .changeset/lucky-cheetahs-relate.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
"trace-deps": minor
"trace-pkg": minor
---

- Feature (`trace-pkg`): Add `conditions` option.
- Feature (`trace-deps`): Add `conditions` parameter to `traceFile`/`traceFiles` to support user runtime loading conditions. (See [#trace-deps/56](https://github.com/FormidableLabs/trace-deps/issues/56))
- BREAKING: Remove `production` and `development` from set of default user conditions in package resolution.
4 changes: 3 additions & 1 deletion packages/trace-deps/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ _Parameters_:

* `srcPath` (`string`): source file path to trace
* `ignores` (`Array<string>`): list of package prefixes to ignore tracing entirely
* `conditions` (`Array<string>`): list of Node.js runtime import [user conditions](https://nodejs.org/api/packages.html#resolving-user-conditions) to trace in addition to our default built-in Node.js conditions of `import`, `require`, `node`, and `default`.
* `allowMissing` (`Object.<string, Array<string>`): Mapping of (1) absolute source file paths and (2) package name or relative file path keys to permitted missing module prefixes values.
* Source file keys must match the entire file path (e.g., `/FULL/PATH/TO/entry.js`) while package keys are the start of package name either alone or with the rest of the relative path to ultimate file (e.g., `lodash`, `@scope/pkg` or `@scope/pkg/some/path/to/file.js`).
* Missing module prefix values may be the package name or any part of the relative path thereafter (e.g., `pkg`, `pkg/some`, `pkg/some/path/to/file.js`)
Expand Down Expand Up @@ -69,6 +70,7 @@ _Parameters_:

* `srcPaths` (`Array<string>`): source file paths to trace
* `ignores` (`Array<string>`): list of package prefixes to ignore
* `conditions` (`Array<string>`): list of Node.js runtime import user conditions to trace.
* `allowMissing` (`Object.<string, Array<string>`): Mapping of source file paths and package names/paths to permitted missing module prefixes.
* `bailOnMissing` (`boolean`): Throw error if missing static import.
* `includeSourceMaps` (`boolean`): Include source map file paths from control comments
Expand Down Expand Up @@ -111,7 +113,7 @@ Examples:
* **Modern Node.js ESM / `package.json:exports` Support**: Node.js v12 and newer now support modern ESM, and `trace-deps` will correctly package your application in any Node.js runtime. Unfortunately, the implementation of how to [resolve an ESM import](https://nodejs.org/api/packages.html) in modern Node.js is quite complex.
* **It's complicated**: For example, for the same import of `my-pkg`, a `require("my-pkg")` call in Node.js v10 might match a file specified in `package.json:main`, while `require("my-pkg")` in Node.js v12 might match a second file specified in `package.json:exports:".":require`, and `import "my-pkg"` in Node,js v12 might match a _third_ file specified in `package.json:exports:".":import`. Then, throw in [conditions](https://nodejs.org/api/packages.html#packages_conditional_exports), [subpaths](https://nodejs.org/api/packages.html#packages_subpath_exports), and even subpath conditions, and it becomes exceedingly difficult to statically analyze what is actually going to be imported at runtime by Node.js ahead of time, which is what `trace-deps` needs to do. 🤯
* **Our solution**: Our approach is to basically give up on trying to figure out the exact runtime conditions that will be used in module resolution, and instead package all reasonable conditions for a given module import. This means that maintain correctness at the cost of slightly larger zip sizes for libraries that ship multiple versions of exports.
* **Our implementation**: When `trace-deps` encounters a dependency, it resolves the file according to old CommonJS (reading `package.json:main`) and then in modern Node.js `package.json:exports` mode with each of the following built-in / suggested official conditions: `import`, `require`, `node`, `default`, `development`, and `production`. (We ignore `browser`).
* **Our implementation / conditions**: When `trace-deps` encounters a dependency, it resolves the file according to old CommonJS (reading `package.json:main`) and then in modern Node.js `package.json:exports` mode with each of the following built-in official conditions: `import`, `require`, `node`, `default`. We do not include any of the suggested user conditions (e.g., `production`, `development`, `browser`) by default. You can add additional user conditions using the `conditions` parameter.
* **Missing Features**: `trace-deps` does not support the deprecated [subpath folder mappings](https://nodejs.org/api/packages.html#packages_subpath_folder_mappings) feature. Some advanced ESM features are still under development.
* **Includes `package.json` files used in resolution**: As this is a Node.js-focused library, to follow the Node.js [module resolution algorithm](https://nodejs.org/api/modules.html#modules_all_together) which notably uses intermediate encountered `package.json` files to determine how to resolve modules. This means that we include a lot of `package.json` files that seemingly aren't directly imported (such as a `const pkg = require("mod/package.json")`) because they are needed for the list of all traced files to together resolve correctly if all on disk together.
Expand Down
38 changes: 31 additions & 7 deletions packages/trace-deps/lib/trace.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,21 +30,33 @@ const RESOLVE_EXTS = [".js", ".json"];
// https://github.com/FormidableLabs/trace-deps/issues/56
const CONDITIONS = [
// Node.js conditions
//
// https://nodejs.org/api/packages.html#packages_conditional_exports
// > Node.js implements the following conditions, listed in order from most specific to
// > least specific as conditions should be defined:
// > - "node-addons"
// > - "node"
// > - "import"
// > - "require
// > - "default"
// NOTE: We don't include `node-addons` but could...
"import",
"require",
"node",
// Try `default` in both CJS + ESM modes.
["default", { require: true }],
["default", { require: false }],
["default", { require: false }]

// Endorsed user conditions.
// https://nodejs.org/api/packages.html#packages_conditions_definitions
//
// Note: We are ignoring
// - `browser`
"development",
"production"
// https://nodejs.org/api/packages.html#community-conditions-definitions
// > - "types"
// > - "deno"
// > - "browser"
// > - "development"
// > - "production"
//
// We do not default include any of these, but can be added with user-specified conditions.
];

// Exports that disable modern ESM.
Expand Down Expand Up @@ -173,6 +185,7 @@ const _recurseDeps = async ({
srcPaths,
depPaths = [],
ignores = [],
conditions = [],
allowMissing = {},
bailOnMissing = true,
includeSourceMaps = false,
Expand Down Expand Up @@ -203,6 +216,7 @@ const _recurseDeps = async ({
const traced = await traceFile({
srcPath: depPath,
ignores,
conditions,
allowMissing,
bailOnMissing,
includeSourceMaps,
Expand All @@ -229,6 +243,7 @@ const _resolveDep = async ({
depName,
basedir,
srcPath,
conditions,
dependencies,
extraDepKeys,
addMisses,
Expand Down Expand Up @@ -339,13 +354,14 @@ const _resolveDep = async ({
}
});

CONDITIONS.forEach((cond) => {
CONDITIONS.concat(conditions).forEach((cond) => {
let resolveOpts;
if (Array.isArray(cond)) {
resolveOpts = cond[1];
cond = cond[0];
}


let relPath;
try {
relPath = resolveExports.resolve(pkg, depName, {
Expand Down Expand Up @@ -431,6 +447,7 @@ const _resolveDep = async ({
depName,
basedir,
srcPath,
conditions,
dependencies,
extraDepKeys,
addMisses,
Expand Down Expand Up @@ -487,6 +504,7 @@ const _resolveDep = async ({
* @param {*} opts options object
* @param {string} opts.srcPath source file path to trace
* @param {Array<string>} opts.ignores list of package prefixes to ignore
* @param {Array<string>} opts.conditions list of additional user conditions to trace
* @param {Object} opts.allowMissing map packages to list of allowed missing package
* @param {boolean} opts.bailOnMissing allow static dependencies to be missing
* @param {boolean} opts.includeSourceMaps include source map paths in output
Expand All @@ -499,6 +517,7 @@ const _resolveDep = async ({
const traceFile = async ({
srcPath,
ignores = [],
conditions = [],
allowMissing = {},
bailOnMissing = true,
includeSourceMaps = false,
Expand Down Expand Up @@ -594,6 +613,7 @@ const traceFile = async ({
depName,
basedir,
srcPath,
conditions,
dependencies,
extraDepKeys,
addMisses,
Expand Down Expand Up @@ -628,6 +648,7 @@ const traceFile = async ({
const recursed = await _recurseDeps({
depPaths,
ignores,
conditions,
allowMissing,
bailOnMissing,
includeSourceMaps,
Expand Down Expand Up @@ -658,6 +679,7 @@ const traceFile = async ({
* @param {*} opts options object
* @param {Array<string>} opts.srcPaths source file paths to trace
* @param {Array<string>} opts.ignores list of package prefixes to ignore
* @param {Array<string>} opts.conditions list of additional user conditions to trace
* @param {Object} opts.allowMissing map packages to list of allowed missing package
* @param {boolean} opts.bailOnMissing allow static dependencies to be missing
* @param {boolean} opts.includeSourceMaps include source map paths in output
Expand All @@ -669,6 +691,7 @@ const traceFile = async ({
const traceFiles = async ({
srcPaths,
ignores = [],
conditions = [],
allowMissing = {},
bailOnMissing = true,
includeSourceMaps = false,
Expand All @@ -682,6 +705,7 @@ const traceFiles = async ({
const results = await _recurseDeps({
srcPaths,
ignores,
conditions,
allowMissing,
bailOnMissing,
includeSourceMaps,
Expand Down
136 changes: 130 additions & 6 deletions packages/trace-deps/test/lib/trace.spec.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2098,12 +2098,10 @@ describe("lib/trace", () => {

expect(dependencies).to.eql(fullPaths([
"node_modules/complicated/default.js",
"node_modules/complicated/development.js",
"node_modules/complicated/import.mjs",
"node_modules/complicated/local/two.mjs",
"node_modules/complicated/main.js",
"node_modules/complicated/package.json",
"node_modules/complicated/production.mjs",
"node_modules/subdep/default.js",
"node_modules/subdep/from-default.js",
"node_modules/subdep/from-main.js",
Expand Down Expand Up @@ -2149,11 +2147,9 @@ describe("lib/trace", () => {

expect(dependencies).to.eql(fullPaths([
"node_modules/complicated/default.js",
"node_modules/complicated/development.js",
"node_modules/complicated/local/one.js",
"node_modules/complicated/main.js",
"node_modules/complicated/package.json",
"node_modules/complicated/production.mjs",
"node_modules/complicated/require.js",
// Note: All of the import paths are to sub-paths, and **not**
// the root package, so no defaults in play.
Expand Down Expand Up @@ -2286,12 +2282,10 @@ describe("lib/trace", () => {
const { dependencies, misses } = await traceFile({ srcPath });

expect(dependencies).to.eql(fullPaths([
"node_modules/complicated/development.js",
"node_modules/complicated/import.mjs",
"node_modules/complicated/local/one.js",
"node_modules/complicated/local/two.mjs",
"node_modules/complicated/package.json",
"node_modules/complicated/production.mjs",
"node_modules/complicated/require.js",
"node_modules/subdep/default.js",
"node_modules/subdep/from-require.js",
Expand Down Expand Up @@ -2601,6 +2595,136 @@ describe("lib/trace", () => {
expect(misses).to.eql({});
});
});

describe("user conditions", () => {
it("handles user conditions", async () => {
mock({
"first.js": `
const one = require("one");
const dynamicTwo = () => import("two");
const second = require("./second");
`,
"second.js": `
const one = require.resolve("one");
(async () => {
await import("three");
})();
`,
node_modules: {
one: {
"package.json": stringify({
name: "one",
main: "index.js",
exports: {
".": {
bespoke: "./bespoke.js",
production: "./production.js",
"default": "./default.js"
}
}
}),
"index.js": "module.exports = 'one';",
"default.js": "module.exports = 'default';",
"production.js": "module.exports = 'production';",
"bespoke.js": "module.exports = 'bespoke';"
},
two: {
"package.json": stringify({
name: "two",
main: "index.js",
exports: {
".": {
bespoke: "./bespoke.js",
production: "./production.js",
require: "./require.js"
}
}
}),
"index.js": "module.exports = 'two';",
"require.js": "module.exports = 'require';",
"production.js": "module.exports = 'production';",
"bespoke.js": "module.exports = 'bespoke';"
},
three: {
"package.json": stringify({
name: "three",
main: "index.mjs",
type: "module",
exports: {
".": {
// NOTE: If first, `import` will override `production` with a `production`
// condition in real Node.js runtime, so production _isn't_ matched here!
"import": "./import.mjs",
production: "./production.mjs"
}
}
}),
"index.mjs": `
import three from "nested-three";
export default three;
`,
"import.mjs": `
import three from "nested-three";
export default three;
`,
node_modules: {
"nested-three": {
"package.json": stringify({
name: "nested-three",
main: "index.mjs",
type: "module",
exports: {
".": {
bespoke: "./bespoke.mjs",
// NOTE: If a custom condition comes _before_ `import`, then it is actually
// matched in real Node.js runtime.
production: "./production.mjs",
"import": "./import.mjs"
}
}
}),
"index.mjs": "export const three = 'nested-three';",
"import.mjs": "export const msg = 'nested-import';",
"production.mjs": "export const msg = 'nested-production';",
"bespoke.mjs": "export const msg = 'nested-bespoke';"
}
}
}
}
});

const { dependencies, misses } = await traceFile({
srcPath: "first.js",
conditions: [
"bespoke",
"production"
]
});
expect(dependencies).to.eql(fullPaths([
"node_modules/one/bespoke.js",
"node_modules/one/default.js",
"node_modules/one/index.js",
"node_modules/one/package.json",
"node_modules/one/production.js",
"node_modules/three/import.mjs",
"node_modules/three/index.mjs",
"node_modules/three/node_modules/nested-three/bespoke.mjs",
"node_modules/three/node_modules/nested-three/import.mjs",
"node_modules/three/node_modules/nested-three/index.mjs",
"node_modules/three/node_modules/nested-three/package.json",
"node_modules/three/node_modules/nested-three/production.mjs",
"node_modules/three/package.json",
"node_modules/two/bespoke.js",
"node_modules/two/index.js",
"node_modules/two/package.json",
"node_modules/two/production.js",
"node_modules/two/require.js",
"second.js"
]));
expect(misses).to.eql({});
});
});
});

describe("traceFiles", () => {
Expand Down
2 changes: 2 additions & 0 deletions packages/trace-pkg/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ Configuration options are generally global (`options.<OPTION_NAME>`) and/or per-
- Can be overridden from CLI with `--concurrency <NUMBER>`
- `options.includeSourceMaps` (`Boolean`): Include source map paths from files that are found during tracing (not inclusion via `include`) and present on-disk. Source map paths inferred but not found are ignored. (default: `false`). Please see [discussion below](#including-source-maps) to evaluate whether or not you should use this feature.
- `options.ignores` (`Array<string>`): A set of package path prefixes up to a directory level (e.g., `react` or `mod/lib`) to skip tracing on. This is particularly useful when you are excluding a package like `aws-sdk` that is already provided for your lambda.
- `options.conditions` (`Array<string>`): list of Node.js runtime import [user conditions](https://nodejs.org/api/packages.html#resolving-user-conditions) to trace in addition to our default built-in Node.js conditions of `import`, `require`, `node`, and `default`.
- `options.allowMissing` (`Object.<string, Array<string>>`): A way to allow certain packages to have potentially failing dependencies. Specify each object key as either (1) an source file path relative to `cwd` that begins with a `./` or (2) a package name and provide a value as an array of dependencies that _might_ be missing on disk. If the sub-dependency is found, then it is included in the bundle (this part distinguishes this option from `ignores`). If not, it is skipped without error.
- `options.dynamic.resolutions` (`Object.<string, Array<string>>`): Handle dynamic import misses by providing a key to match misses on and an array of additional glob patterns to trace and include in the application bundle.
- _Application source files_: If a miss is an application source file (e.g., not within `node_modules`), specify the **relative path** (from the package-level `cwd`) to it like `"./src/server/router.js": [/* array of patterns */]`.
Expand All @@ -88,6 +89,7 @@ Configuration options are generally global (`options.<OPTION_NAME>`) and/or per-
- `packages.<PKG_NAME>.trace` (`Array<string>`): A list of [fast-glob][] glob patterns to match JS files that will be further traced to infer all imported dependencies via static analysis. Use this option to include your source code files that comprises your application.
- `packages.<PKG_NAME>.includeSourceMaps` (`Boolean`): Additional configuration to override value of `options.includeSourceMaps`.
- `packages.<PKG_NAME>.ignores` (`Array<string>`): Additional configuration to merge with `options.ignores`.
- `packages.<PKG_NAME>.conditions` (`Array<string>`): Additional configuration to merge with `options.conditions`.
- `packages.<PKG_NAME>.allowMissing` (`Object.<string, Array<string>>`): Additional configuration to merge with `options.allowMissing`. Note that for source file paths, all of the paths are resolved to `cwd`, so if you provide both a global and package-level `cwd` the relative paths probably won't resolve as you would expect them to.
- `packages.<PKG_NAME>.dynamic.resolutions` (`Object.<string, Array<string>>`): Additional configuration to merge with `options.dynamic.resolutions`.
- `packages.<PKG_NAME>.dynamic.bail` (`Boolean`): Override `options.dynamic.bail` value.
Expand Down
Loading

0 comments on commit 43552d1

Please sign in to comment.