ts-node fails when ES Modules are in the dependency graph in Node.js 13+

[trusktr]

I basically detailed the issue in this comment: #155 (comment)

It's a chicken-and-egg-like problem:

• If we use module: 'commonjs', then if any TS files import ES Modules (indirectly in their dependency graph), then Node throws an error because CommonJS modules can not import ES Modules.
• If we change to module: 'esnext', then the errors from the previous point go away, but now the .js file that is loading ts-node and calling something like require('typescript-entry-point.ts') will have a similar issue, because the call to require('typescript-entry-point.ts') will try to load an ES Module.
• Finally, if we convert the .js file into an ES Module, we can not convert require('typescript-entry-point.ts') into import 'typescript-entry-point.ts' because now ES Modules don't handle .ts extensions (at least not out of the box, and it seems the old require hooks don't operate on these new identifiers)

At the moment, I'm sort of stuck, because I have dependencies in my dependency tree that are ES Modules.

The only workaround I can think of is to compile everything to .js files (ES Modules) and avoid to use ts-node.

I wonder if a combination of allowJs and ignore so that it compiles JS files would help. I haven't tried that yet.

[trusktr]

One way to reproduce this is to take a working application with module: 'commonjs' being passed to the register() call, make sure you're using the latest Node 13, then add an ES Module dependency (namely a .js file inside a node_modules package with type: 'module', as I believe that'll be the prime way that people will publish ES Modules from now on) on the leaf end of any branch of the dependency tree.

[trusktr]

related:

• Support .mjs output [also a TypeScript issue] #436
• Node option for experimental and module resolutions #880

Now that Node 13 is out in the wild, looks like people are starting to run into this without any flags. Eventually someone will have unpaid free time to fix it, but I think it is easier to just pre-compile everything to JS before running Node.

[jkrems]

If we use module: 'commonjs', then if any TS files import ES Modules (indirectly in their dependency graph), then Node throws an error because CommonJS modules can not import ES Modules.

I'm trying to understand this scenario - how did this work before? I assume existing TypeScript running via ts-node was never meant to use real ESM semantics, instead it's more similar to babel-style "using import as sugar for require". So it's de-facto CommonJS and shouldn't be loading ES modules from its dependencies (at least that should be the default behavior afaict).

Is this about existing projects breaking or about existing projects migrating from CJS-"import" to standard imports?

[jkrems]

Ah, I think I got it now: It's about running ts-node for module code. Which would involve compilation within the ES module loader, potentially registering the compiler after node is started up (as opposed to using a CLI flag). The short answer is that the APIs to allow that aren't yet in node, so I don't think ts-node can do anything about this for now. :(

[trusktr]

Not even by hooking in to the experimental load API?

[jkrems]

There's two independent holes in the API surface:

1. Being able to change the source text of something loaded from disk without breaking relative resolution. There is an open PR to address this by @GeoffreyBooth: esm: source hooks nodejs/node#30986
2. Being able to start a simple binary (ts-node) that registers a transformer and then runs user code. Right now the only way to register hooks is using a node flag (node --some-flag=ts-node main.ts). Short of forking a new node process, there's currently no API that allows to implement ts-node main.ts with native module support.

The second point is tricky because it's by design. We don't really want to allow userland code (from node's perspective) being able to switch out parts of the loading pipeline at runtime. It can lead to weird issues where the same global scope contains code expecting different module loaders and thanks to import() that's actually observable (as opposed to require).

One idea brought up by @guybedford was to have an API that can create a new global scope ("context" or "realm") with all bells and whistles, including node standard library, process global, and fresh module system. Tools like ts-node could use it to set up a new context for running the application code. But there are some downsides and questions, e.g. around garbage collection of the original "bootstrap" context and the time it would take to initialize the new context.

[EntraptaJ]

I created a proof of concept for using the current version of the --experimental-loader feature to transpile typescript files right before node loads it and then give node the now ESNext Javascript code with the SyntheticModule VM feature. It's fairly hacky right now but it made for a fun 6 hour session: https://github.com/KristianFJones/TS-ES-Node

[Jamesernator]

^ Just for above there's a new loader hook in node nightly called getSource so the following loader can be used in node nightly. Only caveats are the same as the babel typescript plugin (e.g. no const enum):

Typescript Loader

import fs from 'fs';
import path from 'path';
import babel from '@babel/core';
import urlUtils from 'url';
import babelPluginTransformTypescript from '@babel/plugin-transform-typescript';

function isURL(string) {
    try {
        new URL(string);
        return true;
    } catch {
        return false;
    }
}
const removePrivateTypes = (babel) => {
    return {
        visitor: {
            ClassPrivateProperty(context) {
                context.get('typeAnnotation')?.remove();
            },
        },
    };
}

const BABEL_OPTS = {
    plugins: [
        babelPluginTransformTypescript,
        removePrivateTypes,
    ],
    parserOpts: {
        plugins: [
            'asyncGenerators',
            'bigInt',
            'classProperties',
            'classPrivateProperties',
            'importMeta',
            'nullishCoalescingOperator',
            'numericSeparator',
            'objectRestSpread',
            'optionalCatchBinding',
            'optionalChaining',
            'topLevelAwait',
            'typescript',
        ],
    },
    sourceMaps: 'inline',
};

export async function getSource(urlString, context, getSourceDefault) {
    if (isURL(urlString)) {
        const url = new URL(urlString);
        if (url.pathname.endsWith('.js') && !fs.existsSync(url)) {
            url.pathname = url.pathname.replace(/\.js$/u, '.ts');
            const contents = await fs.promises.readFile(url, 'utf8');
            const { code: source } = await babel.transformAsync(contents, {
                ...BABEL_OPTS,
                sourceFileName: path.basename(urlUtils.fileURLToPath(url)),
            })
            return { source };
        }
    }
    return getSourceDefault(urlString, context, getSourceDefault);
}

export async function resolve(specifier, context, defaultResolve) {
    try {
        const defaultResolution = defaultResolve(specifier, context, defaultResolve);

        try {
            const url = new URL(defaultResolution.url);
            url.pathname = url.pathname.replace(/\.ts$/, '.js');
            return { url: url.href };
        } catch {
            return defaultResolution;
        }
    } catch {
        return { url: new URL(specifier, context.parentURL).href };
    }
}

[GeoffreyBooth]

Hi, I created the getSource hook. There's also a transformSource hook that was intended for things like transpilation. See the transpiler loader example: https://github.com/nodejs/node/blob/master/doc/api/esm.md#transpiler-loader

[EntraptaJ]

Look's like I know what I'm doing this weekend.

[EntraptaJ]

Been busy, here is a proof of concept using the transformSource hook. Supports relative imports of .ts and .tsx without requiring extensions. https://github.com/K-FOSS/TS-ESNode

[EntraptaJ]

Okay, made a few small changes and fixes. It's now able to handle external modules that have the TSLib import helper without requiring refactoring existing imports. The TS-ESNode loader hook can be dropped into existing projects and just work. Hopefully! I just tested it on my main Application template that uses TypeORM & TypeGraphQL with tsconfig.json set to use ESNext as target and modules and it all just works. Transpiled code is all using imports and exports with Node V14 in Module mode.

[EntraptaJ]

Removed the globby requirement. Module has been published on NPM https://www.npmjs.com/package/@k-foss/ts-esnode zero dependencies other then peer dependency on TypeScript. NPM reports unpacked size of 8.66 kB. Should hopefully be a drop in replacement for TS-Node for now.

[blakeembrey]

@KristianFJones If you're interested I'd love to land a PR in ts-node with this functionality.

[EntraptaJ]

I'd love to give that a shot. I'll take a crack at it sometime this weekend.

[ejose19]

@KristianFJones I tried your fork and so far it works without issues, I also added --harmony-top-level-await to try along typescript 3.8 and it worked without issues. Hopefully it can be integrated into ts-node.

Btw, I saw you state it requires node 14 (nightly), but I could use 13.7 without any issues (even TLA), anything lower than that and I start getting all kinds of errors.

[jkrems]

Please note that TLA may look like it's working at first glance but is known to still have some sharp edges. Both the base implementation in V8 and the integration in nodejs. See: nodejs/node#30370. So it's nice to play around with locally but things are kind of expected to break horribly at this point. It's not ready for real-world use yet.

Just in case anybody is tempted to enable that flag somewhere. :)

[EntraptaJ]

@ejose19 Interesting, I had been developing it with Node 13.6, didn't realize the transformSource hooks were added to 13.7. Unless you tried the one on my Account and not the new transformSource one I have on my K-FOSS Organization.

[vansergen]

It seems that ts-node fails to run with "module": "esnext" in the tsconfig.json and "type": "module" in the package.json with the following error:

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /Users/user/test-project/index.ts
    at Loader.defaultGetFormat [as _getFormat] (internal/modules/esm/get_format.js:71:15)
    at Loader.resolve (internal/modules/esm/loader.js:98:42)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at Loader.getModuleJob (internal/modules/esm/loader.js:188:29)
    at Loader.import (internal/modules/esm/loader.js:163:17)

[EntraptaJ]

Well, looks like @K-FOSS/TS-ESNode does work with Node 13.7. That's amazing. I'll get to work on porting this directly into ts-node hopefully sometime this weekend. Work has been busy, not enough time for personal coding.

[ai]

@KristianFJones if you can describe how we can help you here, I can ask my friends to make pull request

[droganov]

can it be related to nodejs/node#32103
?

[EntraptaJ]

@KristianFJones if you can describe how we can help you here, I can ask my friends to make pull request

My hesitation with making a port/adaptation of TS-ESNode for ts-node is I'm not sure if the core of TS-ESNode truly belongs in TS-Node without a full refactor of the existing code base. I could easily just copy over my TS-ESNode work into ts-node/node-hooks and have it work node --loader ts-node/node-hooks but I'm not sure if it makes sense to have almost a fully different approach to loading TypeScript into Node.JS in an existing library.

[EntraptaJ]

can it be related to nodejs/node#32103
?

No, I've written a full TypeScript discovery system to provide node style resolution within ESModule mode using my loader hook to automatically discover .ts,.tsx,.js,.jsx files when no existing file is provided to the hook.

https://github.com/K-FOSS/TS-ESNode/blob/next/src/findFiles.ts

[owenallenaz]

For some people who get to this point, if you add to your tsconfig.json, it allows the .js files to resolve.

{
  "ts-node": {
    "experimentalResolver": true
  }
}

file.ts

export default function foo() {}

another.ts

import foo from "./file.js";

[tom10271]

It is amazing that 3 years after this was reported I am still facing this issue with no solutions tried would solve but hours spent and wasted.

[BrennerSpear]

I still can't import my .mjs files when I'm running a script / seed file :(

[spinlud]

Same error, none of the above solutions worked.

typescript: 4.9.5
ts-node: 10.9.1

[maximemoreillon]

I was encountering the same issue but following this guide and managed to solve my problem:

https://www.typescriptlang.org/docs/handbook/esm-node.html

However, I don't feel good about importing my project files with the .js extension

[menangaTech]

I was encountering the same issue but following this guide and managed to solve my problem:

https://www.typescriptlang.org/docs/handbook/esm-node.html

However, I don't feel good about importing my project files with the .js extension

can you show me how the code examples is?
Thanks

[maximemoreillon]

@menangaTech I was trying to use formadata-node, which in an ESM-only package. Here's a snippet that causes problems:

import { FormData } from "formdata-node"
const form = new FormData()

I fixed it by having "type":"module" in my package.json file and the following tsconfig.json:

{
  "compilerOptions": {
    "target": "es2016",
    "module": "nodenext",
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true
  },
  "ts-node": {
    "esm": true
  }
}

The important parts are the module and ts-node settings

[erick2014]

I was trying to setup this typescript for apollo-server and I was getting the same error, this is how my tsconfig.json file end up:

{
  "compilerOptions": {
    "target": "es2020",
    "module": "esnext",
    "lib": ["es2020"],
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true,
    "moduleResolution": "node",
    "sourceMap": true
  },
  "ts-node": {
    "esm": true
  }
}

[thomas-void0]

mark.

[NeoPrint3D]

I was trying to setup this typescript for apollo-server and I was getting the same error, this is how my tsconfig.json file end up:

{
  "compilerOptions": {
    "target": "es2020",
    "module": "esnext",
    "lib": ["es2020"],
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true,
    "moduleResolution": "node",
    "sourceMap": true
  },
  "ts-node": {
    "esm": true
  }
}

Thanks this actually worked @erick2014

[eiva]

Please, can somebody explain for dummy how to use ts-node imports.
Whatever combination I'm trying to use resulting in some errors either
Cannot use import statement outside a module or any other import problem...

Trivial setup:
package.json

{
  "name": "items",
  "version": "0.1.0",
  "private": true,
  "type": "module",
  "scripts": {
    "test": "ts-node app/translate.ts"
  },
  "dependencies": {
  },
  "devDependencies": {
    "sass": "^1.63.2",
    "ts-node": "^10.9.1",
    "@types/node": "20.2.5",
    "globby": "^13.1.4",
    "typescript": "5.1.3"
  }
}

tsconfig.json

{
  "compilerOptions": {
    "target": "es2020",
    "module": "esnext",
    "lib": ["es2020"],
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true,
    "moduleResolution": "node",
    "sourceMap": true
  },
  "include": ["app/*.ts"],
  "exclude": ["node_modules"],
  "ts-node": {
    "esm": true
  }
}

code app/translate.ts

import sass from "sass"
async function main() {
  const style = await sass.compileAsync('template/style.scss')
  console.log(style)
}
Promise.all([main()])

Error

> ts-node app/translate.ts

(node:27364) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `node --trace-warnings ...` to show where the warning was created)
E:\Development\Items\node_modules\sass\sass.node.js:1
import * as util from "util"
^^^^^^

SyntaxError: Cannot use import statement outside a module

Will really appreciate for help.

[kshep92]

@menangaTech I was trying to use formadata-node, which in an ESM-only package. Here's a snippet that causes problems:

import { FormData } from "formdata-node"
const form = new FormData()

I fixed it by having "type":"module" in my package.json file and the following tsconfig.json:

{
  "compilerOptions": {
    "target": "es2016",
    "module": "nodenext",
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true
  },
  "ts-node": {
    "esm": true
  }
}

The important parts are the module and ts-node settings

Answer after answer on StackOverflow said to remove "type":"module" from package.json, yet that setting (in addition to your tsconfig) solved all my issues. Thank you!

[Prenuer]

Mail.com / ipmartini@gmail.com / darknetencryption@mail.com

[samislam]

Any update on this?
ts-node will always fail when the "type": "module" is in package.json

[Sengulair]

Any update?
"type": "module" in package.json still failing

[samislam]

Any update?
"type": "module" in package.json still failing

You better stop using ts-node and start using tsx unless you really need to use the emit decorators metadata option of typescript if you're using something like TypeORM (you can still use tsx though).

[adrian-afl]

I get it that the error is because some dependency became an ES Module when previously was CommonJS. I have a funny situation now that a project works, but if I remove package-lock.json and reinstall, it stops working. Clearly some sub-dependency has changed into ESM. Is there an easy way to detect which one it is so I can pinpoint it's version and maybe let the authors know?

[owenallenaz]

@achlubek Almost always when I hit this message if you look very carefully at the error you receive it will allow you to track down which specific module is causing the problem.

[adrian-afl]

@achlubek Almost always when I hit this message if you look very carefully at the error you receive it will allow you to track down which specific module is causing the problem.

Yeah I managed to track it down to Chai v5 becoming ESM only. Too bad eh!

[SamuelGaona]

Any update?
"type": "module" in package.json still failing

You better stop using ts-node and start using tsx unless you really need to use the emit decorators metadata option of typescript if you're using something like TypeORM (you can still use tsx though).

Any recommendation about using tsx with emitDecorators? Im having troubles with that.

[samislam]

Any update?
"type": "module" in package.json still failing

You better stop using ts-node and start using tsx unless you really need to use the emit decorators metadata option of typescript if you're using something like TypeORM (you can still use tsx though).

Any recommendation about using tsx with emitDecorators? Im having troubles with that.

tsx can't emit decorators, and I don't think it's possible (during development)

[nabilfreeman]

I fixed this by running npx tsx instead of npx ts-node

Once again, ES Modules are the dumbest thing to ever happen to the JS ecosystem