Compilation is unbelievably slow

[lukashavrlant]

Hi, for some reason we have performance problems when using ts-node in our mocha tests. It takes about 500 ms to compile an empty ts file. I added some console.logs to measure the time to run this very line: const output = service.getEmitOutput(fileName) and this line takes 500 ms to run even though the content of the file is an empty string (the code variable on the line 314 is empty string). Actually, all our tests files take too long to process, while the production files take a few milliseconds to compile. So any *.spec.ts takes about 500 ms to compile, a regular *.ts file takes about 20 ms. Do you have any idea what could be the root cause or how should I debug it more?

We use latest mocha, ts-node and typescript, but we tried some old versions too and the problem persists. The tsconfig.json:

{
  "compileOnSave": false,
  "compilerOptions": {
    "outDir": "dist",
    "rootDir": ".",
    "declaration": true,
    "target": "es2017",
    "lib": ["es2017"],
    "module": "commonjs",
    "moduleResolution": "node"
  },
  "exclude": [
    "node_modules",
    "dist"
  ]
}

[davidgruar]

I have also experienced this issue when running alsatian tests after upgrading to ts-node v7. I've found I can resolve the issue by specifying the --files CLI flag.

The problem seems to stem from the fact that the test framework requires each .test.ts file in turn, and each call to require appears to compile every module in that file's dependency tree from scratch. Specifying --files reverts to the old behaviour of loading all files on startup (and presumably caching them in memory).

@blakeembrey does this sound like a plausible diagnosis, and can you see any better solution?

[cspotcode]

Have you tried --transpile-only mode?

[mazswojejzony]

I've run into similar issue today. After ts-node upgrade from 7.0.1 to 8.0.1 my mocha test suite became very slow (32s vs 4s). Adding TS_NODE_TRANSPILE_ONLY=true resolved my issue. Thanks @cspotcode

[davidgruar]

--transpile-only does speed things up considerably for me. I don't want to lose the type checking though!

[lukashavrlant]

I tried all the proposed solutions. The files: true makes execution significantly faster! Turning off type checking also made things faster. I just had to turn off typeCheck and turn on transpileOnly (typeCheck: false, transpileOnly: true). This is good enough workaround, but it seems that the default behavior is still weird.

[blakeembrey]

That's intereseting, files: false is definitely meant to make things faster rather than slower. Does anyone have a large enough project that's running slow and want to either 1. investigate if the reason @davidgruar mentions appears true or 2. allow me temporary access to look into further myself?

Ideally files: false loads a single file and spiders out so it is possible that this is a slight issue with performance. It's also possible that the previous version was hitting the cache heavily, if you upgraded from v7 to v8 (v8 removed the cache because of type issues). We could also investigate adding caching back but it needs to be much smarter than the previous version to solve transitive type changes.

[garethflowers]

I've hit this issue too after updating from 7.0.1 to 8.0.1.

From what I can tell when files: false the tsconfig.json file gets changed in memory to remove the files and includes attributes.

ts-node/src/index.ts

Line 484 in 544851e

config.files = []

This seems to create an invalid config (when using TypeScript 3.2.4):

error TS18002: The 'files' list in config file 'tsconfig.json' is empty.

Removing the files and includes attributes altogether however seems to resolve the issue.

[blakeembrey]

If anyone can give me temporary access to a slow repo, let me know! I can quickly take a look into the issue and happy to sign any NDA, etc you need. If not, can I get a sense of peoples tsconfig.json that's slow - are you relying on files, include, exclude, rootDir or other to compile with TypeScript?

Let's try with reactions:

• ❤️ files
• 🚀 include
• 👀 exclude
• 😕 rootDir
• 🎉 Other

[garethflowers]

8.0.2 hasn't made any difference for me. The fact it's adding files: [] in to the tsconfig.json still kills performance.

I can't share our project unfortunately - but here's our tsconfig.json:

{
	"compileOnSave": true,
	"compilerOptions": {
		"emitDecoratorMetadata": true,
		"experimentalDecorators": true,
		"forceConsistentCasingInFileNames": true,
		"lib": [
			"es2017"
		],
		"module": "commonjs",
		"moduleResolution": "node",
		"newLine": "LF",
		"noEmitOnError": true,
		"noImplicitReturns": true,
		"noUnusedLocals": true,
		"noUnusedParameters": false,
		"outDir": "dist",
		"removeComments": true,
		"rootDir": ".",
		"sourceMap": true,
		"strict": true,
		"strictFunctionTypes": false,
		"suppressImplicitAnyIndexErrors": true,
		"target": "es6"
	}
}

[blakeembrey]

@garethflowers Are you using mocha or something else that makes the explanation in #754 (comment) make sense?

Alternatively, can you try populating files = ['one.ts file here'] so it's valid, but still overridden, and let me know if that changes anything?

[davidgruar]

@blakeembrey I've made a minimal repro at https://github.com/davidgruar/tsnode-perf-repro - please check it out and have a play. Loading just three test files with fairly few dependencies takes more than a second, and the more imports you add, the longer it takes.

[blakeembrey]

It doesn't seem like a huge overhead when you time the entire program, instead of just requires:

There is a difference, however, so for testing it might make the most sense to enable --files whereas running an application it doesn't - there's only a single input and TypeScript pre-processes everything anyway. I'll look into it more though because the difference is clear and it would be nice to bring the dynamic files number down.

[blakeembrey]

Ok, perfect. It does look like TypeScript makes 4x the number of requests to the filesystem when files: false. Looking into whether there's some issue with how TypeScript is functioning here and how this could be improved.

[jeremy-coleman]

i've found several settings contributing to slow compile times, im pretty sure the issue is related to settings that will delete the default setting of ignore: node_modules. IE: allowJs. Others that may also (just from my memory of guess/check) are removeComments, emitDecoratorMetadata, and a few others. I've also noticed that if you glob without file ext like include: src//* instead of src//*.ts it can cause slowdowns , idk if its chokadir watching fontfiles that may be in there or something. using isolatedModules is the sure-fire way to speed things up though.

[blakeembrey]

@jeremy-coleman Sorry, I couldn't quite follow. Can you please let me know who you/what are responding to here?

Edit: I don't think any of those are related to the root cause of issues mentioned in this thread, that's all.

[lukashavrlant]

@blakeembrey Hi, I also created an example: https://github.com/lukashavrlant/ts-node-perf But I guess it is similar to the previous example. Basically compiling TS files to JS files and then running mocha tests using the generated JS files is faster then using ts-node directly. See readme for more details please. The more imports I use is test files, the slower it gets. For the sake of example I used zero imports in tests files.

[blakeembrey]

Unfortunately this definitely seems to be the case. I'm not 100% sure why in the language services this is so expensive. It appears that changing rootFiles results in the "project out of date" and it starts re-resolving all types and traversing node_modules. This is very bad in the test loading case - I need to find a way to use the previously cached resolutions over trying to re-resolve everything when the root files change. cc @TypeStrong/typescript-team @weswigham in case I'm doing something wrong.

[weswigham]

@sheetalkamat would you know what would need to be done to allow API implementers to persist the resolution cache across builds?

[sheetalkamat]

When host doesn't have resolveModuleName it creates the cache per program. So whenever program changes the new resolution cache is created. (Note this is different from internal resolution cache that tsc and tsserver use that are made aware of which files have changed and hence which resolutions to invalidate) I don't think ModuleResolutionCache that compiler has, is equippd to handle the changes and hence its per program.
I think better option would be to use our WatchAPI instead as that is equipped to handle changes in file and hence invalidating partial resolutions.

[Orokon]

I'm also having issues with speed. I'm having a big Angular Application where I am running protractor tests with the jasmine framework. When I start the tests with protractor flag --troubleshoot i can see where it gets to the point where the files get processed. Usually that takes like +- 3 seconds and after that jasmine starts. Having version 8.0.0 and above its taking up to almost 2 Minutes until the files are processed and jasmine starts.

Sadly I don't know where I could try out the solution with file: false. Any help is appreciated.

[cezaryrk]

It's not a solution, but downgrade to 6.2.0 helped me currently with the test performace issues.

[blakeembrey]

For now, please try using the environment flag from https://github.com/TypeStrong/ts-node#cli-and-programmatic-options - TS_NODE_FILES=true. I'll attempt refactoring to the newer TypeScript watch API and see if it improves performance in the coming weeks.

[miao17game]

ts-node 7.0 -- 2.5s
ts-node 8.0 -- 5.2s

same environment.

[blakeembrey]

@miao17game By same environment, is this a test suite, a script or something else?

[chadbr]

Hi -- for what it's worth --

We have a rather large code base (no way to know really...)

It is on the order of 450 Mocha tests.

7.x was not "snappy" to startup - but it was tolerable.

I upgraded from 7.0.2 to 8.0.1 and the tests will actually never finish on Windows. They work fine on Linux (like 7.x).

After reading through this, I was trying to use the --files true flag. On a smaller codebase it made a big different (the tests ran).

On the code base with 450 tests... it simply never finished. The strange thing is if I set the environment variable TS_NODE_FILES -- set TS_NODE_FILES=true ; the tests start almost immediately.

If it helps:

mocha --recursive --require ts-node/register -R spec ./**/*.spec.ts --no-timeouts --exit

tsconfig.json

{
  "compilerOptions": {
    "target": "es6",
    "outDir": "dist",
    "rootDir": "src",
    "moduleResolution": "node",
    "module": "commonjs",
    "declaration": true,
    "importHelpers": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "sourceMap": true,
    "listFiles": false,
    "traceResolution": false,
    "pretty": true,

    "strictNullChecks": true,
    "forceConsistentCasingInFileNames": true,
    "noFallthroughCasesInSwitch": true,
    "noImplicitAny" : true,
    "noImplicitReturns": true,
    "noImplicitThis": true,
    "noUnusedLocals": false,
    "noUnusedParameters": false,

    "typeRoots": [
      "node_modules/@types"
    ],
    "lib": [
      "es2018",
      "es6",
      "dom",
      "dom.iterable"
    ]
  },
  "include": [
    "src/**/*.ts"
  ],
  "exclude": [
    "node_modules/**"
  ],
  "compileOnSave": false
}

package.json

    "ts-node": "^8.0.2",
    "typescript": "3.3.3"
    "tslib": "^1.9.3",

[kaiza]

We have just started running into this problem after upgrading our Angular app to 7.2.5 and at the same time we also upgraded ts-node to 8.0.2. Since the upgrade running the protractor e2e tests always took about 2 minutes before they would start running. After some investigation I found that it was taking a long time loading the spec files. I've had to downgrade ts-node back to 5.0.1 which fixes it but just wondering if anyone else has had the same problem and if they were able to solve it with the latest ts-node version.

[koliada]

I'm facing this issue when writing tests using mocha + proxyquire.
My solution so far is to mock as many dependencies with proxyquire as possible and use these three options for ts-node (they all seem to positively affect the start-up speed):

transpileOnly: true,
typeCheck: false,
files: false

[khaledosman]

@acro5piano could you explain how you do that please 🙏

@Alexandredc instead of having your nodemon config watch your .ts files and execute ts-node on the entry ts file on changes, You can instead run tsc --watch (optionally with "incremental": true in your tsconfig.json) when the .ts files change to generate the js files in the "outputDir" folder, then run node / nodemon against the generated entry .js file

[acro5piano]

@khaledosman @Alexandredc

My current solution is tsc + nodemon, but I'm still doing the configuration for Ava.
The downside is the complex configuration. ts-node and tsconfig-paths is a simple configuration, works nicely for fs.readFile and ts path mappings and Ava. But tsc + nodemon is 2x speeds up my project.

The basics are:

# Directory structure

repo/
   |- tsconfig.json
   |- package.json
   |- src/
   |    `- index.ts
   `- build/
        `- index.js

// tsconfig.json

{
  "compilerOptions": {
    "outDir": "./build",      // <-- important
    "noEmit": false,     // <-- important
    "target": "es2019",
    "sourceMap": true,
    "incremental": true,
    "isolatedModules": true,
    "moduleResolution": "node",
    "esModuleInterop": true,
    "skipLibCheck": true
  }
}

// package.json

  "scripts": {
    "dev": "concurrently yarn:dev:*",
    "dev:nodemon": "nodemon build/index.js",
    "dev:tsc": "tsc -w"
  },

And the last thing is to add deps:！

yarn add -D concurrently nodemon typescript

edited: You may also needs source-map-support

[azu]

I've met same issue on developing web app/API server using express + ts-node-dev(ts-node).
So, I've created express-lazy-router for reducing this issue.
This is inspired by webpack's lazyCompilation.

express-lazy-router help us to reduce the compilation time at start time by compiled when needed.
📝 It is for webapp using express, it is not for testing

[kfrajtak]

Can you use the power of esbuild? https://esbuild.github.io/

[acro5piano]

@kfrajtak
I've tried esbuild and it is unbelievably fast.
I moved to it. Compile + run is much faster than ts-node. Thanks.

Edit: I'm using esbuild --watch and nodemon build/out.js now

Edit 2: I've tried to use other project using decorator, but its Interpolation is still different from original tsc. so I'm sticking tsc --watch in that project.

[cspotcode]

See also: #1160 which allows opting into the swc compiler via ts-node.

[acro5piano]

Could esbuild-register be an alternative solution?

https://github.com/egoist/esbuild-register

[cspotcode]

I investigated esbuild when implementing #1160 but there were performance limitations in per-file compilation. #1160 gives the same performance benefits, and people are already successfully using it, for example #1276 (reply in thread) I recommend checking it out and giving feedback in the discussion thread.

#1160 implements pluggable transformers.  The swc transformer is built-in, but anyone can write a plugin to use any other transformer. This means your existing TS project configuration is all you need to use a different transformer, and you get the same benefits as ts-node: sourcemap support, ESM loader, etc.

Also, for many people, transpileOnly is all they need.  Often they've made a configuration mistake: they believe transpileOnly is enabled, but it actually is not. We implemented #1243 to make it easier to debug configuration mistakes, enabling more people to use transpileOnly and to better understand their configuration.

[cspotcode]

The new features mentioned above, --transpiler and --show-config, have been released in v10

https://github.com/TypeStrong/ts-node/releases/tag/v10.0.0

There is a dedicated discussion thread linked if you need assistance with the upgrade.

[cspotcode]

Closing because some solutions are already available (--transpiler), other aspects are tracked by other open issues (incremental builds), and unfortunately this thread has not yielded good reproductions with which we can test.

• --transpiler with swc yields the fastest compilation when skipping typechecking, beating out even a fully-cached solution
• Docs have a dedicated page with performance recommendations: https://typestrong.org/ts-node/docs/performance
• Use ts-node with TypeScript incremental build #817, Refactor fs (cache/tracing) layer into self-contained API #1161, and Make --files default? Conditionally when leveraging incremental compilation? #1245 track the research/work necessary to support incremental builds

[O-Q]

In my case with .mocharc.js, replace:
require: ["ts-node/register", ...]
with:
require: ["ts-node/register/transpile-only", ...]
resolved my issue.

[leoblum]

@O-Q thank you, man. That helps me to start test faster!

[multipliedtwice]

Could this be a solution?
https://github.com/a7ul/esbuild-node-tsc

[cspotcode]

@thousandsofraccoons ts-node's "swc" integration is the recommended solution here. It skips typechecking and uses the blazing-fast swc transpiler. You can use it today, check the docs links below for instructions.

https://typestrong.org/ts-node/docs/performance#skip-typechecking
https://typestrong.org/ts-node/docs/transpilers#bundled-swc-integration

It is promoted out of "experimental" status in our next, upcoming release. #1536

[eyalroth]

Closing because some solutions are already available (--transpiler), other aspects are tracked by other open issues (incremental builds), and unfortunately this thread has not yielded good reproductions with which we can test.

• --transpiler with swc yields the fastest compilation when skipping typechecking, beating out even a fully-cached solution
• Docs have a dedicated page with performance recommendations: https://typestrong.org/ts-node/docs/performance

FYI swc seems to be incompatible with sequelize-typescript. I found the issue sec#1160 which mentions this, but it's possible that there are more undocumented issues preventing the two from working together.

Running with --transpile-only alone doesn't seem to help much for our tests execution.

[cspotcode]

We've seen a lot of cases where someone thought they were using transpileOnly, but actually they weren't. So that's always a possibility whenever it doesn't seem like transpileOnly is making much impact: perhaps it's not even turned on.

Is this an issue with cross-file metadata, in other words, is it compatible or incompatible with isolatedModules?

[acro5piano]

@eyalroth

FYI swc seems to be incompatible with sequelize-typescript

Yeah, I believe only tsc (and ts-node without SWC) can handle emitDecoratorMetadata correctly. I tried almost all solutions such as babel, esbuild, swc but they doesn't emit decorator metadata correctly. I checked it by reading compiled code and it seems impossible.

Thus, I stopped using decorators. Moved from mikro-orm to objection, Type-GraphQL to Nexus, remove tsyringe, then moved from ts-node to esbuild. Now things became much better.

[eyalroth]

We've seen a lot of cases where someone thought they were using transpileOnly, but actually they weren't. So that's always a possibility whenever it doesn't seem like transpileOnly is making much impact: perhaps it's not even turned on.

@cspotcode I tried all the options -- cli argument, environment variable and tsconfig.json 🤷

Is this an issue with cross-file metadata, in other words, is it compatible or incompatible with isolatedModules?

I... have no idea. Looks like @acro5piano knows more about this 😄

[cspotcode]

Compilers such as babel, swc, and esbuild do not understand type information. If type information from one file affects the emitted code from another file, it will fail to emit the code you're expecting using these tools. TypeScript has an option isolatedModules which will raise typechecking errors to warn you that your code is incompatible with these tools.

https://www.typescriptlang.org/tsconfig/#isolatedModules

This is an important detail to understand: these errors are raised when using TypeScript to typecheck, not when using babel, swc, or esbuild to compile your code! The assumption here is that you run typechecking as part of your automated testing even when you use a non-typechecking compiler. Think of the typechecker as a lint step.

Unfortunately, the reason I closed this issue still applies here:
Without any reproductions for us to look at, we can't keep guessing at what may or may not be happening in someone's project. Given a reproduction, we can take a look.

[cspotcode]

Another thing worth mentioning:
It's understandable that many users cannot share a proprietary codebase. But they might be able to create a fake codebase with just as many lines of code, using the exact same configuration files, but all the code is copy-pasted nonsense. If a fake codebase exhibits similar performance, it gives us something concrete to look at.

[eyalroth]

It's understandable that many users cannot share a proprietary codebase. But they might be able to create a fake codebase with just as many lines of code, using the exact same configuration files, but all the code is copy-pasted nonsense. If a fake codebase exhibits similar performance, it gives us something concrete to look at.

Of course! I'm planning on creating such a fake repo. I just need time :)

[cspotcode]

Awesome, thank you! Feel free to share it here or create a separate issue if that would help to focus the conversation.

[eyalroth]

The strangest thing happened. I undid all the configuration changes I made yesterday, npm ci-ed, added--transpile-only alone, and now it seems that the tests start running much faster.

Sorry for all the fuss.