File globbing slow with large node_modules folder

[poelstra]

Atom Typescript adds the following filesGlob to tsconfig.json by default:

"./**/*.ts",
"!./node_modules/**/*.ts"

This indeed excludes things from node_modules, but it apparently still causes the full node_modules folder to be traversed.

I've tried replacing the second line with "./node_modules/**" and "./node_modules/" in order to prevent it from entering the directory at all, but that didn't seem to help (or didn't work at all).

We have a very large number of node modules in this particular tree, which makes Atom very slow for many operations (adding a file, renaming) and causes frequent hickups while editting. Haven't timed it exactly, but it's in the order of tens of seconds before e.g. a save of tsconfig.json 'notices' added or removed files.

As an experiment, I moved the tsconfig.json to the subdir that actually contains the TS code (which doesn't have a node_modules dir), and now saving tsconfig,json is 'just' a second or two (it's a fairly large codebase).

Somewhat related to #217, which also got slow due to traversing large number of directories.

Is there e.g. a different glob pattern that may help?

[poelstra]

Seems Microsoft is also trying to do something similar: microsoft/TypeScript#3043

[basarat]

can you try putting the exclude first:

"!./node_modules/**/*.ts",
"./**/*.ts"

[poelstra]

Tried these (note: we need the .d.ts line too, as we generate the .js etc next to the .ts in this project):

        "!./node_modules/**/*.ts",
        "./**/*.ts",
        "!./**/*.d.ts"

        "!./node_modules/**",
        "./**/*.ts",
        "!./**/*.d.ts"

        "!./node_modules/**/*",
        "./**/*.ts",
        "!./**/*.d.ts"

They are all still slow, and worse, they add .ts below node_modules files to the list.

Also tried

        "!./node_modules/**/*.ts",
        "./**/*.ts"

just to see if it was our last .d.ts line, but that still leads to stuff in node_modules to be included (and it's also slow).

[nycdotnet]

Out of curiosity, could you just do '!./node_modules' or '!./node_modules/' ? I don't think there's any reason to drag the files inside node_modules into it...

[poelstra]

I'm not at my work PC right now, but I think I tried that too and that it didn't work.

It was a bit cumbersome to test though, given that I typically had to wait a long time and often needed to kill Atom, so I may have overlooked something.

There also seems to be a difference between starting with 'nothing', a slash, or './', and I can't remember whether I tried them all. I simply moved the tsconfig a subdirectory for now...

[basarat]

Note: Workaround move tsconfig to some sub directory e.g. src/tsconfig.json

[snowcxt]

The workaround works for me. Thanks

[omidkrad]

AFAIK tsconfig.json does not support globes. When you're talking about adding or removing globs from tsconfig.json where is that config file exactly?

[basarat]

When you're talking about adding or removing globes from tsconfig.json

we're not adding / removing globs ... we are adding / removing files in the file system and maintaining that in the files property

[omidkrad]

You mean files/exclude in tsconfig.json is new behavior, and globes/filesGlob mentioned in the previous comments was the old way?

The thing is, even though moving tsconfig.json to src directory works but I prefer it to be on the root of my project. The structure of my project is like:

project
    +-- src
    |   +-- .ts files
    +-- node_modules
    +-- jspm_packages
    |   +-- packages
    |       +-- .d.ts files
    +-- definitions
    |   +-- .d.ts files
    +-- tsconfig.json

Even though I add node_modules to exclude in tsconfig.json, looks like atom-typescript is still loading it.

[snowcxt]

Even the workaround only works for the small projects. Now my project has 40+ ts/tsx files and 20+ d.ts files. Now it's very slow.

[basarat]

Even the workaround only works for the small projects. Now my project has 40+ ts/tsx files and 20+ d.ts files. Now it's very slow.

Shouldn't be the case. We have 300+ files and it works fine.

[snowcxt]

I'm using windows (I tested win7 and win8.1) and atom-typescript 7. Creating .ts file would take minutes (more than 5 minutes) to finish all the pending requesting. During the process, I can't build/format/resolve reference path for the ts files.

[intellix]

Also running into this issue. Was comparing my TS experience between Atom and VSC and it's pretty much unusable in Atom at the moment. As everyone says, I try to get autocomplete on something and it'll queue up requests for getProjectFileDetails and errorsForFile into the hundreds.

The tsconfig.json is inside /src as was suggested here. In fact, to reproduce is quite simple

• Create a project with angular-cli: https://github.com/angular/angular-cli

[tushar-chauhan]

I am also facing the same issue as @intellix (in a project generated by angular-cli). The number of pending requests keep on increasing with every keystroke. It will be great if this gets resolved soon.

[pablodenadai]

@intellix any luck fixing this issue with angular-cli?

[tom-schoener]

Same Problem with a fresh project using angular-cli. (Version 1.0.0-beta.5, node 5.7.1 on Windows 10)

[pablodenadai]

@tttee try adding

  "filesGlob": [
    "main.ts",
    "typings.d.ts",
    "app/**/*.ts",
    "!app/**/*.spec.ts"
  ]

to tsconfig.json file.

[tom-schoener]

@ghpabs Thank you, that works very very well. I don't know what filesGlob does, but I will look into it.

For comparison with filesGlob added to tsconfig.json:

Edit: If someone else stumbled upon this issue and wants to know what filesGlob does (Copied from https://atom.io/packages/atom-typescript):

Atom-TypeScript also supports filesGlob which will expand files for you based on minmatch|glob|regex (similar to grunt).

[Maistho]

@ghpabs Adding that to my tsconfig does not seem to solve the issue. Pending request still increasing each keystroke.
edit: nvm, I edited the wrong tsconfig. Works great!

[intellix]

ok so I just looked a little more into this. I see that defining a "filesGlob" as suggested above fixes it and gets it working again. angular-cli generates a project with tsconfig.json containing only a "files" option, like so:

"files": [
  "main.ts",
  "typings.d.ts"
]

This issue is about this happening if you have a large node_modules folder and so I tried adding this to rectify that:

"exclude": [
  "node_modules"
]

This doesn't solve the problem. It seems the issue is that Atom-Typescript doesn't work at all unless you have a filesGlob specified. Shouldn't it work with just files specified and shouldn't the exclude also help somewhat?

[Maistho]

Also, angular-cli places the tsconfig in src/ which should also exclude node_modules.

I suppose adding support for the files array would mitigate this issue entirely, and make atom-typescript more compatible with other editors

[Maistho]

After some digging around, I think the issue is when trying to edit something which is not exported (directly or re-exported) in a file that exists in the files array in tsconfig.json. If atom-typescript can find the file with filesGlob and add it, everything is fine, but removing filesGlob and editing a file which doesn't exist in the files array already will make it really slow. Only including main.ts and referencing everything from there doesn't work very well.

[jximenes]

Try to run Atom in administrator mode.
Works for me.

[Maistho]

That's not really a solution though. I've decided to take a further look at how this could be resolved in a couple of weeks, when I have more time. The issue doesn't seem to be trivial.

[Najibsaurus]

try add ur tsconfig.json like this :

{
  "compileOnSave": false,
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "moduleResolution": "node",
    "sourceMap": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "removeComments": true,
    "declaration": false,
    "noLib": false,
    "noImplicitAny": false
  },
  "filesGlob": [
    "main.ts",
    "typings.d.ts",
    "app/**/*.ts",
    "!app/**/*.spec.ts"
  ]
}

go to terminal rebuild ur tsconfig.json and then run in Administrator mode. it works for me.

[arlowhite]

Just a thought, but with TypeScript 2.0 coming soon, maybe "filesGlob" and calculating the "files" property should be deprecated.

So if your tsconfig.json has "include", Atom-Typescript goes into TypeScript 2.0 mode
https://github.com/Microsoft/TypeScript-Handbook/blob/master/pages/tsconfig.json.md

Would this be an easier/better solution?

---

I'm using TypeScript 2.0.2 and atom-typescript 10.1.6
I found that adding filesGlob (in addition to include and exclude) improved my rebuild time from 13 seconds to 5 seconds. (click the refresh icon in the TypeScript panel) tsc in the terminal takes 4 seconds.

tsconfig.json

  "include": [
    "app/assets/**/*.ts"
  ],
  "exclude": [
    "node_modules",
    "target"
  ],
  "filesGlob" : [
    "app/assets/**/*.ts",
    "!node_modules/**",
    "!target/**",
  ]

Note: paths are weird because I'm working on a Play project.

[jts-code]

Trying out the different solutions provided here has yielded a few interesting results. When I add the filesGlob to my tsconfig.json or move tsconfig.json to a sub-directory, I get the faster tooling and autocomplete, however, my project now Cannot find name 'module'.

If I remove the changes to tsconfig.json and run Atom as administrator, the error about the module disappears, and the auto complete runs somewhat faster, but not as fast as the filesGlob fix.