-
Notifications
You must be signed in to change notification settings - Fork 12.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JS/TS project supporting large projects (1000+ files) #19326
Comments
From @unional on October 11, 2017 23:25 Info 524 [16:18:3.299] remove project: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/js/pan/network/gp/GlobalProtectPortalsViewer.js
Info 525 [16:18:3.300] Close directory watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/js/pan/network/gp
Info 526 [16:18:3.300] Close directory watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/js/pan/network
Info 527 [16:18:3.300] Close directory watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/js/pan
Info 528 [16:18:3.301] Close directory watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/js
Info 529 [16:18:3.301] Close directory watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs
Info 530 [16:18:3.301] Close directory watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui
Info 531 [16:18:3.301] Close directory watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt
Info 532 [16:18:3.301] Close directory watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src
Info 533 [16:18:3.301] Close directory watcher for: c:/Users/hwong/p4/feature/fs4-kiev
Info 534 [16:18:3.301] Close directory watcher for: c:/Users/hwong/p4/feature
Info 535 [16:18:3.301] Close directory watcher for: c:/Users/hwong/p4
Info 536 [16:18:3.301] Close directory watcher for: c:/Users/hwong
Info 537 [16:18:3.301] Close directory watcher for: c:/Users
Info 538 [16:18:3.302] Project 'c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/tsconfig.json' (Configured) 0 Today I notice this. The folder EDIT: This is probably something different. |
From @unional on October 12, 2017 2:28 Info 12 [15:29:54.281] Add recursive watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/custom-typings
Info 13 [15:29:54.281] c for path c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/custom-typings not found in cache...
Info 14 [15:29:54.281] Starting C:\Program Files\Microsoft VS Code\Code.exe with args ["c:\\Program Files\\Microsoft VS Code\\resources\\app\\extensions\\node_modules\\typescript\\lib/watchGuard.js","c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/custom-typings"]
Info 15 [15:29:54.415] WatchGuard for path c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/custom-typings returned: OK
Info 16 [15:29:54.416] Add recursive watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/js/pan
Info 17 [15:29:54.416] watchDirectory for c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/js/pan uses cached drive information.
Info 18 [15:29:54.416] Add recursive watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/spec
Info 19 [15:29:54.416] watchDirectory for c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/spec uses cached drive information.
Info 20 [15:29:54.417] Add type root watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/node_modules/@types
Info 21 [15:29:54.417] watchDirectory for c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/node_modules/@types uses cached drive information. Noticed this: Info 12 [15:29:54.281] Add recursive watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/custom-typings
Info 13 [19:21:34.595] c for path c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/custom-typings not found in cache...
Info 14 [19:21:34.595] Starting C:\Program Files\Microsoft VS Code\Code.exe with args ["c:\\Program Files\\Microsoft VS Code\\resources\\app\\extensions\\node_modules\\typescript\\lib/watchGuard.js","c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/custom-typings"]
Info 15 [15:29:54.415] WatchGuard for path c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/custom-typings returned: OK While the large folder: Info 16 [15:29:54.416] Add recursive watcher for: c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/js/pan
Info 17 [19:21:34.733] watchDirectory for c:/Users/hwong/p4/feature/fs4-kiev/src/mgmt/ui/htdocs/js/pan uses cached drive information. uses cached drive info. Is there a way to clear the cache to test again? Is it possible that the cache is corrupted? |
From @unional on October 13, 2017 23:21 I try to build and run vscode from source code. But I can't find it in the developer tools when the warning appears. How do I debug into the TypeScript extension? |
From @unional on October 14, 2017 4:4 I dig in and discovered I have one large file that push the That file is a 9MB JS file. However, Also, that value is not configurable. On the other hand, it would be a good idea to fire an event mentioning which file(s) causing the language service to become disabled. |
From @unional on October 19, 2017 0:2 Also, I notice the file sizes are counted twice. I added a line here: https://github.com/Microsoft/TypeScript/blob/29ed92e20d31d637e92584216d55fb16cb80231e/src/server/editorServices.ts#L1374 to print out the Info 12 [16:57:20.882] max (20971520), avail (20971520), total (13548710)
...
Info 40 [16:59:36.825] max (20971520), avail (7422810), total (13548710) You can see the avail is decreased by the values in I have already removed all large items and now I don't have anything more to exclude. Help! 😞 |
This is an experimentally-derived value - at 20 MB of raw JS, we run out of address space in 32-bit nodejs and crash. In ts server, 64-bit node uses approximately twice as much memory for the same work, so this is where we'd be using about 3 GB of memory in 64-bit node and things would really not be responsive anyway. We don't have any good ideas right now for how to support that much JS at once. Having the user do some sort of configuration to reduce the scope of analysis is the only available fix in this situation. |
How about the double dip? It seems like if the double dip can be avoided, then it can support close to 2k files with the same 20MB |
@mjbvz This is a legacy JS and I'm trying to determine if we can migrate to using VSCode and TypeScript. In the smaller component I've got intellisense to partially work, but with some issues. Is this limitation a language service limitation only? Would TypeScript compiler be able to handle a project of this size even if intellisense is not fully functional? |
For me, I manually edit that value to 40MB. It is working fine so far. |
That might be helpful, but I'm trying to get some insights because I can't just simply try it. This legacy JS project has its own custom built module loading that TypeScript can't understand. |
Would love to get access to the source to see if there is something we can optimize here.
yes. unless you have a jsconfig.json (or tsconfig.json) to specify what the project shape is. The dependents are figured out by walking the imports/require statements in your file, and building a trasitive clusre of that.
We have limits on big files, files bigger than 5 MBs, we have found these to be usually minified libraries and not user code. We also have a total limit on whole program size to avoid crashing the server. this number is rather based on huristics and we can optimize it.. but i would need to have a project sample to be able to see what can be done.. our expiernece here is that if you have a tsconfig.json or jsconfig.json all files under this directory will be included, and we have found users who run into this limit usually have some directoriesd that either include dependenices or generated files, and does not happen for user code.
The language service needs to parse the files, and keep them in memory to be able to provide any information. memory is finite, and can be exhausted. As noted, we have found that with the appropriate setup the LS scales relatively well. We would love to look at your project and see if there are additional changes that we need to make.
The LS and the compiler use the same engine. We have projects of similar and bigger sizes, so the size does not seem too big. i would want to ensure we have the correct configuration and project organization though. |
It is an internal project. Probably wouldn't be helpful in the current state. It uses its own proprietary modules loading which doesn't conform to any of the popular module loaders. I'm attempting to do some programmatic refactoring to commonJS format, but there is a lot of work to be done. I've only partially done this as an attempt to assess if this is going to work.
The size of the project I originally listed is with no dependencies.
So I'm a long way from converting this project to actual TS. However, that is a long term potential outcome. However, based on this statement, if intellisense is breaking, then I'm to assume if I were to be able to actually convert this project to TS that the compiler also would have issues compiling it?
Since this is a massive legacy JS application, there isn't much I can do about the organization. The smallest unit of separation I have been able to manage so far would be about 2000+ JS modules that need to be seen as one project. What would be mostly helpful to me at this point is to better understand what is breaking now. I can't tell by the current logging which JS files are not being processed or how many of the 2000 files are being loaded/supported. Maybe the new additions to logging will be helpful. |
we would be happy to sign any required NDAs.
it will help us understand why your inteliisense is not working.
This suggests it is a configuration issue with your project, since the project does not have many dependencies that the engine understands already.
The size is not that out of the ordinary, we have other internal and external projects that are similar or bigger in size.
Converting or not is not the issue here. we should support .js projects regardless..
I do not know why your intellisense is breaking.. so i would not make this assumption.
you can start by looking at the tsserver log file. sharing that would help.. feel free to send it to me by email. |
Will look into what can be shared. How do I use |
VSCode ships with an older version of TypeScript. Please see Using Newer TypeScript Versions documentation for more details on updating your VSCode to use a different version of TypeScript. |
I was able to get the new logging. However, no files are listed with this log entry After this is a list of all of the file watchers. There are 2061 JS files listed there. 2061 is the correct number of JS files for this part of the project. FYI, executing this
|
Probably something broken during refactoring. I'll take a look into it. |
Closing this issue as we do not have enough information to investigate. Please try using the latest typescript@next to see if the problem has been fixed |
From @unional on October 3, 2017 22:36
VSCode: 1.16.1
Original discussion:
microsoft/vscode#9594
I followed this instruction: microsoft/vscode#9594 (comment)
But looking in
tsserver.log
, I don't see anything that stands out.Copied from original issue: microsoft/vscode#35549
The text was updated successfully, but these errors were encountered: