Compiler OOM in watch mode and with --d #21140

aldendaniels · 2018-01-11T16:52:42Z

Process is running out out of memory on TS 2.6.2 in watch mode.

To reproduce:

Run tsc --noemit --watch --project .
Edit a file

The process hangs for a minute or so, then crashes with a node out-of-memory exception.

This does not appear to be related to strict generic checks (see #19662), as the exception still occurs when running tsc --noemit --watch --noStrictGenerics --project . (presuming that the --noStrictGenerics flag overrides the .tsconfig).

This is possibly a dup of #19253. It's a large project (2700+ files) and we're hitting performances challenges generally so project size is likely a factor. I can't share the project as the code is proprietary, but would be happy to jump on a screen share to further diagnose if that would be helpful.

cc: @mhegazy

The text was updated successfully, but these errors were encountered:

mhegazy · 2018-01-11T17:55:27Z

Can you try typescript@next?
Can you share the project otuside of github? we wold be happy to sign any required NDA's to get access to the code base.

aldendaniels · 2018-01-11T18:24:21Z

@mhegazy thanks for the awesome support. In typescript@next the initial watch run never completes (a different problem).

In TS 2.6 it seems that the OOM exception only occurs on certain files (but there it does occur reliably). On other files the incremental watch works, but is slow (10+ seconds for a change that affects the local scope of a single function).

Running MacOS. Trace (although don't expect it to be useful) is:


<--- Last few GCs --->

[8826:0x103000000]   426050 ms: Mark-sweep 1402.7 (1458.8) -> 1402.7 (1455.8) MB, 1498.3 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1498 ms) last resort GC in old space requested
[8826:0x103000000]   427533 ms: Mark-sweep 1402.7 (1455.8) -> 1402.7 (1455.8) MB, 1481.8 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2fc54a1a5ee1 <JSObject>
    2: getRelationKey(aka getRelationKey) [/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:~27132] [pc=0x11326b6b0ec6](this=0x2fc5ccb82311 <undefined>,source=0x2fc551d97001 <Type map = 0x2fc53121d481>,target=0x2fc574935979 <Type map = 0x2fc53121d481>,relation=0x2fc529348871 <Map map = 0x2fc568f048d9>)
    3: isTypeRelatedTo(aka isTypeRelatedTo) [/Users/alden/Projects/krypton/nod...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 5: v8::internal::Factory::NumberToString(v8::internal::Handle<v8::internal::Object>, bool) [/usr/local/bin/node]
 6: v8::internal::Runtime_NumberToString(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
 7: 0x11326a40463d
[1]    8826 abort      tsc --noemit --watch --project .

I've reached out internally to see if we can give you access with an NDA. Will follow up when I hear back.

sheetalkamat · 2018-01-11T18:40:39Z

@aldendaniels Just want to make sure you dont run into this exception when you run tsc without watch option?

aldendaniels · 2018-01-11T18:46:42Z

@sheetalkamat That's correct. The exception only occurs in watch mode.

aldendaniels · 2018-01-11T22:58:24Z

@mhegazy To circle back on getting a repo - preference on our end would be to start digging on a screen-share to get a repro before granting access to the repo due to possible overhead on getting the NDA signed on both ends.

Let me know if this is something you'd be open to and thanks for the help regardless. It's worth noting that this issue isn't critical for us, because the IDE integration still works as well as the non-watch mode. We don't make heavy use of watch internally ATM.

mhegazy · 2018-01-11T23:22:04Z

I think we can arrange for a screen sharing session.

aldendaniels · 2018-01-12T00:11:50Z

Nice - thank you! Email is alden [at] coda [dot] io. Should be available to jump on a call during west-coast business hours.

davydof · 2018-02-13T19:26:32Z

Fixed for me with "typescript": "2.8.0-dev.20180213"

sheetalkamat · 2018-03-09T23:16:10Z

@aldendaniels can you give a try to typescript@next to see if your issue still repros? If the issue still repros can you also try out environment variables and options as mentioned in #21243 to see if any of that helps?

aldendaniels · 2018-03-12T14:43:19Z

@sheetalkamat For starters, my baseline behavior as changed. We've upgraded to 2.7.2 and now when I run tsc --noemit --watch --project . the process hangs indefinitely (and burns CPU) but doesn't crash.

I tried on typescript@2.8.0-dev.20180308 and the process crashed with the following traceback:

7:41:27 AM - Starting compilation in watch mode...


/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:60890
                throw e;
                ^

Error: Debug Failure. False expression: Target symbol and symbol do not have the same name
    at getLiteralTypeFromPropertyName (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:26229:30)
    at Object.map (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:515:29)
    at getLiteralTypeFromPropertyNames (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:26241:36)
    at getIndexType (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:26249:33)
    at instantiateType (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:27069:28)
    at instantiateType (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:27072:91)
    at getTypeOfInstantiatedSymbol (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:23502:32)
    at getTypeOfSymbol (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:23524:24)
    at getTypeOfParameter (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:33998:24)
    at getTypeAtPosition (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:34010:53)

sheetalkamat · 2018-03-12T18:10:12Z

That assert failure is #22403 and is already fixed. Our nightly builds are not being published and is tracked in issue #22455. Once that is resolved you would be able to try out typescript@next

aldendaniels · 2018-03-14T00:23:55Z

@sheetalkamat Looks like that issue is fixed but there's still no build out - please let me know when the build is out and I'll retry

Bnaya · 2018-03-14T09:34:55Z

@aldendaniels looks like there's a new version '2.8.0-dev.20180314': '2018-03-14T06:38:06.997Z'

aldendaniels · 2018-03-14T15:49:03Z

@Bnaya thanks for the heads up. Looks like we get a new error now:

8:47:12 AM - Starting compilation in watch mode...


/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:61401
                throw e;
                ^

Error: Debug Failure. False expression: Deferring unused identifier check twice
    at registerForUnusedIdentifiersCheck (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:36999:26)
    at checkFunctionOrMethodDeclaration (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:36995:13)
    at checkMethodDeclaration (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:36110:13)
    at checkSourceElement (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:39110:28)
    at Object.forEach (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:306:30)
    at checkClassExpressionDeferred (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:38165:16)
    at checkDeferredNodes (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:39293:25)
    at checkSourceFileWorker (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:39317:17)
    at checkSourceFile (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:39300:13)
    at getDiagnosticsWorker (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:39355:17)

sheetalkamat · 2018-03-15T17:56:48Z

@aldendaniels The bug for the exception you were seeing, just got fixed. Please try out build that will be released tonight.

aldendaniels · 2018-03-15T18:03:13Z

@sheetalkamat Good news! Will do.

mohsen1 · 2018-03-15T22:18:04Z

Getting Deferring unused identifier check twice in our project with 2.8 as well

sheetalkamat · 2018-03-22T16:22:46Z

@aldendaniels were you able to try the latest build out to see if the issue is fixed. Thanks

aldendaniels · 2018-03-24T00:46:04Z

Sorry, thanks for the ping. Now I'm getting this on 2.9.0-dev.20180323:

Starting compilation in watch mode...


/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:2424
            throw e;
            ^

Error: Debug Failure. 272
    at bindThisPropertyAssignment (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:18480:30)
    at bindWorker (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:18224:29)
    at bind (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:18128:13)
    at visitNode (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:11188:24)
    at Object.forEachChild (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:11413:24)
    at bindEachChild (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:17085:16)
    at bindChildrenWorker (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:17182:21)
    at bindChildren (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:17055:17)
    at bind (/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:18134:21)
    at /Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:17061:68

weswigham · 2018-05-25T00:45:39Z

@aldendaniels Can you run tsc --extendedDiagnostics --noEmit --project . and post the output (should have times and numbers) so we can get an idea of how large the project is (in terms of our internal objects), or does that crash, too?

aldendaniels · 2018-05-30T00:25:54Z

@weswigham sorry for the delay - this is what I get:


<--- Last few GCs --->

[17219:0x105800000]   163243 ms: Mark-sweep 1417.4 (1493.4) -> 1417.4 (1462.4) MB, 1202.9 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1203 ms) last resort GC in old space requested
[17219:0x105800000]   164514 ms: Mark-sweep 1417.4 (1462.4) -> 1417.4 (1462.4) MB, 1270.3 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x5c68aba57c1 <JSObject>
    1: checkTypeRelatedTo(aka checkTypeRelatedTo) [/Users/alden/Projects/krypton/node_modules/typescript/lib/tsc.js:~28043] [pc=0x9cf9cdb9970](this=0x5c6bf4022d1 <undefined>,source=0x5c655d4f721 <Type map = 0x5c68f4fa3e9>,target=0x5c643be9111 <Type map = 0x5c6b5e90d21>,relation=0x5c643be9b69 <Map map = 0x5c68f4848d9>,errorNode=0x5c6bf4022d1 <undefined>,headMessage=0x5c6bf4022d1 <undefined>,contai...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
 5: v8::internal::Factory::NewCodeRaw(int, bool) [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
 6: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, int) [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
 7: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::TurboAssembler*, v8::internal::EhFrameWriter*, v8::internal::CompilationInfo*, v8::internal::Handle<v8::internal::Object>) [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
 8: v8::internal::compiler::CodeGenerator::FinalizeCode() [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
 9: v8::internal::compiler::PipelineImpl::FinalizeCode() [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
10: v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl() [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
11: v8::internal::Compiler::FinalizeCompilationJob(v8::internal::CompilationJob*) [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
12: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
13: v8::internal::StackGuard::HandleInterrupts() [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
14: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/alden/Projects/krypton/build/.dev-coda-node/bin/node]
15: 0x9cf9bf842fd
16: 0x9cf9cdb9970
[1]    17219 abort      tsc --extendedDiagnostics --noEmit --project . > ~/Desktop/ts-output.txt

It looks like the OOM error might be preventing other useful logging.

weswigham · 2018-05-30T00:35:01Z

Yeah, that OOM isn't related to declarations or watch mode at all (and it runs oom while still typechecking) - without a repro it's going to be incredibly hard to debug! Then again, you said the project was >2700 files, right? Projects of that size are usually incredibly sensitive to increased memory usage on our part (even slightly) because they're already at or near the process limit. Does the project compile if you run the compiler with more memory allocated to node? (eg, node --max_old_space_size=4096 /path/to/tsc.js --otheroptions) If it does then it's just that our memory usage went up a bit and the project is too large for us to build all at once now (but we have some upcoming stuff that could help), if it doesn't, then there's a serious bug causing us to consume exponential memory for something that's going to be hard to fix without a repro of some kind.

aldendaniels · 2018-05-30T15:45:06Z

@weswigham Yes, pumping the memory allotment for node does the trick. Sorry for the red herring. I'd been using an alias a coworker had added for the normal run, so I didn't realize we'd bumped the memory requirement until running in watch mode. You're correct that this isn't watch-mode related.

We have some upcoming stuff that could help

This would be great. Generally, typescript performance is becoming painful for us. We migrated from flow to typescript sometime back - it's been a good conversion on several fronts, but performance has been inferior - admittedly it's not an apples-to-apples comparison as we we've typed more code since the conversion. Still, we noticed the difference immediately. I realize that our project is larger than most and there's optimizations we can make by compartmentalizing our builds, etc.

Examples include:

High memory usage.
10 second TS overhead running a test
Slower webpack builds
Over a minute on a fast laptop to typecheck the repo
For some reason VSCode is responsive but Webstorm is slow

I realize there's no easy fix (feel free to close this issue) but thought I'd drop the feedback as you evaluate prioritization choices.

mhegazy · 2018-05-30T17:25:37Z

@aldendaniels we would love to get access to your sources and debug more to see what is causing the OOM issues. we would be happy to sign an NDA as needed.

aldendaniels · 2018-05-31T00:21:01Z

@mhegazy Thank you! This would be great. Can you shoot me an email (alden at coda dot io) and we can take it from there?

mhegazy · 2018-05-31T00:30:06Z

my email is mhegazy at microsoft dot com.

aldendaniels · 2018-05-31T18:23:26Z

Thanks! Moved to email.

RyanCavanaugh · 2019-03-13T17:43:05Z

@aldendaniels are you still hitting this? mhegazy no longer works on TypeScript so we'll need to get a fresh copy of the repro if so

mhegazy assigned sheetalkamat Jan 11, 2018

mhegazy added Needs More Info The issue still hasn't been fully clarified Bug A bug in TypeScript labels Jan 11, 2018

mhegazy added this to the TypeScript 2.7.1 milestone Jan 11, 2018

mhegazy modified the milestones: TypeScript 2.7.1, TypeScript 2.8 Jan 30, 2018

mhegazy modified the milestones: TypeScript 2.8, TypeScript 2.8.1 Mar 9, 2018

sheetalkamat mentioned this issue Mar 14, 2018

Debug failure in registerForUnusedIdentifiersCheck #22578

Closed

DanielRosenwasser mentioned this issue Mar 23, 2018

[2.8.0-rc] Segfault when running compiler with --d or --watch (out of memory) #22826

Open

mhegazy removed this from the TypeScript 2.8.1 milestone Mar 26, 2018

mhegazy modified the milestones: TypeScript 2.9.2, TypeScript 3.0 May 30, 2018

sheetalkamat removed their assignment Jun 26, 2018

mhegazy modified the milestones: TypeScript 3.0, TypeScript 3.0.1 Jul 2, 2018

mhegazy modified the milestones: TypeScript 3.0.1, TypeScript 3.1 Jul 18, 2018

RyanCavanaugh modified the milestones: TypeScript 3.1, TypeScript 3.2 Aug 23, 2018

DanielRosenwasser modified the milestones: TypeScript 3.2, TypeScript 3.3 Oct 30, 2018

RyanCavanaugh modified the milestones: TypeScript 3.3, TypeScript 3.4.0 Feb 1, 2019

RyanCavanaugh unassigned weswigham Mar 13, 2019

RyanCavanaugh modified the milestones: TypeScript 3.4.0, Backlog Mar 13, 2019

RyanCavanaugh added Needs More Info The issue still hasn't been fully clarified and removed Bug A bug in TypeScript labels Mar 13, 2019

sheetalkamat mentioned this issue Oct 23, 2020

New flag assumeChangesAffectShape for incremental and watch scenario #41219

Closed

RyanCavanaugh closed this as completed Feb 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compiler OOM in watch mode and with --d #21140

Compiler OOM in watch mode and with --d #21140

aldendaniels commented Jan 11, 2018

mhegazy commented Jan 11, 2018

aldendaniels commented Jan 11, 2018

sheetalkamat commented Jan 11, 2018

aldendaniels commented Jan 11, 2018

aldendaniels commented Jan 11, 2018

mhegazy commented Jan 11, 2018

aldendaniels commented Jan 12, 2018

davydof commented Feb 13, 2018

sheetalkamat commented Mar 9, 2018

aldendaniels commented Mar 12, 2018

sheetalkamat commented Mar 12, 2018

aldendaniels commented Mar 14, 2018

Bnaya commented Mar 14, 2018

aldendaniels commented Mar 14, 2018

sheetalkamat commented Mar 15, 2018

aldendaniels commented Mar 15, 2018

mohsen1 commented Mar 15, 2018

sheetalkamat commented Mar 22, 2018

aldendaniels commented Mar 24, 2018

weswigham commented May 25, 2018

aldendaniels commented May 30, 2018

weswigham commented May 30, 2018

aldendaniels commented May 30, 2018

mhegazy commented May 30, 2018

aldendaniels commented May 31, 2018

mhegazy commented May 31, 2018

aldendaniels commented May 31, 2018

RyanCavanaugh commented Mar 13, 2019

Compiler OOM in watch mode and with --d #21140

Compiler OOM in watch mode and with --d #21140

Comments

aldendaniels commented Jan 11, 2018

mhegazy commented Jan 11, 2018

aldendaniels commented Jan 11, 2018

sheetalkamat commented Jan 11, 2018

aldendaniels commented Jan 11, 2018

aldendaniels commented Jan 11, 2018

mhegazy commented Jan 11, 2018

aldendaniels commented Jan 12, 2018

davydof commented Feb 13, 2018

sheetalkamat commented Mar 9, 2018

aldendaniels commented Mar 12, 2018

sheetalkamat commented Mar 12, 2018

aldendaniels commented Mar 14, 2018

Bnaya commented Mar 14, 2018

aldendaniels commented Mar 14, 2018

sheetalkamat commented Mar 15, 2018

aldendaniels commented Mar 15, 2018

mohsen1 commented Mar 15, 2018

sheetalkamat commented Mar 22, 2018

aldendaniels commented Mar 24, 2018

weswigham commented May 25, 2018

aldendaniels commented May 30, 2018

weswigham commented May 30, 2018

aldendaniels commented May 30, 2018

mhegazy commented May 30, 2018

aldendaniels commented May 31, 2018

mhegazy commented May 31, 2018

aldendaniels commented May 31, 2018

RyanCavanaugh commented Mar 13, 2019