Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TSC Regression causing crash (Heap OOM) between version 4.8.4 & 4.9.3 in large project #53087

Open
whitespine opened this issue Mar 4, 2023 · 50 comments
Assignees
Labels
Needs Investigation This issue needs a team member to investigate its status. Rescheduled This issue was previously scheduled to an earlier milestone

Comments

@whitespine
Copy link

Bug Report

When updating typescript within our project from version 4.7.3 to a more recent version, our project began experiencing OOM issues.
Specifically, it is completely unable to build via tsc or even tsc --noEmit.
Ours is a somewhat large project, so I couldn't point to a minimal reproduction (sorry!).
I have 16GB of RAM so I doubt its a "classic" memory error, seems more likely something started infinitely using memory during a version upgrade.

<--- Last few GCs --->

[7915:0x5f6b230]    32781 ms: Mark-sweep (reduce) 2041.6 (2082.5) -> 2040.8 (2083.0) MB, 492.6 / 0.0 ms  (average mu = 0.088, current mu = 0.038) allocation failure; scavenge might not succeed
[7915:0x5f6b230]    33342 ms: Mark-sweep (reduce) 2041.9 (2083.0) -> 2041.1 (2083.2) MB, 556.2 / 0.0 ms  (average mu = 0.047, current mu = 0.009) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb6d470 node::Abort() [node]
 2: 0xa7e0a8  [node]
 3: 0xd46ee0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd47287 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf24645  [node]
 6: 0xf25548 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xf35a53  [node]
 8: 0xf368c8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xf1122e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xf125f7 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
11: 0xef37ca v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
12: 0x12b6d6f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
13: 0x16e89b9  [node]
[1]    7915 abort      tsc --noEmit

🔎 Search Terms

tsc 4.9 4.9.5 Out of Memory Heap Out of Memory Crash Killed Silent Error
(Am i doing this right?)

🕗 Version & Regression Information

Error began occuring when updating from typescript 4.7.3 to any 4.9.3^ version.
Issue persists into Version 5.1.0-dev.20230303 (nightly at time of testing).
Issue is NOT present in v4.8.4, or any prior version we tried (that satisfied our dependencies, anyways.)

  • This is a crash
  • This changed between versions 4.8.4 and 4.9.3

💻 Code

Branch state is: https://github.com/Eranziel/foundryvtt-lancer/tree/oom

Was unable to produce a minimally viable reproduction, as the OOM error gives next to no context as to where the issue occurs or why. Sorry. More than willing to provide more investigative information, but I don't really know what there is to provide.

🙁 Actual behavior

It crashes on more recent versions for no clear reason.

🙂 Expected behavior

I'd rather it didn't.

@liuxingbaoyu
Copy link

FYI: You can try https://github.com/microsoft/TypeScript/wiki/Performance-Tracing, it helped us locate the problem in #50515.

@whitespine
Copy link
Author

FYI: You can try https://github.com/microsoft/TypeScript/wiki/Performance-Tracing, it helped us locate the problem in #50515.

I will attempt this as soon as i have recruited someone with enough ram to run it without crashing :)

@whitespine
Copy link
Author

4.8.4 Diagnostic information:

Files:                        1023
Lines of Library:            28146
Lines of Definitions:       218302
Lines of TypeScript:         25936
Lines of JavaScript:             0
Lines of JSON:                   0
Lines of Other:                  0
Nodes of Library:           118224
Nodes of Definitions:       564026
Nodes of TypeScript:        115815
Nodes of JavaScript:             0
Nodes of JSON:                   0
Nodes of Other:                  0
Identifiers:                301992
Symbols:                    865373
Types:                      208799
Instantiations:             365698
Memory used:               909729K
Assignability cache size:    27600
Identity cache size:          4422
Subtype cache size:           7237
Strict subtype cache size:    9428
Tracing time:                0.17s
I/O Read time:               0.06s
Parse time:                  0.96s
ResolveModule time:          0.08s
ResolveTypeReference time:   0.00s
Program time:                1.27s
Bind time:                   0.62s
Check time:                  2.85s
printTime time:              0.00s
Emit time:                   0.00s
Dump types time:             9.03s
Total time:                  4.74s

5.1.0-dev.20230305 Diagnostic information:

Files:                         1029
Lines of Library:             29397
Lines of Definitions:        218302
Lines of TypeScript:          25916
Lines of JavaScript:              0
Lines of JSON:                    0
Lines of Other:                   0
Identifiers:                 302941
Symbols:                    6242486
Types:                       702542
Instantiations:              417693
Memory used:               5496499K
Assignability cache size:     31947
Identity cache size:           4714
Subtype cache size:            5307
Strict subtype cache size:     7579
Tracing time:                 0.08s
I/O Read time:                0.04s
Parse time:                   0.94s
ResolveModule time:           0.09s
ResolveTypeReference time:    0.00s
Program time:                 1.19s
Bind time:                    0.49s
Check time:                  57.68s
printTime time:               0.00s
Emit time:                    0.00s
Dump types time:              7.27s
Total time:                  59.37s

Notably there is an about 5x increase in memory usage, as well as a >5x increase in the "Types" and "Symbols".

Now that I can actually build on v10 there several new errors, most interesting among them the following:

src/module/item/license-sheet.ts:37:23 - error TS2590: Expression produces a union type that is too complex to represent.

The line is ```TS
for (let d of docs as LancerItem[])

    // Find the assoc frame
    for (let et of [EntryType.FRAME, EntryType.MECH_SYSTEM, EntryType.MECH_WEAPON, EntryType.WEAPON_MOD]) {
      let pack = game.packs.get(`world.${et}`);
      if (pack) {
        let docs = await pack.getDocuments({ "system.license": license.system.key });
        for (let d of docs as LancerItem[]) {
          let rank = (d as any).system.license_level as number;
          while (unlocks.length <= rank) {
            unlocks.push([]);
          }
          unlocks[rank].push(d);
        }
      }
    }

Where docs is of type:
docs: StoredDocument<LancerActor | Scene | LancerItem | JournalEntry | Macro | Playlist | RollTable>[].
Here's where "it's a big project" sort of bites us, in that I can't really minimally represent these types here in this comment. Each is a unique class, all with a shared parent class.

Maybe this is indicative of large union types being the root of our problem?

...

Observing the trace, the file src/module/comp-builder.ts is responsible for an entire 53 SECONDS of the total 57 second Check Time. It produces no errors.
image

Unsure how to proceed from here.

@ritschwumm
Copy link

i had a quite similar issue recently after upgrading typescript. i can't remember the details, but it seems the problem was that under some circumstances tsc started creating a lot of types internally when asked to infer the type of a (typewise) complicated function. in the end, writing const foo:FunctionType = (it) => ... instead of const foo = (it:FunctionArgs):FunctionReturn => ... solved the issue. maybe something similar is happening here?

@RyanCavanaugh RyanCavanaugh added the Needs Investigation This issue needs a team member to investigate its status. label Mar 7, 2023
@RyanCavanaugh RyanCavanaugh added this to the TypeScript 5.1.0 milestone Mar 7, 2023
@Raynos
Copy link

Raynos commented Mar 15, 2023

Ran into this issue myself today on a large project.

@arlyon
Copy link

arlyon commented Mar 17, 2023

Similar regression on 5.0.2. For me, 4.8.3 is ok.

thanks for the exceptional bug report @whitespine!

@karlhorky

This comment was marked as off-topic.

@jakebailey

This comment was marked as off-topic.

@Raynos
Copy link

Raynos commented Mar 20, 2023

I was able to solve the OOM issue for myself with

NODE_OPTIONS=--max-old-space-size=8192 tsc

@JarekToro
Copy link

Since @Raynos solution is really just a stop gap. What is needed to further this ticket. We also have a large project that works fine on 4.8.4 but cant upgrade to 4.9 or 5.0 because tsc gets an OOM error

@JarekToro
Copy link

Here is what I have gathered from running some test.
Test Ran with
NODE_OPTIONS=--max-old-space-size=8192 tsc --noEmit --diagnostics --extendedDiagnostics --incremental false
Here is 4.8

Files:                        2979
Lines of Library:            33311
Lines of Definitions:       233669
Lines of TypeScript:         59133
Lines of JavaScript:             0
Lines of JSON:                 943
Lines of Other:                  0
Nodes of Library:           148246
Nodes of Definitions:       723896
Nodes of TypeScript:        250983
Nodes of JavaScript:             0
Nodes of JSON:                2256
Nodes of Other:                  0
Identifiers:                382980
Symbols:                    597111
Types:                      216450
Instantiations:            7252666
Memory used:               684086K
Assignability cache size:   202376
Identity cache size:          3213
Subtype cache size:          12303
Strict subtype cache size:    4719
I/O Read time:               0.11s
Parse time:                  1.95s
ResolveModule time:          0.58s
ResolveTypeReference time:   0.01s
Program time:                3.04s
Bind time:                   1.00s
Check time:                 16.47s
printTime time:              0.00s
Emit time:                   0.00s
Total time:                 20.50s

Here is 4.9

Files:                         2980
Lines of Library:             33853
Lines of Definitions:        233669
Lines of TypeScript:          59133
Lines of JavaScript:              0
Lines of JSON:                  943
Lines of Other:                   0
Identifiers:                 384065
Symbols:                     571900
Types:                     18112640
Instantiations:             7055141
Memory used:               5726426K
Assignability cache size:  10914208
Identity cache size:           3215
Subtype cache size:           11958
Strict subtype cache size:     4719
I/O Read time:                0.15s
Parse time:                   1.72s
ResolveModule time:           0.47s
ResolveTypeReference time:    0.01s
Program time:                 2.65s
Bind time:                    0.79s
Check time:                 199.89s
printTime time:               0.00s
Emit time:                    0.00s
Total time:                 203.33s

I also generated a trace
image

From looking into the trace. It seems to be stuck in a loop related to the i18next & react-i18next

This is basically the part that loops (if I followed the trace right)

import type { Namespace, TFuncKey } from 'i18next';
import { i18n } from './i18n';

export interface I18nText<N extends Namespace = Namespace> {
  namespace: N;
  keyName: TFuncKey<N>; // <-- TFuncKey seems to be the culprit
}

That lib is for sure complicated typewise. But it worked as it should in 4.8 and breaks compilation in 4.9 and 5.0

Anything else I can provide? @jakebailey

@jakebailey
Copy link
Member

We've made some improvements recently that feel like this issue; can you try the nightly build?

@JarekToro
Copy link

JarekToro commented Mar 28, 2023

Using Version "5.1.0-dev.20230328"

Running NODE_OPTIONS=--max-old-space-size=8192 tsc --noEmit --pretty --diagnostics --extendedDiagnostics --incremental false

Result

/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:114282
      throw e;
      ^

RangeError: Value undefined out of range for undefined options property undefined
    at Map.set (<anonymous>)
    at recursiveTypeRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60159:24)
    at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59590:122)
    at eachTypeRelatedToType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59946:25)
    at unionOrIntersectionRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59768:174)
    at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59590:39)
    at structuredTypeRelatedToWorker (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60559:25)
    at structuredTypeRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60172:21)
    at recursiveTypeRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:60142:19)
    at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59590:122)

Note

This took about 4mins to run before failing

@jakebailey
Copy link
Member

Uh, hm, well, it's certainly not supposed to crash anymore. If you have the time, if you can roll it back by date to find when it started crashing, that would be helpful.

@JarekToro
Copy link

Phew! Ok finally I got it.

Last version that did not crash. 5.0.0-dev.20221103
Which makes the first version that crashed: 5.0.0-dev.20221108

Note that they were still very slow. And still needed NODE_OPTIONS=--max-old-space-size=8192` set.

Small Note

From 5.0.0-dev.20230210 and lower the error message slightly changed from what I posted earlier to

/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:113758
      throw e;
      ^

RangeError: Value undefined out of range for undefined options property undefined
    at Map.set (<anonymous>)
    at getIndexedAccessTypeOrUndefined (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56795:28)
    at getIndexedAccessType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56764:12)
    at /home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56687:68
    at map (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:209:19)
    at distributeIndexOverObjectType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56687:21)
    at getSimplifiedIndexedAccessType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56710:37)
    at getSimplifiedType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:56683:55)
    at getNormalizedType (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:58949:481)
    at isRelatedTo (/home/jarek/Development/project/node_modules/typescript/lib/tsc.js:59262:23)

@jakebailey
Copy link
Member

Oh, yikes, those are the versions before and after the module transform (where things were temporarily broken). Is your project somewhere that I can test it?

@JarekToro
Copy link

It is not. Tomorrow I can perhaps try and create a repo that replicates some of the i18next stuff we are doing with types and see if I can get it to have the same issue.

@JarekToro
Copy link

JarekToro commented Mar 29, 2023

@jakebailey Well hey it wasn't that bad, I was able to replicate it with little code

https://github.com/JarekToro/TSC-Regression-test-5.0

I understand this to be the offending line.
https://github.com/JarekToro/TSC-Regression-test-5.0/blob/main/utils.ts#L15

If you remove that line. Or remove the function in its entirety. It builds correctly.

Running NODE_OPTIONS=--max-old-space-size=8192 tsc --noEmit --pretty --diagnostics --extendedDiagnostics --incremental false  To Test

@jakebailey
Copy link
Member

I'm trying to look at this, but it seems like this thread contains many different OOMs and claims about which versions do and don't work.

The only repro I have is @JarekToro's, but that explicitly says that something changed in 5.0. But, if I go back to TS 4.8, that code fails to compile entirely, saying that it's excessively deep. 4.9 hangs, as does 5.0. Is it supposed to complete?

But, that's still not the original reported issue. I guess what I should be checking is https://github.com/Eranziel/foundryvtt-lancer/tree/oom?

@jakebailey
Copy link
Member

Testing the case from the original issue, a bisect points to #51151. @weswigham

@jakebailey
Copy link
Member

Reverting the above also fixes @JarekToro's case, so at least it seems like all of the cases in this thread with repros are the same thing.

@Raynos
Copy link

Raynos commented Apr 4, 2023

My OOM complaint is not a regression per se, just an artifact of a large codebase, I had multiple different versions & copies of the entire aws-sdk v2 library loaded into my project and each copy of aws-sdk has an insanely large amount of symbols.

I don't have a reproduction example at hand, but npm dedupe aws-sdk also resolved my OOM issue to reduce the amount of symbols.

@JarekToro
Copy link

JarekToro commented Apr 4, 2023

Reverting the above also fixes @JarekToro's case, so at least it seems like all of the cases in this thread with repros are the same thing.

@jakebailey That's good to hear, So yes in regards to my reproduction I was trying to focus on what version the build failed at. Which while related is technically different. Then the original issue. And while I simplified the code base to reproduce the crash. I didn't check to see if it passed on 4.8.4. If I have time I will try to make a separate repro of the slow down difference.

@jakebailey
Copy link
Member

My OOM complaint is not a regression per se, just an artifact of a large codebase, I had multiple different versions & copies of the entire aws-sdk v2 library loaded into my project and each copy of aws-sdk has an insanely large amount of symbols.

I don't have a reproduction example at hand, but npm dedupe aws-sdk also resolved my OOM issue to reduce the amount of symbols.

Yeah, that might just be a different issue. We've had similar problems on DefinitelyTyped regarding that. If you can see something change between two different TS versions in the --extendedDiagnostics, that might be interesting, though.

@xobotyi
Copy link

xobotyi commented Apr 20, 2023

Have same issue with 5.0.4 along with 5.1.0-beta
also using i18next along with react-i18next

@fesieg
Copy link

fesieg commented May 16, 2023

facing the same problem when upgrading to TS 5+

our tsc build slowly grows its memory usage until it eventually crashes with @whitespine 's error

@melroy89

This comment was marked as off-topic.

@jakebailey
Copy link
Member

jakebailey commented Jun 3, 2023

This issue explicitly says something broke between 4.8 and 4.9. If you have a problem going from 5.0 to 5.1, it is almost assuredly another issue and I would suggest filing a new one or looking for another recent issue about your problem which mentions 5.1.

@ernestostifano
Copy link

Updated from 4.9.5 to 5.1.3 today and having the same exact issue. Build hangs until it runs OOM. Using references and paths in a Yarn PnP monorepo.

@jakebailey
Copy link
Member

@ernestostifano As above, this issue is intended to be for a specific regression between 4.8 and 4.9; if you're jumping from 4.9 to 5.1, it's unlikely to be the same thing. Can you please file a new issue and also test 5.0 to better narrow the range?

@melroy89

This comment was marked as spam.

@jakebailey
Copy link
Member

We cannot possibly keep track of things if all OOM errors are being dumped into one thread. It's much easier for us to track and ensure all of these problems are fixed when they are split into multiple issues with supporting reproductions. We'll dedupe them if they are the same.

@fesieg
Copy link

fesieg commented Jun 5, 2023

We cannot possibly keep track of things if all OOM errors are being dumped into one thread. It's much easier for us to track and ensure all of these problems are fixed when they are split into multiple issues with supporting reproductions. We'll dedupe them if they are the same.

You have a point. Going to watch #54517 from now on, even though the error is the same.

@ernestostifano
Copy link

@jakebailey but there are several comments in this thread referring to versions above 5.0.0. Maybe your affected range is not correct. We have a really big project too, everything was working fine on 4.9.5 and just updating TypeScript to 5.1.3 triggered this issue. The funny thing is that the build hangs on a package containing i18next as a dependency, just like the examples above.

The build just take forever until it crashes going OOM.

We downgraded to 5.0.4 and the issue remains.

There is a comment from @JarekToro above reporting that the first affected version is 5.0.0-dev.20221108.

Since ours is a very sensitive project, we will just have to downgrade to 4.9.5 until this issue gets solved.

@ernestostifano
Copy link

We cannot possibly keep track of things if all OOM errors are being dumped into one thread. It's much easier for us to track and ensure all of these problems are fixed when they are split into multiple issues with supporting reproductions. We'll dedupe them if they are the same.

@jakebailey I understand your point and me too will watch #54517 instead, but it really looks like we are talking about the same issue here.

@melroy89
Copy link

melroy89 commented Jun 5, 2023

It's much easier for us to track and ensure all of these problems are fixed when they are split into multiple issues with supporting reproductions.

OK, created a separate issue: #54542

@ashylen
Copy link

ashylen commented Jul 11, 2023

For everyone that are struggling with i18n, bumping packages to following versions seems to fixed issue with OOM for Typescript v4.9.5 for me.

Previous:

"i18next": "22.5.0"
"react-i18next": "12.3.1"

New:

"i18next": "23.2.8"
"react-i18next": "13.0.2"

However, now I've been bombarded with a lot of type errors 😄

@esetnik
Copy link

esetnik commented Jul 11, 2023

@ashylen unfortunately the latest versions of i18next didn't resolve the issue for me. See reproducible example here i18next/i18next#1921 (comment)

@stephenasuncionDEV
Copy link

Still getting this issue. I'm getting the following error:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

when running tsc --pretty --noEmit even if I set the space size like the following:

cross-env NODE_OPTIONS=--max-old-space-size=10240 tsc --pretty --noEmit

@wakaztahir
Copy link

Typescript compiler 4.8.4 works, I downgraded to build my project

@jakebailey
Copy link
Member

This thread has lots and lots of people with likely different issues; I've recently published a tool which makes it easy to bisect issues in TypeScript without knowing how to build TypeScript itself: https://www.npmjs.com/package/every-ts#bisecting

It'd be extremely helpful for those who know which versions do and don't work to use this tool to identify which change in particular broke things; doing so will allow us to categorize these issues.

@RyanCavanaugh RyanCavanaugh added the Rescheduled This issue was previously scheduled to an earlier milestone label Mar 4, 2024
@felixmeziere
Copy link

felixmeziere commented Mar 14, 2024

We have the same issue as original and it is also with i18next, it's been blocking us from upgrading the library for almost a year :/

@jakebailey
Copy link
Member

Do you have a repo that can be cloned and tested consistently?

@felixmeziere
Copy link

Hi, I didn't manage to create a repo reproducing the error but I have found a fix for the issue in i18next, I will report back there with a PR in 2-3 weeks if everything has worked as expected in the meantime, and link it here so that you can analyse wether it's something ts could get better at.
Basically the issue has to do with values and inferences within conditional types being computed when they shouldn't (moving a ternary up the type stack and doing some type caching with const fixed the issue (not sure what I say strictly makes sense as I'm a noob in typing jargon but you get the idea)).

i18next/i18next#2138 (comment)

@felixmeziere
Copy link

In the meantime here is what seems to do the trick, in case someone wants to have a look i18next/i18next@master...GreenGoTech:i18next:master

Aviortheking added a commit to tcgdex/cards-database that referenced this issue May 11, 2024
Signed-off-by: Avior <github@avior.me>
@NomanGul
Copy link

FYI: You can try https://github.com/microsoft/TypeScript/wiki/Performance-Tracing, it helped us locate the problem in #50515.

doing this helped me trace the actual problem. I had some recursive types in my code that were causing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs Investigation This issue needs a team member to investigate its status. Rescheduled This issue was previously scheduled to an earlier milestone
Projects
None yet
Development

No branches or pull requests