OOM on large monorepo project #1606

avatarneil · 2021-06-22T15:15:39Z

Search terms

OOM, Out Of Memory, Heap Allocation, Crash, Performance

Expected Behavior

I expected pnpm typedoc --packages 'packages/*' to run without crashing!

Actual Behavior

pnpm typedoc --packages 'packages/*' crashes when running on >3 packages within our monorepo. It's worth noting, however, that TypeDoc executes perfectly fine when attempting to generate docs for only a handful of packages within the monorepo (e.g. pnpm typedoc --packages 'packages/mds-types' --packages 'packages/mds-agency').

Stack Trace

<--- Last few GCs --->

[45163:0x118008000] 52690 ms: Mark-sweep 4047.0 (4134.4) -> 4036.0 (4139.4) MB, 1383.4 / 0.0 ms (average mu = 0.423, current mu = 0.122) allocation failure scavenge might not succeed
[45163:0x118008000] 54171 ms: Mark-sweep 4052.5 (4140.0) -> 4040.8 (4144.0) MB, 1434.9 / 0.0 ms (average mu = 0.269, current mu = 0.032) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
1: 0x101390285 node::Abort() (.cold.1) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
2: 0x1000c5e59 node::Abort() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
3: 0x1000c5fbf node::OnFatalError(char const*, char const*) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
4: 0x100242fd7 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
5: 0x100242f73 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
6: 0x1003fe015 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
7: 0x10045b674 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
8: 0x100442ceb void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
9: 0x100442816 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
10: 0x100442516 v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
11: 0x1004603ae v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
12: 0x100418102 v8::internal::ItemParallelJob::Task::RunInternal() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
13: 0x100418588 v8::internal::ItemParallelJob::Run() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
14: 0x100444625 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
15: 0x1004441e7 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
16: 0x10042ef87 v8::internal::MarkCompactCollector::Evacuate() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
17: 0x10042c84b v8::internal::MarkCompactCollector::CollectGarbage() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
18: 0x1003fe734 v8::internal::Heap::MarkCompact() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
19: 0x1003fb723 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
20: 0x1003f95cd v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
21: 0x10040722a v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
22: 0x1004072b1 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
23: 0x1003d41dd v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
24: 0x100759ebf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
25: 0x100ad6939 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
ERROR Command was killed with SIGABRT (Aborted): typedoc --packages packages/*

It seems like this could be pretty tricky to trace down, as there are no pointers in the stack trace to TypeDoc code (at least as far as I can tell) :/

I wonder if this could simply be due to over-parallelization, and if that's the case (perhaps this is a dumb suggestion), could there be an option added to generate documentation serially instead of in parallel to lighten the memory pressure?

Steps to reproduce the bug

Apologies in advance for not being able to create a minimal-repro project for this; I believe this is an issue that arises due to the size of a project.

Clone mds-core
git checkout feature/neil/upgrade-typedoc-to-0.21.0
Ensure pnpm is installed (this is a pnpm based project, YMMV using npm/yarn)
pnpm clean && pnpm build (clears any local build artifacts that may exist, installs dependencies, builds JS artifacts)
pnpm typedoc --packages 'packages/*'

Environment

Typedoc version: 0.21.0
TypeScript version: 4.2.4
Node.js version: 15.11.0
OS: MacOS 12.0b1

The text was updated successfully, but these errors were encountered:

Gerrit0 · 2021-06-23T03:42:30Z

Hmm... does typedoc crash before or after creating all the programs? (Does rendering json instead of HTML work?) Unfortunately typedoc relies on having program info throughout the rendering process, so there isn't currently a good way to switch it around to only create one ts.Program at a time. Fixing this would nearly enable generating docs from TypeDoc's json output, which I'd like to be able to someday anyways...

avatarneil · 2021-06-23T16:15:43Z

@Gerrit0 It seems like with the default memory allocated to node on my machine, emitting to JSON also fails; however, when I bump node's memory limit up to ~32GB, after about 10 minutes it almost succeeds! It seems like memory usage begins dropping, but prior to the emission of the JSON I am faced with a different (I imagine likely unrelated) error 😿 :

TypeDoc exiting with unexpected error:
TypeError: Cannot read property 'map' of undefined
    at Object.convertType (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/types.js:254:57)
    at convertType (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/types.js:96:34)
    at /Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/types.js:254:71
    at Array.map (<anonymous>)
    at Object.convertType (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/types.js:254:57)
    at Object.convertType (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/types.js:96:34)
    at Converter.convertType (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/converter.js:61:24)
    at Object.createSignature (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/factories/signature.js:37:41)
    at convertVariableAsFunction (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/symbols.js:441:21)
    at Object.convertVariable (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/symbols.js:404:16)

Thanks for all your great work on this project and for the quick response, we used to use TypeDoc for generating our project documentation (older version of Typescript & pre-0.20.0 TypeDoc), and our team is eager to be able to use TypeDoc again!

Gerrit0 · 2021-06-26T23:52:58Z

Unfortunately the convertType there means that you still haven't gotten through conversion, so TypeDoc is still working on converting one of the projects.

I think fixing this requires doing a couple things:

Refactor such that resolution happens in stages, and after resolution, no references to compiler nodes or symbols are kept. This is necessary since if we don't do this, then we have to keep every program in memory until all resolution is complete
Refactor to create only one ts.Program at a time

This is a rather large change... it's possible, but by no means trivial.

danni · 2022-01-15T09:30:08Z

I've run into what looks like the same bug. Medium-sized mono-repo using project references to build, publishes an acyclic graph of packages. Uses a yarn workspace and lerna.

Runs out of memory in packages mode. It does however complete if I give it a list of entrypoints in resolve mode. It doesn't seem to make any difference if I give a list of packages entrypoints in packages mode.

This project was only recently ported to project references (from a pure lerna setup) and I have memory of my pre-project references setup working correctly. But I foolishly didn't keep that branch, if you're interested I can recreate it.

`typedoc` builds successfully on my local ARM Mac, but runs out of memory on GitHub Actions. I suspect that the limit needs to be bumped slightly to make things go for now. This is a consequence of TypeStrong/typedoc#1606 Also includes a config change of typedoc to stop emitting source code, and to exclude functions marked as `@internal`. Plus, ESLint will now ignore built docs.

jlarmstrongiv · 2022-06-17T16:10:00Z

Any workarounds besides increasing memory?

Is it possible to build each project’s typedoc individually and then merge them at the end?

Giving it a list of entrypoints in resolve mode did not work for me.

Gerrit0 · 2022-06-18T14:06:16Z

Is it possible to build each project’s typedoc individually and then merge them at the end?

Once 0.23 is out, this is what I plan to work on next, and is how I'd like packages mode to always work, but it's not there yet.

jlarmstrongiv · 2022-06-18T16:11:19Z

Cool! Looking forward to it

That’s similar to how merging jest coverage reports with nyc works

matteobruni · 2022-07-15T15:30:24Z

I've reported a test in #1971 replying to a message there, you can read the full comment here: #1971 (comment)

Long story short, tested allocating 25 GB for my monorepo without success.

I'd like to help about that, but I don't know where to start.

Gerrit0 · 2022-07-17T23:04:48Z

Fixing this is going to involve quite a few changes:

We need to be able to deserialize JSON emitted with --json (requires a review of all current serialized properties to ensure that important ones are included)
Events need to be revisited/reworked. Currently, plugins listening to conversion events don't have a way to tell when TypeDoc should be finished with a ts.Program and thus should throw away all references to it, so they do it on conversion end.
(Not strictly required, but I think will make 1 easier) rework any reflection/type models which store a reference to any ts.* object (exception: ts.__String) to not. The big one here is ReferenceType needs to not store a ts.Symbol... not sure what this looks like quite yet.

Gerrit0 · 2022-11-07T02:31:20Z

Bit of a progress update - the beta branch today has working deserialization code - still needs a lot of work in the sense that TypeDoc doesn't know what to do with this, but.... it's progress!

Next steps:

Support .json files as entry points, which get deserialized as previously converted project
Create a merge projects function
Update packages mode to run TypeDoc in each folder, emitting JSON to a cache location, and then merge projects

mikelehen · 2022-11-15T16:50:32Z

I don't know if this is helpful, but we're running into the same issue. We currently have 6 packages in our monorepo (https://github.com/covid-projections/act-now-packages). It's honestly not a ton of code (some of the packages are trivial). What I notice is that:

$ yarn typedoc packages/*/src/index.ts runs in 10 seconds and only needs a 1G node heap.

$ yarn typedoc --entryPointStrategy packages packages/* runs in 50 seconds and needs 5G of node heap.

So not only is there a big memory hit for using the monorepo support, but there's also a big performance hit (even though I'm on an M1 Max MacBook Pro with lots of cores). And because of #1835 the first command actually results in working cross-linking, while the second one does not. Of course the big "downside" is that the generated docs show file paths (e.g. "metrics/src") instead of actual package names (e.g. "@actnowcoalition/metrics") and don't show per-module readme files, etc.

So I find myself wanting the speed / memory usage of not using monorepo support, but the doc formatting of the monorepo support. No idea if that's feasible.

MYoung25 · 2022-12-15T20:47:12Z

Hopefully some of this information is helpful, we have run into the same issue with aws-sdk-js-v3. Special thanks to @eduardomourar for a draft PR that can provide a reproduction: aws/aws-sdk-js-v3#4139

After allocating 250gb of RAM to the build, it continued to crash with a (different!) error: std::bad_alloc

note: the number after the line 1 reference changes on every attempt

An OS max_map_count mmap limit was causing the std::bad_alloc failure, and after increasing the max_map_count, we were able to get a build to succeed after 161 minutes, with >200gb of RAM utilization.

Gerrit0 · 2023-03-06T00:09:28Z

I'm not fully confident it's fully correct yet, but I just released 0.24.0-beta.1 which completely reworks how packages mode works. It will now effectively run TypeDoc in each package folder, then merge the json created for each package.

docs: not written yet, out of time for this week
changelog: https://github.com/TypeStrong/typedoc/blob/beta/CHANGELOG.md

I ran this on the tsParticles repo, which previously required >25gb to work with packages mode, and TypeDoc finished in 120 seconds, using a maximum of 970mb RAM. (One difference: typedoc-plugin-missing-exports was not enabled) diff

stuft2 · 2023-04-05T09:02:37Z

@Gerrit0 Nice work! I'm using 0.24.0-beta.8 and it's working great!

Gerrit0 · 2023-04-08T22:48:04Z

v0.24.0 has released for real now :)

akphi · 2023-04-08T23:24:01Z

@Gerrit0 works breezily well with my project! The only one thing I saw warning about lodash typings in a module where I created utils wrapper around lodash utils, not sure if this has something to do with exclude not working well with entryPointStrategy: packages or not? If you don't have any idea off top your head, I'll find time to file a bug.

But really appreciate your effort man 🙏

Gerrit0 · 2023-04-08T23:26:47Z

Please do open an issue, I thought I'd ironed out all of the cross-module issues, but it's certainly possible I missed something

avatarneil added the bug Functionality does not match expectation label Jun 22, 2021

avatarneil mentioned this issue Jun 22, 2021

[documentation] database schema lacuna-tech/mds-core#624

Open

Gerrit0 mentioned this issue Jan 15, 2022

Typedoc confuses any error types. #1828

Closed

This was referenced Mar 18, 2022

chore(docs): Increase Node memory limit for docs inikulin/parse5#467

Merged

Improve readability of source code #1889

Merged

This was referenced Jul 1, 2022

Excessive memory/time usage (12GB) #1974

Closed

Bug when creating a single documentation for an entire monorepo #1971

Closed

matteobruni mentioned this issue Jul 15, 2022

2.2.0 tsparticles/tsparticles#4341

Merged

matteobruni mentioned this issue Aug 12, 2022

TypeDoc has been loaded multiple times happening in a subfolder #2034

Closed

stuft2 mentioned this issue Oct 4, 2022

OOM Generating Documentation byu-oit/openapi-js#3

Open

eduardomourar mentioned this issue Nov 2, 2022

Bump typedoc to v0.23.x aws/aws-sdk-js-v3#4134

Closed

2 tasks

mikelehen mentioned this issue Nov 15, 2022

Investigate 'Javascript heap out of memory' error act-now-coalition/act-now-packages#356

Closed

mikelehen mentioned this issue Jan 10, 2023

Reenable typedoc with workarounds to avoid out-of-memory crash. act-now-coalition/act-now-packages#399

Merged

Gerrit0 mentioned this issue Mar 10, 2023

v0.24 Beta Release #2197

Closed

8 tasks

Gerrit0 added this to the v0.24 milestone Mar 11, 2023

Gerrit0 closed this as completed Apr 8, 2023

timothymcmackin added a commit to timothymcmackin/js-sdk that referenced this issue Jun 22, 2023

Upgrade typedoc to avoid TypeStrong/typedoc/issues/1606

c6e0904

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OOM on large monorepo project #1606

OOM on large monorepo project #1606

avatarneil commented Jun 22, 2021

Gerrit0 commented Jun 23, 2021

avatarneil commented Jun 23, 2021

Gerrit0 commented Jun 26, 2021

danni commented Jan 15, 2022

jlarmstrongiv commented Jun 17, 2022 •

edited

Loading

Gerrit0 commented Jun 18, 2022

jlarmstrongiv commented Jun 18, 2022

matteobruni commented Jul 15, 2022

Gerrit0 commented Jul 17, 2022

Gerrit0 commented Nov 7, 2022

mikelehen commented Nov 15, 2022

MYoung25 commented Dec 15, 2022

Gerrit0 commented Mar 6, 2023

stuft2 commented Apr 5, 2023

Gerrit0 commented Apr 8, 2023

akphi commented Apr 8, 2023

Gerrit0 commented Apr 8, 2023

OOM on large monorepo project #1606

OOM on large monorepo project #1606

Comments

avatarneil commented Jun 22, 2021

Search terms

Expected Behavior

Actual Behavior

Steps to reproduce the bug

Environment

Gerrit0 commented Jun 23, 2021

avatarneil commented Jun 23, 2021

Gerrit0 commented Jun 26, 2021

danni commented Jan 15, 2022

jlarmstrongiv commented Jun 17, 2022 • edited Loading

Gerrit0 commented Jun 18, 2022

jlarmstrongiv commented Jun 18, 2022

matteobruni commented Jul 15, 2022

Gerrit0 commented Jul 17, 2022

Gerrit0 commented Nov 7, 2022

mikelehen commented Nov 15, 2022

MYoung25 commented Dec 15, 2022

Gerrit0 commented Mar 6, 2023

stuft2 commented Apr 5, 2023

Gerrit0 commented Apr 8, 2023

akphi commented Apr 8, 2023

Gerrit0 commented Apr 8, 2023

jlarmstrongiv commented Jun 17, 2022 •

edited

Loading