Bug when creating a single documentation for an entire monorepo

[matteobruni]

Search terms

monorepo, multiple packages

Question

I have a monorepo with a central core package and a lot of plugins that use it.

I'd like to have a single docs website that covers all of them, but I don't understand using the existing documentation how to achieve that.

For the monorepo I'm using Lerna + NX, without workspaces (I'm having issues with any kind of workspaces).

[Gerrit0]

You probably want to use typedoc --entryPointStrategy packages 'packages/*' which will:

1. Tell TypeDoc to use packages mode, which will look for a separate package.json file for each entry point and use that to find the module's entry point.
2. Tell TypeDoc to look for packages under packages/*

Monorepos aren't something I have a ton of experience with, if you're able to set up a PR for https://github.com/TypeStrong/typedoc-repros which creates a similar structure to your real project, I can probably give better advice.

[matteobruni]

You can see the structure in my lerna.json file here

Not all of them are necessary for TypeDoc, once I understand how to work with the entryPointStrategy packages I can figure it out myself.

It looks something that I can also document.

[matteobruni]

@Gerrit0 I've found a way to specify all packages to use for the typedoc output but I'm receiving this error now when I try running typedoc:

% yarn build:docs
yarn run v1.22.19
$ rimraf -f docs && typedoc
Loaded plugin typedoc-plugin-carbon-ads
Loaded plugin typedoc-plugin-clarity
Loaded plugin typedoc-plugin-coverage
Loaded plugin typedoc-plugin-keyword
Loaded plugin typedoc-plugin-missing-exports

<--- Last few GCs --->

[70446:0x138008000]    34301 ms: Mark-sweep 4049.3 (4140.5) -> 4036.6 (4143.6) MB, 1304.1 / 0.0 ms  (average mu = 0.097, current mu = 0.028) allocation failure scavenge might not succeed
[70446:0x138008000]    35562 ms: Mark-sweep 4052.6 (4143.6) -> 4040.3 (4147.3) MB, 1222.0 / 0.0 ms  (average mu = 0.062, current mu = 0.031) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100d2c2dc node::Abort() [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
 2: 0x100d2c464 node::errors::TryCatchScope::~TryCatchScope() [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
 3: 0x100e7bb90 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
 4: 0x100e7bb24 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
 5: 0x100fff23c v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
 6: 0x100ffdd5c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
 7: 0x1010090a4 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
 8: 0x101009138 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
 9: 0x100fdbfcc v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
10: 0x1013140d0 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
11: 0x10162874c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
12: 0x101604e48 Builtins_MapIteratorPrototypeNext [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
13: 0x105f25f78 
14: 0x105c6e988 
15: 0x105f26dcc 
16: 0x105c5e064 
17: 0x105c3aadc 
18: 0x105c52bac 
19: 0x1060a1320 
20: 0x106136838 
21: 0x105d8d27c 
22: 0x10687a650 
23: 0x105c53d34 
24: 0x105c48500 
25: 0x105da5ba8 
26: 0x10666d980 
27: 0x106636e74 
28: 0x105c53320 
29: 0x105c48500 
30: 0x105da5ba8 
31: 0x10666d980 
32: 0x105c48938 
33: 0x105da5ba8 
34: 0x10666d980 
35: 0x1065f991c 
36: 0x105d9d700 
37: 0x1068950ac 
38: 0x105d8893c 
39: 0x1060e91f4 
40: 0x105c6b5e0 
41: 0x105f36000 
42: 0x106722080 
43: 0x105d8a7fc 
44: 0x105f233b8 
45: 0x105fe39c8 
46: 0x1068535b8 
47: 0x106850670 
48: 0x1067ad810 
49: 0x106857104 
50: 0x10684dc54 
51: 0x1068513d4 
52: 0x106859bc0 
53: 0x1067ae724 
54: 0x10684fee4 
55: 0x1067ae84c 
56: 0x1067aeb10 
57: 0x106067900 
58: 0x10689369c 
59: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
60: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
61: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
62: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
63: 0x1015e93e4 Builtins_FlatMapIntoArray [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
64: 0x1015e9c54 Builtins_ArrayPrototypeFlatMap [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
65: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
66: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
67: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
68: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
69: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
70: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
71: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
72: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
73: 0x1015bc3d8 Builtins_InterpreterEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
74: 0x1015ba1cc Builtins_JSEntryTrampoline [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
75: 0x1015b9e64 Builtins_JSEntry [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
76: 0x100f8ba24 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
77: 0x100f8b0b8 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
78: 0x100e98548 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
79: 0x100cfc07c node::ExecuteBootstrapper(node::Environment*, char const*, std::__1::vector<v8::Local<v8::String>, std::__1::allocator<v8::Local<v8::String> > >*, std::__1::vector<v8::Local<v8::Value>, std::__1::allocator<v8::Local<v8::Value> > >*) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
80: 0x100cfd0e0 node::StartExecution(node::Environment*, char const*) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
81: 0x100cfcf8c node::StartExecution(node::Environment*, std::__1::function<v8::MaybeLocal<v8::Value> (node::StartExecutionCallbackInfo const&)>) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
82: 0x100c7da24 node::LoadEnvironment(node::Environment*, std::__1::function<v8::MaybeLocal<v8::Value> (node::StartExecutionCallbackInfo const&)>) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
83: 0x100d656b0 node::NodeMainInstance::Run(int*, node::Environment*) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
84: 0x100d653ac node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
85: 0x100cff2e0 node::Start(int, char**) [/Users/matteo/.nvm/versions/node/v16.15.1/bin/node]
86: 0x10596d08c 
/bin/sh: line 1: 70446 Abort trap: 6           typedoc
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

The repository is this: https://github.com/matteobruni/tsparticles/tree/dev

This is the typedoc.json file: https://github.com/matteobruni/tsparticles/blob/dev/typedoc.json

The commands for building the whole repository are (from the project root):

yarn
npx lerna bootstrap
npx lerna run build

But I think only yarn could be enough since the global TypeDoc is not using Lerna.

For building the broken TypeDoc configuration (from the project root):

yarn build:docs

[Gerrit0]

https://github.com/matteobruni/tsparticles/blob/5db88726ea9992f3df140ac3f4ba7d2d776589df/typedoc.json#L5-L70

... that's 65 entry points. In packages mode, that means that's 65 concurrent programs TypeDoc needs to construct. It's no surprise you don't have enough memory for it, that'd probably need 20-25gb of memory to complete.

I want to re-work packages mode so that TypeDoc only creates one ts.Program at a time, but haven't gotten around to it. This is being tracked under #1606.

[matteobruni]

https://github.com/matteobruni/tsparticles/blob/5db88726ea9992f3df140ac3f4ba7d2d776589df/typedoc.json#L5-L70

... that's 65 entry points. In packages mode, that means that's 65 concurrent programs TypeDoc needs to construct. It's no surprise you don't have enough memory for it, that'd probably need 20-25gb of memory to complete.

I want to re-work packages mode so that TypeDoc only creates one ts.Program at a time, but haven't gotten around to it. This is being tracked under #1606.

Just for test, I've tried allocating 25GB to the node process but the heap message still happened. At least it worked for 30 minutes, I have 32GB total, but I'm not going to try more than 25GB unless it could help the debug process.

This is the configuration used, maybe someone will have more luck than me.

node --max-old-space-size=25600 node_modules/.bin/typedoc