JavaScript heap out of memory

[bignass04]

I am using Pattern lab Node v2.8.0 on Mac with Node v6.9.1.

ERROR

I get the following error when my app is building the handlebars patterns:
Security context: 0x81bb64cfb51
2: string(aka string) [/Users/mnasser/ha50/node_modules/json5/lib/json5.js:~218] [pc=0x1dd9da48b83e] (this=0x81bb6404381 )
3: object(aka object) [/Users/mnasser/ha50/node_modules/json5/lib/json5.js:~433] [pc=0x1dd9da48ee26] (this=0x81bb6404381 )
4: object(aka object) [/Users/mnasser/ha50/node_modules/json5/lib/json5.js:~433] [pc=0x1dd9da48eed4] (this=0x81bb64043...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/Users/mnasser/.nvm/versions/node/v6.9.1/bin/node]
2: node::FatalException(v8::Isolate, v8::Localv8::Value, v8::Localv8::Message) [/Users/mnasser/.nvm/versions/node/v6.9.1/bin/node]
3: v8::internal::V8::FatalProcessOutOfMemory(char const, bool) [/Users/mnasser/.nvm/versions/node/v6.9.1/bin/node]
4: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/mnasser/.nvm/versions/node/v6.9.1/bin/node]
5: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object*, v8::internal::Isolate) [/Users/mnasser/.nvm/versions/node/v6.9.1/bin/node]
6: 0x1dd9d9f079a7
sh: line 1: 97019 Abort trap: 6 node scripts/patternlab.js build

RESOLUTION

I found that I can resolve the error if I go to patternlab-node/core/lib/patternlab.js and change line 428 from
allFooterData = JSON5.parse(JSON5.stringify(patternlab.data));
to
allFooterData = JSON.parse(JSON.stringify(patternlab.data));

and change line 373 from
allData = JSON5.parse(JSON5.stringify(patternlab.data));
to
allData = JSON.parse(JSON.stringify(patternlab.data));

BENCHMARKS

The build times were recorded with one time stamp displayed immediately after line 32 of patternlab-node/core/lib/patternlab.js as the start time and one immediately after the build process is completed as the end time.

• Number of patterns: 76
• Build time with JSON5 (in seconds): 35.3
• Build time with JSON (in seconds): 7.08

[bmuenzenmeyer]

Additional info.

JSON5 was added with #293
Coincidentally this is also when #316 was first reported. I want to look into this angle a bit more, and think that we can likely support both via configuration. (Speed vs listed json parsing)

I also recall that handlebars engine might have a bigger object structure than others, but I think this is still a valid thread to tug at.

cc @bramsmulders

[bmuenzenmeyer]

my own benchmarking:

node: 6.9.5
PL Node 2.8.0
data: 68 lines <--- almost none
patterns: 120 mixed between mustache and handlebars
build time with JSON5 (in seconds): 4.36s
build time with JSON (in seconds): 3.64s (17% faster)
adding more data, say, by including the starterkit-mustache-demo json, did not have an appreciable difference.

@bignass04, what is the total size of your data.json file, or your overall data footprint? do you have lots of pattern-specific data, for instance?

[simonknittel]

We did a quick fix for our CI servers by creating an file named fix-build.sh containing:

echo $(uname)

if [ "$(uname)" = 'Linux' ]; then
  echo 'Linux detected.'
  sed -i "s/allFooterData = JSON5.parse(JSON5.stringify(patternlab.data));/allFooterData = JSON.parse(JSON.stringify(patternlab.data));/g" node_modules/patternlab-node/core/lib/patternlab.js
  sed -i "s/allData = JSON5.parse(JSON5.stringify(patternlab.data));/allData = JSON.parse(JSON.stringify(patternlab.data));/g" node_modules/patternlab-node/core/lib/patternlab.js
elif [ "$(uname)" = 'Darwin' ]; then
  echo 'macOS detected.'
  sed -i "" "s/allFooterData = JSON5.parse(JSON5.stringify(patternlab.data));/allFooterData = JSON.parse(JSON.stringify(patternlab.data));/g" node_modules/patternlab-node/core/lib/patternlab.js
  sed -i "" "s/allData = JSON5.parse(JSON5.stringify(patternlab.data));/allData = JSON.parse(JSON.stringify(patternlab.data));/g" node_modules/patternlab-node/core/lib/patternlab.js
fi

And then adding it to the postinstall script of npm: "postinstall": "sh fix-build.sh"

[bmuenzenmeyer]

Wow that's a decently involved fix - I will prioritize a better solution.

[bignass04]

@bmuenzenmeyer The total data.json file is 714KB. I don't think we have much pattern-specific data but I haven't done much work on this application yet.

[bmuenzenmeyer]

for reference, mine above was 2KB

Quite the difference.

[simonknittel]

Our data.json is currently at 78KB, but with 440 individual/pattern specific JSON files.

[bmuenzenmeyer]

I've generated some test data and will be using that for further analysis

[bmuenzenmeyer]

Testing:

• Node: 6.9.5
• Pattern Lab Node 2.8.0
• Edition Gulp `1.401
• Starterkit Mustache Demo latest
• Test data as mentioned in JavaScript heap out of memory #637 (comment)
• "cleanPublic": false

---

1. JSON5.parse(JSON5.stringify(patternlab.data));

   gulp patternlab:build
   completed in 34 seconds

2. JSON.parse(JSON.stringify(patternlab.data));

   gulp patternlab:build
   completed in 8.68 seconds

3. jsonCopy(patternlab.data, 'config.paths.source.data'); <-- new helper file, which lints the JSON and then uses lodash cloneDeep

   gulp patternlab:build
   completed in 11 seconds

4. jsonCopy(patternlab.data, 'config.paths.source.data'); <-- new helper file, only uses lodash cloneDeep, does not lint (which could allow malformed JSON into rendering)

   gulp patternlab:build
   completed in 10 seconds

[bmuenzenmeyer]

I am going with a pure JSON.stringify(JSON.parse(data)) implementation, and going to retain the try catch for other applications as they may arise. proper linting should occur at time of read, not copy.

[bmuenzenmeyer]

I think I have fixed this with PL Node 2.9.X

[bmuenzenmeyer]

resolved (i think) with 2.9.0