Solve "Out of memory" during generation by limiting the number of concurrent tasks

[ppoffice]

What does it do?

Allow users to set the number of files to be generated in parallel.
Users who are struggling with the "out of memory" problem (e.g., #3350 #3311) can limit Hexo's memory usage by hexo g --concurrency <number>.

How to test

1. hexo init a new blog.
2. change the hexo dependency to ppoffice/hexo#master in package.json
3. npm install
4. Add some posts git clone https://github.com/SukkaLab/hexo-many-posts.git source/_posts/tests
5. hexo g -c <number of concurrent tasks>

Testings

Environment

• hexo: https://github.com/ppoffice/hexo#master
• theme: landscape
• hexo-cli: 2.0.0
• os: Linux 5.0.0-21-generic linux x64
• node: 12.4.0
• node_flags: --max-old-space-size=8192

Test 1: hexo clean && hexo g

• No concurrency limit
• 659 files generated in 24 s
• Memory usage: 500 ~ 660 MB
  

Test 2: hexo clean && hexo g -c 20

• 20 concurrent tasks at most
• 659 files generated in 30 s
• Memory usage: 450 ~ 550 MB
  

Test 3: hexo clean && hexo g -c 10

• 10 concurrent tasks at most
• 659 files generated in 34 s
• Memory usage: 480 ~ 550 MB
  

Test 4: hexo clean && hexo g -c 5

• 5 concurrent tasks at most
• 659 files generated in 34 s
• Memory usage: 420 ~ 550 MB
  

Pull request tasks

• Add test cases for the changes.
• Passed the CI test.

[coveralls]

Coverage increased (+0.004%) to 97.157% when pulling 7a51bb4 on ppoffice:master into 48b506d on hexojs:master.

[curbengh]

LGTM, I don't mind --concurrency since it reflects the option in Promise.map(), but I figure some might prefer -p, --parallel.

Edit: -p, --parallel could be confused with -p, --path.

[tcrowe]

I have reviewed and tested it myself. Thank you @ppoffice and @curbengh

--concurrency -c is appropriate.

mkdir concurrency-test
cd concurrency-test
hexo init
npm install 'ppoffice/hexo#master'
git clone https://github.com/SukkaLab/hexo-many-posts.git source/_posts/tests
hexo gen
hexo clean
hexo gen -c 8
hexo clean
hexo gen --concurrency 1
hexo clean
hexo gen -c 200

[curbengh]

Related issues: #1763 #1735 #1529

[chenshiforever]

but I donn't kown what i can do ,this step2. change the hexo dependency to ppoffice/hexo#master in package.json is not understand . i think i have nothing to do except wait for it to update to 4.0.

[curbengh]

That is using ppoffice's fork, you can use this repository instead.

package.json
-  "hexo": "^3.9.0",
+  "hexo": "hexojs/hexo",

Modify the line with "hexo" in your "package.json" file, change the value from "^3.9.0", to "hexojs/hexo",

Example

This is akin to using bleeding-edge release.

[chenshiforever]

But this approach doesn't seem to be of much use. It does add more than a hundred posts, but exceeding that number will crash, and here's my error message
中文版:但是这个方法对我来说似乎没有起太大的作用。确实可以增加一百多的post数量，但是超过了这个数字就会崩溃，下面是我的报错信息

INFO  1968 files generated in 6.87 min
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node_module_register
 2: v8::internal::FatalProcessOutOfMemory
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::Factory::NewRawTwoByteString
 5: v8::internal::Smi::SmiPrint
 6: v8::internal::StackGuard::HandleInterrupts
 7: v8::String::WriteUtf8
 8: v8_inspector::V8InspectorClient::currentTimeMS
 9: node::Buffer::New
10: node::Buffer::New
11: 00000199789F17C6
npm ERR! code ELIFECYCLE
npm ERR! errno 3
npm ERR! hexo@0.0.1 push: `hexo cl && hexo g -c 30 && gulp && gulp mv && hexo s`
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the hexo@0.0.1 push script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

[ppoffice]

@chenshiforever
Please try to use a smaller concurrency value such as 10. By the way, what theme are you using? It may also have something to do with its optimization.