WPEN is not scrapeable anymore in 16GB of RAM

[kelson42]

See https://farm.openzim.org/tasks/5d5c519f1339d1137618f4b1
An additional effort should be done on memory consumption

[kelson42]

@ISNIT0 @mgautierfr Any idea how we could do better?

[kelson42]

Here is the monitoring at that time

[kelson42]

@ISNIT0 I'm not so sure anymore mwoffliner has crashed because of lack of memory considering that we had still a lot of memory cached.

[kelson42]

@ISNIT0 I have asked @rgaudin to restart a scrape, the error is quite weird to me. I want to see if the problem is reproducable.

[ISNIT0]

@kelson42 Any news?

Looks like it fails when trying to resize the favicon, not sure what your monitoring says:

Failed to run mwoffliner after [116020s]: {
	"stack": "Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:366:11)\n    at Object.spawn (child_process.js:551:9)\n    at exec2 (/tmp/mwoffliner/node_modules/imagemagick/imagemagick.js:24:25)\n    at Object.exports.convert (/tmp/mwoffliner/node_modules/imagemagick/imagemagick.js:253:10)\n    at Promise (/tmp/mwoffliner/lib/mwoffliner.lib.js:423:47)\n    at new Promise (<anonymous>)\n    at resizeFavicon (/tmp/mwoffliner/lib/mwoffliner.lib.js:422:28)\n    at /tmp/mwoffliner/lib/mwoffliner.lib.js:439:28\n    at Generator.next (<anonymous>)\n    at /tmp/mwoffliner/lib/mwoffliner.lib.js:9:71\n    at new Promise (<anonymous>)\n    at __awaiter (/tmp/mwoffliner/lib/mwoffliner.lib.js:5:12)\n    at saveFavicon (/tmp/mwoffliner/lib/mwoffliner.lib.js:419:20)\n    at /tmp/mwoffliner/lib/mwoffliner.lib.js:337:23\n    at Generator.next (<anonymous>)\n    at fulfilled (/tmp/mwoffliner/lib/mwoffliner.lib.js:6:58)\n    at process._tickCallback (internal/process/next_tick.js:68:7)",
	"message": "spawn ENOMEM",
	"errno": "ENOMEM",
	"code": "ENOMEM",
	"syscall": "spawn"
}


**********

spawn ENOMEM

**********

MWO has scraped all the content, but post-processing the favicon seems to crash...

[rgaudin]

Failed again in the favicon resize. https://farm.openzim.org/tasks/5d611a341339d1137618f565

[ISNIT0]

Looks like it's related to this: nodejs/node#25382

[kelson42]

@ISNIT0 You told me that the problems comes probably from the fact that a process fork happens here at at time the Node.js process is already quite big. Maybe the solution is simply to move this image resize to the start of the scraping process?

[ISNIT0]

It seems imagemin (our image compression library) also suffers from this problem, so we could potentially save even more memory by finding a node configuration variable. nodejs/node#25382 (comment)

The favicon is done at the end because favicon downloading etc. is handled the same way any other media file is downloaded.

[ISNIT0]

This PR may help #968

[kelson42]

@ISNIT0 Still same problem

T:91842; 16000/19032
T:91844; 17000/19032
T:91846; 18000/19032
T:91848; 19000/19032
T:91900;  write checksum
T:91900; finish
[log] [2019-09-09T15:10:53.780Z] Summary of scrape actions: {
	"files": {
		"success": 30889,
		"fail": 0
	},
	"articles": {
		"success": 5925375,
		"fail": 11
	},
	"redirects": {
		"written": 8408528
	}
}
[log] [2019-09-09T15:10:53.787Z] Finished dump
[log] [2019-09-09T15:10:53.794Z] Doing dump
[log] [2019-09-09T15:10:53.798Z] Writing zim to [/output/wikipedia_en_all_nopic_2019-09.zim]
[log] [2019-09-09T15:10:53.799Z] Flushing Redis file store
[log] [2019-09-09T15:10:54.098Z] Found [5] stylesheets to download
[log] [2019-09-09T15:10:54.098Z] Downloading stylesheets and populating media queue
[log] [2019-09-09T15:10:54.166Z] Downloaded stylesheets
[log] [2019-09-09T15:10:54.167Z] Saving favicon.png...
Failed to run mwoffliner after [102343s]: {
	"stack": "Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:366:11)\n    at Object.spawn (child_process.js:551:9)\n    at exec2 (/tmp/mwoffliner/node_modules/imagemagick/imagemagick.js:24:25)\n    at Object.exports.convert (/tmp/mwoffliner/node_modules/imagemagick/imagemagick.js:253:10)\n    at Promise (/tmp/mwoffliner/lib/mwoffliner.lib.js:417:47)\n    at new Promise (<anonymous>)\n    at resizeFavicon (/tmp/mwoffliner/lib/mwoffliner.lib.js:416:28)\n    at /tmp/mwoffliner/lib/mwoffliner.lib.js:433:28\n    at Generator.next (<anonymous>)\n    at /tmp/mwoffliner/lib/mwoffliner.lib.js:9:71\n    at new Promise (<anonymous>)\n    at __awaiter (/tmp/mwoffliner/lib/mwoffliner.lib.js:5:12)\n    at saveFavicon (/tmp/mwoffliner/lib/mwoffliner.lib.js:413:20)\n    at /tmp/mwoffliner/lib/mwoffliner.lib.js:331:23\n    at Generator.next (<anonymous>)\n    at fulfilled (/tmp/mwoffliner/lib/mwoffliner.lib.js:6:58)\n    at process._tickCallback (internal/process/next_tick.js:68:7)",
	"message": "spawn ENOMEM",
	"errno": "ENOMEM",
	"code": "ENOMEM",
	"syscall": "spawn"
}

[kelson42]

Maybe using a lib like https://github.com/lovell/sharp would fix the pb?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.