V8 crash during MarkCompact GC

[papandreou]

• Version: 6.2.1 (V8 5.0.71.52)
• Platform: Ubuntu Linux 16.04, x86_64
• Subsystem: V8, GC, Buffer

What steps will reproduce the problem?

Check out https://github.com/gustavnikolaj/node6-gc-overflow and run https://github.com/gustavnikolaj/node6-gc-overflow/blob/4766e784cadbf795f7be4665012a68de1d20ab13/index.js
It will fail reliably with node 6.2.1, but it works fine with 5.11.1 (v8 version 4.6.85.31).

What is the expected output?

A single line of "DONE!" to stdout.
Like below:

$ nvm use 5.11.1 && node --max-old-space-size=6000 index.js 4000
Now using node v5.11.1 (npm v3.8.6)
DONE!

What do you see instead?

$ nvm use 6.2.1 && node --max-old-space-size=6000 index.js 4000
Now using node v6.2.1 (npm v3.9.3)

#
# Fatal error in ../deps/v8/src/heap/spaces.h, line 1516
# Check failed: size_ >= 0.
#

==== C stack trace ===============================

1: V8_Fatal
2: 0xc56387
3: v8::internal::MarkCompactCollector::SweepSpaces()
4: v8::internal::MarkCompactCollector::CollectGarbage()
5: v8::internal::Heap::MarkCompact()
6: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags)
7: v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags)
8: v8::internal::Heap::HandleGCRequest()
9: v8::internal::StackGuard::HandleInterrupts()
10: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*)
11: 0xf767a506338
Illegal instruction

Background

We use assetgraph-builder to build production versions of our web application. It's a tool that will take a web application and transform the code in various ways.
It will find style sheets and concatenate them, pick up on images that is referenced and optimize them etc. Because it loads all the assets into memory we have had to use the --max-old-space-size flag to avoid running out of memory.

When we attempted to upgrade one of our larger web applications to node.js v6 we ran into the error described above. From what we have gathered, it happens when the garbage collector is attempting to clear more than ~2.2GB of memory using the MarkCompact strategy.

node/deps/v8/src/heap/spaces.h

Lines 1510 to 1517 in c7b0d06

	// Shrink the space by removing available bytes. Since shrinking is done
	// during sweeping, bytes have been marked as being in use (part of the size)
	// and are hereby freed.
	void ShrinkSpace(int size_in_bytes) {
	capacity_ -= size_in_bytes;
	size_ -= size_in_bytes;
	CHECK(size_ >= 0);
	}

Our theory is that the signed 32 bit signed ints are overflowed and thus cause the check to fail. The only difference in the ShrinkSpace function between v8 4.6 and v8 5.0 is that the check has been promoted from a debug-check to a run time check, so we have most likely been exposed to this error longer than we were aware. The commit message suggests that the V8 team is already trying to root cause the problem: v8/v8@1682935

We have attempted to patch deps/v8 ourselves by naively changing the int types to size_t but without luck, so we suspect that it's a bug in the actual calculation algorithm.

Reproducing the bug with V8

We've also tried and failed to reduce the code snippet to something that will also trigger the bug in v8_shell, which means that Buffer cannot be used. None of our attempts have been able to trigger a MarkCompact GC, though, but that might be due to a lack of understanding of the heuristics employed by V8. This is the reason why we're reporting the bug here initially.

[bnoordhuis]

Our theory is that the signed 32 bit signed ints are overflowed and thus cause the check to fail.

I agree, it seems to stem from V8 using int (32 bits) in a number of places where it should be using intptr_t (64 bits in 64 bits builds) - the static_cast<int>(area_end() - area_start()) in MemoryChunk::area_size() in particular looks suspect.

It looks fairly straightforward to fix but I don't have time to write up a patch. Does someone from @nodejs/v8 want to take this on?

[bnoordhuis]

@papandreou You should file a bug over at https://bugs.chromium.org/p/v8/issues/list too.

[papandreou]

@bnoordhuis Thanks a lot for taking a look. I filed a sister issue here: https://bugs.chromium.org/p/v8/issues/detail?id=5094

[papandreou]

Fixed for V8 5.6, according to @mlippautz on https://bugs.chromium.org/p/v8/issues/detail?id=5094#c9

I'll rerun the experiment once a node.js built against a fixed V8 is available.

[targos]

@papandreou the latest nightly builds have V8 5.6: https://nodejs.org/download/nightly/

[papandreou]

@targos, thanks for the ping.

I dusted off the experiment, and it actually doesn't fail in node.js 6.5.0 and above. V8 was updated to 5.1.281.75 in that release (#8054).

It also works with the latest nightly (v8.0.0-nightly20170302172be50fe1).

Seems like it's time to close this issue ;)

[dandv]

Saw a similar error in node v7.10.1 with a long-running script that had been running just fine for months:

Fatal error in ../deps/v8/src/heap/spaces.cc, line 2769
# Check failed: (size)>=(0).

[ofrobots]

@dandv Node 7.x release line is no longer supported, you should move to one of the supported release lines.