Fatal Error (Abort) creating string larger than 268435440 bytes

[juliangruber]

Given this script:

var kMaxLength = process.binding('smalloc').kMaxLength;
console.log(kMaxLength, 'kMaxLength');

var len = 268435440;
var blob = new Buffer(len).toString('utf8');
console.log(len, 'ok');

len = 268435441;
blob = new Buffer(len).toString('utf8');
console.log(len, 'ok');

The output should be:

1073741823 'kMaxLength'
268435440 'ok'
268435441 'ok'

But it actually is:

1073741823 'kMaxLength'
268435440 'ok'


#
# Fatal error in ../deps/v8/src/handles.h, line 48
# CHECK(location_ != NULL) failed
#

==== C stack trace ===============================

 1: ??
 2: ??
 3: ??
 4: ??
 5: ??
 6: ??
 7: ??
 8: ??
 9: ??
Illegal instruction: 4

[juliangruber]

this is ~256mb

[petkaantonov]

@trevnorris @chrisdickinson and I are already working on this FWIW

[petkaantonov]

If your file is just using ascii, you can work around this currently by specifying 'ascii' as encoding

[Fishrock123]

I feel like this has already been reported, but I can't find the issue..

[juliangruber]

here is a stacktrace i get on linux:

==== C stack trace ===============================

 1: V8_Fatal
 2: v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::String::NewStringType, int)
 3: node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding)
 4: node::Buffer::Utf8Slice(v8::FunctionCallbackInfo<v8::Value> const&)
 5: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&))
 6: ??
 7: ??
Illegal instruction (core dumped)

[juliangruber]

updated the title, the issue is obviously the .toString()

[Fishrock123]

@juliangruber ah. Is this related to #649?

[petkaantonov]

@Fishrock123 yes, there is even a same issue reported deep in the comments.

It's not a v8 issue, while v8 does have a smallish limit for strings allocated in the js-heap (the 268mb or so), it is our fault for trying to allocate the string in js-heap in the first place.

[brendanashworth]

This doesn't seem to be an issue with the new Buffer rewrite / removal of smalloc:

brendanashworth in ~ $ node -v
v2.1.0
brendanashworth in ~ $ node test
1073741823 'kMaxLength'
268435440 'ok'
FATAL ERROR: invalid array length Allocation failed - process out of memory
Abort trap: 6

brendanashworth in ~ $ node -v
v3.0.0
brendanashworth in ~ $ node test
268435440 'ok'
268435441 'ok'

Closing this for now, feel free to reopen if I'm wrong! :)

[trevnorris]

This isn't an issue of the smalloc change, but it is an issue. The fix is just about to be landed in #2402

[trevnorris]

Fixed by 617ee32.

[trevnorris]

@rvagg Adding this to the v4.0.0 milestone because it's a patch that prevents a possible segfault.

[mhart]

I believe this should be reopened. Output under v4.1.1 is:

268435440 'ok'
buffer.js:378
    throw new Error('toString failed');
    ^

Error: toString failed
    at Buffer.toString (buffer.js:378:11)
    at Object.<anonymous> (/Users/michael/github/rss/data/test.js:6:24)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:475:10)
    at startup (node.js:117:18)
    at node.js:951:3

[bnoordhuis]

@mhart This issue is about node aborting on large strings. It no longer does, it throws a catchable exception.

[mhart]

OK, I was going by the expected output in the issue description.

[arjunkori]

i am getting the same issue ,

  buffer.js:378
    throw new Error('toString failed');^
Error: toString failed
    at Buffer.toString (buffer.js:378:11)