Buffer encoding regression.

[ChALkeR]

var s0 = 'тест ';
s0 += new Array(1031913 + 1 - s0.length).join('Z');
var s1 = new Buffer(s0, 'ucs2').toString('ucs2');
var s2 = new Buffer(s1, 'utf-8').toString('utf-8');
console.log(s0 === s2);
console.log(s0.substr(0, 200));
console.log(s2.substr(0, 200));

This works under node 0.10 (s0 equals to s2, the example logs 2 equal strings), but breaks in iojs 1.2.0 and 1.4.2 (s0 is not equal to s2, the second logged string is not correct).

When broken, s2.substr(0, 200) somewhy equals to new Buffer(s0, 'ucs2').toString('utf-8').substr(0, 200).

1031913 is the critical length, non-latin strings with length greater or equal to 1031913 trigger the bug, strings with length equal or less than 1031912 do not trigger the bug.

[Fishrock123]

cc @trevnorris?

[ChALkeR]

Reproducible in nodejs 0.12 with different critical string length: 515957, which is exactly 1/2 of the iojs critical string length. Using nodejs 0.12.0-3 from Archlinux x86_64 packages.

I mean that the max safe length in iojs is 1031912, the max safe length is nodejs 0.12 is 515956, 515956 * 2 == 1031912.

[trevnorris]

Must have to do with the use of externalized strings. I'll investigate.

[bnoordhuis]

The tipping point is exactly EXTERN_APEX bytes so yes, must be something to do with externalized (two-byte?) strings. I'll take a look as well, I already spotted at least one (possibly unrelated) bug.

[bnoordhuis]

Okay, here is a simplified test case:

var assert = require('assert');

var a = Array(1031914).join('x');
var b = Buffer(a, 'ucs2').toString('ucs2');
var c = Buffer(b, 'utf8').toString('utf8');

assert.equal(a.length, b.length);
assert.equal(b.length, c.length);
assert.equal(a, b);
assert.equal(b, c);

And when you run it, the last assert fails like this, making it quite obvious what's going on.

$ out/Release/iojs tmp/issue1024-2.js

assert.js:87
  throw new assert.AssertionError({
        ^
AssertionError: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx == 'x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x\u0000x
    at Object.<anonymous> (/home/bnoordhuis/src/v1.x/tmp/issue1024-2.js:10:8)
    at Module._compile (module.js:444:26)
    at Object.Module._extensions..js (module.js:462:10)
    at Module.load (module.js:339:32)
    at Function.Module._load (module.js:294:12)
    at Function.Module.runMain (module.js:485:10)
    at startup (node.js:112:16)
    at node.js:863:3

The call to Buffer(b, 'utf8') creates a buffer that looks like 78 00 78 00 78 00 .. but only when b is an externalized two-byte string.

[bnoordhuis]

Found it: https://github.com/iojs/io.js/blob/6433ad1/src/string_bytes.cc#L305-312

StringBytes::Write() does a plain memcpy() when is_extern is true but that's wrong when the source is two-byte and the destination is one-byte or UTF-8 (and probably the other way around as well.)

[vkurchatkin]

related nodejs/node-v0.x-archive#8683 ?

[bnoordhuis]

@vkurchatkin Yes, looks to be the same issue. I'll try to come up with a patch later today.

[bnoordhuis]

Fixed in 1640ded.

[ChALkeR]

Thanks!