Invalid string length exception on large reads

[genzgd]

Crashes when reading 50,000,000 numbers (at least on my Mac M1).

Code to reproduce (against localhost/local docker):

const { createClient } = require('@clickhouse/client')

(async () => {
    const client = createClient();
    console.log(Date.now() / 1000)
    const result = await client.query({
        query: 'SELECT number FROM numbers(50000000)',
        format: "JSONCompactEachRow"
    });
    console.log(Date.now() / 1000);
    const stuff = await result.json();
    console.log(stuff.length);
    console.log(Date.now() / 1000);

})()

Exception:

1664586660.882
1664586660.895
AbortError: The operation was aborted
    at Object.destroyer (node:internal/streams/destroy:305:11)
    at createAsyncIterator (node:internal/streams/readable:1141:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'ABORT_ERR'
}
/Users/geoff/projects/nodetests/node_modules/@clickhouse/client/dist/utils/stream.js:16
        result += textDecoder.decode(chunk, { stream: true });
                              ^

RangeError: Invalid string length
    at getAsText (/Users/geoff/projects/nodetests/node_modules/@clickhouse/client/dist/utils/stream.js:16:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Rows.text (/Users/geoff/projects/nodetests/node_modules/@clickhouse/client/dist/rows.js:37:17)
    at async Rows.json (/Users/geoff/projects/nodetests/node_modules/@clickhouse/client/dist/rows.js:50:45)
    at async /Users/geoff/projects/nodetests/index.js:47:19

Node.js v18.9.0

[slvrtrn]

Unfortunately, that seems to be a hard Node.js limitation. We cannot use strings larger than 512Mb at the moment. If I understand it correctly, the max size was increased from 256Mb to 1Gb in the past and then reduced to 512Mb.

See

• https://stackoverflow.com/questions/44533966/v8-node-js-increase-max-allowed-string-length
• How to deal with strings longer than String::kMaxValue nodejs/help#712 (comment)
• v8/v8@ea56bf5

My tests showed that it indeed crashes at around 512Mb of size, with the resulting string's length around 536870888.