child_process stdout being truncated #19218
Comments
addaleax
added
child_process
I assume, you mean the output is the truncated one?
I think Do you get the same issue with other scripts, e.g. when trying to run |
addaleax
added
confirmed-bug
|
@addaleax Yes, the same output as the truncated one. I'll update the post. I'll try with that script and report back. |
|
@santigimeno That this also occurs with non-Node child processes makes that sound unlikely (assuming that that’s correct)? |
|
@addaleax I just tested with the |
|
@santigimeno I asked the OP to also check with |
|
I think his response was he'll try and report back |
|
BTW, |
|
@santigimeno Thanks for testing that for me. |
|
I've tried blocking the process, but that doesn't seem to work either. Same truncated output. Code as follows: |
|
This is easily reproducible in MAC through: const cp = require('child_process')
const op = {maxBuffer: 100 * 1024 * 1024}
if (process.argv[2]) {
child = cp.spawn('node', ['-e', 'process.stdout.write(Buffer.alloc(10 * 1024 * 1024))'], op)
}
else
child = cp.spawn('node', ['-e', 'console.log(Buffer.alloc(10 * 1024 * 1024))'], op)
let buf = ''
child.stdout.on('data', (c) => {
buf += c
})
child.stdout.on('end', () => {
console.log('ENDED')
console.log(buf.length)
})#node 19218.js #node 19218.js true
And the root cause identified as the fact that I don't know the latest on this, also don't know the best practice / workaround. So /cc @nodejs/process @nodejs/child_process |
I think that's not working because you are making blocking the parent side of the pipe. For this to work the child process would need to make stdout/stderr blocking on its side (That's the reason why applying #784 (comment) to node would fix this issue) |
|
@gireeshpunathil Your scripts can not reproduce the bug. It should use
|
|
@killagu - thanks for checking. So basically the difference between yours and mine is that you add a Nevertheless, no need to investigate this further as the underlying bug is well understood, and we have open issues on the same. The effort be better directed towards looking at how do we fix it. Does that sound good to you? |
|
@gireeshpunathil If you use The data is not truncated. If you do not use So the script can not reproduce the bug. |
can you please expand on that? what that number 164 signifies, according to you? |
|
never mind, I got what you are saying. So the stringified version of the buffer object was passed across rather than the whole content. thanks for clarifying. How about this version? const cp = require('child_process')
const op = {maxBuffer: 100 * 1024 * 1024}
if (process.argv[2] === 'child') {
const buf = Buffer.alloc(10).toString()
for(var i=0; i< 1024 * 1024; i++)
console.log(buf)
process.exit(0)
} else {
const child = cp.spawn(process.execPath, [__filename, 'child'])
let buf = ''
child.stdout.on('data', (c) => {
buf += c
})
child.stdout.on('end', () => {
console.log(buf.length)
})
}shows random length if you run multiple times - both on linux and mac. |
|
If you use Not use You can run the script @gireeshpunathil |
|
@gireeshpunathil The new script works well. Thanks for the reply. |
|
I can reproduce this too (Node v10.11.0, v10.13.0, v11.0.0), the const cp = require("child_process");
const op = { maxBuffer: 100 * 1024 * 1024 };
if (process.argv[2] === "child") {
const buf = Buffer.alloc(10).toString();
for (var i = 0; i < 1024 * 1024; i++) console.log(buf);
process.exit(0);
} else {
const child = cp.spawn(process.execPath, [__filename, "child"]);
child.stdout._handle.setBlocking(true);
let buf = "";
child.stdout.on("data", c => {
buf += c;
});
child.stdout.on("end", () => {
console.log(buf.length);
});
}Is there another workaround at the moment? |
|
Remove the |
|
@sam-github Is there a way to do this and still be able to dictate the exit code when Node eventually exits? |
|
You can do https://nodejs.org/api/process.html#process_process_exitcode |
|
This problem still exists in child_process and in almost all of the libraries that build on it or that attempt to replace it. There's one library that The maintainers of |
|
It looks like In the example below, if you limit the output to 10 bytes, it will still show the whole 200 kb output: const out = []
const p = childProcess.spawn('curl', ['https://stackoverflow.com'], {maxBuffer: 10})
p.stdout.on('data', line => {
out.push(String(line))
})
p.on('close', () => {
console.log(out.join(''))
})Env:Version: 15.14.0 |
|
@aleksey-hoffman |
|
@benjie I see, only sync version supports it. Thanks. |
|
I'm still seeing this problem when using exec and spawn on macOS Monterey 12.0.1. Is there a status on the fix? |
|
Having this problem reading stdout on a large command output using exec() in macOS Monterey |
|
Still happens on macOS 14.4 and Node v18.19.1. This is a really serious bug in a very basic part of Node that I'm quite surprised didn't receive any attention... 😢 |
|
Though in my case there isn't any |
It seems like
spawn()andexec()is truncatingstdout, even whenmaxBufferis set. I've been trying to capture the output ofjest --help, which the full output looks like the following:spawn()
Here's an example
spawn()script:And the captured output:
Notice that about half the output is missing.
exec()
Another attempt using
exec().The output is the same as the truncated above.
Execa
Another attempt using
execa.The output is the same as the truncated above.
The text was updated successfully, but these errors were encountered: