-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
child_process: only stop readable side of stream passed to process #27373
Closed
Closed
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
9cd75c7
child_process: only stop readable side of stream passed to proc
addaleax 5ac726d
fixup! child_process: only stop readable side of stream passed to proc
addaleax 2467b1f
fixup! fixup! child_process: only stop readable side of stream passed…
addaleax a67dd6f
fixup! child_process: only stop readable side of stream passed to proc
addaleax File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
30 changes: 30 additions & 0 deletions
30
test/parallel/test-child-process-stdio-merge-stdouts-into-cat.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
'use strict'; | ||
const common = require('../common'); | ||
const assert = require('assert'); | ||
const { spawn } = require('child_process'); | ||
|
||
// Regression test for https://github.com/nodejs/node/issues/27097. | ||
// Check that (cat [p1] ; cat [p2]) | cat [p3] works. | ||
|
||
const p3 = spawn('cat', { stdio: ['pipe', 'pipe', 'inherit'] }); | ||
const p1 = spawn('cat', { stdio: ['pipe', p3.stdin, 'inherit'] }); | ||
const p2 = spawn('cat', { stdio: ['pipe', p3.stdin, 'inherit'] }); | ||
p3.stdout.setEncoding('utf8'); | ||
|
||
// Write three different chunks: | ||
// - 'hello' from this process to p1 to p3 back to us | ||
// - 'world' from this process to p2 to p3 back to us | ||
// - 'foobar' from this process to p3 back to us | ||
// Do so sequentially in order to avoid race conditions. | ||
p1.stdin.end('hello\n'); | ||
p3.stdout.once('data', common.mustCall((chunk) => { | ||
assert.strictEqual(chunk, 'hello\n'); | ||
p2.stdin.end('world\n'); | ||
p3.stdout.once('data', common.mustCall((chunk) => { | ||
assert.strictEqual(chunk, 'world\n'); | ||
p3.stdin.end('foobar\n'); | ||
p3.stdout.once('data', common.mustCall((chunk) => { | ||
assert.strictEqual(chunk, 'foobar\n'); | ||
})); | ||
})); | ||
})); |
27 changes: 27 additions & 0 deletions
27
test/parallel/test-child-process-stdio-reuse-readable-stdio.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
'use strict'; | ||
const common = require('../common'); | ||
const assert = require('assert'); | ||
const { spawn } = require('child_process'); | ||
|
||
// Check that, once a child process has ended, it’s safe to read from a pipe | ||
// that the child had used as input. | ||
// We simulate that using cat | (head -n1; ...) | ||
|
||
const p1 = spawn('cat', { stdio: ['pipe', 'pipe', 'inherit'] }); | ||
const p2 = spawn('head', ['-n1'], { stdio: [p1.stdout, 'pipe', 'inherit'] }); | ||
|
||
// First, write the line that gets passed through p2, making 'head' exit. | ||
p1.stdin.write('hello\n'); | ||
p2.stdout.setEncoding('utf8'); | ||
p2.stdout.on('data', common.mustCall((chunk) => { | ||
assert.strictEqual(chunk, 'hello\n'); | ||
})); | ||
p2.on('exit', common.mustCall(() => { | ||
// We can now use cat’s output, because 'head' is no longer reading from it. | ||
p1.stdin.end('world\n'); | ||
p1.stdout.setEncoding('utf8'); | ||
p1.stdout.on('data', common.mustCall((chunk) => { | ||
assert.strictEqual(chunk, 'world\n'); | ||
})); | ||
p1.stdout.resume(); | ||
})); |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of this, we could also use
stream._stdio.push(null)
and disable reading permanently, instead of until it’s explicitly started again. Any thoughts on that? Generally, the use case of “let another process read from this pipe, once that process is done we’ll continue reading ourselves” is not absurd or anything – shell scripts do that a lot.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I never saw it myself but I can understand why just pausing the stream might be useful for piping logs or similar if the process crashes. If there's no technical drawback it sounds strictly better in term of enabled use cases, compared to closing it for good.
A few questions to better understand how it works:
readStop
do, particularly on writable streams? Is it just a noop?pause
exactly do? Keep the fd open but stop polling it for data for readable streams, and buffer what's written into it for writable streams?pause
andreadStop
, how is it meant to be resumed? Is it justresume()
, or is there another function that should be called to counteractreadStop
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@arcanis Okay, thanks! I don’t think there are any technical drawbacks to this.
It stops libuv from listening for readable events on the fd until
readStart()
is called again. In partiuclar, noread()
system call will occur during that time.pause()
makes the stream stop emitting'data'
events. According to the documentation, it does not stop emittingreadable
events, but I guess in that case it’s up to the user to take care of not calling.read()
anyway.Thanks for pointing this out – yes, one thing more is necessary here: Adding
stream.handle.reading = false;
, so that._read()
will callreadStart()
again:node/lib/net.js
Line 522 in eac8f50
_read()
actually gets called once we resume the stream. (None of these things are signs of a great streams implementation… we should probably get rid ofhandle.reading
altogether, but that’s not really a concern for this PR.)(Both calling
.resume()
explicitly and using.on('data')
to start reading again should work.)I’ve added those things, and also added a test file for this scenario.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a lot of flags, eheh! 😄
Btw, why isn't
readStop
enough? If the libuv doesn't listen for readable events anymore, I guess that thedata
events won't be triggered, right?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@arcanis The issue that I’m seeing is, that at the time that this code is executed, there might be an existing
'data'
listener or similar that reads data from the stream. When the readable stream buffer drains, the streams implementation calls_read()
, and for sockets that leads to ahandle.readStart()
call, undoing the effects of thehandle.readStop()
call here.I mean, sure, it’s somewhat questionable to just ignore existing listeners that read data from the stream, but at this point I feel like that’s a reasonable solution?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see, that makes sense, thanks for the explanation!