-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stream: avoid drain for sync streams #32887
Conversation
85dc602
to
6062917
Compare
Previously a sync writable receiving chunks larger than highwatermark would unecessarily ping pong needDrain.
6062917
to
abc1d0b
Compare
Not sure if this should be a semver-major. |
This looks great if it does not cause issues in user-land (it will almost certainly break ws tests). |
I'll run a CITGM. |
CITGM looks good apart from |
It should be just a matter of fixing those two failing tests (not sure how), core functionality should not be broken. |
Perhaps wait until #32780 lands so I don't have to fix them twice. |
@nodejs/streams |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
||
// We must ensure that previous needDrain will not be reset to false. | ||
if (!ret) | ||
state.needDrain = true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would add a few comments about this change here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what kind of comment?
I tagged this as "dont-land" on everything but v14.x. |
Do not rely on the `'drain'` event for synchronous writes. Refs: nodejs/node#32887
I've fixed the failing tests: websockets/ws@18d773d. |
Previously a sync writable receiving chunks larger than highwatermark would unecessarily ping pong needDrain. PR-URL: #32887 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
Landed in 003fb53 |
Previously a sync writable receiving chunks larger than highwatermark would unecessarily ping pong needDrain. PR-URL: #32887 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
Previously a sync writable receiving chunks larger than highwatermark would unecessarily ping pong needDrain. PR-URL: #32887 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
@ronag This PR (found after bisecting on v14.x) seems to be causing issues for |
@mscdex +1 in reverting this. I would love to have a regression test added (even in ssh2 and we can add it to citgm) so that we do not regress again. |
I'm ok with revert + a comment in the code. Just a note here, reverting this might have performance impact e.g. when piping from a file stream (which has a larger highwater mark) to a sync transform stream (which has a smaller highwatermark) e.g. for hashing. It's a bit strange to me that this would cause breakage. Is ssh2 doing something funky with streams? |
Hi, |
None of those examples are self-contained :/, they need an active ssh server to run. |
I've stripped down the relevant Source code'use strict';
const { Duplex, Transform } = require('stream');
const { connect, createServer } = require('net');
const { inherits } = require('util');
function Channel(protoStream, socket) {
const streamOpts = {
highWaterMark: 2 * 1024 * 1024,
allowHalfOpen: false
};
Duplex.call(this, streamOpts);
socket.on('drain', () => {
console.log('ondrain()', 'waitClientDrain', this._waitSocketDrain);
if (this._waitSocketDrain) {
this._waitSocketDrain = false;
if (this._chunk)
this._write(this._chunk, null, this._chunkcb);
else if (this._chunkcb)
this._chunkcb();
}
});
this._protoStream = protoStream;
// outgoing data
this._waitSocketDrain = false;
this._chunk = undefined;
this._chunkcb = undefined;
}
inherits(Channel, Duplex);
Channel.prototype._read = function(n) {};
Channel.prototype._write = function(data, encoding, cb) {
const protoStream = this._protoStream;
const packetSize = 64 * 1024;
const len = data.length;
let p = 0;
while (len - p > 0) {
let sliceLen = len - p;
if (sliceLen > packetSize)
sliceLen = packetSize;
const slice = data.slice(p, p + sliceLen);
const ret = protoStream.sendData(slice);
console.log(`Channel._write() ret = ${ret} after writing ${slice.length} byte(s)`);
p += sliceLen;
if (!ret) {
this._waitSocketDrain = true;
this._chunk = undefined;
this._chunkcb = cb;
break;
}
}
console.log(`Channel._write() outside of loop; p=${p} len=${len} waitSocketDrain=${this._waitSocketDrain}`);
if (len - p > 0) {
if (p > 0) {
// partial
let buf = Buffer.allocUnsafe(len - p);
data.copy(buf, 0, p);
this._chunk = buf;
} else {
this._chunk = data;
}
this._chunkcb = cb;
return;
}
if (!this._waitSocketDrain)
cb();
};
function ProtocolStream() {
Transform.call(this, { highWaterMark: 32 * 1024 });
}
inherits(ProtocolStream, Transform);
ProtocolStream.prototype.sendData = function(data) {
return this.push(data);
};
createServer(function(socket) {
this.close();
}).listen(0, '127.0.0.1', function() {
const { address, port } = this.address();
connect(port, address, function() {
console.log('Client connected');
const protoStream = new ProtocolStream();
const channel = new Channel(protoStream, this);
channel.on('finish', () => {
console.log('============ Channel finish');
this.destroy();
});
channel.write(Buffer.alloc(128 * 1024));
protoStream.pipe(this).pipe(protoStream);
channel.end(Buffer.alloc(128 * 1024));
});
}); Executing the test on node v10.19.0 results in the following output: $ node-v10.19.0 stream-issue.js
Client connected
Channel._write() ret = false after writing 65536 byte(s)
Channel._write() outside of loop; p=65536 len=131072 waitSocketDrain=true
ondrain() waitClientDrain true
Channel._write() ret = false after writing 65536 byte(s)
Channel._write() outside of loop; p=65536 len=65536 waitSocketDrain=true
ondrain() waitClientDrain true
Channel._write() ret = false after writing 65536 byte(s)
Channel._write() outside of loop; p=65536 len=131072 waitSocketDrain=true
ondrain() waitClientDrain true
Channel._write() ret = false after writing 65536 byte(s)
Channel._write() outside of loop; p=65536 len=65536 waitSocketDrain=true
ondrain() waitClientDrain true
============ Channel finish
$ Executing the test on node v14.1.0 or later results in the following output: $ node-v14.7.0 stream-issue.js
Client connected
Channel._write() ret = false after writing 65536 byte(s)
Channel._write() outside of loop; p=65536 len=131072 waitSocketDrain=true (... and the process never exits) |
Looks like it ends up making an incorrect assumption regarding whether or not I think this is weird/incorrect/bug usage and node streams are behaving correctly. However, I guess it is is breaking change so if no one feels strongly about the performance loss I think we just revert it. |
wrt to ssh2 and being hard to spot. My testing found this error to be
(or appear to be) intermittent. i.e. I could run the same test 5 times
and it would not fail until the 5th time (with same test input), but
sometimes it would fail on the first test, or second etc. When I tried
testing with input data sizes which increased by 10k per run, if I
started with a relatively small size e.g. 90k, it would not fail (I wold
kill the test when it reached 50Mb). If on the other hand, I started
with 200k or 500k, it would usually fail immediately.
I also found that if I was connecting with a destination of localhost,
rather than a remote sftp server or one running on a vbox image, I did
not see the failure, though I didn't test as extensively with this
target.
It is quite likely the existing tests just didn't trigger the issue.
So fixing in ssh2 will be a little challenging given how difficult it is
to know if the issue is fixed or just hasn't been triggered. Reverting
and putting into v15 would at least buy some time to try and find the
right fix for ssh2. Just my 2c - ssh2/ssh2-streams author may have a
different view. Happy to assist with testing if that is at all helpful.
|
Not to do the "+1" thing here, but we've seen this regression in the wild with the (quite popular) Is there any chance of getting this reverted? |
Previously a sync writable receiving chunks
larger than highwatermark would unecessarily
ping pong needDrain.
300% improvement for sync streams when chunks are bigger than HWM.
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passes