-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stream: do not unconditionally call _read()
on resume()
#26965
Conversation
`readable.resume()` calls `.read(0)`, which in turn previously set `needReadable = true`, and so a subsequent `.read()` call would call `_read()` even though enough data was already available. This can lead to elevated memory usage, because calling `_read()` when enough data is in the readable buffer means that backpressure is not being honoured. Fixes: nodejs#26957
@addaleax Sadly, an error occurred when I tried to trigger a build. :( |
/cc @nodejs/streams PTAL – does this look like the right approach to you? I’m particularly unsure if this bit makes sense in the first place: Lines 440 to 441 in 2c73868
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This makes sense to me. However this is so complex that we might found a bug in 6 months ;). |
🙌 this fixes the buffering part of #25057. (I believe it's still technically possible for |
May I suggest we fast-track this? |
@ronag This is ready to be merged anyway – I’d be careful about backporting a change like this to LTS faster than usual. I understand that this is an important fix, but at the same time we’ve run into issues with streams changes breaking people’s code much more often than we’d like. /cc @nodejs/lts Landed in 20c3ac2 |
`readable.resume()` calls `.read(0)`, which in turn previously set `needReadable = true`, and so a subsequent `.read()` call would call `_read()` even though enough data was already available. This can lead to elevated memory usage, because calling `_read()` when enough data is in the readable buffer means that backpressure is not being honoured. Fixes: #26957 PR-URL: #26965 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
`readable.resume()` calls `.read(0)`, which in turn previously set `needReadable = true`, and so a subsequent `.read()` call would call `_read()` even though enough data was already available. This can lead to elevated memory usage, because calling `_read()` when enough data is in the readable buffer means that backpressure is not being honoured. Fixes: #26957 PR-URL: #26965 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
`readable.resume()` calls `.read(0)`, which in turn previously set `needReadable = true`, and so a subsequent `.read()` call would call `_read()` even though enough data was already available. This can lead to elevated memory usage, because calling `_read()` when enough data is in the readable buffer means that backpressure is not being honoured. Fixes: #26957 PR-URL: #26965 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Signed-off-by: Beth Griggs <Bethany.Griggs@uk.ibm.com>
`readable.resume()` calls `.read(0)`, which in turn previously set `needReadable = true`, and so a subsequent `.read()` call would call `_read()` even though enough data was already available. This can lead to elevated memory usage, because calling `_read()` when enough data is in the readable buffer means that backpressure is not being honoured. Fixes: #26957 PR-URL: #26965 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Signed-off-by: Beth Griggs <Bethany.Griggs@uk.ibm.com>
@nodejs/releasers would you mind bacporting this in the next Node 10 release? thanks! |
@mcollina any news here? This is rather serious for us and we have no LTS to fallback to... |
@ronag what's the question? Are you asking for this to be included in the next version of Node v10? |
@nodejs/lts would you mind backporting this? |
`readable.resume()` calls `.read(0)`, which in turn previously set `needReadable = true`, and so a subsequent `.read()` call would call `_read()` even though enough data was already available. This can lead to elevated memory usage, because calling `_read()` when enough data is in the readable buffer means that backpressure is not being honoured. Fixes: #26957 PR-URL: #26965 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
`readable.resume()` calls `.read(0)`, which in turn previously set `needReadable = true`, and so a subsequent `.read()` call would call `_read()` even though enough data was already available. This can lead to elevated memory usage, because calling `_read()` when enough data is in the readable buffer means that backpressure is not being honoured. Fixes: #26957 PR-URL: #26965 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Notable changes: - **deps**: upgrade openssl sources to 1.1.1c (Sam Roberts) [#28212](#28212) - **stream**: do not unconditionally call `\_read()` on `resume()` (Anna Henningsen) [#26965](#26965) - **worker**: fix nullptr deref after MessagePort deser failure (Anna Henningsen) [#25076](#25076) PR-URL: #28731
Notable changes: - **deps**: upgrade openssl sources to 1.1.1c (Sam Roberts) [#28212](#28212) - **stream**: do not unconditionally call `\_read()` on `resume()` (Anna Henningsen) [#26965](#26965) - **worker**: fix nullptr deref after MessagePort deser failure (Anna Henningsen) [#25076](#25076) PR-URL: #28731
Notable changes: - **deps**: upgrade openssl sources to 1.1.1c (Sam Roberts) [#28212](#28212) - **stream**: do not unconditionally call `\_read()` on `resume()` (Anna Henningsen) [#26965](#26965) - **worker**: fix nullptr deref after MessagePort deser failure (Anna Henningsen) [#25076](#25076) PR-URL: #28731
Notable changes: - **deps**: upgrade openssl sources to 1.1.1c (Sam Roberts) [#28212](#28212) - **stream**: do not unconditionally call `\_read()` on `resume()` (Anna Henningsen) [#26965](#26965) - **worker**: fix nullptr deref after MessagePort deser failure (Anna Henningsen) [#25076](#25076) PR-URL: #28731
readable.resume()
calls.read(0)
, which in turn previously setneedReadable = true
, and so a subsequent.read()
call wouldcall
_read()
even though enough data was already available.This can lead to elevated memory usage, because calling
_read()
when enough data is in the readable buffer means that backpressure
is not being honoured.
Fixes: #26957
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passes