Ensure async iteration works with destroyed streams #23785

mcollina · 2018-10-20T13:09:15Z

This PR is twofold:

~~makes sure that .destroy() works correctly on zlib streams (before it was not cleaning up the _handle)~~ done in zlib: do not leak on destroy #23734
fixes support for async-iterating destroyed streams.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

nodejs-github-bot · 2018-10-20T13:09:16Z

@mcollina build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/1317/pipeline

test/parallel/test-stream-readable-async-iterators.js

mafintosh · 2018-10-20T14:20:28Z

@mcollina does this supersede my zlib destroy leak PR?

Hakerh400 · 2018-10-20T15:23:37Z

https://github.com/nodejs/node/blob/0983ed818e5b93b6db794b04b58425d8d4ffc43d/lib/internal/streams/async_iterator.js#L90-L91

Nit: destroy(err)

mcollina · 2018-10-20T15:32:12Z

@mafintosh which one? Very likely that should be merged first.

devsnek

It seems this relies on a race condition? I won't pretend to know anything about our crazy streams but I want to just verify what's going on here is what is intended.

devsnek · 2018-10-20T15:39:36Z

lib/internal/streams/async_iterator.js

+    if (this[kStream].destroyed) {
+      // this is needed because if .destro(err) is called, the error
+      // will be emitted via nextTick
+      return new Promise((resolve, rejects) => {


any reason for rejects rather than reject?

none, I’ll fix it.

mcollina · 2018-10-20T16:12:14Z

@devsnek this is normalizing a race condition that we cannot get rid of inside our stream machinery :/.

mafintosh · 2018-10-20T16:45:42Z

@mcollina this one #23734

mcollina · 2018-10-20T17:41:29Z

Ah I didn’t know we were working on essentially the same issue!

Can you pick up the change that I did inside processCallback? It might even be worth a test on its own. I’ll rebase this on top of yours.

lib/internal/streams/async_iterator.js

addaleax · 2018-10-21T01:50:09Z

lib/zlib.js

+
+Zlib.prototype._destroy = function(err, cb) {
+  const handle = this._handle;
+  if (handle && handle.buffer !== null) {


Can you add a comment here describing what this condition means?

addaleax · 2018-10-21T01:50:39Z

lib/zlib.js

-  _close(this, callback);
-  this.destroy();
+Zlib.prototype.close = function(callback) {
+  this.destroy(null, callback);


This changes timing for the close callback… is that intentional?

mcollina · 2018-10-23T22:54:49Z

CI: https://ci.nodejs.org/job/node-test-pull-request/18090/

PTAL, this should be ready for review.

mcollina · 2018-10-24T09:30:09Z

CI: https://ci.nodejs.org/job/node-test-pull-request/18100/

mcollina · 2018-10-24T09:30:47Z

@lpinca @addaleax is this ok to land for you?

mafintosh · 2018-10-24T09:35:43Z

lib/internal/streams/async_iterator.js

@@ -158,6 +174,9 @@ const createReadableStreamAsyncIterator = (stream) => {
  stream.on('readable', onReadable.bind(null, iterator));
  stream.on('end', onEnd.bind(null, iterator));
  stream.on('error', onError.bind(null, iterator));
+  // needed because some streams will not emit 'end', e.g. Zlib.
+  // https://github.com/nodejs/node/issues/23730
+  stream.on('close', onEnd.bind(null, iterator));


could use stream.finished to simplify in the future, but 👍

lpinca · 2018-10-24T09:36:36Z

test/parallel/test-stream-readable-async-iterators.js

+    const passthrough = new PassThrough();
+    const err = new Error('kaboom');
+    pipeline(readable, passthrough, common.mustCall((e) => {
+      assert.strictEqual(err, e);


Suggested change

assert.strictEqual(err, e);

assert.strictEqual(e, err);

lpinca · 2018-10-24T09:36:51Z

test/parallel/test-stream-readable-async-iterators.js

+    try {
+      await readable[Symbol.asyncIterator]().next();
+    } catch (e) {
+      assert.strictEqual(err, e);


Suggested change

assert.strictEqual(err, e);

assert.strictEqual(e, err);

addaleax · 2018-10-24T10:21:06Z

@mcollina There’s nothing speaking against landing this as far as I am concerned :)

Fixes nodejs#23730.

mcollina · 2018-10-24T11:54:38Z

CI: https://ci.nodejs.org/job/node-test-pull-request/18110/

PTAL, I've switched to use finished instead.

lpinca · 2018-10-24T13:17:46Z

Still LGTM.

mcollina · 2018-10-24T13:24:56Z

Landed in 3ec8cec

Fixes #23730. PR-URL: #23785 Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Matheus Marchini <mat@mmarchini.me>

mafintosh · 2018-10-24T13:26:57Z

(Also still LGTM for the record)

Fixes #23730. PR-URL: #23785 Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Matheus Marchini <mat@mmarchini.me>

MylesBorins · 2018-11-26T21:39:48Z

Landed in 10.x with 398418d and b1e1fe4

Please lmk if it should be backed out

mcollina · 2018-11-26T21:47:46Z

go for it! Il giorno lun 26 nov 2018 alle 22:41 Myles Borins <notifications@github.com> ha scritto:

…

Landed in 10.x with 398418d <398418d> and b1e1fe4 <b1e1fe4> Please lmk if it should be backed out — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#23785 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADL4-2fOTTylhaBVvajyK70fXegKpuvks5uzGAUgaJpZM4Xx6oH> .

Fixes #23730. PR-URL: #23785 Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Matheus Marchini <mat@mmarchini.me>

mcollina requested review from mafintosh, jasnell, addaleax and devsnek October 20, 2018 13:09

nodejs-github-bot added the zlib Issues and PRs related to the zlib subsystem. label Oct 20, 2018

mcollina mentioned this pull request Oct 20, 2018

Stream destruction breaks async-iteration and ends nodejs event loop #23730

Closed

jasnell reviewed Oct 20, 2018

View reviewed changes

test/parallel/test-stream-readable-async-iterators.js Show resolved Hide resolved

devsnek approved these changes Oct 20, 2018

View reviewed changes

devsnek added stream Issues and PRs related to the stream subsystem. experimental Issues and PRs related to experimental features. labels Oct 20, 2018

lpinca reviewed Oct 20, 2018

View reviewed changes

lib/internal/streams/async_iterator.js Outdated Show resolved Hide resolved

addaleax reviewed Oct 21, 2018

View reviewed changes

addaleax mentioned this pull request Oct 21, 2018

zlib: do not leak on destroy #23734

Closed

4 tasks

mcollina force-pushed the fix-23730 branch from c13cc44 to 0d029f6 Compare October 23, 2018 22:52

mcollina force-pushed the fix-23730 branch from 0d029f6 to 80487d0 Compare October 24, 2018 09:27

mafintosh reviewed Oct 24, 2018

View reviewed changes

mafintosh approved these changes Oct 24, 2018

View reviewed changes

lpinca approved these changes Oct 24, 2018

View reviewed changes

addaleax removed the zlib Issues and PRs related to the zlib subsystem. label Oct 24, 2018

stream: async iteration should work with destroyed stream

14586b0

Fixes nodejs#23730.

mcollina force-pushed the fix-23730 branch from 80487d0 to 14586b0 Compare October 24, 2018 11:08

mmarchini approved these changes Oct 24, 2018

View reviewed changes

mmarchini added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 24, 2018

mcollina closed this Oct 24, 2018

mcollina deleted the fix-23730 branch October 24, 2018 13:25

This was referenced Oct 26, 2018

Stream async iteration breaks when the loop body calls .destroy(null) #23890

Closed

Multiple fixes for async iterators #23901

Closed

targos mentioned this pull request Oct 27, 2018

Release proposal: v11.1.0 #23922

Merged

MylesBorins added dont-land-on-v8.x labels Nov 26, 2018

codebytere mentioned this pull request Nov 27, 2018

v10.13.1 proposal #24675

Closed

codebytere mentioned this pull request Nov 29, 2018

v10.14.2 proposal #24727

Merged

jaydenseric mentioned this pull request Jan 10, 2019

Errors in tests appearing for Node.js >= v10.14.2 jaydenseric/graphql-upload#130

Closed

rektide mentioned this pull request Mar 6, 2019

Advance Readable[Symbol.asyncIterator] out of experimental #26479

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure async iteration works with destroyed streams #23785

Ensure async iteration works with destroyed streams #23785

mcollina commented Oct 20, 2018 •

edited

Loading

nodejs-github-bot commented Oct 20, 2018

mafintosh commented Oct 20, 2018

Hakerh400 commented Oct 20, 2018

mcollina commented Oct 20, 2018

devsnek left a comment

devsnek Oct 20, 2018

mcollina Oct 20, 2018

mcollina commented Oct 20, 2018

mafintosh commented Oct 20, 2018

mcollina commented Oct 20, 2018

addaleax Oct 21, 2018

addaleax Oct 21, 2018

mcollina commented Oct 23, 2018

mcollina commented Oct 24, 2018

mcollina commented Oct 24, 2018

mafintosh Oct 24, 2018

lpinca Oct 24, 2018

lpinca Oct 24, 2018

addaleax commented Oct 24, 2018

mcollina commented Oct 24, 2018

lpinca commented Oct 24, 2018

mcollina commented Oct 24, 2018

mafintosh commented Oct 24, 2018

MylesBorins commented Nov 26, 2018

mcollina commented Nov 26, 2018 via email

Ensure async iteration works with destroyed streams #23785

Ensure async iteration works with destroyed streams #23785

Conversation

mcollina commented Oct 20, 2018 • edited Loading

Checklist

nodejs-github-bot commented Oct 20, 2018

mafintosh commented Oct 20, 2018

Hakerh400 commented Oct 20, 2018

mcollina commented Oct 20, 2018

devsnek left a comment

Choose a reason for hiding this comment

devsnek Oct 20, 2018

Choose a reason for hiding this comment

mcollina Oct 20, 2018

Choose a reason for hiding this comment

mcollina commented Oct 20, 2018

mafintosh commented Oct 20, 2018

mcollina commented Oct 20, 2018

addaleax Oct 21, 2018

Choose a reason for hiding this comment

addaleax Oct 21, 2018

Choose a reason for hiding this comment

mcollina commented Oct 23, 2018

mcollina commented Oct 24, 2018

mcollina commented Oct 24, 2018

mafintosh Oct 24, 2018

Choose a reason for hiding this comment

lpinca Oct 24, 2018

Choose a reason for hiding this comment

lpinca Oct 24, 2018

Choose a reason for hiding this comment

addaleax commented Oct 24, 2018

mcollina commented Oct 24, 2018

lpinca commented Oct 24, 2018

mcollina commented Oct 24, 2018

mafintosh commented Oct 24, 2018

MylesBorins commented Nov 26, 2018

mcollina commented Nov 26, 2018 via email

mcollina commented Oct 20, 2018 •

edited

Loading