stream: don't emit drain if ended

[ronag]

Don't emit 'drain' if the stream has been ended. 'drain' should only be emitted if the stream wants more data.

See, https://nodejs.org/api/stream.html#stream_event_drain.

the 'drain' event will be emitted when it is appropriate to resume writing data to the stream.

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

NOTE TO SELF: After this is merged look into emitting 'drain' after state.length < hwm instead of state.length === 0.

[Trott]

@nodejs/streams

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/24970/

[Trott]

Looks fine to me, but it would be good to have someone on @nodejs/streams vet it.

Doesn't seem to have semver implications to me but there may be something subtle I'm missing?

Doesn't seem like it would require benchmarking, but again, maybe I'm wrong?

[mcollina]

LGTM with a good CITGM run

[mcollina]

I’m good with this being a patch.

[Trott]

I’m good with this being a patch.

Do you still want a CITGM run? (I'm happy to do it if we want to be extra cautious, but also happy to skip it if it's not necessary.)

[Trott]

Making my previous tentative LGTM more official.

[mcollina]

Having a CITGM run would be better, yes Il giorno sab 17 ago 2019 alle 08:43 Rich Trott <notifications@github.com> ha scritto:

…

***@***.**** approved this pull request. Making my previous tentative LGTM more official. — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#29086?email_source=notifications&email_token=AAAMXY4HD7EQDW626LU4VFDQE6MXNA5CNFSM4IK4BWZKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCB3OAZQ#pullrequestreview-276226150>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAMXYZOR53HD7F4GTCUBMTQE6MXNANCNFSM4IK4BWZA> .

[Trott]

CITGM (queued): https://ci.nodejs.org/job/citgm-smoker/1958/

[Trott]

Those CITGM results seem OK to me after a quick review. You too @mcollina?

[Trott]

Extra-super-cautious second CITGM: https://ci.nodejs.org/job/citgm-smoker/1969/

[Trott]

Landed in 605d7c4

[isaacs]

I was mistaken. bdf07f4 is the problem, not this commit. Original comment hidden in details.

This commit is causing problems. I think it should be reverted.

While the docs for 'drain' mention that it's appropriate to write more data, in practice throughout the ecosystem, 'drain' means "all the written data has been consumed", whether the stream is now ended or not.

As a result, .once('drain') handlers are not removed when they should be, and pipelines can even back up if they depend on this behavior. We're seeing this in npm, where users get a flood of "excess event handler" warnings.

Thanks for your attention.

[mcollina]

Thanks, we’ll revert this asap.

[mcollina]

@isaacs have you got a module that broke/had issue about this? I'd love to add a unit test or something to CITGM so we can avoid regressions.

[ronag]

Makes sense. Is there anything I can do here to follow up? Update the docs to reflect the actual behaviour?

Though I still think the behaviour of this PR is the correct one. Currently it is theoretically possible to cause an write after end in otherwise valid code. I guess we could work around that by updating the docs and also asking the user to check the .writable property after 'drain' and before write(). Not optimal, but maybe sufficient compromise? Or do we revert the revert in a semver-major?

[isaacs]

@mcollina Run npm install against https://github.com/ljharb/es-abstract and you'll see a flood of warnings.

I don't have an example of it backup up a stream and causing problems that way. It'd be harder to reproduce in the wild, because it'd depend on some very specific behavior and timing, but could probably be synthetically created in a test.

[ronag]

Run npm install against https://github.com/ljharb/es-abstract and you'll see a flood of warnings.

Isn't that the same issue as #29239? i.e. something else?

[isaacs]

@ronag It could be! The issue I saw is that drain isn't getting emitted when it should be, so it led me to this commit, and reverting seems to make the problem go away on my system. But it's possible that it's both, or neither, or something else :)

I'm not sure the best way to go about getting this done, but personally, I'd really like a more massive overhaul of the streams implementation if we're talking about semver major changes. The drain sematic is really not the worst problem.

We could improve performance and vastly simplify the codebase, with a minimal disruption to userland code, by moving to something like minipass. If streams were synchronous, and the class hierarchy rooted on a passthrough implementation (rather than squishing together Readable and Writable into Duplex), then the entire system would be much less brittle. I'm not sure the best way to go about proposing that these days, but I'd be happy to help however I can.

[ronag]

@isaacs: Could you try not reverting this commit and instead revert bdf07f4? I'm just curious whether that makes a difference for you.

[isaacs]

@ronag Will do. I do still think this behavior change is a mistake in a minor version, so probably both need to be rolled back.

[isaacs]

Confirmed @ronag's hunch. Reverting bdf07f4 (or both bdf07f4 and 605d7c4) makes the warnings go away, and reverting only 605d7c4 does not.