Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream: Writable should not invoke _destroy while pending ops #29044

Closed
ronag opened this issue Aug 8, 2019 · 8 comments
Closed

stream: Writable should not invoke _destroy while pending ops #29044

ronag opened this issue Aug 8, 2019 · 8 comments

Comments

@ronag
Copy link
Member

ronag commented Aug 8, 2019

I don't think it is a good idea that a call to Writable.destroy will invoke Writable._destroy while there are still active/pending _write & _writev (or _read for that matter). Since these might be working on resources that might be closed and cause invalid state.

We have already encountered this problem with fs writable streams (#2006). The solution there is a bit of a "hack". I'm sure the same problem can occur in other userland stream implementations as well as other edge cases (including core streams).

I think the "proper" way to resolve this is to invok Writable._destroy after all pending operations are complete (e.g. pendingcb === 0).

I don't have a obvious way right now to do this change as it would require inlining the destroyImpl helper.

I think this applies to Readable as well.

@ronag
Copy link
Member Author

ronag commented Aug 8, 2019

ping @mcollina thoughts?

@ronag ronag changed the title stream: Writable should not invoke _destroy while pending callbacks stream: Writable should not invoke _destroy while pending ops Aug 8, 2019
@mcollina
Copy link
Member

mcollina commented Aug 8, 2019

destroy() is an immediate action. I’m -1 in having it to wait.

@ronag
Copy link
Member Author

ronag commented Aug 8, 2019

@mcollina: What about pending operations? I would argue that we cannot make destroy an immediate action in the cases where the resources to be destroyed are in active use (especially since we don't have synchronous cancellation). The fs example illustrates this.

@mcollina
Copy link
Member

mcollina commented Aug 8, 2019

destroy() is an action that will guarantee that there will be no pending operations. The problem with waiting for pending writes is that they might never come: the current model ensures that. Waiting for pending writes might cause memory leaks in implementations that are buggy.

I might be ok in adding an option to write { waitPending: true } so that _destroy() is called after that. However this is going to be tricky as we would have to start erroring while a stream is being destroyed.

@ronag
Copy link
Member Author

ronag commented Aug 8, 2019

destroy() is an action that will guarantee that there will be no pending operations.

But how? We can't guarantee that since we can't cancel already dispatched _write & _writev?

The problem with waiting for pending writes is that they might never come: the current model ensures that. Waiting for pending writes might cause memory leaks in implementations that are buggy.

Maybe some kind of timeout? Which is the responsibility of the _write implementor.

I understand how you would like it to work. But I don't see how we can make it work like that.

#29028 will only ensure that buffered requests are not dispatched. Not that already dispatched requests will be cancelled.

@ronag
Copy link
Member Author

ronag commented Aug 8, 2019

I would like to refer to this issue #2006 which illustrates the problem. The way it was solved is kind of a hack where destroy doesn't actually destroy (https://github.com/nodejs/node/blob/master/lib/internal/fs/streams.js#L392).

@mcollina
Copy link
Member

mcollina commented Aug 8, 2019

But how? We can't guarantee that since we can't cancel already dispatched _write & _writev?

Let's take a fs.WriteStream as an example. destroy() responsibility is to close the file descriptor as soon as it's possible. It guarantees that there are no leaked file descriptor. destroy() is not a "graceful" action, but rather a disruptive action to be called to tear down state. The normal behavior for writables is to use end() - that implements what you are asking for already.

@ronag
Copy link
Member Author

ronag commented Aug 8, 2019

Hm, that makes sense. Thank you for explaining.

@ronag ronag closed this as completed Aug 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants