Skip to content
This repository has been archived by the owner on Apr 22, 2023. It is now read-only.

Implement stream.Writable._flush #7631

Closed
TomFrost opened this issue May 15, 2014 · 3 comments
Closed

Implement stream.Writable._flush #7631

TomFrost opened this issue May 15, 2014 · 3 comments

Comments

@TomFrost
Copy link

stream.Writable should have a _flush function similar to stream.Transport, to support the use case of a target data store only being able to be written in batches.

This topic was mentioned in Issue #5315, but was dropped due to the use case allowing for a different solution.

Use case examples:

  • Inserting incoming data into a database in large batches rather than flood the DB with numerous one-off INSERT statements.
  • Pushing data to an external API in batches to save bandwidth and avoid third party rate limiting

The challenge, currently, is that the Writable can support these use cases with backpressure by maintaining an internal buffer of data, then when some limit is reached, flush that buffer and call the _write callback when it's complete. However, that leaves the Writable with a remaining buffer that will never be written, because there is no call to indicate that it should flush.

Here are the previously suggested workarounds, and why they do not fit this use case:

  • Listen for 'finish', then flush and emit 'close': 'finish' is defined as being emitted when the Writable's data has been flushed. That is not the case here. Also, 'close' is defined as being emitted when the resources have been cleaned up, which is not what this operation is doing.
  • Use a Transport stream to collect the input into the necessary batches before pushing that to the Writable: The Writable object is what knows the limitations of the endpoint that it's writing to; spreading this logic out to a separate object which must also have that knowledge violates encapsulation of that concern.
  • Use a Transport stream as though it were a Writable: Transport streams do not emit the same events that a Writable does, and coercing it to do so would be an ugly hack.
  • Define some other event type to emit when the data is flushed: This makes it so that our Writable cannot be interchanged with other Writables, as abstracted code will not know to listen for this new event. Also, 'finish' is still defined as being called when the data is flushed, so this still breaks the definition of 'finish'.
@vkurchatkin
Copy link

Duplicate of #7348

@markstos
Copy link

@TomFrost and others: related conversation is now continuing in the io.js issue tracker: nodejs/readable-stream#112

@chrisdickinson
Copy link

This functionality is available via .cork, .uncork, and ._writev.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants