You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Apr 22, 2023. It is now read-only.
stream.Writable should have a _flush function similar to stream.Transport, to support the use case of a target data store only being able to be written in batches.
This topic was mentioned in Issue #5315, but was dropped due to the use case allowing for a different solution.
Use case examples:
Inserting incoming data into a database in large batches rather than flood the DB with numerous one-off INSERT statements.
Pushing data to an external API in batches to save bandwidth and avoid third party rate limiting
The challenge, currently, is that the Writable can support these use cases with backpressure by maintaining an internal buffer of data, then when some limit is reached, flush that buffer and call the _write callback when it's complete. However, that leaves the Writable with a remaining buffer that will never be written, because there is no call to indicate that it should flush.
Here are the previously suggested workarounds, and why they do not fit this use case:
Listen for 'finish', then flush and emit 'close': 'finish' is defined as being emitted when the Writable's data has been flushed. That is not the case here. Also, 'close' is defined as being emitted when the resources have been cleaned up, which is not what this operation is doing.
Use a Transport stream to collect the input into the necessary batches before pushing that to the Writable: The Writable object is what knows the limitations of the endpoint that it's writing to; spreading this logic out to a separate object which must also have that knowledge violates encapsulation of that concern.
Use a Transport stream as though it were a Writable: Transport streams do not emit the same events that a Writable does, and coercing it to do so would be an ugly hack.
Define some other event type to emit when the data is flushed: This makes it so that our Writable cannot be interchanged with other Writables, as abstracted code will not know to listen for this new event. Also, 'finish' is still defined as being called when the data is flushed, so this still breaks the definition of 'finish'.
The text was updated successfully, but these errors were encountered:
stream.Writable should have a _flush function similar to stream.Transport, to support the use case of a target data store only being able to be written in batches.
This topic was mentioned in Issue #5315, but was dropped due to the use case allowing for a different solution.
Use case examples:
The challenge, currently, is that the Writable can support these use cases with backpressure by maintaining an internal buffer of data, then when some limit is reached, flush that buffer and call the _write callback when it's complete. However, that leaves the Writable with a remaining buffer that will never be written, because there is no call to indicate that it should flush.
Here are the previously suggested workarounds, and why they do not fit this use case:
The text was updated successfully, but these errors were encountered: