-
-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Expose streaming query as (typed) AsyncIterable #426
Labels
Comments
I wonder about the safety of the following approach: async function* streamMARCsByLibraryID(
connection: DatabaseConnection,
libraryId: number,
) {
const stream = await new Promise<Readable>((resolve, reject) =>
connection
.stream(
sql.unsafe`SELECT id, marc
FROM unitas_library.harvested_record
WHERE library_id = ${libraryId}`,
(stream) => resolve(stream),
)
.catch((reason) => reject(reason)),
);
for await (const { row } of stream) {
yield <{ id: string; marc: string }>row;
}
stream.destroy();
} |
FYI now the stream results are typed. |
Typed |
🎉 This issue has been resolved in version 37.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It would be pretty nifty to be able to use the
for await (const datum of <something>) {}
construction with streaming queries, with proper types.Desired Behavior
Motivation
For slower computations where storing all data in memory at once (or fetching it over the network at once) is not feasible, AsyncIterables offer convenient data processing facilities, while still being typed.
The
stream
method offers similar facilities, but seems to not have any typing due toReadable
not beingReadable<T>
.Implementation
The most low-effort way to me seems to offer the possibility of
StreamHandler
to be anasync
function, in which case building my ownasTypedIterable
on top of this will be trivial.One issue is that the user somehow has to make sure that the Readable is actually closed/destroyed. This is done automagically when one runs a
for await (const _ of <readable>){}
, but the Readable obviously does not get closed/destroyed if there is no such loop.OTOH, this is also currently the case for the existing
stream
, so we might consider this a case of foot meet gun.The text was updated successfully, but these errors were encountered: