You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Given an Ion file of a series of records (e.g. an object in S3), we'd like to be able to stream read/parse this file, ultimately creating an async (Tokio) Stream of Ion records. Due to memory constraints (Lambda, Fargate, etc), we cannot read the entire file into memory and then iterate through the records.
The text was updated successfully, but these errors were encountered:
To clarify a bit: the Reader is already capable of reading a file without pulling the entire thing into memory. However, if the input source it is reading from contains incomplete data, the Reader would either block (e.g. waiting on a socket) or fail (e.g. at the end of a Vec<u8>).
We'd like to offer an API that allows the Reader to be resilient to such a use case. For example, calls to next() or read_string() might return an IonError::InsufficientData (or IonError::WouldBlock?), indicating that the user should try again once more data is available.
We now have non-blocking text and binary readers, but they can't be constructed with the ReaderBuilder API yet. Once that's done (see #484), we can close this issue.
Given an Ion file of a series of records (e.g. an object in S3), we'd like to be able to stream read/parse this file, ultimately creating an async (Tokio)
Stream
of Ion records. Due to memory constraints (Lambda, Fargate, etc), we cannot read the entire file into memory and then iterate through the records.The text was updated successfully, but these errors were encountered: