You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Mar 25, 2024. It is now read-only.
The current implementation of Iterator for Deserializer created by from_reader constructor is eager. Upon calling next() for the first time, it uses Loader to load all the contents via read_to_end() on reader.
Such implementation is unfortunate for two reasons:
it does not allow for online processing of documents on readers without EOF
it wastes space and time, e.g. if someone needs to load only a single or few first documents from a large reader
I suggest loading only a single document from the reader at the time when calling next(). It could replace the current implementation or be optional feature accessible only by, for example, from_reader_lazy() constructor
The text was updated successfully, but these errors were encountered:
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The current implementation of
Iterator
forDeserializer
created byfrom_reader
constructor is eager. Upon callingnext()
for the first time, it uses Loader to load all the contents viaread_to_end()
on reader.Such implementation is unfortunate for two reasons:
I suggest loading only a single document from the reader at the time when calling
next()
. It could replace the current implementation or be optional feature accessible only by, for example,from_reader_lazy()
constructorThe text was updated successfully, but these errors were encountered: