Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LoadFileMixin always opens the whole file #459

Open
jmilloy opened this issue Apr 9, 2021 · 2 comments
Open

LoadFileMixin always opens the whole file #459

jmilloy opened this issue Apr 9, 2021 · 2 comments
Assignees

Comments

@jmilloy
Copy link
Collaborator

jmilloy commented Apr 9, 2021

Description

The LoadFileMixin is used by rasterio, csv, and xarray nodes for both local and remote files. As written, the it always loads the whole file. The Mixin should be updated to only load the whole file in certain cases, and otherwise open the dataset directly from the source path.

Describe the solution you'd like

Distinguish between local and remote sources. Only read the file into memory if necessary for caching.

Additional Notes

The Rasterio node should be modified to not use the mixin. This will make the read_from_source unnecessary.

@jmilloy
Copy link
Collaborator Author

jmilloy commented Apr 12, 2021

I updated the Rasterio node for 3.0.0, and I'll do the rest of this for 3.1.0

@jmilloy
Copy link
Collaborator Author

jmilloy commented Aug 7, 2021

I'm going to try to do this for 8/13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant