Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate large graph warnings #102

Open
gjoseph92 opened this issue Dec 3, 2021 · 0 comments
Open

Investigate large graph warnings #102

gjoseph92 opened this issue Dec 3, 2021 · 0 comments

Comments

@gjoseph92
Copy link
Owner

gjoseph92 commented Dec 3, 2021

I've noticed large graph warnings from Dask when working with reasonably-sized stackstac DataArrays, like

UserWarning: Large object of size 1.73 MiB detected in task graph:

One thought: what if this is a situation like dask/dask#8008? When we turn the asset table into a Dask array, we're making one chunk per element. What if each of these embedded elements aren't actually size one, but reference the entire memory of the asset table? That would make the serialized size of the asset table N^2!

Naw, I think that's unlikely. Serialization isn't dumb enough to copy the entire buffer even when it's not needed. There's probably some other cruft in there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant