Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fallback when reading sharded data from a host that does not support range requests #619

Open
d-v-b opened this issue Jul 21, 2024 · 2 comments

Comments

@d-v-b
Copy link
Contributor

d-v-b commented Jul 21, 2024

Apropos #606, not every host supports byte range requests to the same degree (or at all). When fetching chunks from a sharded datasource, if byte range requests fail, Neuroglancer displays no data at all. But in this case Neuroglancer could fall back to fetching the entire shard file and reading chunks in memory. This will fail for large shards, but with the same familiar failure mode triggered by any object that's too large to fetch. For small shards, however, fetching a full shard as a fallback would allow neuroglancer to display data that it otherwise could not.

@jbms
Copy link
Collaborator

jbms commented Jul 24, 2024

I can imagine that in some cases this could be useful. But on the other hand, it isn't clear how often it would really be useful --- I imagine in many cases having to download the entire shard would lead to a very poor experience if it fits in memory at all. Certainly if this were supported, it would be important to surface a warning to the user somehow, to indicate that it is operating in a degraded mode and that they should fix the web server.

A simpler improvement would be to surface a more informative error when range requests aren't supported but e.g. a HEAD request succeeds. Related to this, it would be useful to surface the chunk download errors outside of the javascript console...

@d-v-b
Copy link
Contributor Author

d-v-b commented Jul 24, 2024

Yes, the net utility of falling back to fetching the whole shard depends on the empirical question of what the typical shard size is. If nearly everyone deploys 10 GB shards, then it's not practical to consider fetching that. But 100 MB shards, on a local connection, would be technically feasible, albeit not a great user experience.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants