Skip to content
This repository has been archived by the owner on Aug 2, 2021. It is now read-only.

extend http downloader with asynchronous lookahead #1524

Closed
zelig opened this issue Jun 27, 2019 · 1 comment · Fixed by #1876
Closed

extend http downloader with asynchronous lookahead #1524

zelig opened this issue Jun 27, 2019 · 1 comment · Fixed by #1876
Assignees

Comments

@zelig
Copy link
Member

zelig commented Jun 27, 2019

The http reader is suboptimal since it downloads in syncronous batches translating to range queries for the LazyChunkReader. Previously we optimised for buffersize in this reader, however, this still leaves longer latencies since each batch is blocking until the slowest chunk arrives.

A low hanging fruit to improve is to extend this reader with an asyncronous lookahead that brings concurrent network retrieval across batches and thereby significantly reduce download latencies.

@FantasticoFox
Copy link

This should not be in here. The specifications need to go into a SWIP the timeline is a result of the issues which are assigned to the user-story. In This case it's the 200 MB / 500 MB iterative file-size support increase and the improvement on stability.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
2 participants