httpstream provides parallel download of a single http file that can be streamed incrementally. It uses multiple http connections and 'byte range' GET calls to do so. The downloaded portions of the file are then assembled into their original order and streamed as the contiguous run of blocks (from the start of the file) is received. This allows for a simple reader to consume the data as if it were read serially. Typical use of the package is:
dl := httpstream.New(ctx, url)
_, err := io.Copy(os.Stdout, dl)A command line tool is also provided, github.com/cosnicolaou/httpstream/cmd/httpstream,
which can be used as follows:
$ go run ./cmd/httpstream --url=https://dumps.wikimedia.org/wikidatawiki/entities/20191202/wikidata-20191202-all.json.bz2 --output=20191202.bz2 --md5=316e7d034c50f072a1b2738ef366bd76
The combination of parallel download and streaming allow for significantly
reducing the latency of large file downloads run on multi-core machines that
are the first step in a more complex pipeline. For example, this package
can be used in conjunction with github.com/cosnicolaou/pbzip2 to overlap
download and decompression. For the example file above, the download completes
in around 30 minutes vs 2-3 hours. As always, some degree of tuning will be required
for each site. The number of parallel connections can be controlled as can the size
of each byte range request. Different values for these can have dramatic effects
on the achieved througput. Larger values also decrease the frequency of status
updates and hence any progress bar updates.
The package provides support for:
- verifying the md5 or sha1 of the downloaded file
- implementing a progress bar by sending progress updates over a channel.
A planned additional feature is the ability to checkpoint and resume a given download.