feat(lapis2): stream data from SILO

[fengelniederhammer]

resolves #744

PR Checklist

• [ ] All necessary documentation has been adapted.
• The implemented feature is covered by an appropriate test.

Summary

The streaming still happens in the same thread as the request handling (by not using StreamingResponseBody). We'll have to observe whether this causes any issues in the future. I couldn't properly get using StreamingResponseBody work with the way we implemented compression. We have to close the compressions streams here, but then the lazy stream can't write anymore.

I profiled LAPIS with two datasets.

TLDR

• Memory consumption when downloading 3.5k sequences goes down from 500 - 900 MB to a 130 - 300 MB.
• Memory consumption when downloading 1M sequence details goes down from 14.5 GB (peak) goes down to a stable usage of ~100MB for the whole download.

Ebola

I executed this a couple of times with the data of this branch and observed the memory consumption:

curl -X 'GET' 'http://localhost:8090/sample/alignedNucleotideSequences' \
                   -H 'accept: text/x-fasta' > /dev/null

Memory consumption on main:

Memory consumption on this branch:

The open Covid dataset on the testserver

I connected my local LAPIS to the SILO running on the server and downloaded metadata:

curl -X 'GET' 'http://localhost:8090/sample/details?dataFormat=csv&limit=1000000'

This is the memory consumption on the current main:

Also the download was incredibly slow. It took 3.5 minutes to downlaod 426 MB:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  426M  100  426M    0     0  2054k      0  0:03:32  0:03:32 --:--:-- 97.7M

This is the same for the current branch:

The download isn't fast, but still almost twice as fast:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  426M    0  426M    0     0  3462k      0 --:--:--  0:02:06 --:--:--  2618

Note: The SILO version that this has been tested with does not stream any data lazily yet. It still loads the full result into memory. Only LAPIS handles the response stream from SILO lazily.

[JonasKellerer]

LGTM

[fengelniederhammer]

Merging despite red docs tests, since this didn't change anything there. It's some infrastructure problem with GitHub / Microsoft and/or Ubuntu.