Impl. size-dependent request body compression #4283
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pvannierop
force-pushed
the
sizedependent_req_body_compression
branch
from
016124a to
8a1d3f3
Compare
Luke-Sikina
reviewed
Jun 3, 2022
| body = pako.gzip(JSON.stringify(body)).buffer; | ||
| if (method === 'POST') { | ||
| var bodyString = JSON.stringify(body); | ||
| if (bodyString.length > REQ_BODY_SIZE_CHAR_LIMIT) { |
There was a problem hiding this comment.
How expensive would it be to do new Blob([bodyString]).size? Using char count to estimate size is a bit unreliable.
There was a problem hiding this comment.
@Luke-Sikina it's just a heuristic right? and why is it innacurate?
There was a problem hiding this comment.
@Luke-Sikina I also reasoned that a precise estimate is not needed here. We just need the least resource-intensive estimate of request size.
pvannierop
force-pushed
the
sizedependent_req_body_compression
branch
from
a1f3e6e to
8a1d3f3
Compare
For both Public and Internal API
pvannierop
force-pushed
the
sizedependent_req_body_compression
branch
from
8a1d3f3 to
e2fe4da
Compare
alisman
approved these changes
Jun 7, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem 1
When loading studies with large number of samples there are a large number of endpoints that try to send extended lists of sample identifiers to the backend. In uncompressed for this can take minutes for ~100k samples. This bottlenet can be removed by compressing the request body before sending. At present this is implemented for specific endpoints. I found out that certain (new?) endpoints are not listed for compression and therefore lock the frontend in an inresponsive state. Therefore, the mechanism of selecting specific endpoints for compression does not effectively solve the problem of large request bodies when the configuration only partially covers problematic endpoints.
Problem 2
The current request body compression is only applied to endpoints of the public API. Certain endpoints of the internal API are very slow.
Solution
I propose to evaluate body of every POST request for size and compress the body when the number of characters in the JSON string exceeds a certain threshold (now arbitrarily set at
500characters). The compression mechanism covers both public and internal API's.