Hopefully preventing server crashes from running out of memory #3276
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(hopefully) fixes #3224
This PR improves the performance of two of our biggest API calls: /adminapi/labels/cvMetadata and /attributesWithLabels. Instead of trying to convert all of the labels to JSON in one go like we had before, which forces us to load everything into memory, we are now writing 10k labels at a time to JSON files. We can then transfer the JSON files with much less memory.
I tested this out on my dev environment by artificially lowering the amount of memory available to Java so that I could test with smaller datasets. This seems to be solving the issue. Will only know for sure once it is on production.
Additionally, I only did this with two API endpoints to start. If we determine that the method is working, it's pretty easy to add that workflow to other endpoints!
In the saved file we include the timestamp for when it was created, which will allow us to serve multiple concurrent API requests for the same API.
Things to check before submitting the PR