Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IFC-751 save diff in smaller pieces to prevent OOM #4544

Merged
merged 2 commits into from
Oct 6, 2024

Conversation

ajtmccarty
Copy link
Contributor

fixes #4511
IFC-751

refactors logic to save an enriched diff so that it uses multiple separate queries instead of one big one to prevent an out-of-memory error when loading a gigantic serialized diff into the database driver

new process is

  • create new enriched diff root nodes (one for main -> branch, one for main -> main)
  • load diff nodes in batches of 100 (one query per batch)
  • link hierarchical nodes with the diff

@github-actions github-actions bot added the group/backend Issue related to the backend (API Server, Git Agent) label Oct 4, 2024
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the changes in here are mostly just refactoring what used to be in save_query.py, but I also renamed the file, so git counts this as all new stuff, which it is not

@ajtmccarty ajtmccarty marked this pull request as ready for review October 5, 2024 20:49
@ajtmccarty ajtmccarty requested a review from a team October 5, 2024 20:49
@dgarros
Copy link
Collaborator

dgarros commented Oct 6, 2024

Please can you add a fragment for the changelog before merging the PR

Copy link

codspeed-hq bot commented Oct 6, 2024

CodSpeed Performance Report

Merging #4544 will degrade performances by 92.09%

Comparing ajtm-10042024-save-big-diff-IFC-751 (4cb5d48) with stable (01e37a6)

Summary

❌ 3 regressions
✅ 7 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark stable ajtm-10042024-save-big-diff-IFC-751 Change
test_get_schema 197.7 ms 805.1 ms -75.45%
test_base_schema_duplicate_CoreProposedChange 1.5 ms 18.4 ms -92.09%
test_schemabranch_process 347 ms 2,925.1 ms -88.14%

@ajtmccarty ajtmccarty merged commit bf8e85e into stable Oct 6, 2024
30 of 31 checks passed
@ajtmccarty ajtmccarty deleted the ajtm-10042024-save-big-diff-IFC-751 branch October 6, 2024 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
group/backend Issue related to the backend (API Server, Git Agent)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants