-
-
Notifications
You must be signed in to change notification settings - Fork 558
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve loop for getAllQueries from FTL's memory #2163
Conversation
Has been confirmed working in the linked discourse topic. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are merge conflicts.
I think it's worth noting that this removed a feature of the API that allows combining multiple requests in the same API call. I don't really know if it is every used for this endpoint, but we use it elsewhere. Example: Requests like api.php?summary&getAllQueries
change behavior in this PR (summary
is now ignored)
Signed-off-by: Christian König <ckoenig@posteo.de>
Signed-off-by: Christian König <ckoenig@posteo.de>
Signed-off-by: Christian König <ckoenig@posteo.de>
Signed-off-by: Christian König <ckoenig@posteo.de>
Merge conflicts resolved. DL6ER made a good point. I searched for My last commit (moving the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks fine.
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there: https://discourse.pi-hole.net/t/pi-hole-ftl-v5-15-web-v5-12-and-core-v5-10-released/54987/1 |
This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there: https://discourse.pi-hole.net/t/confusion-from-the-main-display-screen/56102/4 |
git rebase
)git commit --signoff
)What does this PR aim to accomplish?:
This PR reduces the memory needed by
api.php?getAllQueries
to process every requested record.It's basically a copy of #2114 but for the current in-memory data served by FTL.
Usually we have seen the out-of-memory error when requesting large amounts of data from the long-term database, but currently there is an issue on discourse where the in-memory data is also too huge to process
getAllQueries
https://discourse.pi-hole.net/t/all-queries-are-not-working/54711
How does this PR accomplish the above?:
The old code used to generate and return the request adds all records into a single array. This array grows for each record (and there are many many records).
Only at the end (and if memory holds up), it transforms the entire array into JSON and sends all at once as a string.
The new code outputs each record as a string immediately.
This way, there is no array and the memory for each iteration is always the same.
The web server is responsible for sending the "text stream", nothing else needs to change.