Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trace: add flush worker and optimize memory usage #967

Merged
merged 4 commits into from
Oct 6, 2023
Merged

Conversation

sreimers
Copy link
Member

@sreimers sreimers commented Sep 27, 2023

This reduces trace memory usage from ~137 MByte (1.000.000 entries) to 13,7 MByte (100.000 entries). Every second the trace buffer is flushed asynchronously if more than 1000 entries reached (TRACE_FLUSH_THRESHOLD).

All values can be overriden on compile time:

-DCMAKE_C_FLAGS="-DTRACE_BUFFER_SIZE=10000 -DTRACE_FLUSH_THRESHOLD=100 -DTRACE_FLUSH_TMR=1000"

TRACE_FLUSH_TMR is in ms.

@cspiel1
Copy link
Collaborator

cspiel1 commented Sep 28, 2023

We should be careful that the flush does not cause massive CPU load. This may influence the result.

@sreimers
Copy link
Member Author

Flush should mostly I/O bound and runs on a different thread, but I can think about some optimizations, like running tmr more periodically and flush smaller amount of entries.

@sreimers sreimers marked this pull request as ready for review October 6, 2023 09:19
@sreimers sreimers merged commit 33599fa into main Oct 6, 2023
44 checks passed
@sreimers sreimers deleted the trace_flush branch October 6, 2023 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants