-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve performance of fscache #1926
Improve performance of fscache #1926
Conversation
* splitting up the cache entries across multiple threads so there isn't | ||
* any overlap between threads anyway. | ||
*/ | ||
struct fscache { |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
@benpeart have you seen these failures?
I also see a build failure e.g. with
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you don't beat me to it, I will investigate further why the mem_pool
is already discarded when we try to use it to allocate memory.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two more things, to let Linux32 build the code. The failure in the Windows job is a bogus one: t3305's scratch directory could not be removed, for whatever reason. The tests passed, though.
Add tracing around initializing and discarding mempools. In discard report on the amount of memory unused in the current block to help tune setting the initial_size. Signed-off-by: Ben Peart <benpeart@microsoft.com>
Add cache hit/miss statistics to the fscache for lstat() and opendir(). The statistics are printed out when the cache is disabled and cleared and only if GIT_TRACE_FSCACHE is set. Signed-off-by: Ben Peart <benpeart@microsoft.com>
Update enable_fscache() to take an optional initial size parameter which is used to initialize the hashmap so that it can avoid having to rehash as additional entries are added. Add a separate disable_fscache() macro to make the code clearer and easier to read. Signed-off-by: Ben Peart <benpeart@microsoft.com>
The threading model for fscache has been to have a single, global cache. This puts requirements on it to be thread safe so that callers like preload-index can call it from multiple threads. This was implemented with a single mutex and completion events which introduces contention between the calling threads. Simplify the threading model by making fscache thread specific. This allows us to remove the global mutex and synchronization events entirely and instead associate a fscache with every thread that requests one. This works well with the current multi-threading which divides the cache entries into blocks with a separate thread processing each block. At the end of each worker thread, if there is a fscache on the primary thread, merge the cached results from the worker into the primary thread cache. This enables us to reuse the cache later especially when scanning for untracked files. In testing, this reduced the time spent in preload_index() by about 25% and also reduced the CPU utilization significantly. On a repo with ~200K files, it reduced overall status times by ~12%. Signed-off-by: Ben Peart <benpeart@microsoft.com>
Now that the fscache is single threaded, take advantage of the mem_pool as the allocator to significantly reduce the cost of allocations and frees. With the reduced cost of free, in future patches, we can start freeing the fscache at the end of commands instead of just leaking it. Signed-off-by: Ben Peart <benpeart@microsoft.com>
The FSCache feature [was optimized to become faster](git-for-windows/git#1926). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This patch series encompasses a set of changes to the fscache that underlies git's lstat() and opendir(). In addition to some additional tracing, the fscache is enhanced to reduce thread contention and take advantage of the new mem_pool heap manager. The net result is ~25% reduction to preload_index() vs the old fscache.