add: use preload-index and fscache for performance

[jeffhostetler]

Teach "add" to use preload-index and fscache features
to improve performance on very large repositories.

During an "add", a call is made to run_diff_files()
which calls check_remove() for each index-entry. This
calls lstat(). On Windows, the fscache code intercepts
the lstat() calls and builds a private cache using the
FindFirst/FindNext routines, which are much faster.

Somewhat independent of this, is the preload-index code
which distributes some of the start-up costs across
multiple threads.

Signed-off-by: Jeff Hostetler jeffhost@microsoft.com

[dscho]

Excellent!

[dscho]

Oops, I just realized something... read_cache_preload() does nothing else than read_cache() followed by some quick updates regarding the up-to-date state of the index: https://github.com/git/git/blob/v2.10.2/preload-index.c#L106-L113

(Please note that read_cache() is nothing else than read_index(&the_index), a hold-over from the time when Git's original working title was still known: dircache.)

So maybe the best course of action is to keep the original read_cache() and add the read_cache_preload(&pathspec); after parsing the pathspecs... That should have the same performance, maybe even better, because the second call will not really read the index but realize that it was already read, and perform only the stat updates: https://github.com/git/git/blob/v2.10.2/preload-index.c#L34-L68

I am thinking about something like this: dscho@8f26c6a

What do you think @jeffhostetler ?