Consider walking directories in up-to-date checks

[rainersigwald]

@ccastanedaucf has observed (#3547 (comment)) significant time spent in up-to-date checks in https://github.com/Microsoft/msbuild/blob/680224d8ecf9a7da82c001d95db336fd9cfd12d8/src/Build/BackEnd/Components/RequestBuilder/TargetUpToDateChecker.cs#L937

Git for Windows got a big speedup by implementing some caching around checks for individual file timestamps (with the core.fscache setting). See https://github.com/git-for-windows/git/blob/master/compat/win32/fscache.c and its history.

The gist of the idea is that it's faster to enumerate all files in a directory and get every file's modification timestamp than it is to get the timestamps on an individual basis. Since there's probably a fairly high degree of overlap in the directories of files we're checking for timestamps, I suspect we could get some benefit from this technique too.

[AndyGerlicher]

@ccastanedaucf can you take a look at implementing this? Might help get Windows closer to Linux after #3547 is in.

[cdmihai]

Would be nice if you could implement it over the existing IFileSystem :)

Also, this might be a very good use case for the fast allocation free directory enumeration introduced by @JeremyKuhne https://blogs.msdn.microsoft.com/jeremykuhne/2018/03/09/custom-directory-enumeration-in-net-core-2-1/

[4creators]

The above problems are a huge burden in some dotnet repos see #2843

[rainersigwald]

@4creators This will probably help with that, but I'd be shocked if it was a very high percentage of the overall incremental build time in those repos.

[ccastanedaucf]

@cdmihai small bench comparing methods of getting last write time for 10 files on Windows. Right now we use the equivalent of File.GetLastWriteTimeUtc.

                       Method |      Mean |     Error |    StdDev |
----------------------------- |----------:|----------:|----------:|
     File_GetLastWriteTimeUtc | 257.60 us | 0.7359 us | 0.5745 us |
 DirectoryInfo_EnumerateFiles |  53.60 us | 1.1738 us | 1.4416 us |
         FileSystemEnumerable |  44.68 us | 0.8567 us | 0.8414 us |

[JeremyKuhne]

small bench comparing methods of getting last write time for 10 files on Windows

Allocations would be interesting to measure as well. Using DirectoryInfo requires allocating a FileSystemInfo around every result, which will also allocate strings to do so. With FileSystemEnumerable you can get zero allocations (outside of the enumerable itself).

[ccastanedaucf]

1000 files in same directory + memory allocations. Looks like FileSystemEnumerable has a constant memory allocation for a given directory.
Edit: wasn't actually enumerating past the first item on FileSystemEnumerable which explains why runtime didn't change, updated times/allocation. Still a huge difference and allocation stays constant.

                       Method |         Mean |       Error |      StdDev |   Gen 0 |  Gen 1 |  Gen 2 | Allocated |
----------------------------- |-------------:|------------:|------------:|--------:|-------:|-------:|----------:|
     File_GetLastWriteTimeUtc | 45,740.60 us | 666.3182 us | 590.6736 us |       - |      - |      - | 109.38 KB |
 DirectoryInfo_EnumerateFiles |  1,036.26 us |  19.6407 us |  18.3719 us | 56.6406 |      - |      - | 242.71 KB |
         FileSystemEnumerable |     357.7 us |    5.823 us |    7.151 us |       - |      - |      - |     296 B |

[JeremyKuhne]

1000 files in same directory + memory allocations

Cool. Shows a pretty big difference. :)

I'm curious about the Allocated (1.71K) number. Is it measuring GC.GetAllocatedBytesForCurrentThread or GetTotalmemory? It is just a somewhat odd total for me to place.

[cdmihai]

Looks like it's ETW allocation events: https://github.com/dotnet/BenchmarkDotNet/blob/36bf7c4ee053134d0bfeaf5d757c12afa332f1c2/src/BenchmarkDotNet.Diagnostics.Windows/MemoryDiagnoser.cs#L74

[rainersigwald]

#3608 is an implementation of this that should be revisited if we move forward with this bug.

[rainersigwald]

Another possible implementation to help with this: #2848; it should also be looked at.

[mfkl]

Has there been more data to help with a design for a solution to this problem?

Looking at previous comments/PRs, native calls vs DirectoryInfo_EnumerateFiles vs FileSystemEnumerable are some of the possible options I guess.

[AR-May]

We are waiting for final measurements for PR #6974 from @rokonec. It is likely that this PR the issue with up-to-date checks would be mostly resolved and that further room for improvements would be too small.

[ladipro]

Closing, assuming that #6974 will be merged soon. Please reopen otherwise.