Optimize glob evaluation #6151
Merged
Optimize glob evaluation #6151
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ladipro
force-pushed
the
6069-globbing-improvements
branch
from
e0b1252 to
e4d7133
Compare
ladipro
commented
Feb 12, 2021
ladipro
force-pushed
the
6069-globbing-improvements
branch
from
e4d7133 to
8bb636c
Compare
donJoseLuis
reviewed
Feb 16, 2021
ladipro
force-pushed
the
6069-globbing-improvements
branch
2 times, most recently
from
4ac817f to
04974ae
Compare
ladipro
commented
Feb 16, 2021
ladipro
commented
Feb 17, 2021
src/Build.UnitTests/Definition/ProjectEvaluationContext_Tests.cs
Outdated
Show resolved
Hide resolved
ladipro
force-pushed
the
6069-globbing-improvements
branch
from
04974ae to
e4e1371
Compare
rokonec
approved these changes
Feb 18, 2021
Forgind
reviewed
Feb 18, 2021
cdmihai
reviewed
Feb 20, 2021
cdmihai
reviewed
Feb 20, 2021
cdmihai
reviewed
Feb 20, 2021
cdmihai
approved these changes
Feb 20, 2021
Forgind
approved these changes
Feb 22, 2021
Forgind
added
the
merge-when-branch-open
cdmihai
reviewed
Feb 22, 2021
cdmihai
approved these changes
Feb 22, 2021
ladipro
commented
Feb 23, 2021
ladipro
force-pushed
the
6069-globbing-improvements
branch
from
a45184b to
c5afa7c
Compare
This was referenced Mar 4, 2021
ladipro
added a commit
that referenced
this pull request
Mar 10, 2021
### Context #6151 introduced a regression where `FileMatcher` takes the pattern `*.*` too literally and returns only files that have a dot in the name. `*.*` should be special cased to mean all files in the directory, with or without an extension. ### Changes Made Fixed the regression by explicitly testing for `*.*` and added test coverage. ### Testing Existing and modified unit tests, repro project from dotnet/sdk#16185 (comment). ### Notes Testing for both `*` and `*.*` is already happening elsewhere in the class. MSBuild calls `Directory.EnumerateFileSystemEntries` which under the covers uses `MatchType.Win32` and causes this behavior of unifying `*.*` with `*` on all platforms.
|
Hey, folks, could this be related to dotnet/sdk#17976? Could someone please take a look? |
This was referenced May 31, 2021
rainersigwald
pushed a commit
to rainersigwald/msbuild
that referenced
this pull request
Jun 8, 2021
Fixes dotnet#6502 Summary This change fixes a regression in glob matching where files without extension are erroneously not matched when taking a specific globbing code path. Customer impact Any customer who uses a glob pattern susceptible to the bug and has files without extensions in their source tree is affected. The bug was reported by external customers. Regression? Yes, caused by dotnet#6151 where glob matching was optimized which internally made it take a different code path. Changes Made Fixes the regression by properly handling `*.*` to mean all files, not just files with a dot in the name. This convention is used in .NET APIs on all platforms and matches the pre-regression behavior. Testing Added unit test coverage. Also verified locally with the repro provided by the original bug reporter. Risk Low. The star patterns are special-cased to mean all files, other patterns are unaffected.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #6069
Context
Glob expansion has been identified as one of the performance bottlenecks when building .NET Core projects. The cost is not negligible with a simple Hello World project and gets progressively worse when the project directory (transitively) contains more files, be it source files or completely unrelated files. Since glob expansion is part of project evaluation, it directly impacts Visual Studio performance as well, especially solution load performance.
Changes Made
This PR is a collection of several changes to the globbing logic aiming to make .NET Core project evaluation faster.
Testing
Existing unit tests for correctness, ETL traces for performance.
Performance:
The wins in terms of the Microsoft-Build/ExpandGlob event duration range from 16% (Hello World project built with
MSBuild.exe) to 48% (Hello World project with ~4000 files in ~600 folders, all non-.cs and non-.resx, built withdotnet build). These numbers were measured with incremental build on the command line, the project was a .NET Core project in all cases, just with a different build engine. Here's a table with all numbers:In terms of the overall build time, we're looking at a 1-2% improvement for a Hello World project and 8-10% for a project with thousands of extra files in the source tree.
Notes
This PR is recommended to be reviewed commit by commit.
FileMatcheris a complex class and there is non-trivial breaking potential as projects may depend on all kinds of corner case behavior. Which is the primary reason for not rewriting the entire thing. The most impactful change in this PR - caching files to avoid repeated filesystem API calls - is behind a change wave condition, just in case.