Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CreateDirectory on Windows #66754

Closed
wants to merge 22 commits into from

Conversation

deeprobin
Copy link
Contributor

@deeprobin deeprobin commented Mar 17, 2022

Fixes #61954

As described in #61954 (comment) this only provides minor improvements.
Here it must be clearly decided which commits we take over and which are reverted again (Decision between micro improvement and code readability).
Personally, I also find it unnecessary that we use a List. However, I have not yet found a better way to get around this.
Even if the benchmarks don't show a big improvement, the interop calls in case of an error have been reduced and so in case of a throw the performance is improved (even if this is not a hot path)

In any case, an improvement of the memory allocation can be seen, since a lot has been spanified (see commit 6faa0e1).

Benchmarks

Before

Method Mean Error StdDev Median Min Max Allocated
CreateDirectory 195.6 us 18.80 us 20.90 us 191.3 us 174.4 us 247.5 us 497 B

After

Method Mean Error StdDev Median Min Max Allocated
CreateDirectory 187.7 us 9.26 us 10.29 us 186.1 us 171.3 us 204.9 us 455 B

/cc @danmoseley
/cc @adamsitnik
/cc @tmds

@ghost ghost added the community-contribution Indicates that the PR has been added by a community member label Mar 17, 2022
@ghost
Copy link

ghost commented Mar 17, 2022

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

⚠️ This is a draft - unexpected changes can take place at any time.

As described in #61954 (comment) this only provides minor improvements.
Here it must be clearly decided which commits we take over and which are reverted again (Decision between micro improvement and code readability).
Personally, I also find it unnecessary that we use a List. However, I have not yet found a better way to get around this.
Even if the benchmarks dont show a big improvement, the interop calls in case of an error have been reduced and so in case of a throw` the performance is improved (even if this is not a hot path)

In any case, an improvement of the memory allocation can be seen, since a lot has been spanified (see commit 6faa0e1).

Benchmarks

Before

Method Mean Error StdDev Median Min Max Allocated
CreateDirectory 180.1 us 6.94 us 7.43 us 177.6 us 169.2 us 195.9 us 497 B

After

Method Mean Error StdDev Median Min Max Allocated
CreateDirectory 173.7 us 3.70 us 3.64 us 173.9 us 166.8 us 178.7 us 450 B

/cc @danmoseley
/cc @adamsitnik
/cc @tmds

Author: deeprobin
Assignees: -
Labels:

area-System.IO, community-contribution

Milestone: -

@deeprobin
Copy link
Contributor Author

I have updated the benchmarks.

@danmoseley
Copy link
Member

github won't let me add this comment, but

EndsWithPeriodOrSpace and EndsWithPeriodOrSpaceSlim are each only essentially used in 1 place in the tree now and are both simple. I would inline them both and remove the methods.

@deeprobin
Copy link
Contributor Author

github won't let me add this comment, but

EndsWithPeriodOrSpace and EndsWithPeriodOrSpaceSlim are each only essentially used in 1 place in the tree now and are both simple. I would inline them both and remove the methods.

I think there are problems on the part of GitHub. I just could not adjust the description of another PR.

grafik


internal static bool EndsWithPeriodOrSpaceSlim(string path)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this method can be turned into a single line and produce the same code:
https://sharplab.io/#v2:EYLgxg9gTgpgtADwGwBYA0AXEBDAzgWwB8ABAJgEYBYAKGIGYACMhgYQYG8aHufuBLAHYYYUAdgA2TckgbAIESQEEAFMXIAGBgAdsGABYBKLrx6dqJi0wDs23XoDaO/QDoAMjAEBzfQzgNyALoMALzBDADkEQyEhLb6jnZuHt56vv5BoRHO4QDcxhYAvvkMxbyCwqISUjJyCgwAQqoacYalpm0WYHrYUAxgIS0JLu5ePn6BeeaWvMQ2/ZmRkTF9IWHh2ZPTRdQFQA===
(checked both 32 and 64 bit)

Given that, I suggest inlining path[path.Length - 1] == ' ' || path[path.Length - 1] == '.' into EndsWithPeriodOrSpace and EnsureExtendedPrefixIfNeeded. This eliminates a method that isn't really useful, but also is a trap for anyone in future that calls it without checking the length. It's perfectly clear when inlined.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, we have slightly changed the behavior here. Previously EndsWithPeriodOrSpace would return false for " " and now it will return true. Is that OK? I'm actually not sure.

If it is, then EndsWithPeriodOrSpace could be even less code: return path.Length > 0 && (path[path.Length - 1] == ' ' || path[path.Length - 1] == '.'); generates less code. But, I don't suggest changing that here.

Copy link
Contributor Author

@deeprobin deeprobin May 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have only two direct usages, which could break:

PathInternal.EnsureExtendedPrefixIfNeeded

        [return: NotNullIfNotNull("path")]
        internal static string? EnsureExtendedPrefixIfNeeded(string? path)
        {
            if (path != null && (path.Length >= MaxShortPath || EndsWithPeriodOrSpaceSlim(path)))
            {
                return EnsureExtendedPrefix(path);
            }
            else
            {
                return path;
            }
        }

and

FileSystemInfo.NormalizedPath

        internal string NormalizedPath
            => PathInternal.EndsWithPeriodOrSpace(FullPath) ? PathInternal.EnsureExtendedPrefix(FullPath) : FullPath;

@tmds @jozkee Are you comfortable with this change (Tests are passing, but behavior might slightly change)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@deeprobin I'd like to err on the side of caution here and avoid the subtle behavior change.

internal static int FillAttributeInfoSlim(string? path, ref Interop.Kernel32.WIN32_FILE_ATTRIBUTE_DATA data, bool returnErrorOnNotFound)
{
int errorCode = Interop.Errors.ERROR_SUCCESS;
string? prefixedString = PathInternal.EnsureExtendedPrefixIfNeeded(path);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The FindFirstFilePrefixed path here is the rare case, right? (<1%) Where the file is locked.

Given that, I don't think you're really saving anything by ensuring you only call EnsureExtendedPrefixIfNeeded once. I would reverse this and leave GetFileAttributesExPrivate private. That avoids creating a way for future code to accidentally call GetFileAttributesEx without prefixing. And it leaves this code a bit simpler.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was marked resolved but I don't see the feedback addressed. Was it commented on somewhere else?

@ghost ghost added the no-recent-activity label Apr 28, 2022
@ghost
Copy link

ghost commented Apr 28, 2022

This pull request has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

@ghost ghost removed the no-recent-activity label May 1, 2022
deeprobin and others added 2 commits May 2, 2022 20:01
Co-Authored-By: Dan Moseley <danmose@microsoft.com>
@deeprobin deeprobin marked this pull request as ready for review May 2, 2022 18:44
@jeffhandley jeffhandley self-assigned this May 30, 2022
@jeffhandley
Copy link
Member

@deeprobin I got this branch caught back up with main and reran the checks; there are some tests failing related to the long paths and the affected code. On my local Windows 11 machine, when I set HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled to 0, the failing tests reproduce locally.

~\git\dotnet\runtime\src\libraries\System.IO.FileSystem\tests [issue-61954-rev ≡] # dotnet test --filter "System.IO.Tests.DirectoryInfo_Create.DirectoryLongerThanMaxDirectoryAsPath_Succeeds"
  Determining projects to restore...
  All projects are up-to-date for restore.
  Microsoft.Interop.SourceGeneration -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\Microsoft.Interop.SourceGeneration\Debug\netstandard2.0\Microsoft.Interop.SourceGeneration.dll
  LibraryImportGenerator -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\LibraryImportGenerator\Debug\netstandard2.0\Microsoft.Interop.LibraryImportGenerator.dll
  TestUtilities -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\TestUtilities\Debug\net6.0\TestUtilities.dll
  System.Diagnostics.EventLog.Messages -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.Diagnostics.EventLog.Messages\Debug\netstandard2.0\System.Diagnostics.EventLog.Messages.dll
  System.Runtime -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.Runtime\ref\Debug\net7.0\System.Runtime.dll
  System.Security.Claims -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.Security.Claims\ref\Debug\net7.0\System.Security.Claims.dll
  System.Collections.NonGeneric -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.Collections.NonGeneric\ref\Debug\net7.0\System.Collections.NonGeneric.dll
  System.Security.Principal.Windows -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.Security.Principal.Windows\ref\Debug\net7.0\System.Security.Principal.Windows.dll
  System.Diagnostics.EventLog -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.Diagnostics.EventLog\ref\Debug\net7.0\System.Diagnostics.EventLog.dll
  System.Security.AccessControl -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.Security.AccessControl\ref\Debug\net7.0\System.Security.AccessControl.dll
  System.IO.FileSystem.AccessControl -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.IO.FileSystem.AccessControl\ref\Debug\net7.0\System.IO.FileSystem.AccessControl.dll
  System.IO.FileSystem.AccessControl -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.IO.FileSystem.AccessControl\Debug\net7.0-windows\System.IO.FileSystem.AccessControl.dll
  System.Diagnostics.EventLog -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.Diagnostics.EventLog\Debug\net7.0-windows\System.Diagnostics.EventLog.dll
  System.ServiceProcess.ServiceController -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.ServiceProcess.ServiceController\ref\Debug\net7.0\System.ServiceProcess.ServiceController.dll
  System.ServiceProcess.ServiceController -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.ServiceProcess.ServiceController\Debug\net7.0-windows\System.ServiceProcess.ServiceController.dll
  StreamConformanceTests -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\StreamConformanceTests\Debug\net7.0\StreamConformanceTests.dll
  System.IO.FileSystem.Tests -> C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.IO.FileSystem.Tests\Debug\net7.0-windows\System.IO.FileSystem.Tests.dll
Test run for C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.IO.FileSystem.Tests\Debug\net7.0-windows\System.IO.FileSystem.Tests.dll (.NETCoreApp,Version=v7.0)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:01:05.87]     System.IO.Tests.DirectoryInfo_Create.DirectoryLongerThanMaxDirectoryAsPath_Succeeds [FAIL]
  Failed System.IO.Tests.DirectoryInfo_Create.DirectoryLongerThanMaxDirectoryAsPath_Succeeds [478 ms]
  Error Message:
   Assert.All() Failure: 3 out of 3 items in the collection did not pass.
[2]: Item: C:\Users\jeffhand\AppData\Local\Temp\#DirectoryInfo_Create_lwiaguzz.mxc\DirectoryLongerThanMaxDirectoryAsPath_Succeeds_293_b25p5hmt\686c58d928f441b285b1de79365f0e77gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
     System.IO.PathTooLongException: The path 'C:\Users\jeffhand\AppData\Local\Temp\#DirectoryInfo_Create_lwiaguzz.mxc\DirectoryLongerThanMaxDirectoryAsPath_Succeeds_293_b25p5hmt\686c58d928f441b285b1de79365f0e77gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg' is too long, or a component of the specified path is too long.
        at System.IO.FileSystem.CreateDirectory(String fullPath, Byte[] securityDescriptor) in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\Common\src\System\IO\FileSystem.DirectoryCreation.Windows.cs:line 145
        at System.IO.DirectoryInfo.Create() in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\System.Private.CoreLib\src\System\IO\DirectoryInfo.cs:line 89
        at System.IO.Tests.DirectoryInfo_Create.Create(String path) in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\System.IO.FileSystem\tests\DirectoryInfo\Create.cs:line 15
        at System.IO.Tests.Directory_CreateDirectory.<DirectoryLongerThanMaxDirectoryAsPath_Succeeds>b__29_0(String path) in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\System.IO.FileSystem\tests\Directory\CreateDirectory.cs:line 296
        at Xunit.Assert.<>c__DisplayClass11_0`1.<All>b__0(T item, Int32 index) in /_/src/xunit.assert/Asserts/CollectionAsserts.cs:line 34
        at Xunit.Assert.All[T](IEnumerable`1 collection, Action`2 action) in /_/src/xunit.assert/Asserts/CollectionAsserts.cs:line 61
[1]: Item: C:\Users\jeffhand\AppData\Local\Temp\#DirectoryInfo_Create_lwiaguzz.mxc\DirectoryLongerThanMaxDirectoryAsPath_Succeeds_293_b25p5hmt\cc4253b47386458684bd29ae0f88d49fggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
     System.IO.PathTooLongException: The path 'C:\Users\jeffhand\AppData\Local\Temp\#DirectoryInfo_Create_lwiaguzz.mxc\DirectoryLongerThanMaxDirectoryAsPath_Succeeds_293_b25p5hmt\cc4253b47386458684bd29ae0f88d49fggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg' is too long, or a component of the specified path is too long.
        at System.IO.FileSystem.CreateDirectory(String fullPath, Byte[] securityDescriptor) in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\Common\src\System\IO\FileSystem.DirectoryCreation.Windows.cs:line 145
        at System.IO.DirectoryInfo.Create() in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\System.Private.CoreLib\src\System\IO\DirectoryInfo.cs:line 89
        at System.IO.Tests.DirectoryInfo_Create.Create(String path) in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\System.IO.FileSystem\tests\DirectoryInfo\Create.cs:line 15
        at System.IO.Tests.Directory_CreateDirectory.<DirectoryLongerThanMaxDirectoryAsPath_Succeeds>b__29_0(String path) in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\System.IO.FileSystem\tests\Directory\CreateDirectory.cs:line 296
        at Xunit.Assert.<>c__DisplayClass11_0`1.<All>b__0(T item, Int32 index) in /_/src/xunit.assert/Asserts/CollectionAsserts.cs:line 34
        at Xunit.Assert.All[T](IEnumerable`1 collection, Action`2 action) in /_/src/xunit.assert/Asserts/CollectionAsserts.cs:line 61
[0]: Item: C:\Users\jeffhand\AppData\Local\Temp\#DirectoryInfo_Create_lwiaguzz.mxc\DirectoryLongerThanMaxDirectoryAsPath_Succeeds_293_b25p5hmt\214cfc26777e4378876d8ed508d2fee7gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
     System.IO.PathTooLongException: The path 'C:\Users\jeffhand\AppData\Local\Temp\#DirectoryInfo_Create_lwiaguzz.mxc\DirectoryLongerThanMaxDirectoryAsPath_Succeeds_293_b25p5hmt\214cfc26777e4378876d8ed508d2fee7gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg' is too long, or a component of the specified path is too long.
        at System.IO.FileSystem.CreateDirectory(String fullPath, Byte[] securityDescriptor) in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\Common\src\System\IO\FileSystem.DirectoryCreation.Windows.cs:line 145
        at System.IO.DirectoryInfo.Create() in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\System.Private.CoreLib\src\System\IO\DirectoryInfo.cs:line 89
        at System.IO.Tests.DirectoryInfo_Create.Create(String path) in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\System.IO.FileSystem\tests\DirectoryInfo\Create.cs:line 15
        at System.IO.Tests.Directory_CreateDirectory.<DirectoryLongerThanMaxDirectoryAsPath_Succeeds>b__29_0(String path) in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\System.IO.FileSystem\tests\Directory\CreateDirectory.cs:line 296
        at Xunit.Assert.<>c__DisplayClass11_0`1.<All>b__0(T item, Int32 index) in /_/src/xunit.assert/Asserts/CollectionAsserts.cs:line 34
        at Xunit.Assert.All[T](IEnumerable`1 collection, Action`2 action) in /_/src/xunit.assert/Asserts/CollectionAsserts.cs:line 61
  Stack Trace:
     at System.IO.Tests.Directory_CreateDirectory.DirectoryLongerThanMaxDirectoryAsPath_Succeeds() in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\System.IO.FileSystem\tests\Directory\CreateDirectory.cs:line 294
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) in C:\Users\jeffhand\git\dotnet\runtime\src\libraries\System.Private.CoreLib\src\System\Reflection\MethodInvoker.cs:line 64
Results File: C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.IO.FileSystem.Tests\Debug\net7.0-windows\TestResults\jeffhand_JEFFHAND-DESK_2022-06-30_23_56_51.trx
Html test results file : C:\Users\jeffhand\git\dotnet\runtime\artifacts\bin\System.IO.FileSystem.Tests\Debug\net7.0-windows\TestResults\TestResult_jeffhand_JEFFHAND-DESK_20220630_235651.html

Failed!  - Failed:     1, Passed:     0, Skipped:     0, Total:     1, Duration: 15 ms - System.IO.FileSystem.Tests.dll (net7.0)

@jeffhandley jeffhandley added the needs-author-action An issue or pull request that requires more info or actions from the author. label Jul 1, 2022
@deeprobin
Copy link
Contributor Author

I take a look

@ghost ghost removed the needs-author-action An issue or pull request that requires more info or actions from the author. label Jul 4, 2022
Copy link
Member

@stephentoub stephentoub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this, but I'm unclear at this time what improvements the PR is concretely providing. Where are the cited allocation improvements coming from?

{
if (string.IsNullOrEmpty(path))
return false;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are the changes in this file necessary?

/// </summary>
internal static bool EndsInDirectorySeparator(ReadOnlySpan<char> path) =>
path.Length > 0 && IsDirectorySeparator(path[path.Length - 1]);

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are the changes in this file necessary?

@@ -986,7 +986,7 @@ private static string GetRelativePath(string relativeTo, string path, StringComp
/// <summary>
/// Returns true if the path ends in a directory separator.
/// </summary>
public static bool EndsInDirectorySeparator(ReadOnlySpan<char> path) => PathInternal.EndsInDirectorySeparator(path);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are the changes in this file necessary?


if (!IsPathUnreachableError(errorCode))
using SafeFindHandle handle = Interop.Kernel32.FindFirstFile(prefixedString!, ref findData);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of this change?

(lastError == 0) &&
(data.dwFileAttributes != -1) &&
((data.dwFileAttributes & Interop.Kernel32.FileAttributes.FILE_ATTRIBUTE_DIRECTORY) != 0);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding these "Slim" functions seems to be an attempt to improve throughput, but the perf results shared don't really show a throughput improvement. Seems like these changes should be reverted?

internal static int FillAttributeInfoSlim(string? path, ref Interop.Kernel32.WIN32_FILE_ATTRIBUTE_DATA data, bool returnErrorOnNotFound)
{
int errorCode = Interop.Errors.ERROR_SUCCESS;
string? prefixedString = PathInternal.EnsureExtendedPrefixIfNeeded(path);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was marked resolved but I don't see the feedback addressed. Was it commented on somewhere else?

@danmoseley
Copy link
Member

I'm going to close this as it's unclear what the benefits are. Feel free to reopen if you address feedback and can show evidence of improvement making the change worthwhile. Thanks.

@danmoseley danmoseley closed this Jul 19, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Aug 18, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.IO community-contribution Indicates that the PR has been added by a community member
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve CreateDirectory on Windows
5 participants