Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make PhysicalFilesWatcher exclusively use polling when pollForChanges=true #41426

Merged
merged 12 commits into from
Nov 12, 2020

Conversation

ericstj
Copy link
Member

@ericstj ericstj commented Aug 27, 2020

Fixes #37664

@ericstj ericstj added NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it area-Extensions-FileSystem labels Aug 27, 2020
@ericstj ericstj self-assigned this Aug 27, 2020
@ghost
Copy link

ghost commented Aug 27, 2020

Tagging subscribers to this area: @maryamariyan
See info in area-owners.md if you want to be subscribed.

@maryamariyan
Copy link
Member

The PR is giving a repro of the error.

https://dev.azure.com/dnceng/public/_build/results?buildId=789722&view=ms.vss-test-web.build-test-results-tab&runId=24915252&resultId=144507&paneView=debug

System.IO.IOException : The configured user limit (1024) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.

Stack trace

at System.IO.FileSystemWatcher.StartRaisingEvents() in /_/src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs:line 46 at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed() in /_/src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.cs:line 638 at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value) in /_/src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.cs:line 151 at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher() in /_/src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PhysicalFilesWatcher.cs:line 377 at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter) in /_/src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PhysicalFilesWatcher.cs:line 133 at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter) in /_/src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PhysicalFileProvider.cs:line 335 at Microsoft.Extensions.FileProviders.PhysicalFileProviderTests.PollingFileProviderShouldntConsumeINotifyInstances() in /_/src/libraries/Microsoft.Extensions.FileProviders.Physical/tests/PhysicalFileProviderTests.cs:line 113 at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__140_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1883

cc: @pranavkm

@maryamariyan
Copy link
Member

maryamariyan commented Aug 27, 2020

@pranavkm do you see any possible risk with making this fix? e8669aa

@ericstj you mentioned earlier that since polling would be the only source of events we could introduce some latency as well. Would there be a guarantee though that when we introduce latency this error gets avoided altogether?

@pranavkm
Copy link
Contributor

@pranavkm do you see any possible risk with making this fix? e8669aa

That looks great 👍

@ericstj ericstj removed NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it labels Aug 27, 2020
@ericstj ericstj marked this pull request as ready for review August 27, 2020 23:21
@ericstj ericstj changed the title Test out INotify exhaustion Make PhysicalFilessWatcher exclusively use polling when pollForChanges=true Aug 27, 2020
@ericstj
Copy link
Member Author

ericstj commented Aug 27, 2020

@pranavkm how likely do you imagine folks are to be regressed by this change? Since we aren't enabling the watcher it means that the only events will be raised by polling. This could result in latency or different events entirely if polling doesn't capture all types of changes that a FileSystemWatcher would. What do you think about the likelihood of this and overall general risk of making this change (at this point in the release)?

@pranavkm
Copy link
Contributor

@ericstj we've advertised the flag that enables polling as a resort for apps that are deployed to environments that do not trigger the .NET file watcher (e.g. mounted drives, network shares etc). For such users, this change would make no visible difference.

@ericstj ericstj assigned eerhardt and unassigned eerhardt Sep 1, 2020
@ericstj ericstj requested a review from eerhardt September 1, 2020 02:08
@ericstj ericstj assigned carlossanlop and unassigned carlossanlop Sep 1, 2020
@ericstj ericstj requested a review from carlossanlop September 1, 2020 05:09
@ericstj ericstj changed the title Make PhysicalFilessWatcher exclusively use polling when pollForChanges=true Make PhysicalFilesWatcher exclusively use polling when pollForChanges=true Sep 1, 2020
This is less breaking that ignoring the FileSystemWatcher when polling
is set as it still lets the PhysicalFileWatcher work in FSW + Polling
mode.

Now when a FileSystemWatcher is not passed to the PhysicalFileWatcher
we'll permit construction (if polling is enabled) and have it behave as
exclusively polling.

The PhysicalFileProvider will use this mode when both
UsePollingFileWatcher and UseActivePolling are set which is what happens
when the DOTNET_USE_POLLING_FILE_WATCHER environment variable is set.
@ericstj ericstj requested a review from maryamariyan September 4, 2020 16:36
Copy link
Member

@eerhardt eerhardt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

Copy link
Member

@stephentoub stephentoub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the existing comment on guard check placement, LGTM.

# Conflicts:
#	src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PhysicalFilesWatcher.cs
@ericstj
Copy link
Member Author

ericstj commented Sep 18, 2020

Test failures as follows:

Unhandled Exception:
System.IO.DirectoryNotFoundException: Could not find a part of the path '/tmp/aexy3v3e.mis'.
   at System.IO.Enumeration.FileSystemEnumerator`1[[System.IO.FileSystemInfo, System.IO.FileSystem, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].CreateDirectoryHandle(String path, Boolean ignoreNotFound) in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs:line 79
   at System.IO.Enumeration.FileSystemEnumerator`1[[System.IO.FileSystemInfo, System.IO.FileSystem, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].Init() in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs:line 39
   at System.IO.Enumeration.FileSystemEnumerator`1[[System.IO.FileSystemInfo, System.IO.FileSystem, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]]..ctor(String directory, Boolean isNormalized, EnumerationOptions options) in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerator.cs:line 41
   at System.IO.Enumeration.FileSystemEnumerable`1.DelegateEnumerator[[System.IO.FileSystemInfo, System.IO.FileSystem, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]]..ctor(FileSystemEnumerable`1 enumerable, Boolean isNormalized) in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerable.cs:line 66
   at System.IO.Enumeration.FileSystemEnumerable`1[[System.IO.FileSystemInfo, System.IO.FileSystem, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]]..ctor(String directory, FindTransform transform, EnumerationOptions options, Boolean isNormalized) in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerable.cs:line 38
   at System.IO.Enumeration.FileSystemEnumerableFactory.FileSystemInfos(String directory, String expression, EnumerationOptions options, Boolean isNormalized) in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerableFactory.cs:line 203
   at System.IO.DirectoryInfo.InternalEnumerateInfos(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options) in /_/src/libraries/System.IO.FileSystem/src/System/IO/DirectoryInfo.cs:line 190
   at System.IO.DirectoryInfo.EnumerateFileSystemInfos(String searchPattern, EnumerationOptions enumerationOptions) in /_/src/libraries/System.IO.FileSystem/src/System/IO/DirectoryInfo.cs:line 172
   at System.IO.DirectoryInfo.EnumerateFileSystemInfos(String searchPattern, SearchOption searchOption) in /_/src/libraries/System.IO.FileSystem/src/System/IO/DirectoryInfo.cs:line 169
   at Microsoft.Extensions.FileSystemGlobbing.Abstractions.DirectoryInfoWrapper.EnumerateFileSystemInfos()+MoveNext() in /_/src/libraries/Microsoft.Extensions.FileSystemGlobbing/src/Abstractions/DirectoryInfoWrapper.cs:line 38
   at System.Collections.Generic.List`1[[Microsoft.Extensions.FileSystemGlobbing.Abstractions.FileSystemInfoBase, Microsoft.Extensions.FileSystemGlobbing, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].InsertRange(Int32 index, IEnumerable`1 collection) in /_/src/libraries/System.Private.CoreLib/src/System/Collections/Generic/List.cs:line 742
   at System.Collections.Generic.List`1[[Microsoft.Extensions.FileSystemGlobbing.Abstractions.FileSystemInfoBase, Microsoft.Extensions.FileSystemGlobbing, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].AddRange(IEnumerable`1 collection) in /_/src/libraries/System.Private.CoreLib/src/System/Collections/Generic/List.cs:line 243
   at Microsoft.Extensions.FileSystemGlobbing.Internal.MatcherContext.Match(DirectoryInfoBase directory, String parentRelativePath) in /_/src/libraries/Microsoft.Extensions.FileSystemGlobbing/src/Internal/MatcherContext.cs:line 67
   at Microsoft.Extensions.FileSystemGlobbing.Internal.MatcherContext.Execute() in /_/src/libraries/Microsoft.Extensions.FileSystemGlobbing/src/Internal/MatcherContext.cs:line 53
   at Microsoft.Extensions.FileSystemGlobbing.Matcher.Execute(DirectoryInfoBase directoryInfo) in /_/src/libraries/Microsoft.Extensions.FileSystemGlobbing/src/Matcher.cs:line 166
   at Microsoft.Extensions.FileProviders.Physical.PollingWildCardChangeToken.CalculateChanges() in /_/src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PollingWildCardChangeToken.cs:line 108
   at Microsoft.Extensions.FileProviders.Physical.PollingWildCardChangeToken.get_HasChanged() in /_/src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PollingWildCardChangeToken.cs:line 98
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.RaiseChangeEvents(Object state) in /_/src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PhysicalFilesWatcher.cs:line 437
   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 627
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 589
   at System.Threading.TimerQueueTimer.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 573
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 646
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 41
   at System.Threading.Thread.StartCallback() in /_/src/mono/netcore/System.Private.CoreLib/src/System/Threading/Thread.Mono.cs:line 278
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.DirectoryNotFoundException: Could not find a part of the path '/tmp/aexy3v3e.mis'.
   at System.IO.Enumeration.FileSystemEnumerator`1[[System.IO.FileSystemInfo, System.IO.FileSystem, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].CreateDirectoryHandle(String path, Boolean ignoreNotFound) in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs:line 79
   at System.IO.Enumeration.FileSystemEnumerator`1[[System.IO.FileSystemInfo, System.IO.FileSystem, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].Init() in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerator.Unix.cs:line 39
   at System.IO.Enumeration.FileSystemEnumerator`1[[System.IO.FileSystemInfo, System.IO.FileSystem, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]]..ctor(String directory, Boolean isNormalized, EnumerationOptions options) in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerator.cs:line 41
   at System.IO.Enumeration.FileSystemEnumerable`1.DelegateEnumerator[[System.IO.FileSystemInfo, System.IO.FileSystem, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]]..ctor(FileSystemEnumerable`1 enumerable, Boolean isNormalized) in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerable.cs:line 66
   at System.IO.Enumeration.FileSystemEnumerable`1[[System.IO.FileSystemInfo, System.IO.FileSystem, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]]..ctor(String directory, FindTransform transform, EnumerationOptions options, Boolean isNormalized) in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerable.cs:line 38
   at System.IO.Enumeration.FileSystemEnumerableFactory.FileSystemInfos(String directory, String expression, EnumerationOptions options, Boolean isNormalized) in /_/src/libraries/System.IO.FileSystem/src/System/IO/Enumeration/FileSystemEnumerableFactory.cs:line 203
   at System.IO.DirectoryInfo.InternalEnumerateInfos(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options) in /_/src/libraries/System.IO.FileSystem/src/System/IO/DirectoryInfo.cs:line 190
   at System.IO.DirectoryInfo.EnumerateFileSystemInfos(String searchPattern, EnumerationOptions enumerationOptions) in /_/src/libraries/System.IO.FileSystem/src/System/IO/DirectoryInfo.cs:line 172
   at System.IO.DirectoryInfo.EnumerateFileSystemInfos(String searchPattern, SearchOption searchOption) in /_/src/libraries/System.IO.FileSystem/src/System/IO/DirectoryInfo.cs:line 169
   at Microsoft.Extensions.FileSystemGlobbing.Abstractions.DirectoryInfoWrapper.EnumerateFileSystemInfos()+MoveNext() in /_/src/libraries/Microsoft.Extensions.FileSystemGlobbing/src/Abstractions/DirectoryInfoWrapper.cs:line 38
   at System.Collections.Generic.List`1[[Microsoft.Extensions.FileSystemGlobbing.Abstractions.FileSystemInfoBase, Microsoft.Extensions.FileSystemGlobbing, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].InsertRange(Int32 index, IEnumerable`1 collection) in /_/src/libraries/System.Private.CoreLib/src/System/Collections/Generic/List.cs:line 742
   at System.Collections.Generic.List`1[[Microsoft.Extensions.FileSystemGlobbing.Abstractions.FileSystemInfoBase, Microsoft.Extensions.FileSystemGlobbing, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].AddRange(IEnumerable`1 collection) in /_/src/libraries/System.Private.CoreLib/src/System/Collections/Generic/List.cs:line 243
   at Microsoft.Extensions.FileSystemGlobbing.Internal.MatcherContext.Match(DirectoryInfoBase directory, String parentRelativePath) in /_/src/libraries/Microsoft.Extensions.FileSystemGlobbing/src/Internal/MatcherContext.cs:line 67
   at Microsoft.Extensions.FileSystemGlobbing.Internal.MatcherContext.Execute() in /_/src/libraries/Microsoft.Extensions.FileSystemGlobbing/src/Internal/MatcherContext.cs:line 53
   at Microsoft.Extensions.FileSystemGlobbing.Matcher.Execute(DirectoryInfoBase directoryInfo) in /_/src/libraries/Microsoft.Extensions.FileSystemGlobbing/src/Matcher.cs:line 166
   at Microsoft.Extensions.FileProviders.Physical.PollingWildCardChangeToken.CalculateChanges() in /_/src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PollingWildCardChangeToken.cs:line 108
   at Microsoft.Extensions.FileProviders.Physical.PollingWildCardChangeToken.get_HasChanged() in /_/src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PollingWildCardChangeToken.cs:line 98
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.RaiseChangeEvents(Object state) in /_/src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PhysicalFilesWatcher.cs:line 437
   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 627
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 589
   at System.Threading.TimerQueueTimer.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 573
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 646
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 41
   at System.Threading.Thread.StartCallback() in /_/src/mono/netcore/System.Private.CoreLib/src/System/Threading/Thread.Mono.cs:line 278

This indicates a problem with the polling file watcher if the directory it is watching is deleted underneath it. This is a reliability issue and one we should probably address. Without it the tests I added will be flaky.

We have to avoid exclusively using FileSystemWatcher since it looks like
FSW misses file deletes that happen as a result of directory delete on OSX.
@ericstj
Copy link
Member Author

ericstj commented Nov 11, 2020

Ok, the flakiness in this test should be fixed now. Can I get another review?

Copy link
Member

@eerhardt eerhardt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just had one tiny nit. LGTM.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
6 participants