NetworkInformationException DirectoryNotFoundException: Could not find a part of the path '/proc/sys/net/ipv4/conf/lo/forwarding' with NET 7.0.2 on gVisor

[ElanHasson]

Description

Original Issue I filed in the dotnet/orleans repo: dotnet/orleans#8271
Issue Picked up by gVisor team: google/gvisor#8414 🥳

I'll try to keep the information here more specific to the runtime issue than to gVisor and Orleans.

The Orleans Call Site is line 351 on the call to NetworkInterface.GetAllNetworkInterfaces() in #76383.

Best guess with suspect is in the runtime: https://github.com/dotnet/runtime/pull/76383/files, timing checks out.
The comments directly above indicate we shouldn't fail but return false

Stack Trace:

2023-01-20T18:21:24.006086878Z Web Scheduler terminated unexpectedly.
2023-01-20T18:21:24.087335495Z System.AggregateException: One or more errors occurred. (An error was encountered while querying information from the operating system.)
  2023-01-20T18:21:24.087379874Z  ---> System.Net.NetworkInformation.NetworkInformationException (2): An error was encountered while querying information from the operating system. ---> System.IO.DirectoryNotFoundException: Could not find a part of the path '/proc/sys/net/ipv4/conf/lo/forwarding'.
2023-01-20T18:21:24.087413687Z    at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
2023-01-20T18:21:24.087426200Z    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Func`4 createOpenException)
2023-01-20T18:21:24.087467392Z    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
2023-01-20T18:21:24.087474969Z    at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
2023-01-20T18:21:24.087523064Z    at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
2023-01-20T18:21:24.087553229Z    at System.IO.File.ReadAllText(String path, Encoding encoding)
2023-01-20T18:21:24.087620365Z    at System.Net.NetworkInformation.StringParsingHelpers.ReadAllText(String filePath)
2023-01-20T18:21:24.087625410Z    at System.Net.NetworkInformation.StringParsingHelpers.ParseRawIntFile(String filePath)
2023-01-20T18:21:24.087659756Z    at System.Net.NetworkInformation.LinuxIPv4InterfaceProperties.GetIsForwardingEnabled()
2023-01-20T18:21:24.087688157Z    at System.Net.NetworkInformation.LinuxIPInterfaceProperties..ctor(LinuxNetworkInterface lni, LinuxNetworkInterfaceSystemProperties systemProperties)
2023-01-20T18:21:24.087695819Z    at System.Net.NetworkInformation.LinuxNetworkInterface.GetLinuxNetworkInterfaces()
2023-01-20T18:21:24.087699797Z    at Orleans.Runtime.Configuration.ConfigUtilities.ResolveIPAddress(String addrOrHost, Byte[] subnet, AddressFamily family) in /_/src/Orleans.Core/Configuration/ConfigUtilities.cs:line 351
2023-01-20T18:21:24.087727244Z    --- End of inner exception stack trace ---
2023-01-20T18:21:24.087732445Z    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
2023-01-20T18:21:24.087758859Z    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
2023-01-20T18:21:24.087803031Z    at Orleans.Hosting.EndpointOptionsExtensions.ConfigureEndpoints(ISiloBuilder builder, String hostname, Int32 siloPort, Int32 gatewayPort, AddressFamily addressFamily, Boolean listenOnAnyHostAddress) in /_/src/Orleans.Runtime/Hosting/EndpointOptionsExtensions.cs:line 132
2023-01-20T18:21:24.087808822Z    at Orleans.Hosting.EndpointOptionsExtensions.ConfigureEndpoints(ISiloBuilder builder, Int32 siloPort, Int32 gatewayPort, AddressFamily addressFamily, Boolean listenOnAnyHostAddress) in /_/src/Orleans.Runtime/Hosting/EndpointOptionsExtensions.cs:line 152
2023-01-20T18:21:24.087815990Z    at WebScheduler.Server.Program.ConfigureSiloBuilder(HostBuilderContext context, ISiloBuilder siloBuilder) in /home/runner/work/web-scheduler/web-scheduler/Source/WebScheduler.Server/Program.cs:line 84
2023-01-20T18:21:24.087823095Z    at Microsoft.Extensions.Hosting.GenericHostExtensions.<>c__DisplayClass0_0.<UseOrleans>b__0(HostBuilderContext context, IServiceCollection services) in /_/src/Orleans.Runtime/Hosting/GenericHostExtensions.cs:line 39
2023-01-20T18:21:24.087841718Z    at Orleans.Hosting.SiloBuilder.InvokeConfigureServicesDelegates(HostBuilderContext ctx, IServiceCollection services) in /_/src/Orleans.Runtime/Hosting/SiloBuilder.cs:line 32
2023-01-20T18:21:24.087850346Z    at Orleans.Hosting.SiloBuilder.<.ctor>b__4_0(HostBuilderContext ctx, IServiceCollection services) in /_/src/Orleans.Runtime/Hosting/SiloBuilder.cs:line 22
2023-01-20T18:21:24.087857676Z    at Microsoft.Extensions.Hosting.HostBuilder.InitializeServiceProvider()
2023-01-20T18:21:24.088156194Z    at Microsoft.Extensions.Hosting.HostBuilder.Build()
2023-01-20T18:21:24.088166643Z    at WebScheduler.Server.Program.Main(String[] args) in /home/runner/work/web-scheduler/web-scheduler/Source/WebScheduler.Server/Program.cs:line 34

Reproduction Steps

This should be reproducible via a call to [NetworkInterface.GetAllNetworkInterfaces()](https://github.com/dotnet/orleans/blob/3.x/src/Orleans.Core/Configuration/ConfigUtilities.cs#L342-L356) from a .NET 7.0.2 app running in a gVisor-sandboxed container.

Expected behavior

If the file or path isn't found, false should be returned, which was previous behavior.

Actual behavior

A System.Net.NetworkInformation.NetworkInformationException is thrown with an inner exception of System.IO.DirectoryNotFoundException: Could not find a part of the path '/proc/sys/net/ipv4/conf/lo/forwarding'.

Regression?

I believe this to be a regression.

See #75883 and #63696.

The same code I have works with .NET 6, but not 7.

Known Workarounds

No response

Configuration

No response

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Original Issue I filed in the dotnet/orleans repo: dotnet/orleans#8271
Issue Picked up by gVisor team: google/gvisor#8414 🥳

I'll try to keep the information here more specific to the runtime issue than to gVisor and Orleans.

The Orleans Call Site is line 351 on the call to NetworkInterface.GetAllNetworkInterfaces() in #76383.

Best guess with suspect is in the runtime: https://github.com/dotnet/runtime/pull/76383/files, timing checks out.
The comments directly above indicate we shouldn't fail but return false

Stack Trace:

2023-01-20T18:21:24.006086878Z Web Scheduler terminated unexpectedly.
2023-01-20T18:21:24.087335495Z System.AggregateException: One or more errors occurred. (An error was encountered while querying information from the operating system.)
  2023-01-20T18:21:24.087379874Z  ---> System.Net.NetworkInformation.NetworkInformationException (2): An error was encountered while querying information from the operating system. ---> System.IO.DirectoryNotFoundException: Could not find a part of the path '/proc/sys/net/ipv4/conf/lo/forwarding'.
2023-01-20T18:21:24.087413687Z    at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
2023-01-20T18:21:24.087426200Z    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Func`4 createOpenException)
2023-01-20T18:21:24.087467392Z    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Func`4 createOpenException)
2023-01-20T18:21:24.087474969Z    at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
2023-01-20T18:21:24.087523064Z    at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
2023-01-20T18:21:24.087553229Z    at System.IO.File.ReadAllText(String path, Encoding encoding)
2023-01-20T18:21:24.087620365Z    at System.Net.NetworkInformation.StringParsingHelpers.ReadAllText(String filePath)
2023-01-20T18:21:24.087625410Z    at System.Net.NetworkInformation.StringParsingHelpers.ParseRawIntFile(String filePath)
2023-01-20T18:21:24.087659756Z    at System.Net.NetworkInformation.LinuxIPv4InterfaceProperties.GetIsForwardingEnabled()
2023-01-20T18:21:24.087688157Z    at System.Net.NetworkInformation.LinuxIPInterfaceProperties..ctor(LinuxNetworkInterface lni, LinuxNetworkInterfaceSystemProperties systemProperties)
2023-01-20T18:21:24.087695819Z    at System.Net.NetworkInformation.LinuxNetworkInterface.GetLinuxNetworkInterfaces()
2023-01-20T18:21:24.087699797Z    at Orleans.Runtime.Configuration.ConfigUtilities.ResolveIPAddress(String addrOrHost, Byte[] subnet, AddressFamily family) in /_/src/Orleans.Core/Configuration/ConfigUtilities.cs:line 351
2023-01-20T18:21:24.087727244Z    --- End of inner exception stack trace ---
2023-01-20T18:21:24.087732445Z    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
2023-01-20T18:21:24.087758859Z    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
2023-01-20T18:21:24.087803031Z    at Orleans.Hosting.EndpointOptionsExtensions.ConfigureEndpoints(ISiloBuilder builder, String hostname, Int32 siloPort, Int32 gatewayPort, AddressFamily addressFamily, Boolean listenOnAnyHostAddress) in /_/src/Orleans.Runtime/Hosting/EndpointOptionsExtensions.cs:line 132
2023-01-20T18:21:24.087808822Z    at Orleans.Hosting.EndpointOptionsExtensions.ConfigureEndpoints(ISiloBuilder builder, Int32 siloPort, Int32 gatewayPort, AddressFamily addressFamily, Boolean listenOnAnyHostAddress) in /_/src/Orleans.Runtime/Hosting/EndpointOptionsExtensions.cs:line 152
2023-01-20T18:21:24.087815990Z    at WebScheduler.Server.Program.ConfigureSiloBuilder(HostBuilderContext context, ISiloBuilder siloBuilder) in /home/runner/work/web-scheduler/web-scheduler/Source/WebScheduler.Server/Program.cs:line 84
2023-01-20T18:21:24.087823095Z    at Microsoft.Extensions.Hosting.GenericHostExtensions.<>c__DisplayClass0_0.<UseOrleans>b__0(HostBuilderContext context, IServiceCollection services) in /_/src/Orleans.Runtime/Hosting/GenericHostExtensions.cs:line 39
2023-01-20T18:21:24.087841718Z    at Orleans.Hosting.SiloBuilder.InvokeConfigureServicesDelegates(HostBuilderContext ctx, IServiceCollection services) in /_/src/Orleans.Runtime/Hosting/SiloBuilder.cs:line 32
2023-01-20T18:21:24.087850346Z    at Orleans.Hosting.SiloBuilder.<.ctor>b__4_0(HostBuilderContext ctx, IServiceCollection services) in /_/src/Orleans.Runtime/Hosting/SiloBuilder.cs:line 22
2023-01-20T18:21:24.087857676Z    at Microsoft.Extensions.Hosting.HostBuilder.InitializeServiceProvider()
2023-01-20T18:21:24.088156194Z    at Microsoft.Extensions.Hosting.HostBuilder.Build()
2023-01-20T18:21:24.088166643Z    at WebScheduler.Server.Program.Main(String[] args) in /home/runner/work/web-scheduler/web-scheduler/Source/WebScheduler.Server/Program.cs:line 34

Reproduction Steps

This should be reproducible via a call to [NetworkInterface.GetAllNetworkInterfaces()](https://github.com/dotnet/orleans/blob/3.x/src/Orleans.Core/Configuration/ConfigUtilities.cs#L342-L356) from a .NET 7.0.2 app running in a gVisor-sandboxed container.

Expected behavior

If the file or path isn't found, false should be returned, which was previous behavior.

Actual behavior

A System.Net.NetworkInformation.NetworkInformationException is thrown with an inner exception of System.IO.DirectoryNotFoundException: Could not find a part of the path '/proc/sys/net/ipv4/conf/lo/forwarding'.

Regression?

I believe this to be a regression.

See #75883 and #63696.

The same code I have works with .NET 6, but not 7.

Known Workarounds

No response

Configuration

No response

Other information

No response

Author:	ElanHasson
Assignees:	-
Labels:	area-System.Net
Milestone:	-

[am11]

Best guess with suspect is in the runtime: https://github.com/dotnet/runtime/pull/76383/files, timing checks out.
The comments directly above indicate we shouldn't fail but return false

This was exactly the issue fixed in #76383. The fix was not backported to release/7 branch #75883 (comment).

@rzikm, @wfurt, worth a backport for orleans consumers?

[wfurt]

I think there are other systems when this may be needed. Since it is regression I would support backport of #76383 - especially since it is pretty trivial.

[rzikm]

I initiated backport #81157

[ElanHasson]

Thanks folks! Much appreciated.

[ElanHasson]

This should probably be closed as it's been released.

[wfurt]

fixed in 7.0.4

[karelz]

Fixed in main (8.0) in PR #76383 and in 7.0.4 in PR #81157.