System.Threading.Tests.ThreadLocalTests.RunThreadLocalTest8_Values failing on Mono on SLES 12 AMD64

[runfoapp]

Runfo Tracking Issue: System.Threading.Tests.ThreadLocalTests.RunThreadLocalTest8_Values failing on Mono on SLES 12 AMD64

Build	Definition	Kind	Run Name

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

[lambdageek]

Apparently SUSE has some cgroup based limits on the number of threads that can be created. Which for some services is as low as 512. And for user accounts the limit is 12288 threads across all logins. https://www.suse.com/releasenotes/x86_64/SUSE-SLES/12-SP2/#fate-320358

Unclear why only MonoVM is affected. Perhaps we spawn more threads?

[ViktorHofer]

Just saw that this happened again in. @akoeplinger @steveisok can someone please take a look.

[steveisok]

@lambdageek Can you look further?

[stephentoub]

(And can we disable the test against mono if it can't be fixed quickly? :)

[ViktorHofer]

I think we should.

[akoeplinger]

Test was disabled with #44357, removing blocking-clean-ci.

[ericstj]

It looks like this is failing in release/5.0 where the test wasn't disabled. Should this be disabled there as well?

[lambdageek]

This might be fixed by #44124 - we now throw a ThreadStartException which the threadpool catches - so thread creation failure shouldn't crash the runtime.

[VincentBu]

Failed in run runtime 20210121.44

Failed test : System.Threading.Tests.ThreadLocalTests.RunThreadLocalTest8_Values

net6.0-Linux-Release-x64-Mono_release-SLES.12.Amd64.Open

Error message:

System.Threading.Tasks.TaskSchedulerException : An exception was thrown by a TaskScheduler.
---- System.Threading.ThreadStartException : Thread failed to start.
-------- System.ExecutionEngineException : 0x0


Stack trace
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1709
   at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].StartNew(Task parent, Func`1 function, CancellationToken cancellationToken, TaskCreationOptions creationOptions, InternalTaskOptions internalOptions, TaskScheduler scheduler) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 336
   at System.Threading.Tasks.TaskFactory.StartNew[Int32](Func`1 function, CancellationToken cancellationToken, TaskCreationOptions creationOptions, TaskScheduler scheduler) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskFactory.cs:line 628
   at System.Threading.Tests.ThreadLocalTests.RunThreadLocalTest8_Values() in /_/src/libraries/System.Threading/tests/ThreadLocalTests.cs:line 253
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) in /_/src/mono/netcore/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs:line 378
----- Inner Stack Trace -----
   at System.Threading.Thread.ThrowThreadStartException(Exception ex) in /_/src/mono/netcore/System.Private.CoreLib/src/System/Threading/Thread.Mono.cs:line 246
   at System.Threading.Thread.StartCore() in /_/src/mono/netcore/System.Private.CoreLib/src/System/Threading/Thread.Mono.cs:line 250
   at System.Threading.Thread.Start(Object parameter, Boolean captureContext) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Thread.cs:line 194
   at System.Threading.Thread.UnsafeStart(Object parameter) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Thread.cs:line 174
   at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask(Task task) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs:line 49
   at System.Threading.Tasks.TaskScheduler.InternalQueueTask(Task task) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskScheduler.cs:line 237
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1681
----- Inner Stack Trace -----

[akoeplinger]

Opened a PR to disable the test again: #47324

[lambdageek]

Ah interesting. This is a good stack trace: we're starting a long-running task by creating a new thread and it throws a ThreadStartException (good - Marek's PR is working).

@kouvel should the portable threadpool catch a ThreadStartException here:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs

Line 49 in c7fda3b

new Thread(s_longRunningThreadWork) { IsBackground = true }.UnsafeStart(task);

What happens on CoreCLR with the non-portable threadpool if you start 1000 long running tasks on SLES?

[kouvel]

should the portable threadpool catch a ThreadStartException here

This thread creation is not part of the thread pool. It probably shouldn't catch these exceptions, currently the thread pool catches the thread-start exception and OOM when creating worker threads but the consensus from discussion about that before was to let the exception propagate, even if it may crash the process, since that is more easily discoverable and recoverable than a hung state where a thread cannot be started for an extended amount of time.

What happens on CoreCLR with the non-portable threadpool if you start 1000 long running tasks on SLES?

Haven't tried on that system, but I believe thread creation failure would typically result in an OOM exception (that is when CreateThread fails in the PAL), and if thread creation succeeds but initial setup allocation fails in the started thread, it looks like Thread.Start would result in ThreadStartException. In both cases it looks like the exception would propagate back to the test and fail the test.