OOM on outerloop-linux runs of runtime-libraries-coreclr pipeline on System.Runtime.Tests and System.Runtime.Numerics.Tests

[akoeplinger]

The System.Runtime and System.Runtime.Numerics work items consistently fail on the linux outerloop pipeline:

/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Runtime.Numerics.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Numerics.Tests (found 7 of 562 test cases)
  Starting:    System.Runtime.Numerics.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 168:    22 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Numerics.Tests.runtimeconfig.json --depsfile System.Runtime.Numerics.Tests.deps.json xunit.console.dll System.Runtime.Numerics.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Fri 07 Jul 2023 11:41:50 AM UTC ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed eg by kill

Looking further down in the log we can see that they've been killed by the Out of Memory (OOM) killer:

[ 2661.445879] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/6a407bb075622c6a7f6a9a7b88ddd30217d8294911afde258ca18d342721bf3b,task=dotnet,pid=25681,uid=1000
[ 2661.445924] Out of memory: Killed process 25681 (dotnet) total-vm:273925828kB, anon-rss:7218652kB, file-rss:0kB, shmem-rss:14728kB, UID:1000 pgtables:14564kB oom_score_adj:0

This has been going on for as long as there are logs in AzDO (June 19th).

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The System.Runtime and System.Runtime.Numerics work items consistently fail on the linux outerloop pipeline:

/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Runtime.Numerics.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Numerics.Tests (found 7 of 562 test cases)
  Starting:    System.Runtime.Numerics.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 168:    22 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Numerics.Tests.runtimeconfig.json --depsfile System.Runtime.Numerics.Tests.deps.json xunit.console.dll System.Runtime.Numerics.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Fri 07 Jul 2023 11:41:50 AM UTC ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed eg by kill

Looking further down in the log we can see that they've been killed by the Out of Memory (OOM) killer:

[ 2661.445879] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/6a407bb075622c6a7f6a9a7b88ddd30217d8294911afde258ca18d342721bf3b,task=dotnet,pid=25681,uid=1000
[ 2661.445924] Out of memory: Killed process 25681 (dotnet) total-vm:273925828kB, anon-rss:7218652kB, file-rss:0kB, shmem-rss:14728kB, UID:1000 pgtables:14564kB oom_score_adj:0

Author:	akoeplinger
Assignees:	-
Labels:	os-linux, area-Infrastructure, runtime-coreclr, blocking-clean-ci-optional, blocking-outerloop
Milestone:	8.0.0

[ericstj]

Unlikely a product bug - issue with test that needs to be better conditioned for its environment.