System.Net.Security.Tests crashed on Alpine.arm32 and OSX.1200 #72227
Labels
Milestone
Comments
ghost
added
the
|
Tagging subscribers to this area: @dotnet/ncl, @vcsjones Issue DetailsNoticed it in one of the runs in PR: #72082 /root/helix/work/correlation/dotnet exec --runtimeconfig System.Net.Security.Tests.runtimeconfig.json --depsfile System.Net.Security.Tests.deps.json xunit.console.dll System.Net.Security.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
Discovering: System.Net.Security.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Net.Security.Tests (found 296 of 485 test cases)
Starting: System.Net.Security.Tests (parallel test collections = on, max threads = 4)
System.Net.Security.Tests.SslStreamEKUTest.SslStream_SelfSignedClientEKUClientAuth_Ok [SKIP]
Condition(s) not met: "IsRootCertificateInstalled"
System.Net.Security.Tests.SslStreamEKUTest.SslStream_ServerEKUClientAuth_Fails [SKIP]
Condition(s) not met: "IsRootCertificateInstalled"
System.Net.Security.Tests.SslStreamEKUTest.SslStream_ClientEKUServerAuth_Fails [SKIP]
Condition(s) not met: "IsRootCertificateInstalled"
System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUServerAuth_Ok [SKIP]
Condition(s) not met: "IsRootCertificateInstalled"
System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUClientAuth_Ok [SKIP]
Condition(s) not met: "IsRootCertificateInstalled"
System.Net.Security.Tests.SslStreamCertificateTrustTest.SslStream_SendCertificateTrust_ThrowsOnUnsupportedPlatform [SKIP]
Condition(s) not met: "DoesNotSupportSendingCustomCANamesInTls"
./RunTests.sh: line 168: 21 Segmentation fault (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Security.Tests.runtimeconfig.json --depsfile System.Net.Security.Tests.deps.json xunit.console.dll System.Net.Security.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Thu Jul 14 01:05:06 UTC 2022 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.There is a how-to-debug.md in DevOps attachment: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-72082-merge-93051a65a35d46238a/System.Net.Security.Tests/1/how-to-debug-dump.md?helixlogtype=result I am having some issues with my RPi device, and couldn't get the callstack from dump myself. Also, FYI @janvorli
|
|
.NET has some problems running under QEMU (see e.g. #13648), this needs to be debugged on an ARM machine. Coincidentally, I have one at home, I will try to look at the dump when I have time. |
|
Does not seem to be specific to arm32 (removing label). Also, the crash happened also in release/6.0 branch, so perhaps it is not a regression? |
karelz
changed the title
|
Marking it 7.0 as it now tracks also OSX -- though it is not 100% clear it is the same problem. |
|
I was not able to reproduce this on either Mac or ARM64 linux. It also has not reappeared on main since 08/07 and the runs I found did not have a dump attached. Recent crashes on PRs seem all to be related to the changes in that PRs respectively |
|
Agreed with @rzikm above -- it seems there was only Given the lower impact now, moving it to 8.0. Let's observe and then we can decide if it is gone or not yet. |
|
The crash on 8/15 | PR #67049 | RedHat.7.Amd64.Open is due to an infinite recursion, and at some point the process crashed on OutOfMemoryException Likely #72381 is related (Edit. it isn't, the crash was caused by the changes in PR) Stacktrace |
|
It is possible this has same root cause as #73621 and #69125 and fixed by #73972. Now, that would not be applicable to OSX. We should look if that looks similar or if that is completely unrelated issue. |
|
Should we split the issue then? |
|
I'm not sure. I was looking at similar issue recently and specially on PRs the failures are often unrelated. Simply putting all "BadExist" together is not sufficient IMHO. It would be great if query provides links to consoles and dumps. I also have access to arm 32/64 CI machine at the moment. I can give it also try and perhaps close if there is no repro and wait for resurfacing in main or recent reasonable PRs. on the QEMU, I had mixed results. running .NET build fails randomly with some strange error in msbuild but running .NET tests in container seems fine. dotnet/msquic#100 |
At least for arm32 it seems that running .NET under QEMU is unreliable. I get NREs both from MSBuild and test runner all the time in random places. I also seemed to get a MSBuild crash similar to #71098 that was fixed approximately two weeks ago. |
|
@filipnavara it is the same for arm64 qemu. I have been investigating that since yesterday when running arm64 stuff in docker on x64 (reported in #27190). I hope to get some understanding of the real culprit. |
|
Similar for me. We had ARM64 pipeline for a while on msquic and if was failing occasionally on msbuild. Running tests seems to be ok. |
|
Looks like this issue disappeared after recent fixes/improvements on runtime/jit on ARM platforms, the only crashes on ARM32/64 in last month are from jitstress and are likely unrelated to this issue. Closing for now, we can reopen if the issue resurfaces. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Occurrences ?-8/24 (incl. PRs):
8/22-8/23PR #7436339x failures in System.Net.Security.Tests, System.Net.Security.Unit.Tests and System.Net.Security.Enterprise.Tests- most likely specific to the PR8/15PR #67049RedHat.7.Amd64.Open- unrelated problem - see #72227 (comment) for detailsconsole: https://helix.dot.net/api/2019-06-17/jobs/93051a65-a35d-4623-8a64-a27a6b8e6467/workitems/System.Net.Security.Tests/console
Noticed it in one of the runs in PR: #72082
There is a how-to-debug.md in DevOps attachment: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-72082-merge-93051a65a35d46238a/System.Net.Security.Tests/1/how-to-debug-dump.md?helixlogtype=result
I am having some issues with my RPi device, and couldn't get the callstack from dump myself. Also,
docker run arm32v7/ubuntu(which uses QEMU) failed to rundotnet7 tool install --global dotnet-soscommand (with and withoutDOTNET_EnableWriteXorExecute=0).FYI @janvorli
The text was updated successfully, but these errors were encountered: