QUIC failures on linux

[AndyAyersMS]

I have seen various QUIC failures on arm64 linux in different PRs and in different pipelines. Symptoms vary, but here is a crash.

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=187390
Build error leg or test failing: System.Net.Quic.Functional.Tests.WorkItemExecution
Pull request: #82752

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "Segmentation fault      (core dumped) \"$RUNTIME_PATH/dotnet\" exec --runtimeconfig System.Net.Quic.Functional.Tests",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Test	Pull Request
225373	dotnet/runtime	System.Net.Quic.Functional.Tests.WorkItemExecution	#84207
220949	dotnet/runtime	System.Net.Quic.Functional.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	2

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have seen various QUIC failures on arm64 linux in different PRs and in different pipelines. Symptoms vary, but here is a crash.

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=187390
Build error leg or test failing: System.Net.Quic.Functional.Tests.WorkItemExecution
Pull request: #82752

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "Segmentation fault      (core dumped) \"$RUNTIME_PATH/dotnet\" exec --runtimeconfig System.Net.Quic.Functional.Tests",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Author:	AndyAyersMS
Assignees:	-
Labels:	blocking-clean-ci, untriaged, area-System.Net.Quic, Known Build Error
Milestone:	-

[wfurt]

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-82752-merge-e1a10947e940484c88/System.Net.Quic.Functional.Tests/1/console.d04fe702.log?helixlogtype=result

claims to be amd64. And it seems like there is no crash dump so it will be hard to see what is happening.

[AndyAyersMS]

Ooops yes that one is amd64. There is a crash dump though.

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-82752-merge-e1a10947e940484c88/System.Net.Quic.Functional.Tests/1/core.1000.14745?helixlogtype=result

[AndyAyersMS]

This is another one, probably different issue as it's a time out: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-heads-main-919e7e57ce2b4e4ab5/System.Net.Quic.Functional.Tests/1/console.03606f93.log?helixlogtype=result

[wfurt]

(lldb) bt
* thread #1, name = 'dotnet', stop reason = signal SIGSEGV
  * frame #0: 0x00007eff8a02c85b libmsquic.so.2`CxPlatRefDecrement at platform_posix.c:339:5
    frame #1: 0x00007eff8a02c83c libmsquic.so.2`CxPlatRefDecrement(RefCount=0x0000001100000124) at platform_posix.c:325
    frame #2: 0x00007ebe44073ee0

It does seems like some MsQuic or Quic issue.

[wfurt]

and msquic version

(lldb) dumpobj 00007ebefe0c9368
Name:        System.String
MethodTable: 00007eff13f5aed8
EEClass:     00007eff13fe0760
Size:        184(0xb8) bytes
File:        /datadisks/disk1/work/A2890920/p/shared/Microsoft.NETCore.App/8.0.0/System.Private.CoreLib.dll
String:      libmsquic.so version=2.1.7.329711 commit=2db78a2cd72d0c95111d5f4884fbc17307a4238d

so this is not the updated msquic @ManickaP

[AndyAyersMS]

Still seeing something like this -- any ETA on a fix?

https://dev.azure.com/dnceng-public/public/_build/results?buildId=211005&view=ms.vss-test-web.build-test-results-tab&runId=3906200&resultId=197690&paneView=debug

[wfurt]

I don't. I was trying to reproduce it running tests in loop but it did not happen. It is interesting that it seems to happen on coreclr runs. Is there something special about them?

[AndyAyersMS]

There is some customization -- I believe we use a checked runtime and perhaps run in a different machine pool? I don't know for sure.

At least we aren't setting any crazy DOTNET env vars.

[ManickaP]

So I tried to look for some recent crashes to get my hands on any dump, but all are already gone.
Query:

WorkItems
| where FriendlyName == "System.Net.Quic.Functional.Tests" and Status == "BadExit" and ConsoleUri has "main" and ExitCode == 139
| join kind=leftouter Jobs on JobId
| order by Finished desc 

Also, some of the linked PRs have different failures, e.g.: System.Net.Quic.QuicException : Stream aborted by peer (654321)..
I'll look into that problem, but I'm closing this for now. If you encounter the Seg Fault again, feel free to re-open.