Flaky test: TupleWithElementNamedWithDefaultName

[Youssef1313]

Saw this test failing a couple of times.

https://dev.azure.com/dnceng/public/_build/results?buildId=1090590&view=logs&jobId=8225d539-f709-56d9-e8fb-4b6141b087c9&j=8225d539-f709-56d9-e8fb-4b6141b087c9&t=60710b2d-694c-5f9a-9f24-31a79ea198f9

[xUnit.net 00:00:16.12]     Microsoft.CodeAnalysis.CSharp.UnitTests.CodeGen.CodeGenTupleTests.TupleWithElementNamedWithDefaultName [FAIL]
  Failed Microsoft.CodeAnalysis.CSharp.UnitTests.CodeGen.CodeGenTupleTests.TupleWithElementNamedWithDefaultName [3 ms]
  Error Message:
   Assert.Equal() Failure
Expected: 0
Actual:   -1
  Stack Trace:
     at Microsoft.CodeAnalysis.CSharp.UnitTests.CodeGen.CodeGenTupleTests.TupleWithElementNamedWithDefaultName() in /_/src/Compilers/CSharp/Test/Emit/CodeGen/CodeGenTupleTest.cs:line 28143

Runfo Tracking Issue: indexofunderlyingfieldsintuple9 flaky

Build	Definition	Kind	Run Name
1566269	roslyn-CI	Rolling	Test Windows CoreClr Debug Single Machine

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	1

Build	Definition	Kind	Run Name
1366490	roslyn-CI	PR 56453	Test_Windows_CoreClr_Release_Windows.10.Amd64.Open
1366007	roslyn-CI	PR 54969	Test_Windows_CoreClr_Release_Windows.10.Amd64.Open
1363479	roslyn-CI	PR 56397	Test_Windows_CoreClr_Release_Windows.10.Amd64.Open
1361977	roslyn-CI	PR 56383	Test_Windows_CoreClr_Release_Windows.10.Amd64.Open
1356530	roslyn-CI	PR 56333	Test_Windows_Desktop_Release_32_Windows.10.Amd64.Open
1353263	roslyn-CI	PR 56186	Test_Windows_CoreClr_Release_Windows.10.Amd64.Open
1349899	roslyn-CI	Rolling	Test_Windows_CoreClr_Release_Windows.10.Amd64.Open
1347540	roslyn-CI	Rolling	Test_Windows_CoreClr_Release_Windows.10.Amd64.Open
1346713	roslyn-CI	PR 56221	Test_Windows_Desktop_Spanish_Release_32_Windows.10.Amd64.Server19H1.ES.Open
1346620	roslyn-CI	Rolling	Test_Windows_Desktop_Release_64_Windows.10.Amd64.Open
1341617	roslyn-CI	PR 56176	Test_Windows_CoreClr_Release_Windows.10.Amd64.Open
1338355	roslyn-CI	PR 56124	Test_macOS_Debug_OSX.1014.Amd64.Open
1338355	roslyn-CI	PR 56124	Test_Windows_CoreClr_Debug_Windows.10.Amd64.Open
1337670	roslyn-CI	Rolling	Test_Windows_CoreClr_Release_Windows.10.Amd64.Open
1335951	roslyn-CI	Rolling	Test_Windows_CoreClr_Debug_Windows.10.Amd64.Open

[Youssef1313]

Also IndexOfUnderlyingFieldsInTuple9 has failed with the same issue.

See https://github.com/dotnet/roslyn/runs/2355802903.

@jaredpar I'm guessing this could be a product issue, not a test-only issue.

[jaredpar]

@jcouv could this be related to the wrapped tuple element field change? That is the only recent change in this area I can think of. The test is decently flaky now, about 1.5 failures per day. If a fix isn't imenninent we should consider disabling until we can dig into it.

[jaredpar]

@jcouv this is the only current active flaky test in main. Would it help if we flipped the assert here to AssertOrFailFast so we got a dump when it failed?

[jaredpar]

@jcouv this is firing with some regularity in CI. Is the failure alone enough to investigate this or do we need a heap dump when it happens? If it's the latter we could send a PR to use AssertOrFailFast in the appropriate place.

[jcouv]

I've instrumented the test with AssertOrFailFast last week. I mean to bring this up in our 1-1 today as I was not able to find resulting crash dumps.

[jaredpar]

Where is the PR where you added that? I think it may not be getting hit on this particular failure. Want to see if the PR timeline lines up with the more recent test failures though. If it does line up want to dig into why dumps aren't getting published for it.

[Youssef1313]

Where is the PR where you added that? I think it may not be getting hit on this particular failure. Want to see if the PR timeline lines up with the more recent test failures though. If it does line up want to dig into why dumps aren't getting published for it.

#56029

[jaredpar]

@jcouv and i dug through this today and it's the very last Assert.Equal call in that test that is failing now

                Assert.Equal(1, item2Underlying.TupleElementIndex);

[jcouv]

Status update: The last occurrence was Sep 16th (two weeks ago). I'm waiting for a new occurrence to grab a crash dump from AssertOrFailFast.

[jcouv]

No occurrence in the last two weeks (compared to a prior average of one per day) leads me to think this was fixed as a side-effect of a recent tuple fix.
I'll go ahead and close. We can re-open if this re-occurs.

[jmarolf]

ran into this again in: https://dev.azure.com/dnceng/public/_build/results?buildId=1462048&view=ms.vss-test-web.build-test-results-tab&runId=42007772&resultId=161964&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

[jasonmalinowski]

@jcouv I think I hit this in:

https://dev.azure.com/dnceng/public/_build/results?buildId=1578889&view=logs&jobId=85444f1b-759e-5cca-5123-eed49cf93063&j=85444f1b-759e-5cca-5123-eed49cf93063&t=17125aaf-307c-5a57-0474-fb68c5b5fee2

What I don't get is I see a dump file being created and it being copied to the places we're supposed to copy dump files, but it looks like it didn't make it anywhere after that?