Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Apple Silicon intermittent priority1 failures #46365

Closed
sdmaclea opened this issue Dec 23, 2020 · 5 comments · Fixed by #50889
Closed

Fix Apple Silicon intermittent priority1 failures #46365

sdmaclea opened this issue Dec 23, 2020 · 5 comments · Fixed by #50889
Labels
arch-arm64 area-VM-coreclr disabled-test The test is disabled in source code against the issue os-macos-bigsur (macOS11) tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly
Milestone
6.0.0

Comments

@sdmaclea
Copy link
Contributor

Resolve intermittent failures

100 runs of the priority1 tests showed the following intermittent failures. Run used a checked build of 3a276d5

   1 GC.Regressions.Github.runtime_32848.runtime_32848.runtime_32848.sh                                                                                       
   1 JIT.Generics.Coverage.chaos56200037cs_o.chaos56200037cs_o.sh                                                                                             
   5 JIT.Generics.Coverage.chaos65204782cs.chaos65204782cs.sh                                                                                                 
  13 JIT.Generics.Coverage.chaos65204782cs_o.chaos65204782cs_o.sh                                                                                             
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest10.Generated10.Generated10.sh   
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1002.Generated1002.Generated1002.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1005.Generated1005.Generated1005.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1006.Generated1006.Generated1006.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest101.Generated101.Generated101.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1022.Generated1022.Generated1022.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1023.Generated1023.Generated1023.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1029.Generated1029.Generated1029.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1040.Generated1040.Generated1040.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1052.Generated1052.Generated1052.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1057.Generated1057.Generated1057.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1075.Generated1075.Generated1075.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1091.Generated1091.Generated1091.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1098.Generated1098.Generated1098.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1114.Generated1114.Generated1114.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1132.Generated1132.Generated1132.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1133.Generated1133.Generated1133.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1135.Generated1135.Generated1135.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1143.Generated1143.Generated1143.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1152.Generated1152.Generated1152.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1165.Generated1165.Generated1165.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1186.Generated1186.Generated1186.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest119.Generated119.Generated119.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1197.Generated1197.Generated1197.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1198.Generated1198.Generated1198.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1203.Generated1203.Generated1203.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1220.Generated1220.Generated1220.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1221.Generated1221.Generated1221.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1222.Generated1222.Generated1222.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1228.Generated1228.Generated1228.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1230.Generated1230.Generated1230.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1231.Generated1231.Generated1231.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1239.Generated1239.Generated1239.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1260.Generated1260.Generated1260.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1293.Generated1293.Generated1293.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1295.Generated1295.Generated1295.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1301.Generated1301.Generated1301.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1313.Generated1313.Generated1313.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1325.Generated1325.Generated1325.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1333.Generated1333.Generated1333.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1336.Generated1336.Generated1336.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1352.Generated1352.Generated1352.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1362.Generated1362.Generated1362.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1378.Generated1378.Generated1378.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1382.Generated1382.Generated1382.sh
   3 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1383.Generated1383.Generated1383.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1390.Generated1390.Generated1390.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1394.Generated1394.Generated1394.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1411.Generated1411.Generated1411.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1429.Generated1429.Generated1429.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1449.Generated1449.Generated1449.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1455.Generated1455.Generated1455.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest15.Generated15.Generated15.sh   
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest151.Generated151.Generated151.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest156.Generated156.Generated156.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest174.Generated174.Generated174.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest189.Generated189.Generated189.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest216.Generated216.Generated216.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest234.Generated234.Generated234.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest27.Generated27.Generated27.sh   
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest282.Generated282.Generated282.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest370.Generated370.Generated370.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest378.Generated378.Generated378.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest38.Generated38.Generated38.sh   
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest380.Generated380.Generated380.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest381.Generated381.Generated381.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest393.Generated393.Generated393.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest394.Generated394.Generated394.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest396.Generated396.Generated396.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest400.Generated400.Generated400.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest419.Generated419.Generated419.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest425.Generated425.Generated425.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest427.Generated427.Generated427.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest436.Generated436.Generated436.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest441.Generated441.Generated441.sh
   3 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest442.Generated442.Generated442.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest462.Generated462.Generated462.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest464.Generated464.Generated464.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest47.Generated47.Generated47.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest487.Generated487.Generated487.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest496.Generated496.Generated496.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest50.Generated50.Generated50.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest500.Generated500.Generated500.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest503.Generated503.Generated503.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest504.Generated504.Generated504.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest517.Generated517.Generated517.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest519.Generated519.Generated519.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest523.Generated523.Generated523.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest525.Generated525.Generated525.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest538.Generated538.Generated538.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest54.Generated54.Generated54.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest55.Generated55.Generated55.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest568.Generated568.Generated568.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest582.Generated582.Generated582.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest591.Generated591.Generated591.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest63.Generated63.Generated63.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest647.Generated647.Generated647.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest649.Generated649.Generated649.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest65.Generated65.Generated65.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest654.Generated654.Generated654.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest667.Generated667.Generated667.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest669.Generated669.Generated669.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest676.Generated676.Generated676.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest716.Generated716.Generated716.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest730.Generated730.Generated730.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest770.Generated770.Generated770.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest774.Generated774.Generated774.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest803.Generated803.Generated803.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest806.Generated806.Generated806.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest81.Generated81.Generated81.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest820.Generated820.Generated820.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest83.Generated83.Generated83.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest831.Generated831.Generated831.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest84.Generated84.Generated84.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest848.Generated848.Generated848.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest891.Generated891.Generated891.sh
   4 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest902.Generated902.Generated902.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest913.Generated913.Generated913.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest914.Generated914.Generated914.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest917.Generated917.Generated917.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest925.Generated925.Generated925.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest929.Generated929.Generated929.sh
   3 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest931.Generated931.Generated931.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest938.Generated938.Generated938.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest939.Generated939.Generated939.sh
   2 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest952.Generated952.Generated952.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest956.Generated956.Generated956.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest977.Generated977.Generated977.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest980.Generated980.Generated980.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest99.Generated99.Generated99.sh
   4 profiler.transitions.transitions.transitions.sh

4 JITStress=1 or JITStress=2 priority1 test runs had similar intermittent failures

   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1099.Generated1099.Generated1099.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1111.Generated1111.Generated1111.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest118.Generated118.Generated118.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1198.Generated1198.Generated1198.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1223.Generated1223.Generated1223.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest1386.Generated1386.Generated1386.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest517.Generated517.Generated517.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest64.Generated64.Generated64.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest752.Generated752.Generated752.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest956.Generated956.Generated956.sh

The 11 GCStress=3 priority1 test runs had a slightly different set of intermittent failures

   1 JIT.Generics.Coverage.chaos65204782cs.chaos65204782cs.sh
   9 JIT.Methodical.Arrays.huge._il_dbghuge_objref._il_dbghuge_objref.sh
   3 JIT.Methodical.Arrays.huge._il_relhuge_objref._il_relhuge_objref.sh
   2 JIT.Methodical.Arrays.huge._il_relhuge_struct._il_relhuge_struct.sh
   2 JIT.Methodical.VT.port._il_relhuge_gcref._il_relhuge_gcref.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest153.Generated153.Generated153.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest17.Generated17.Generated17.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest181.Generated181.Generated181.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest189.Generated189.Generated189.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest459.Generated459.Generated459.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest523.Generated523.Generated523.sh
   1 Loader.classloader.TypeGeneratorTests.TypeGeneratorTest68.Generated68.Generated68.sh

/cc @janvorli @mangod9 @sandreenko
@sdmaclea sdmaclea added this to the 6.0.0 milestone Dec 23, 2020
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Dec 23, 2020
@mangod9 mangod9 removed the untriaged New issue has not been triaged by the area owner label Dec 23, 2020
@mangod9
Copy link
Member

mangod9 commented Dec 23, 2020

Assume there a handful of issues (and/or asserts) causing multiple test failures?

@sdmaclea
Copy link
Contributor Author

Symptoms look like GC holes to me. The TypeGeneratorTests typically are failing with a message something like object set to a null reference. I still have stdout/stderr for every failure.

@sdmaclea
Copy link
Contributor Author

I ran GCStress=3 and will fix the hard failure reported there.

I am working on enabling GCStress=C on Apple Silicon. I am hoping that may turn (at least some of) these intermittent failures into hard failures. That would make them easier to debug.

@sdmaclea sdmaclea mentioned this issue Dec 26, 2020
Closed
31 tasks
@sdmaclea sdmaclea mentioned this issue Feb 15, 2021
Closed
@BruceForstall BruceForstall mentioned this issue Mar 3, 2021
Merged
@sdmaclea sdmaclea mentioned this issue Mar 22, 2021
Closed
@sdmaclea sdmaclea added the disabled-test The test is disabled in source code against the issue label Apr 8, 2021
@sdmaclea
Copy link
Contributor Author

sdmaclea commented Apr 8, 2021 •
edited
Loading

This issue seems resolved on my local machine. I recently upgraded to macOS 11.3 Beta 6. I ran all the failing tests 100 times. There were no failures.

Opened #50889 to reenable the tests in CI. First run had one failure Loader.classloader.TypeGeneratorTests.TypeGeneratorTest130.Generated130.Generated130.sh.

Ran the test locally (with my build) 10K times w/o failure.

I guess I'll pull the helix payload and try that too to be more sure.

Update:
I tried the helix payload on my M1 machine. I ran the Generated130 test 10K times. All passed.

I retried the #50889 outerloop and the Generated130 failed in the second run too.

I am pretty confident the fix is update to macOS 11.3 Beta 6 or better.

@sdmaclea sdmaclea added the tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly label Apr 8, 2021
@sdmaclea
Copy link
Contributor Author

sdmaclea commented Apr 8, 2021

Waiting on https://github.com/dotnet/core-eng/issues/11937#issuecomment-816110773 or an issue to be opened.

@sdmaclea sdmaclea mentioned this issue May 3, 2021
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label May 3, 2021
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jun 9, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Jul 9, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-arm64 area-VM-coreclr disabled-test The test is disabled in source code against the issue os-macos-bigsur (macOS11) tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly
Projects
None yet
Milestone
6.0.0
Development

Successfully merging a pull request may close this issue.

Reenable resolve Apple Silicon tests sdmaclea/runtime
3 participants
@sdmaclea @Dotnet-GitSync-Bot @mangod9