Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assertion failed: (*card_word)==0 in DynamicGenerics tests #76801

Closed
jkotas opened this issue Oct 10, 2022 · 12 comments
Closed

Assertion failed: (*card_word)==0 in DynamicGenerics tests #76801

jkotas opened this issue Oct 10, 2022 · 12 comments
Assignees
Labels
area-GC-coreclr blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Milestone

Comments

@jkotas
Copy link
Member

jkotas commented Oct 10, 2022

Failed test: nativeaot_SmokeTests::_DynamicGenerics_DynamicGenerics_DynamicGenerics_cmd (nativeaot\\SmokeTests\\DynamicGenerics\\DynamicGenerics\\DynamicGenerics.cmd)
Assertion failed: (*card_word)==0, file D:\a\_work\1\s\src\coreclr\gc\gc.cpp, line 13092

Failed in #76799 (windows.x64.Checked). Full log: https://dev.azure.com/dnceng-public/public/_build/results?buildId=45701&view=logs&jobId=58dc7ccb-0414-5dd3-62a5-bf2e63258b7c&j=ea7d6d51-d18d-5306-47bd-e3b7db04467e&t=3c9d9518-9046-50a7-9208-2e9137da71b6

{
  "ErrorMessage": "(*card_word)==0",
  "BuildRetry": false
}

Report

Build Definition Step Name Console log Pull Request
58837 dotnet/runtime Run tests in single file mode Log #76985
59002 dotnet/runtime Run tests in single file mode Log #62863
58981 dotnet/runtime Run tests in single file mode Log #77137
58748 dotnet/runtime Run tests in single file mode Log #77289
58346 dotnet/runtime Run tests in single file mode Log #77272
58445 dotnet/runtime Run tests in single file mode Log #77280
58398 dotnet/runtime Run tests in single file mode Log #76901
58362 dotnet/runtime Run tests in single file mode Log #77277
58366 dotnet/runtime Run tests in single file mode Log #77277
58323 dotnet/runtime Run tests in single file mode Log #77078
58278 dotnet/runtime Run tests in single file mode Log #77267
57679 dotnet/runtime Run tests in single file mode Log #77237
58192 dotnet/runtime Run tests in single file mode Log #73472
58181 dotnet/runtime Run tests in single file mode Log #77213
57257 dotnet/runtime Run tests in single file mode Log #77213
58029 dotnet/runtime Run tests in single file mode Log #77245
57955 dotnet/runtime Run tests in single file mode Log #77239
57923 dotnet/runtime Run tests in single file mode Log #77243
57916 dotnet/runtime Run tests in single file mode Log #77189
57869 dotnet/runtime Run tests in single file mode Log #62689
57894 dotnet/runtime Run tests in single file mode Log #76981
57856 dotnet/runtime Run tests in single file mode Log #76985
57711 dotnet/runtime Run tests in single file mode Log #77228
57651 dotnet/runtime Run tests in single file mode Log #74963
57635 dotnet/runtime Run tests in single file mode Log #74157
57589 dotnet/runtime Run tests in single file mode Log #74820
57537 dotnet/runtime Run tests in single file mode Log #77231
57502 dotnet/runtime Run tests in single file mode Log #77011
57470 dotnet/runtime Run tests in single file mode Log #76981
57458 dotnet/runtime Run tests in single file mode Log #75140
56859 dotnet/runtime Run tests in single file mode Log #77196
57079 dotnet/runtime Run tests in single file mode Log #77176
57047 dotnet/runtime Run tests in single file mode Log #77214
56766 dotnet/runtime Run tests in single file mode Log #77197
56623 dotnet/runtime Run tests in single file mode Log #77189
56287 dotnet/runtime Run tests in single file mode Log #77052
56168 dotnet/runtime Run tests in single file mode Log #77102
54722 dotnet/runtime Run tests in single file mode Log #76924
55967 dotnet/runtime Run tests in single file mode Log #70941
55952 dotnet/runtime Run tests in single file mode Log #77166
55933 dotnet/runtime Run tests in single file mode Log #77147
55914 dotnet/runtime Run tests in single file mode Log #77163
55901 dotnet/runtime Run tests in single file mode Log #74820
55803 dotnet/runtime Run tests in single file mode Log #77151
55653 dotnet/runtime Run tests in single file mode Log #77107
55265 dotnet/runtime Run tests in single file mode Log #77106
55251 dotnet/runtime Run tests in single file mode Log #77132
55243 dotnet/runtime Run tests in single file mode Log #76981
55062 dotnet/runtime Run tests in single file mode Log #75542
55028 dotnet/runtime Run tests in single file mode Log #76942
54886 dotnet/runtime Run tests in single file mode Log #73472
54750 dotnet/runtime Run tests in single file mode Log #77043
54700 dotnet/runtime Run tests in single file mode Log #77112
54551 dotnet/runtime Run tests in single file mode Log #77113
54522 dotnet/runtime Run tests in single file mode Log #77107
54457 dotnet/runtime Run tests in single file mode Log #77103
54175 dotnet/runtime Run tests in single file mode Log #77065
54303 dotnet/runtime Run tests in single file mode Log #77103
54153 dotnet/runtime Run tests in single file mode Log #77102
53820 dotnet/runtime Run tests in single file mode Log #77065
53751 dotnet/runtime Run tests in single file mode Log #77080
53655 dotnet/runtime Run tests in single file mode Log #76779
53449 dotnet/runtime Run tests in single file mode Log #77056
53287 dotnet/runtime Run tests in single file mode Log #77067
53170 dotnet/runtime Run tests in single file mode Log #77062
52849 dotnet/runtime Run tests in single file mode Log #76851
52807 dotnet/runtime Run tests in single file mode Log #77052
52590 dotnet/runtime Run tests in single file mode Log #77042
52728 dotnet/runtime Run tests in single file mode Log #70941
52655 dotnet/runtime Run tests in single file mode Log #75140
52585 dotnet/runtime Run tests in single file mode Log #77003
52413 dotnet/runtime Run tests in single file mode Log #76904
52315 dotnet/runtime Run tests in single file mode Log #77035
52242 dotnet/runtime Run tests in single file mode Log #76997
52128 dotnet/runtime Run tests in single file mode Log #77027
52118 dotnet/runtime Run tests in single file mode Log #76985
51963 dotnet/runtime Run tests in single file mode Log #76981
50357 dotnet/runtime Run tests in single file mode Log #76971
50156 dotnet/runtime Run tests in single file mode Log #76950
51782 dotnet/runtime Run tests in single file mode Log #68994
51604 dotnet/runtime Run tests in single file mode Log #76890
51454 dotnet/runtime Run tests in single file mode Log #77003
51463 dotnet/runtime Run tests in single file mode Log #76981
51434 dotnet/runtime Run tests in single file mode Log #77011
51394 dotnet/runtime Run tests in single file mode Log #73472
51277 dotnet/runtime Run tests in single file mode Log #76811
51266 dotnet/runtime Run tests in single file mode Log #76848
50960 dotnet/runtime Run tests in single file mode Log #76985
50629 dotnet/runtime Run tests in single file mode Log #76972
50557 dotnet/runtime Run tests in single file mode Log #74963
50518 dotnet/runtime Run tests in single file mode Log #76975
50008 dotnet/runtime Run tests in single file mode Log #76941
49938 dotnet/runtime Run tests in single file mode Log #74157
49860 dotnet/runtime Run tests in single file mode Log #76883
49784 dotnet/runtime Run tests in single file mode Log #76933
49747 dotnet/runtime Run tests in single file mode Log #76848
49477 dotnet/runtime Run tests in single file mode Log #76901
49169 dotnet/runtime Run tests in single file mode Log #62863
48098 dotnet/runtime Run tests in single file mode Log #76339
48968 dotnet/runtime Run tests in single file mode Log #76705
Displaying 100 of 125 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
3 63 125
@jkotas jkotas added area-GC-coreclr Known Build Error Use this to report build issues in the .NET Helix tab labels Oct 10, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Oct 10, 2022
@ghost
Copy link

ghost commented Oct 10, 2022

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details
Failed test: nativeaot_SmokeTests::_DynamicGenerics_DynamicGenerics_DynamicGenerics_cmd (nativeaot\\SmokeTests\\DynamicGenerics\\DynamicGenerics\\DynamicGenerics.cmd)
Assertion failed: (*card_word)==0, file D:\a\_work\1\s\src\coreclr\gc\gc.cpp, line 13092

Failed in #76799. Full log: https://dev.azure.com/dnceng-public/public/_build/results?buildId=45701&view=logs&jobId=58dc7ccb-0414-5dd3-62a5-bf2e63258b7c&j=ea7d6d51-d18d-5306-47bd-e3b7db04467e&t=3c9d9518-9046-50a7-9208-2e9137da71b6

{
  "ErrorMessage": "(*card_word)==0",
  "BuildRetry": false
}
Author: jkotas
Assignees: -
Labels:

area-GC-coreclr, Known Build Error

Milestone: -

@jkotas
Copy link
Member Author

jkotas commented Oct 10, 2022

@dotnet/gc I suspect that this may be related to regions and frozen objects. Could you please review verify_card_bundles logic for any obvious problems?

@EgorBo
Copy link
Member

EgorBo commented Oct 10, 2022

I assume it started to appear after #76737 ?

@jkotas jkotas added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Oct 10, 2022
@mangod9
Copy link
Member

mangod9 commented Oct 11, 2022

Assigning to Maoni assuming its related to enabling regions on NativeAOT?

@mangod9 mangod9 removed the untriaged New issue has not been triaged by the area owner label Oct 11, 2022
@mangod9 mangod9 added this to the 8.0.0 milestone Oct 11, 2022
@Maoni0
Copy link
Member

Maoni0 commented Oct 11, 2022

Andrew is looking at this.

@trylek
Copy link
Member

trylek commented Oct 13, 2022

I also think there's something wrong with reporting - I saw this in my recent PR testing and while detailed scrutiny of the log ultimately revealed the assertion failure, the summary at the end of the log says

cmdLine:D:\a\_work\1\s\artifacts\tests\coreclr\windows.x64.Checked\nativeaot\SmokeTests\DynamicGenerics\DynamicGenerics\DynamicGenerics.cmd Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 10/12/2022 8:44:52 PM, end: 10/12/2022 9:14:52 PM)
Test Harness Exitcode is : -100

A possible explanation is that in my testing the assertion failure happened very close to the 30 minute timeout for the test.

@cshung
Copy link
Member

cshung commented Oct 13, 2022

I also think there's something wrong with reporting - I saw this in my recent PR testing and while detailed scrutiny of the log ultimately revealed the assertion failure, the summary at the end of the log says

cmdLine:D:\a_work\1\s\artifacts\tests\coreclr\windows.x64.Checked\nativeaot\SmokeTests\DynamicGenerics\DynamicGenerics\DynamicGenerics.cmd Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 10/12/2022 8:44:52 PM, end: 10/12/2022 9:14:52 PM)
Test Harness Exitcode is : -100
A possible explanation is that in my testing the assertion failure happened very close to the 30-minute timeout for the test.

I noticed that when the assertion happened on my machine, it stuck at a dialog box, so that might explain the timeout.

@trylek
Copy link
Member

trylek commented Oct 13, 2022

Thanks Andrew, I guess that explains a lot, in CoreCLR testing we're going a long way towards suppressing all such dialogs by disabling WER UI and by other means, it might be the case that the relatively recent NativeAOT testing is still missing some of these measures.

@cshung
Copy link
Member

cshung commented Oct 15, 2022

Still stuck with this one, but I found something interesting.

It appears that when the assertion happens, the line being printed is always this:
Running Test: B282745.testMDArrayWith3Dimensions

@jkotas
Copy link
Member Author

jkotas commented Oct 15, 2022

NativeAOT on Windows is building with hardware write watch currently. It does not build with FEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP and FEATURE_MANUALLY_MANAGED_CARD_BUNDLES defined. Can this config difference lead to this assertion failure?

@Maoni0
Copy link
Member

Maoni0 commented Oct 18, 2022

I was surprised to see that these 2 defines are not defined for AoT. it does expose a bug with regions (when we make used larger for regions we are not using the OS writewatch to watch the extended portion of the card table so we are not getting updated from that part), however, I care more about making AoT have these 2 defined on windows. @cshung will open an issue for AoT folks. we'd like to get rid of the code that's still using the OS writewatch completely.

@cshung
Copy link
Member

cshung commented Oct 22, 2022

Before #77224, I am able to reproduce the bug locally in a few minutes. After the fix, I am unable to repro it locally for 4 hours. Closing this issue opportunistically to see if that repro again. Feel free to reopen if the bug repros.

@cshung cshung closed this as completed Oct 22, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Nov 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-GC-coreclr blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Projects
None yet
Development

No branches or pull requests

6 participants