Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WasmTestOnBrowser-System.Private.Xml.Tests - GENERAL_FAILURE #76134

Closed
karelz opened this issue Sep 24, 2022 · 7 comments
Closed

WasmTestOnBrowser-System.Private.Xml.Tests - GENERAL_FAILURE #76134

karelz opened this issue Sep 24, 2022 · 7 comments
Assignees
@vargaz
Labels
arch-wasm WebAssembly architecture area-VM-meta-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Milestone
7.0.x

Comments

@karelz
Copy link
Member

karelz commented Sep 24, 2022 •
edited by build-analysis bot
Loading

Runfo failures in last 30 days

  • Platform: net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-(Windows.Amd64.Server2022.Open)windows.amd64.server2022.open
Day Run Details
9/23 Rolling run 28100 Console - InvalidOperationException
9/21 Rolling run 24630 Console - InvalidOperationException
9/20 2x Rolling run 24096 Console & Console - InvalidOperationException
9/20 Rolling run 23279 Console - InvalidOperationException
9/19 Rolling run 22721 Console - InvalidOperationException
9/18 Rolling run 21535 Console - ChannelClosedException
9/17 Rolling run 21283 Console - ChannelClosedException
9/16 Rolling run 20550 Console - ChannelClosedException
9/16 Rolling run 19690 Console - ChannelClosedException
9/15 Rolling run 18995 Console - ChannelClosedException

Failure examples:

InvalidOperationException - Console on 9/23:

[23:37:33] info: Starting:    System.Private.Xml.Tests.dll
[23:38:40] fail: 224964280
[23:38:40] fail: 224964280
[23:38:40] fail: 224964280
[23:38:40] fail: 225333728
[23:38:40] info: WASM EXIT 1
[23:38:40] info: Waiting to flush log messages with a timeout of 120 secs ..
[23:38:40] info: WASM EXIT 1
[23:38:40] fail: Application has finished with exit code TESTS_FAILED but 0 was expected
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HMKTNI0N1333", Request id "0HMKTNI0N1333:00000001": An unhandled exception was thrown by the application.
      System.InvalidOperationException: Message processor is not running. Make sure to call RunAsync first
         at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestMessagesProcessor.InvokeAsync(String message, Boolean isError) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/WasmTestMessagesProcessor.cs:line 94
         at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmBrowserTestRunner.RunConsoleMessagesPump(WebSocket socket, CancellationToken token) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/Browser/WasmBrowserTestRunner.cs:line 196
         at Microsoft.DotNet.XHarness.CLI.Commands.WebServer.TestWebServerStartup.<>c__DisplayClass3_0.<<Configure>b__2>d.MoveNext() in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WebServer.cs:line 163
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
XHarness exit code: 71 (GENERAL_FAILURE)

C:\helix\work\workitem\e\wasm_build\AppBundle>set EXIT_CODE=71 
----- end Fri 09/23/2022 23:38:44.97 ----- exit code 71 ----------------------------------------------------------

ChannelClosedException - Console on

[23:13:04] info: Starting:    System.Private.Xml.Tests.dll
[23:14:16] fail: 245597000
[23:14:16] fail: 245597000
[23:14:16] fail: 245597000
[23:14:16] fail: 245597120
[23:14:16] info: WASM EXIT 1
[23:14:16] info: Waiting to flush log messages with a timeout of 120 secs ..
[23:14:16] info: WASM EXIT 1
[23:14:16] fail: Flushing log messages failed with: System.InvalidOperationException: An attempt was made to transition a task to a final state when it had already completed.
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestMessagesProcessor.ProcessMessage(String message, Boolean isError) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/WasmTestMessagesProcessor.cs:line 200
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestMessagesProcessor.RunAsync(CancellationToken token) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/WasmTestMessagesProcessor.cs:line 82
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestMessagesProcessor.RunAsync(CancellationToken token) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/WasmTestMessagesProcessor.cs:line 82
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestMessagesProcessor.CompleteAndFlushAsync(Nullable`1 timeout) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/WasmTestMessagesProcessor.cs:line 106. Ignoring.
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HMKPPD0JB2H6", Request id "0HMKPPD0JB2H6:00000001": An unhandled exception was thrown by the application.
      System.Threading.Channels.ChannelClosedException: The channel has been closed.
         at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmBrowserTestRunner.RunConsoleMessagesPump(WebSocket socket, CancellationToken token) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/Browser/WasmBrowserTestRunner.cs:line 196
         at Microsoft.DotNet.XHarness.CLI.Commands.WebServer.TestWebServerStartup.<>c__DisplayClass3_0.<<Configure>b__2>d.MoveNext() in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WebServer.cs:line 163
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
[23:14:16] fail: [out of order message from the browser]: http://127.0.0.1:49195/dotnet.js 4:1872 "[main] websocket closed: [object CloseEvent]" CloseEvent
[23:14:16] info: Could not process message: {"method":"console.log","payload":"WASM EXIT 1","arguments":["WASM EXIT 1"]}
[23:14:16] info: Application has finished with exit code TESTS_FAILED. But the log processor failed with GENERAL_FAILURE.
[23:14:16] fail: Application has finished with exit code GENERAL_FAILURE but 0 was expected
XHarness exit code: 71 (GENERAL_FAILURE)

C:\helix\work\workitem\e\wasm_build\AppBundle>set EXIT_CODE=71 
----- end Sun 09/18/2022 23:14:20.41 ----- exit code 71 ----------------------------------------------------------

Report

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0
@karelz karelz added the arch-wasm WebAssembly architecture label Sep 24, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Sep 24, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost
Copy link

ghost commented Sep 24, 2022

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Runfo failures in last 30 days

  • Platform: net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-(Windows.Amd64.Server2022.Open)windows.amd64.server2022.open
Day Run Details
9/23 Rolling run 28100 Console - InvalidOperationException
9/21 Rolling run 24630 Console - InvalidOperationException
9/20 2x Rolling run 24096 Console & Console - InvalidOperationException
9/20 Rolling run 23279 Console - InvalidOperationException
9/19 Rolling run 22721 Console - InvalidOperationException
9/18 Rolling run 21535 Console - ChannelClosedException
9/17 Rolling run 21283 Console - ChannelClosedException
9/16 Rolling run 20550 Console - ChannelClosedException
9/16 Rolling run 19690 Console - ChannelClosedException
9/15 Rolling run 18995 Console - ChannelClosedException

Failure examples:

InvalidOperationException - Console on 9/23:

[23:37:33] info: Starting:    System.Private.Xml.Tests.dll
[23:38:40] fail: 224964280
[23:38:40] fail: 224964280
[23:38:40] fail: 224964280
[23:38:40] fail: 225333728
[23:38:40] info: WASM EXIT 1
[23:38:40] info: Waiting to flush log messages with a timeout of 120 secs ..
[23:38:40] info: WASM EXIT 1
[23:38:40] fail: Application has finished with exit code TESTS_FAILED but 0 was expected
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HMKTNI0N1333", Request id "0HMKTNI0N1333:00000001": An unhandled exception was thrown by the application.
      System.InvalidOperationException: Message processor is not running. Make sure to call RunAsync first
         at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestMessagesProcessor.InvokeAsync(String message, Boolean isError) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/WasmTestMessagesProcessor.cs:line 94
         at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmBrowserTestRunner.RunConsoleMessagesPump(WebSocket socket, CancellationToken token) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/Browser/WasmBrowserTestRunner.cs:line 196
         at Microsoft.DotNet.XHarness.CLI.Commands.WebServer.TestWebServerStartup.<>c__DisplayClass3_0.<<Configure>b__2>d.MoveNext() in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WebServer.cs:line 163
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
XHarness exit code: 71 (GENERAL_FAILURE)

C:\helix\work\workitem\e\wasm_build\AppBundle>set EXIT_CODE=71 
----- end Fri 09/23/2022 23:38:44.97 ----- exit code 71 ----------------------------------------------------------

ChannelClosedException - Console on

[23:13:04] info: Starting:    System.Private.Xml.Tests.dll
[23:14:16] fail: 245597000
[23:14:16] fail: 245597000
[23:14:16] fail: 245597000
[23:14:16] fail: 245597120
[23:14:16] info: WASM EXIT 1
[23:14:16] info: Waiting to flush log messages with a timeout of 120 secs ..
[23:14:16] info: WASM EXIT 1
[23:14:16] fail: Flushing log messages failed with: System.InvalidOperationException: An attempt was made to transition a task to a final state when it had already completed.
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestMessagesProcessor.ProcessMessage(String message, Boolean isError) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/WasmTestMessagesProcessor.cs:line 200
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestMessagesProcessor.RunAsync(CancellationToken token) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/WasmTestMessagesProcessor.cs:line 82
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestMessagesProcessor.RunAsync(CancellationToken token) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/WasmTestMessagesProcessor.cs:line 82
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestMessagesProcessor.CompleteAndFlushAsync(Nullable`1 timeout) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/WasmTestMessagesProcessor.cs:line 106. Ignoring.
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HMKPPD0JB2H6", Request id "0HMKPPD0JB2H6:00000001": An unhandled exception was thrown by the application.
      System.Threading.Channels.ChannelClosedException: The channel has been closed.
         at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmBrowserTestRunner.RunConsoleMessagesPump(WebSocket socket, CancellationToken token) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/Browser/WasmBrowserTestRunner.cs:line 196
         at Microsoft.DotNet.XHarness.CLI.Commands.WebServer.TestWebServerStartup.<>c__DisplayClass3_0.<<Configure>b__2>d.MoveNext() in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WebServer.cs:line 163
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
[23:14:16] fail: [out of order message from the browser]: http://127.0.0.1:49195/dotnet.js 4:1872 "[main] websocket closed: [object CloseEvent]" CloseEvent
[23:14:16] info: Could not process message: {"method":"console.log","payload":"WASM EXIT 1","arguments":["WASM EXIT 1"]}
[23:14:16] info: Application has finished with exit code TESTS_FAILED. But the log processor failed with GENERAL_FAILURE.
[23:14:16] fail: Application has finished with exit code GENERAL_FAILURE but 0 was expected
XHarness exit code: 71 (GENERAL_FAILURE)

C:\helix\work\workitem\e\wasm_build\AppBundle>set EXIT_CODE=71 
----- end Sun 09/18/2022 23:14:20.41 ----- exit code 71 ----------------------------------------------------------
Author: karelz
Assignees: -
Labels:

arch-wasm
Milestone: -

@karelz karelz added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Sep 24, 2022
@radical
Copy link
Member

radical commented Sep 26, 2022

The xharness exception can be ignored. The real issue is #74302 .

@runfoapp runfoapp bot mentioned this issue Sep 26, 2022
Closed
@lewing
Copy link
Member

lewing commented Sep 26, 2022 •
edited
Loading

looks like a manifestation of #74302

@lewing lewing removed the untriaged New issue has not been triaged by the area owner label Sep 26, 2022
@lewing lewing modified the milestones: 7.0.0, 7.0.x Sep 26, 2022
@carlossanlop
Copy link
Member

carlossanlop commented Oct 5, 2022 •
edited
Loading

It's affecting various System.Xml tests. They all show this error message:

[21:36:12] warn: Blocking on the main thread is very dangerous, see https://emscripten.org/docs/porting/pthreads.html#blocking-on-the-main-browser-thread
[21:51:05] fail: Tests timed out. Killing driver service pid 4661
[21:51:06] fail: Application has finished with exit code TIMED_OUT but 0 was expected
XHarness exit code: 71 (GENERAL_FAILURE)

@radical
Copy link
Member

radical commented Oct 5, 2022

Ignore all the failures in these two:

  • Build Browser wasm Linux Release LibraryTests_Threading
  • Build Browser wasm Linux Release LibraryTests_Threading_PerfTracing

The job is orange because we want to ignore the test failures. It will be red for build failures though, and those we are interested in.

@carlossanlop carlossanlop mentioned this issue Oct 5, 2022
Merged
@vargaz vargaz mentioned this issue Oct 7, 2022
Merged
vargaz added a commit to vargaz/runtime that referenced this issue Oct 11, 2022
@vargaz
[mono][interp] Fix an issue with deopt and interpreter tiering.
95654d0 
If a method is tiered while being run from interp_run_clause_with_il_state (),
the clause_args argument to interp_exec_method () still contains the old IL
offsets confusing the EH code, i.e. this line:
```
if (clause_args && frame == clause_args->exec_frame && context->handler_ip >= clause_args->end_at_ip)
```

Clear out clause_args at the beginning to avoid this.

Hopefully fixes
dotnet#76134
dotnet#74302
@lewing lewing closed this as completed in a071887 Oct 14, 2022
github-actions bot pushed a commit that referenced this issue Oct 14, 2022
@vargaz
[mono][interp] Fix an issue with deopt and interpreter tiering.
25da681 
If a method is tiered while being run from interp_run_clause_with_il_state (),
the clause_args argument to interp_exec_method () still contains the old IL
offsets confusing the EH code, i.e. this line:
```
if (clause_args && frame == clause_args->exec_frame && context->handler_ip >= clause_args->end_at_ip)
```

Clear out clause_args at the beginning to avoid this.

Hopefully fixes
#76134
#74302
@lewing
Copy link
Member

lewing commented Oct 14, 2022

we think this is fixed in mail now so please make noise if you see it in a test

carlossanlop pushed a commit that referenced this issue Nov 10, 2022
@github-actions @vargaz
[release/7.0] [mono][interp] Fix an issue with deopt and interpreter …
d933222 
…tiering. (#77059)

* [mono][interp] Fix an issue with deopt and interpreter tiering.

If a method is tiered while being run from interp_run_clause_with_il_state (),
the clause_args argument to interp_exec_method () still contains the old IL
offsets confusing the EH code, i.e. this line:
```
if (clause_args && frame == clause_args->exec_frame && context->handler_ip >= clause_args->end_at_ip)
```

Clear out clause_args at the beginning to avoid this.

Hopefully fixes
#76134
#74302

* [mono][interp] Avoid tiering up methods while running clauses.

The IL offsets in the clause_args argument become out-of-date after tiering up.

Co-authored-by: Zoltan Varga <vargaz@gmail.com>
@ghost ghost locked as resolved and limited conversation to collaborators Nov 14, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@vargaz vargaz

Labels
arch-wasm WebAssembly architecture area-VM-meta-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Projects
None yet
Milestone
7.0.x
Development

No branches or pull requests
6 participants
@radical @lewing @vargaz @carlossanlop @karelz @jeffschwMSFT