Test standalone/io/test_runner_test can fail because process_test times out instead of crashing

[sgjesse]

See http://build.chromium.org/p/client.dart/builders/vm-win32-debug/builds/373/steps/checked_tests/logs/stdio

[madsager]

Set owner to @whesse.
Added Accepted label.

[a-siva]

Failed again at:

http://chromegw.corp.google.com/i/client.dash/builders/vm-win32-release/builds/2232/steps/checked_tests/logs/stdio

[a-siva]

Can we skip this test until the flakiness is fixed or at least mark it as
PASS, FAIL.

[whesse]

The root cause is that the test of a crashing process, process_test.exe 0 0 1 1, is sometimes timing out, rather than crashing. This sometimes also leads to an error on a code path with a boolean operator applied to null, where the null comes from a map lookup on a nonexistent key. Fixing by adding the key to the map, and increasing the timeout on the crash test.

The test should be fixed by adding 'verbose': false to the dummy configurations in TestRunnerTest, but it is not clear to me why this doesn't always fail in checked mode, and the most recent error is unclear as well.

The error is in a line:
    if (testCase.output.unexpectedOutput && testCase.configuration['verbose']) {

when the map 'configuration' has no key 'verbose'.
The error we should get, and that we do get in run vm-win32-release/builds/2184 is
stderr:
Unhandled exception:
type 'Null' is not a subtype of type 'bool' of 'boolean expression'.
 0. Function: 'RunningProcess.testComplete' url: 'file:///e:/b/build/slave/vm-win32-release/build/dart/tools/testing/dart/test_runner.dart' line:551 col:67
 1. Function: 'RunningProcess.stepExitHandler' url: 'file:///e:/b/build/slave/vm-win32-release/build/dart/tools/testing/dart/test_runner.dart' line:584 col:19
 
The error I am unclear on is

NullPointerException : method: 'get:_verbose@1bfad15'
Receiver: null
Arguments: []
 0. Function: 'Object.noSuchMethod' url: 'bootstrap' line:717 col:3
 1. Function: 'RunningProcess.testComplete' url: 'file:///e:/b/build/slave/vm-win32-release/build/dart/tools/testing/dart/test_runner.dart' line:551 col:66

from vm-win32-release/builds/2232.
What is this _verbose@1bfad15? And why would get be called on it? Is is some sort of run-time optimization?

[efortuna]

get:_verbose@1bfad15 was a result of attempting to fix the null/boolean bug: http://code.google.com/p/dart/issues/detail?id=2625 You can ignore that error, it should not be occurring in the future.

However, this seems only to be a symptom of the failure. The failure has still happened in traces like here: http://chromegw.corp.google.com/i/client.dart/builders/vm-win32-release/builds/2356/steps/tests/logs/stdio

[whesse]

The failure is still happening.

http://chromegw.corp.google.com/i/client.dart/builders/vm-linux-debug/builds/1945/steps/tests/logs/stdio

[madsager]

Added this to the M1 milestone.

[whesse]

This failure: process_test.exe 0 0 1 1 timing out, is still happening. Most recent failure is on http://build.chromium.org/p/client.dart/builders/vm-linux-debug/builds/2968

Looking at replacing this with a different way of crashing, because this needs to be fixed.

[whesse]

Removed Area-IO label.
Added Area-Test label.

[whesse]

I cannot easily find another way of making a crashing process - calling exit() with a negative argument does not work. We will stop the test failures by allowing the test process_test 0 0 1 1 to time out, as well as crash, until we can figure this out.

---

Removed this from the M1 milestone.
Added this to the Later milestone.
Removed Priority-Medium label.
Added Priority-Low label.
Changed the title to: "Test standalone/io/test_runner_test can fail because process_test times out instead of crashing".

[whesse]

This is still occuring - the suppression was removed from standalone.status as part of the V2 library upgrade, and the flakiness showed up on the buildbot again.

Adding PASS, FAIL back to standalone.status.

[whesse]

Removed the owner.
Added Triaged label.

[sgjesse]

The suppression is no longer in standalone.status.

It is only there for simarm and simmips - tracked as issue #12413.

---

Added AssumedStale label.