TypedMessageTest on dartium-lucid64-inc is flaky

[peter-ahe-google]

The log contains "PASS

EOF" so the test should be passing, I assume.

It looks like there is more output after #EOF. Perhaps this is a race condition in testing of isolates. So my guess is that either the test is flaky, or the framework doesn't handle isolates correctly.

FAILED: dartium release_ia32 TypedMessageTest
Expected: pass
Actual: fail

stdout:
Content-Type: text/plain
[print]: Starting log server.
PASS

EOF

[print]: Log [0, 1, 2, 3, 4]
[print]: Stopping log server.

Command line: python /mnt/data/b/build/slave/dartium-lucid64-inc/build/src/dart/tools/testing/drt-trampoline.py /mnt/data/b/build/slave/dartium-lucid64-inc/build/src/out/Release/DumpRenderTree --no-timeout --dart-flags=--ignore-unrecognized-flags --enable_type_checks --enable_asserts /mnt/data/b/build/slave/dartium-lucid64-inc/build/src/out/Release_ia32/generated_tests/dartium/tests_language_src_TypedMessageTest-/test.html

[sigmundch]

Currently we check that a test passes if PASS is the first line right after 'Content-Type:text/plain'. In this example it seems that the log message ([print]: ...) happened while the page was being printed.

I think we should simply avoid printing anything in our tests.

[peter-ahe-google]

Sorry, Siggi, I couldn't disagree more :-)

It is important that our tests can test every thing a user throws at it, and it is important that we can debug test failures.

We must be allowed to use print, just as we must be allowed to use scripts without library names.

Any feature we disallow in tests should be considered to not be implemented correctly, because we aren't testing it.

[sigmundch]

Ok, let's make the test framework more robust to avoid these issues. We basically need to change unittest and test.dart so we can ignore any printed content (except for the "PASS" or "FAIL" indicator).

Since this is not a test for 'print', I'm thinking that we should anyways remove the prints on this particular test :-)

[a-siva]

The test should fail consistently on all architectures not just
dartium_lucid64-inc if the problem is just the extra 'print'
statements. Is that true. does it always fail consistently on all
runs?

[sigmundch]

The problem in this case is that the print was interleaved with the output of the page.

We ask DumpRenderTree to print the page in a text format so we can extract the status of the test.

Content-Type:text/plain
PASS
#EOF

As long as the prints occur before or after, we are good. In chrome, all the prints occur before the text output, for some reason in dartium the prints are flushed later and show up interleaved.

[peter-ahe-google]

Although this isn't a test of print, the print statements may be instrumental in debugging test failures.

Here is my take:

* the test framework must support any language feature or library feature we throw at it (except for deleting everything, etc)

* the test framework should not suppress output. You can spend a lot of time debugging why there is not output :-)

[efortuna]

+whesse, for test.dart change

---

cc @whesse.

[DartBot]

This comment was originally written by whe...@gmail.com

---

We could change the browser tests so they accept stuff between text/plain and PASS and #EOF, but then what if we get output between PA and SS?
I don't see why DumpRenderTree is outputting the page while Dart content is still running - shouldn't the outputting of the page be atomic? Maybe DumpRenderTree could put all of its page output together, and print it to stdout atomically.

I think making test.dart more tolerant in what it accepts might fix the current problems, but there is a real problem in the way Dartium DRT is outputting the page.

[sigmundch]

Added Isolates label.

[whesse]

Added this to the Later milestone.

[sigmundch]

Added Library-Isolates label.

[sigmundch]

Removed Isolates label.

[dgrove]

Added Library-Isolate label.

[dgrove]

Removed Library-Isolates label.

[ricowind]

We already ignore stuff interleaved, closing this

[ricowind]

Added Fixed label.