Jobs with flaky tests show as green

[mhdawson]

See discussion in nodejs/node#8065 (comment)

Looks like jobs that have tests marked as flaky show as green instead of the expected yellow. At the same time the post of results shows that tests failed.

[joaocgreis]

There are two issues here:

1. Jobs that have tests marked as flaky show as green instead of the expected yellow.
2. Posting results back to PRs shows failure when the job is a success.

[joaocgreis]

About 1:
Jobs need a post step to mark the job as unstable. Only the windows job had that, so I copied it to all others. Seems to have worked in all but osx and arm fanned: https://ci.nodejs.org/job/node-test-commit/4563/ . However, this seems to be a problem with test.py, not Jenkins: note that tests are not marked as flaky in the console output (cc @jbergstroem @Trott )

About 2:
Could be a bug in the code to do this in the AIX job. Didn't check carefully, but seems to be different from others (e.g. test-linux). EDIT: Checked carefully and made the job similar to others, might be fixed, but there are no AIX servers to test now.

[Trott]

Possibly/likely related: Most jobs respect tests that are marked as flaky, but the Raspberry Pi arm-fanned job seemingly does not. This is quite a problem for the recent test-crypto-timing-safe-equal.

[not-an-aardvark]

I think the issue is that the Raspberry Pi build is the only one that's failing; it might not be the only job that's not respecting flaky tests. (edit: never mind, see below)

[Trott]

@not-an-aardvark That's what I mean. (Or maybe I'm misunderstanding you?) On the other jobs, when that test fails, it turns them yellow rather than red. That's the result of that test being marked flaky. But on the Raspberry Pi, that test failing turns the job red.

[not-an-aardvark]

Never mind, I hadn't realized some of the other tests turned yellow. I had thought the Raspberry Pi tests were the only ones that failed.

[mhdawson]

@joaocgreis In terms of the jobs being green instead of yellow, that is now fixed so this issue may be able to be closed.

[joaocgreis]

This issue still exists for osx and arm-fanned. I suspect everything is OK on the Jenkins side and the failure is in test.py, but was not yet able to investigate further.

[joaocgreis]

The FLAKY_TESTS_MODE variable was not passed on those two jobs. Fixed now. Credit to @rvagg , thanks!