debug failing RosAssert tests on Debian Stretch

[dirk-thomas]

No description provided.

[dirk-thomas]

The same hash which passed before on melodic/stretch fails the RosAssert tests in recent builds as well as this PR CI.

[dirk-thomas]

Testing in a local Debian Stretch docker container passed the tests 😒

[mikepurvis]

This is just mindless brainstorming, but some observations:

• They're all death tests, which fork and then from the child assert a particular termination condition for the parent.
• In all cases, there's an expected textual output which isn't being seen.

Given how operating system-y this stuff is, I have to ask what kernel you're running locally vs. what's on the buildfarm workers?

Also, if this is a race issue (possibly exacerbated by difference compiler optimization defaults?) might hypervisor load have something to do with it? I don't know how to shake out this hypothesis, but it's bizarre how consistent it is on the CI machine.

[mikepurvis]

Not as consistent as I thought though. Of the recent non-PR builds which have been kicked off as a consequence of me merging stuff, visible at http://build.ros.org/job/Mdev_ds__ros_comm__debian_stretch_amd64/:

• 13, 14, 15, and 16 have the RosAssert failures.
• 17 has no test failures, but includes the GCC warning committed by mistake in Implement Bag encryption/decryption. #1206.
• 18 has the spurious RandomPlay failure.

[dirk-thomas]

I have to ask what kernel you're running locally vs. what's on the buildfarm workers?

I am running 16.04 locally (currently 4.4.0-119). The buildfarm is using a standard Ubuntu / Debian docker image on top of a Ubuntu 16.04 host (so also a 4.4.0 kernel).

[dirk-thomas]

Closing since not being investigated.