enable rviz's “visual tests” on Linux using a small wm and a framebuffer

[wjwwood]

I've been told that a patch like this should allow rviz's visual tests to be run on our Linux CI, even on the headless EC2 instances we use. It works by installing a minimal window manager (matchbox) and a frame buffer device to which to render.

I'll run a CI job to see if actually works, but in the meantime any review of the proposed change is welcome:

• Linux:
• Linux-aarch64:

[mikaelarguedas]

Looks like it works! Tests built and ran and passing 🎉 (no idea if it's testing the right thing but it looks promising)

Not sure I have enough insight to really review this but given that it works I'm in 👍

[wjwwood]

I'll wait for comments from some rviz contributors before deciding to merge it. @mikaelarguedas in your experience does this significantly change the job time?

[mikaelarguedas]

doesnt look like it.

+3 min on amd
+10min on arm
But as we don't know how many other tests have been repeated it's not really meaningful.

A better way to know would be to:

• run jobs with an without that patch that:
  • build only up to rviz*
  • test only rviz* without retries

[anhosi]

It is looking good. The "display tests" did run successfully. 🎆
The visual tests (graphical end to end tests using screenshots as asserts) did not run as they need to be started explicitely with -DEnableVisualTests=True.

We restarted a single CI run with visual tests enabled.

[anhosi]

The CI setup is also working for the visual tests! 👍

The test run https://ci.ros2.org/job/ci_linux/4937/ shows 2 errors. One (image_display_visual_test) is listed twice. We already have a fix ready for this one. The other one (odometry_display_visual_test) looks like a parsing problem of the logs as it is passing. In our experience such artifacts go away when there are no real failures.

The runtime of the visual tests is around 8 minutes. It is probably not necessary to have them run for every ros2 build as if other projects break some rviz functionality the other tests should already pick up on that. We would suggest to use the visual tests for rviz pull requests, though.

[wjwwood]

CI building all packages, testing only rviz packages, without this patch:

• Linux (36 min)
• Linux-aarch64 (50 min)
• macOS (1hr 22 min)
• Windows (41 min)

CI building all packages, testing only rviz packages, with this patch:

• Linux (41 min)
• Linux-aarch64 (57 min)
• macOS (1hr 11min)
• Windows (49 min)

I'll compare run times after they complete.

[wjwwood]

Looks like linux is affected by about 4-7 minutes with this patch (updated previous comment). However, the other jobs are also slower (macOS and Windows), which is weird. I looked the jobs I ran again, but the only difference is which ci branch is used (test settings look the same). So there could be a good deal of variability within the jobs themselves.

I'll merge this and since I'm buildfarmer I'll watch for regressions and issues. We'll still manually add the -DEnableVisualTests=True when desired.

[dirk-thomas]

These new test failures seem to be related to this: https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/489/testReport/

[wjwwood]

@anhosi (or someone else maybe) is looking into this failure. I'll give them another day to see if they can find the issue and then revert this if not.

[wjwwood]

I thought those were because I ran with the "rerun failures" option to check the timing (so I thought they were flakey).

[greimela-si]

@wjwwood @dirk-thomas Are these test failures still occurring?

[wjwwood]

@greimela-si yes, here's last nights: https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/496/testReport/

I'll try to follow up on the rviz pr's that might be related.

[wjwwood]

@greimela-si any ideas about this issue, the test failures are still there. I don't think any of the outstanding rviz pull requests will address it (though I will try to get to those soon).

[clalancette]

@greimela-si Any updates for these test failures? They are the cause of most of the test failures we see when running on aarch64 (last night: https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/624)

[greimela-si]

@anhosi @Martin-Idel-SI might have details.