[JENKINS-64516] Add support for Robot Framework 4.0

[asimell]

Robot Framework 4.0 adds a new test status "SKIP" and removes test criticality. RF 4.0 also removes <tags></tags> and <arguments></arguments> tags from output.xml, so parser needs to be able to handle new output.xml formats.

This PR

• Adds support for SKIP status to job list view, test result overview, Blue Ocean, test result details view
• Adds support for reading both RF 3.x and 4.x styled output.xml files
• Improves some Java syntax related little things
• Updates documentation with fresh screenshots and information on how to configure the plugin for RF 4.0 (use onlyCritical: false)
• Improves readability for the list column view by adding a 1 pixel white stroke around the text
• Adds unit tests to test parsing RF 4.x output.xml files.

[Tattoo]

Let's finish the unresolved questions

[Tattoo]

LGTM!

[emakaay]

Nice work, updating the plugin to support robotframework 4.0

I did some initial tests, and found a glitch.

I first tested the plugin(v3.0.0) using un-adjusted robot projects (using robotframework v3.2.2 and robotframework-pabot v1.10.1). That worked perfect (the generated report was processed correctly. Rebot was used creating the report.

Then I tested the plugin(v3.0.0) using un-adjusted robot projects(using robotframework v4.0.0 and robotframework-pabot v1.11). The generated report (created using rebot) was generated without a problem, But the plugin did not pickup the correct values for passed and failed tests

The project had 333 tests in total, 326 passed and 7 failed.

the plugin gave the following result:
total, failed, passed, skipped, pass%
critical tests 0 0 0 0 100%
all tests 337 7 330 0 97.9

in short:
robotframework 3.3.2 without pabot (using rebot) is working
robotframework 4.0.0 without pabot (using rebot) is working
robotframework 3.3.2 with pabot <1.11 (using rebot) is working
robotframework 4.0.0 with pabot =1.11 (using rebot) is not working

I don't know why and i can't share the report for debugging, I'm sorry to say

[Tattoo]

@emakaay ,

Tried with the same version combos and smaller test count, and it seems to work correctly. Could you try running your setup with smaller test set to see if the counts work?

Another thing that puzzles me, is why are you using rebot together with pabot? pabot AFAIK already creates you the combined output.xml, log.html & report.html, you shouldn't need to do it separately.

[emakaay]

@Tattoo ,

I will try to run a smaller set of tests, to see if that results to different outcome. That will take a couple of days though.

I know pabot already creates the mentioned files. The reason we use rebot, is because we have tests that sometimes fail. Sometimes caused by fragile tests, but most of the time environment issues cause a perfect test to fail. Those failing test will be rerun (with the robot option--rerunfailed) to get a perfect 'green' test result...

To combine the multiple files we use rebot!

[Tattoo]

@emakaay ,

Ok, that makes sense. Just to double-check, I would advise you to also take a look that each different version of output.xml is named differently and what version output.xml is actually configured to the plugin -- maybe it picks up the wrong output.xml after all? Diligently naming all output.xmls with --output or putting them in separate folders with --outputdir let's you be sure of this.

[emakaay]

@Tattoo
You're absolutely right about the output files. Each file should have a unique file name, to prevent problems.
This process is created in a way unique files are created, and only after the rebot phase, the output.xml file created.
This process has been is use for more then a year, and has not changed to test this new plugin.

I don't think that is causing any problems

[emakaay]

I recently said that the new plugin with robotframework v4.0 was working (no pabot).
But there IS a difference how the report is shown.

Before v3.0 the plugin would always present 2 lines (1 with critical tests, and another with All Tests)

With v3.0 the first line (critical test) is there, but always filled with 0,0,0, 100%

When using the plugin in combination with pabot this is also true

[Tattoo]

Yes, this is by design as explained in the plugin documentation, as there is no criticality anymore in RF 4.0