Add Junit output support

[b4nst]

I've added some code to be able to output a JUnit XML file. Thought you could find this interesting. I've added the --junit PATH_TO_FILE option to the command line args. If the option is present, an XML file is create to PATH_TO_FILE. I've used the cElementTree module, included in the python lib since 2.5.

This is a little bit raw yet, but I can make it more robust if you consider adding this feature to pyresttest.

Cheers !
Bastien.

[svanoort-jenkins]

Can one of the admins verify this patch?

[netjunki]

@bastienar I have a similar implementation to this in my fork that doesn't require a file name to be provided (generated based on yaml test provided). Might be nice to have the file name be optional.

pyresttest '' $TESTFILE --junit

(the empty url is a hack related to the fact that my test suites use a mix of http/https and multiple domains.)

[b4nst]

@netjunki Good idea, I've improved my code a little bit. Now there is an optional --junit-path arg available. If you do not precise --junit-path when --junit option is set, the output file will be written next to the test file, following the name test-file-name-w/o-ext-test-results.xml

pyresttest $URL /dir/sample-test.yaml --junit

Will write junit output to /dir/sample-test-test-results.xml

pyresttest $URL /dir/sample-test.yaml --junit --junit-path /other-dir/result/
or
pyresttest $URL /dir/sample-test.yaml --junit --junit-path /other-dir/result

Will write junit output to /other-dir/result/sample-test-test-results.xml only if /other-dir/result/ exists

pyresttest $URL /dir/sample-test.yaml --junit --junit-path ../other-dir/result/junit.xml

Will write junit output to ../other-dir/result/junit.xml only if ../other-dir/result exists

Relative paths are resolved from the test file directory

[svanoort]

@bastienar Thank you for your contribution, this is definitely a long-desired and highly useful feature; unfortunately it is colliding/conflicting with the work in progress intended to support this & build towards parallel test execution and other parts of the PyrestTest 2.0 roadmap.

Please, can you rebase this to be a PR on top of #171 and use the callback mechanism added there?

With that PR JUnit logging should be as simple as adding the bundle of callbacks that log to JUnit output (unit testing that they behave as expected). This represents the "quick and dirty" solution from #43 - though we should make sure it covers the JUnit <--> PyRestTest mappings described there. The alternative was full unittest integration (more work though, I think).

Passing in a callbacks bundle should make this change simpler & shorter (no duplication of test logic, just output), easier to test (we just verify the logging bits), and easier to maintain.

As @netjunki mentioned, they also have something similar along these lines, which might make sense to combine with this where appropriate.

Thank you!
Sam

PS: I know that PR includes some unfinished bits from #170 around the refactoring of test/benchmark execution, but by using the callbacks, the logging is decoupled and you shouldn't have to worry about that. My hope is to finish & merge that to master by end of this month (due to recent work commitments I have had less side project time available nights/weekends than expected).

[b4nst]

@svanoort ok, i will work on that.

[b4nst]

I ran into a problem on importing module macros when trying to execute the script with python3 :

ImportError: No module named 'pyresttest.macros'

Any ideas of what could cause such an error ?

[svanoort]

@bastienar Are you doing import by from .macros import * ?

[svanoort]

@bastienar Also, I'd prefer to close this PR and create one that is applied to the top of the branch for the other one - that way in reviewing the PR, we only see the deltas versus that branch (simplifies merging), and it won't get merge conflicts when that branch updates. Also, we can merge into that branch, and then I can resolve existing merge conflicts with master so you don't have to :)

[svanoort]

Closing this PR so we can open one that targets branch refactor-execution - this greatly simplifies review and merging.

[b4nst]

Yep, I agree with that. And for the import, I'm doing import with from pyresttest.macros import *. If I do import with from .macros import * :

SystemError: Parent module '' not loaded, cannot perform relative import

and with from macros import *, same error as previous, but in module macros.

No problem with python2.7.

[svanoort]

@bastienar Where does your file live? If it's under the pyresttest folder with the other classes, the relative import works in 2.6, 2.7 and 3.x (tested, it's the best solution I've found).

The issue may be how your file is being loaded/run. Resttest.py does some dirty hacks to allow running it as a script but still load imports correctly.

If you want to run a file directly from the top directly (or hit unit tests), you need to give it a module reference : https://github.com/svanoort/pyresttest/blob/refactor-execution/run_tests.sh