Reporter: test_case_end no longer fired after test case restart

[u3shit]

Description

Since b802a58, the reporter's test_case_end is no longer called when restarting a test case due to subcases. Unfortunately the commit message is not too informative about why did this happen and how can we get back the old behavior.
Looking at the code I don't see any obvious way to detect this situation from the reporter, except in case of terminating a test case due to an exception. Is it possible to get back the old behavior? Or if it's not possible, adding a new event like 'test_case_restart`, so the reporter can handle it?

Extra information

• doctest version: v2.3.4 (updating from 2.0.0)
• Operating System: Gentoo Linux amd64
• Compiler+version: clang 8.0.1 + libc++

[onqtam]

You are perhaps the earliest user of the reporter interface! I added support for it in version 2.0 but I finalized it along with a full xml reporter implementation in 2.3 and there was a breaking change indeed. Before this users had to check a flag called should_reenter in order to determine if test_case_end has been called for the last time or that the test case will be re-entered because of unfinished subcases. I decided that was ugly and moved to the current behavior - test_case_start is called only once and test_case_end is called only once at the end after all re-entering.

Since you are asking for this then it is indeed important to at least some people so I'll change something. Adding the old behavior/interface is ugly to me so I'll indeed add a new method in the interface which notifies about a re-enter because of subcases.

Thanks for being so thorough (tracking down the versions/commits/etc.) and sorry for the uninformative commit messages - I'll strive for better in the future. The fix will go in the following few hours in the dev branch but I can't guarantee when it will go into master as an official 2.3.5 version.

[u3shit]

Thanks! The reporter I've written is available here: https://github.com/u3shit/libshit/blob/8d0c89f70012cc8cbfe2c531f17ab90d505f5d51/test/main.cpp#L36 (probably not a good example of how to write a reporter, but it was good enough for my needs)
It was my attempt to create something that jenkins' junit parser understands (it was the no. 1 issue that prevented me from switching from catch to doctest previously). Since junit doesn't support subcases, I went with saving each rerun as a different junit testcase, encoding the subcases in the testcase name. It's probably the best I can do without writing a jenkins plugin, which I'd like to avoid if possible :)

[onqtam]

I love your naming preferences in terms of username and projects :D

doctest would benefit from a junit reporter provided out-of-the-box and it is in the roadmap. Other people have expressed interest as well:
#75 (comment)

If you're up for it you could make a PR with an implementation - hopefully it will be similar to that of Catch in terms of output. It can go directly in doctest/parts/doctest.cpp right after the xml reporter - I assume they will be pretty similar. But it shouldn't depend on boost and it could probably reuse the xml writer class in doctest. Anyway - if you're up for it :)

[u3shit]

Thanks :D Unfortunately not everyone is so positive about my naming scheme, but I can live with that.

I could look into it, but in that case I should probably get rid of the ConsoleReporter dependency, which is more like a hack than a proper solution. It looks like I only used boost for xml escaping, which would be hopefully unnecessary with the xml writer class.
I had problems with the junit logger of Catch not supporting subcases properly, so the output format is intentionally different. The format I used is probably overkill, but something like testcase/subcase 0/subcase 1/subcase 2/... could work.

[onqtam]

releasing 2.3.5