SUBCASE in function not behaving as expected

[zimtkeks]

Description

I have code with multiple ways to generate a piece of data. I also have a set of checks that should be applied to such a piece of data, no matter how it was generated. I don't want to duplicate the checks for each way of generation, and instead put the checks (implemented in SUBCASEs) into a function, which is then called where needed.

So this would be what I want to avoid, but what works as expected:

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"
#include <iostream>

TEST_CASE("Nested")
{
    DOCTEST_SUBCASE("generate data variant 1")
    {
        int data(44);
        
        // checks
        DOCTEST_SUBCASE("check data 1") { REQUIRE(data % 2 == 0); std::cout << "#"; }
        DOCTEST_SUBCASE("check data 2") { REQUIRE(data % 4 == 0); std::cout << "#"; }
    }
    DOCTEST_SUBCASE("generate data variant 1")
    {
        int data(80);
        
        // checks (identical in both variants)
        DOCTEST_SUBCASE("check data 1") { REQUIRE(data % 2 == 0); std::cout << "#"; }
        DOCTEST_SUBCASE("check data 2") { REQUIRE(data % 4 == 0); std::cout << "#"; }
    }
}

Output:

[doctest] doctest version is "2.3.4"
[doctest] run with "--help" for options
####===============================================================================
[doctest] test cases:      1 |      1 passed |      0 failed |      0 skipped
[doctest] assertions:      4 |      4 passed |      0 failed |
[doctest] Status: SUCCESS!

Note the 4 #, indicating the executed SUBCASEs.

Instead, I want the checks in a separate function:

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"
#include <iostream>

void checks(int data)
{
    DOCTEST_SUBCASE("check data 1") { REQUIRE(data % 2 == 0); std::cout << "#"; }
    DOCTEST_SUBCASE("check data 2") { REQUIRE(data % 4 == 0); std::cout << "#"; }
}

TEST_CASE("Nested")
{
    DOCTEST_SUBCASE("generate data variant 1")
    {
        int data(44);
        
        // checks
        checks(data);
    }
    DOCTEST_SUBCASE("generate data variant 1")
    {
        int data(80);
        
        // checks (identical in both variants)
        checks(data);
    }
}

However, this results in:

[doctest] doctest version is "2.3.4"
[doctest] run with "--help" for options
##===============================================================================
[doctest] test cases:      1 |      1 passed |      0 failed |      0 skipped
[doctest] assertions:      2 |      2 passed |      0 failed |
[doctest] Status: SUCCESS!

I.e., the checks are run only once, not twice.
This is pretty suprising for me*. How can I make that work as desired?

* Originally, I started writing my tests in Catch, where I didn't have this issue of putting SECTIONs into functions. Now I want to switch to doctest since I need thread-safety.

Steps to reproduce

Copy + paste the code into your online-demo

Thanks for your help!

[onqtam]

Well.. this is a legit bug, thanks for catching and reporting it! I'll look into fixing it as soon as I can - perhaps in a week or two...

EDIT: I'm actually not sure if it is a bug, or an improvement request, since this use case was never thought of, but thanks nonetheless!

[zimtkeks]

Bug or improvement request - I find the option of moving lots of checks into functions very valuable to avoid code duplication.
If there are other (non-hacky) means to achieve the same goal, I'd be happy to learn about them.

[onqtam]

Out of curiosity - do you actually need subcases for the checking? I mean... in the example provided it makes sense for the generation of the data, but not for the checking - that could just be a function with asserts. Or maybe your real use case is more complicated...

[zimtkeks]

Well, I need them only as much as need subcases. They're not required for anything I do but they're neat because they let you represent tree-like state changes with minimal code.
On a given piece of data I apply operations A, B or C. And after each one of them, I apply some follow-up operations A.1, B.1 and C.1. So this tree of operations documents how a given piece of data can be used but also it is being used to check the output of various data generation methods. I hope this makes sense ;-)

I'm not really sure that's the best way to organize my tests but subcases certainly help a lot to avoid redundant code - as does the ability put them into functions.

FYI, the nesting depth of my subcases is max. around 10.

I also moved back to catch for the time being since I noticed that threads in tests do seem to work well if they their execution is serialized (which is natural and inherently ensured in my tests).

[onqtam]

Thanks for the context. I do plan on fixing this - hopefully this weekend, but cannot promise a date.

[onqtam]

fixed and pushed in the dev branch - will release an official version soon.