compile time unit test templates

[arturbac]

Hi I taught You may be interested as idea of templated tests.
When i started working with ut 1.x I needed something more that just runtime tests.
So I utilized Your ut as runtime component and I wrote consteval part
short:

using buffered_string_type_list
  = metatests::type_list<coll::string, coll::u8string, coll::u16string, coll::u32string, coll::wstring>;

 test_result result;
  "basic_string_constr"_test = [&]
  {
    auto fn_tmpl = []<typename string_type>(string_type const *) -> metatests::test_result
    {
      using st = string_type;
      using size_type = typename string_type::size_type;
      using char_type = typename string_type::char_type;
[..]
      constexpr_test(vs == text_short.view());
      constexpr_test(size(vs) == text_short.size());
      constexpr_test(capacity(vs) >= text_short.size());
      constexpr_test(is_null_termianted(vs));
     [...]
      return {};
    };

    result |= run_consteval_test<string_type_list>(fn_tmpl);
    result |= run_constexpr_test<string_type_list>(fn_tmpl);
  };

template<compatibile_test_result test_result_type>
constexpr test_result
  constexpr_test(test_result_type expression, source_location const location = source_location::current())
  {
  if(std::is_constant_evaluated())
    {
    if(!expression)
      throw;
    return test_result{expression};
    }
  else
    {
    boost::ut::expect(static_cast<bool>(expression), location);
    return test_result{expression};
    }
  }

example
https://github.com/arturbac/small_vectors/blob/master/unit_tests/string_ut.cc
all functionality in
https://github.com/arturbac/small_vectors/blob/master/unit_tests/ut_core/unit_test_core.h

[kris-jusiak]

Interesting, I'm all for compile time testing and like the constexpr_test, I think expect could be extended this way.
I'm very interested in exploring the space and extend ut with some awesomeness in this space.

I've done some experimenting - https://godbolt.org/z/jxehbcYK8, based on simple static_assert/assert principle.
I think there a few things to look for.

• Ideally the compile time and run-time tests would be written the same way.
• Compile-time tests should be able to be executed at run-time if needed (for coverage).
• There should be ability to filter test at compile-time similar to the way it can be done at run-time.
• Most importantly, there should be a way to get nice error messages at compile-time (I think that's the hardest).

Any way, I'm more than happy to explore ideas. I think the if consteval idea is the way to go for expect. Any thoughts about other points, especially about the error messages? Thanks for sharing and great work.

[arturbac]

so the first 2 points I got ;-) when I develop and get UB then compile time may be hard so if I cant solve compile time error I turn off consteval_* commenting it out and then do debuging at runtime
//result |= run_consteval_test<string_type_list>(fn_tmpl);
result |= run_constexpr_test<string_type_list>(fn_tmpl);
Relating to error messages ( in this topic), would be nice if U try -Weverything -Werror
I have to "condomise" ut ;-)

#if defined(__clang__)
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wsign-conversion"
#pragma clang diagnostic ignored "-Wextra-semi"
#pragma clang diagnostic ignored "-Wreserved-identifier" //added with 2.x
#endif

#include <boost/ut.hpp>

#if defined(__clang__)
#pragma clang diagnostic pop
#endif

getting back to main topic -> concepts are the only way, putting requirement on functions and classes significantly improve error messages for me.

with the filter idea - some options templete param options{ .enabled=some_constexpr_var }, then You can disable at compile time tests<options{.enabled ...}> or expects<>