Capturing warning on successful test

[yairlenga]

I’m looking for a good way to represent “warning” in tests for a large, complex real time calculation engines. The expectation is that there is a “reasonable” number of accepted errors. I want to differentiate between “clean” success, or “warning”.

Main categories are “warning” are “partial” (most calculation done), timing (getting close to max execution time), “close enough” (result are acceptable, but getting close to error margin).

Is there away to add statuses to the preconfigured list (success, broken, failed, …), or other way to tag this state at the various levels (suite, sub-suites, …). My end goal is to be able to show them visually (e.g. light green), and to filter.

[baev]

Great idea!

In Allure 3, we're already working on the concept of creating report-level warnings/errors if some criteria aren't met: e.g., if you have more failures than you expect at max, a warning will appear on the report overview.

Our other concept is to mark concrete test results by criteria. A good example is to mark slow tests. Since your idea is similar to slow tests, we can probably think of a more generic way to process that.