Question: verbose use case

[felipefoz]

Hi,

Let me describe my use case.

I am creating a test VI with multiple test assertions. This test VI concerns one VI.

When I am doing iterative programming (TDD) by running test and changing the VI, I get myself probing the outputs of the VI sometimes.

I know that the Verbose mode in Caraya can provide some information on the asserted values, which can be extremely useful during the iterative process, and inspecting the report window in the failed cases.

This description of what went wrong and the highlight of the problem is the default in some frameworks (i.e. python unittest).

1. Wouldn't it make sense the verbose mode to be default in Caraya?

2. If that is not the case, couldn't we include the verbose as option in define test, so we could enable that when needed?

Regards,

[francois-normandin]

@felipefoz

1. Verbose mode is inherently slower execution, so it was decided that it would not be defaulting to True.
2. You can define that as part of the Test Suite.

[felipefoz]

1. Verbose mode is inherently slower execution, so it was decided that it would not be defaulting to True.

Does it not run only when there is a failure?

2. You can define that as part of the Test Suite.

I thought about this, but isn't this function to be used in higher level call? I mean, calling a bunch of "define tests", my use case is specially for running locally in the test vi.

For running this, in the project level, for now I modified the QD plugin to allow using shift to set to verbose mode, but sometimes is cumbersome to run all the tests in the project all the time, that's why I came up with this question.

[francois-normandin]

@felipefoz Caraya is an assertion-based framework. Assertions can be called without defining any tests, so the base unit of "Assert" must be as fast as it can be, even in cases where it fails.
That's the reason we should use wrappers when we want specific behavior.

I've started a formalized version of an extensible Caraya wrapper, but it is not yet ready for primetime. **All the screenshots are partly blurred... so not to steal disclose the original naming convention, I intent this to be open source too... ** ;-)

Anyhow, here is what it looks like in the abstract:

• Each Caraya Assertion is wrapped into an equivalent namespaced Assertion with its icon's color changed.
• I retrieve a Strategy object that can be configured individually for each type of assertion (stored in a factory, globally or locally-retrievable)
• I inject the strategy into a wrapper of the Caraya Assertion Properties, which allows me to execute a different preparation for each type of assertion. (e.g Assert True verbosity is different from Assert Almost Equal verbosity)
• Strategy can be extended to include project-defined actions, such as mapping to RequirementMap, etc.

Assert True wrapper:

Execution on the strategy for Assert True:

Because the settings can be Global or Local, we can run as a "project-scoped" test by defining a Test Suite:

Or we can run a "locally-scoped" test by simply calling the Define case and the default strategy is implemented.
Note that your default strategy would be saved in labview.ini or some other setting, so that you can configure it for your whole team/company through known settings.

Either way, my report contains all the verbosity or other strategies I have globally or locally defined.

[felipefoz]

Caraya is an assertion-based framework. Assertions can be called without defining any tests, so the base unit of "Assert" must be as fast as it can be, even in cases where it fails.

IMHO , I will have to disagree, I would like to have as much as possible of information on why my test have failed, I probably then do not care about performance anymore. This is true when developing locally, and even more true in a CI environment. But just my opinion, not that matters much here.

I've started a formalized version of an extensible Caraya wrapper, but it is not yet ready for primetime. **All the screenshots are partly blurred... so not to steal disclose the original naming convention, I intent this to be open source too... ** ;-)

Is this something related to what you described in #182? It looks like, then I am looking forward to this, as this can be a good alternative. Meanwhile, I can work with what I have here.

Thanks for the feedback.

Regards,

[francois-normandin]

IMHO , I will have to disagree, I would like to have as much as possible of information on why my test have failed, I probably then do not care about performance anymore.

Although your use case is definitely how most people use Caraya for, it is not the only use case.
The core assertion VIs are not Unit Tests, but really Assertions in the same sense as the VIs in the "Assert Type" palette.

If you set the nodes to "Assert Only" = TRUE, then you can use it in production code as this example:
(Note that I do not call the Define Test method)

In this case, I think it is obvious that verbose assertions would be detrimental to performance. Since this is baked into the core of Caraya, we cannot change the default values within having real-world impact on production code.

I totally understand your request and how it would make it useful for the majority of people to have it verbose by default, and having to switch it off for production... I'm not very keen on adding more inputs on Connector Panes (because we can use interfaces and dependency-injection mechanisms if we build on the core components) and of course, we need to be very careful about backwards-compatibility.

There might be a reflection to make about the impacts of setting Verbose = True by default WHEN Assert Only is False... but to be honest, I think the implementation of the verbose behavior is simplistic as it is. I would rather have its implementation be user-defined rather than fixed for the whole community.

Is this something related to what you described in #182? It looks like, then I am looking forward to this, as this can be a good alternative. Meanwhile, I can work with what I have here.

Yes it is related to that.
I don't want to promise anything, but I might be able to publish a draft in the coming weeks. If I can't, should I reach out and invite you to a private preview?

[felipefoz]

If you set the nodes to "Assert Only" = TRUE, then you can use it in production code as this example:
(Note that I do not call the Define Test method)
In this case, I think it is obvious that verbose assertions would be detrimental to performance. Since this is baked into the core of Caraya, we cannot change the default values within having real-world impact on production code.

Ah, didn't know about that. That's something new I have learned about Caraya. Then I understand that performance is relevant.

I don't want to promise anything, but I might be able to publish a draft in the coming weeks.

No rush. We are in holidays now, so I wouldn't expect anything.

If I can't, should I reach out and invite you to a private preview?

Yes, definitely. If this is possible, then I will be glad for.