allow for vendor defined test case variables

[erikarn]

Hi!

I'd like to add some more variables to our internal ATF (in C) test cases so they can have more verbose descriptions, link to bug ids, etc. I'm sure the same thing would be useful in FreeBSD. However, if I just go and add extra variables, kyua gets upset with me.

Would it be possible to have kyua not complain if it encounters variables it doesn't know?

[ngie-eign]

+1000

[jmmv]

Prefix the variables with X- and they will be accepted.

[erikarn]

Cool! Where in the source is it parsing the fields from atf-c again?

-a

On 2 July 2015 at 17:00, Julio Merino notifications@github.com wrote:

Prefix the variables with X- and they will be accepted.

—
Reply to this email directly or view it on GitHub
#144 (comment).

[jmmv]

In Kyua 0.11 that was in testers/atf_list.c under the parse_test_case function, and then the output gets handled in model/metadata.cpp. This was "a bit" confusing.

In HEAD that happens in engine/atf_list.cpp under the parse_atf_metadata function. I'd suggest looking at the code in head at this point. I expect to be able to publish a new release very soon now and there have been major changes internally.

But basically, defining properties in ATF tests as X- should get them propagated all the way from the test program to the results database and to any reports generated. E.g. kyua report --verbose displays the custom properties, and JUnit reports should do the same as well.

What's really missing is a way to personalize the reports so that they can display and do useful stuff with custom properties. (E.g. if you had a X-FreeBSD-Bug=bin/12345 property, it'd be nice if the reports magically turned that into a hyperlink to Bugzilla.) I do not have any specific ideas on how to achieve this, but I suspect some level of customization via Lua hooks would do the trick.

[erikarn]

Hi,

Cool, this works!

Is tihs mentioned in the documentation anywhere?

-a

On 2 July 2015 at 17:13, Julio Merino notifications@github.com wrote:

In Kyua 0.11 that was in testers/atf_list.c under the parse_test_case
function, and then the output gets handled in model/metadata.cpp. This
was "a bit" confusing.

In HEAD that happens in engine/atf_list.cpp under the parse_atf_metadata
function. I'd suggest looking at the code in head at this point. I expect
to be able to publish a new release very soon now and there have been major
changes internally.

But basically, defining properties in ATF tests as X- should get them
propagated all the way from the test program to the results database and to
any reports generated. E.g. kyua report --verbose displays the custom
properties, and JUnit reports should do the same as well.

What's really missing is a way to personalize the reports so that they can
display and do useful stuff with custom properties. (E.g. if you had a
X-FreeBSD-Bug=bin/12345 property, it'd be nice if the reports magically
turned that into a hyperlink to Bugzilla.) I do not have any specific ideas
on how to achieve this, but I suspect some level of customization via Lua
hooks would do the trick.

—
Reply to this email directly or view it on GitHub
#144 (comment).

[jmmv]

It was briefly mentioned in kyuafile(5) and in atf-test-case(4). I'm going to improve and extend the wording a little bit.

[jmmv]

I've improved the documentation in Kyua and in ATF (the manpages referenced above). Hope this helps!