[improvement] Add to @Test interface fields IDs and issues

[shaburov]

As an auto tester, I want to be able to associate an autotest with a test case in the test management system and associate the test with an issue in the task tracker system.

For autotest management, it is worth adding auxiliary fields to the @Test interface so that the end user can manage the presentation of the results in the checklists.

Fields:

• String[] ids = default {}; - test case IDs. For example in testrail, kanoah or another test management system.
• String[] issues = default {}; - features or/and bugs in the task tracker system. For example in jira, mantis, YouTrack, etc.

[shaburov]

@juherr need consent or dialogue.

[juherr]

For our own tests, we use the description for that.
But my advice for a more robust solution will be a custom annotation.

[shaburov]

@juherr a lot of test annotations are very bad practice for autotests.
Your product is very well adapted to auto-tests development but is not intended to develop auto-tests on checklists for BDD. Autotest must be user-friendly in development and support.

For example:
@Test description is used to describe the test.
@Details can be avoided by adding fields to the @Test annotation. Adding another annotation for each test is redundant.

@Suite(service = TestRail.class, interfaze = API.class, task = "case_fields_operations")
public class CaseFieldTests extends BaseCorvusTest {

    @Test(description = "Expecting a successful receive of the existing case fields")
    @Details(ids = "C2312", issue = "QA-573")
    public void test_20190106014139() {
        List<TRCaseField> fields = CLIENT.getTRCaseFields();
        assertThat(fields).isNotEmpty();
        for (TRCaseField field : fields) {
            assertThat(field.getAdditionalProperties()).isEmpty();
        }
    }
}

[juherr]

I'm not feeling good to add new attributes to the @Test annotation because there will always be new needs.
That's why I advise annotation like yours @Details.
JUnit has a similar approach too.

Another option could be to support @Test aliases but it will be a headache for the IDE support.

[shaburov]

Your suggestion for Support @Test aliases is, in my opinion, a very reasonable solution.
The developers of plug-ins for IDEs in my memory rather quickly fixed defects in integration with third-party plug-ins. At least IntelliJ IDE developers.

I admit honestly, junit for me is a very clumsy solution, as with the flexibility they have problems in older versions, that remained even in version 5. And in my opinion, this decision should not be taken as an example. In my experience in developing autotests, TestNG is much better than junit.

[shaburov]

You can also add the field String [] details = default {}; so that the end user can fill it according to his rules and then parse in the listeners.

For example:

public class ExampleTest {

    @Test(description = "description", details = {"id_C2312", "issue_QA-573"})
    public void test_20190106014139() { }

}

[juherr]

I think a generic attribute like details is a good compromise.

[krmahadevan]

Why not do something like this

Quoting for completeness. We could just have an annotation that represents a key/value pair and then enrich @Test to be able to accept an array of these key/value pair contained elements. That way, anyone is free to add any semantics to the key/value pair.

public @interface TableMapping {
  public String dbName();
  public String tableName();
}

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
public @interface column {
    public TableMapping[] table();
}

@column(table={
  @TableMapping(dbName="dbName", tableName="tableName"),
  @TableMapping(dbName="db2", tableName="table2")
})
public String userId = "userid";

[shaburov]

@krmahadevan It is not possible to make a general solution that would allow the end user to add their own implementation. In your example, this is the TableMapping annotation.

The examples below do not work.

• @interface members may not have type parameters

public @interface Test {
    <A extends Details> A details();
}

• @interface may not have extends list

public @interface DetailsImpl extends Details { 
}

• @interface may not have type parameters

public @interface Test<A extends Details> {
    A details();
}

[krmahadevan]

@shaburov - I wasn't talking about allowing users to inject their own types into a TestNG annotation.

I was talking about enriching the @Test annotation to support an array of key/value pairs which users can use to represent various informations (of course in string format) just as how you had hinted at
#2003 (comment) and what @juherr stated in
#2003 (comment) as a generic attribute.

I am pushing it one step above and suggesting that TestNG expose a key/value pair so that a end-user is free to add his/her own semantics.

Here's a sample of what I am talking about.

package org.testng.annotations;

import static java.lang.annotation.ElementType.METHOD;
import static java.lang.annotation.ElementType.TYPE;

import java.lang.annotation.Retention;
import java.lang.annotation.Target;

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD, TYPE})
public @interface CustomAttributes {
  String name();

  String value();
}

Here's the modified @Test annotation

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD, TYPE})
public @interface Test {

  CustomAttributes[] attributes() default {};
//remaining code omitted for brevity.
}

Heres the test class

package com.rationaleemotions.annotations;

import java.util.Arrays;
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
import org.testng.annotations.CustomAttributes;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(SampleTestClass.class)
public class SampleTestClass implements IInvokedMethodListener {

  @Test(
      attributes = {
        @CustomAttributes(name = "foo", value = "fooValue"),
        @CustomAttributes(name = "nar", value = "barValue")
      })
  public void testMethod() {
    System.err.println("Hello World");
  }

  @Override
  public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
    Test test =
        method.getTestMethod().getConstructorOrMethod().getMethod().getAnnotation(Test.class);
    System.err.println("Printing attributes");
    Arrays.stream(test.attributes())
        .forEach(
            customAttributes -> {
              System.err.println("Key = " + customAttributes.name());
              System.err.println("Value = " + customAttributes.value());
            });
  }
}

And here's the output

Hello World
Printing attributes
Key = foo
Value = fooValue
Key = nar
Value = barValue

===============================================
Default Suite
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================


Process finished with exit code 0

[shaburov]

@krmahadevan I like your decision.
value -> to array and default values

    String name() default "";
    String[] values() default "";

@juherr what do you say? In my opinion, Krishnan offered a good solution.

[juherr]

LGTM 👍

[wenijinew]

For me, it's enough to set description value with customized format and parse it in the corresponding customized utility.
In addition, @test should not be used for so specific requirement. It's used to mark class or method as part of the test. The method with the annotation could be a test scenario. Specific test case could be data-driven. That means one @test method could be used for many different test cases with different input parameters.
Thus, why not use method parameter for customized test information such as id, issue mentioned above.

[juherr]

@WengM I agree with you but @Test has already too many parameters we won't be able to remove without breaking changes. So, adding a new generic parameter is not a big deal if it can help.

Meta-annotation is a fashion alternative I'd prefer to use but need more work :) Maybe later.

[chengyinjie]

@shaburov - I wasn't talking about allowing users to inject their own types into a TestNG annotation.

I was talking about enriching the @Test annotation to support an array of key/value pairs which users can use to represent various informations (of course in string format) just as how you had hinted at
#2003 (comment) and what @juherr stated in
#2003 (comment) as a generic attribute.

I am pushing it one step above and suggesting that TestNG expose a key/value pair so that a end-user is free to add his/her own semantics.

Here's a sample of what I am talking about.

package org.testng.annotations;

import static java.lang.annotation.ElementType.METHOD;
import static java.lang.annotation.ElementType.TYPE;

import java.lang.annotation.Retention;
import java.lang.annotation.Target;

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD, TYPE})
public @interface CustomAttributes {
  String name();

  String value();
}

Here's the modified @Test annotation

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD, TYPE})
public @interface Test {

  CustomAttributes[] attributes() default {};
//remaining code omitted for brevity.
}

Heres the test class

package com.rationaleemotions.annotations;

import java.util.Arrays;
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
import org.testng.annotations.CustomAttributes;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(SampleTestClass.class)
public class SampleTestClass implements IInvokedMethodListener {

  @Test(
      attributes = {
        @CustomAttributes(name = "foo", value = "fooValue"),
        @CustomAttributes(name = "nar", value = "barValue")
      })
  public void testMethod() {
    System.err.println("Hello World");
  }

  @Override
  public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
    Test test =
        method.getTestMethod().getConstructorOrMethod().getMethod().getAnnotation(Test.class);
    System.err.println("Printing attributes");
    Arrays.stream(test.attributes())
        .forEach(
            customAttributes -> {
              System.err.println("Key = " + customAttributes.name());
              System.err.println("Value = " + customAttributes.value());
            });
  }
}

And here's the output

Hello World
Printing attributes
Key = foo
Value = fooValue
Key = nar
Value = barValue

===============================================
Default Suite
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================


Process finished with exit code 0

smart idea!