First attempt for api test

[GabrielXia]

Contains

A first attempt to #2317. In general, with @Cervator and @iaronaraujo, we will try to detect all the api method changes in a PR and detect the correct scroll to increment.

In this PR, I wrote a first test for org.terasology.engine.ComponentFieldUri. It'll do:

1. Test all public constructors and methods, if there is a change in their signatures (modifiers, parameters, return type and exceptions) or a remove of a method , majorIncreaseTest() print "Major increase required" with details
2. If the class has a new public method & constructor, minorIncreaseTest() will print "Minor increase requierd"
3. There are also other tests generated by Evosuite for code coverage. If they are not passed a major increment might be needed
4. If all the api-tests pass, but the test in regular tests in engine-tests don't 100% pass, that might be a minor increment
5. If all the tests pass, then that's a patch increment

In most cases, tests generated by Evosuite will detect method signature changes, too. E.g. If we change a parameter in a method, some tests will not be complied. Even if these tests don't detect the method changes, majorIncreaseTest() will detect it.

How to test

In org.terasology.engine.ComponentFieldUri, try to do some changes in the method and run the tests

• If you change the the parameters or return types of a public method or remove a method, normally the tests can't be compiled. If you modified the modifiers, e.g. public to public final, majorIncreaseTest() will detect it
• If you add a new public method & constructor, minorIncreaseTest() will detect it

Outstanding before merging

• Need to consider the workflow of the API test
• Need to evaluate the MethodSignature detection
• Need to evaluate whether the tests generated by Evosuite has meanings.

Future work

• Add more meaningful tests
• Tests to all API classes
• A scraper to show what API we have covered by the API test

Do I miss something? Thanks for comments :-)

[GooeyHub]

Hooray Jenkins reported success with all tests good!

[GabrielXia]

I'm not sure if "evosuite-standalone-runtime" is necessary. In this test, only verifyException() method is used from the library. Maybe we can just remove these tests with verifyException() if we don't want use evosuite library

[GabrielXia]

@Cervator If in a pr someone changes the code in a method, then the method returns a different value. In this case, some auto-generated test will fail. It needs a major increase or a minor increase? IMHO it's a major increase.

[Cervator]

Tricky situation. Based solely on SemVer I don't think its spec cares one way or another: so long as the signatures match you pass SemVer. Logic changes are assumed to be benevolent (working toward fixing bugs or providing new features). A failed unit test either means you introduced a bug or failed to update a unit test, which should block a PR for a reason unrelated to SemVer.

So: broken unit test = fix your code! No merge for you until then :-) SemVer doesn't really matter until the PR qualifies as merge-able.

[Cervator]

Just to make sure I understand the current state of the approach here: the example test prepares a mockup of the target class dynamically, then validates that it still matches public methods etc right? Wouldn't we want to load in a stored set of data from a prior build? Or is that more or less the next step and some of this stuff makes it easy to switch over to do so? :-)

[GooeyHub]

Hooray Jenkins reported success with all tests good!

[oniatus]

Interesting approach 👍 I think this is feature-complete as experiment so probably it is okay to close it @GabrielXia? I would like to finalize the discussion and strategy about API testing on #2317 and see what we can come up with. There is one different approach we could try out before selecting one way to go.

[Cervator]

Bump - #3253 has now been merged, which uses the other approach that may be simpler than this.

@GabrielXia can you play around with the other system and compare it to your own here to see if any further improvements can be made? There are also some follow-up issues.

If this is complete and remaining an experiment it would be nice to close it, I just want to make sure we don't lose anything nice and useful. And in either case the work is hugely appreciated :-)

[syntaxi]

I'm going to close this as superseded by #3253. Thanks for the experimental work 👍