Manual testing of Release 2022.12.1 Part 1

[tyuldashev]

Possible values:

• Project: Maven, Gradle, Intellij
• JDK: 1.8, 11, 17
• Framework: JUnit 4, JUnit 5, TestNG
• Parametrization: On, Off

Suggestion:

Let's do independent validation of "JDK <-> Framework" and "JDK <-> Framework <-> Parametrization". Project type will equally distributed between testers.

JDK <-> Framework Pair:

All possible combinations of JDK and Framework are selected.
Project type is iterated independently.
Parametrization is OFF.

Scenario:

1. Crete new project with specified project type, JDK.
2. Remove default test dependency from Maven or Gradle configuration and refresh project structure.
3. Copy paste org.utbot.examples.exceptions.ExceptionExamples and org.utbot.examples.controlflow.Conditions from UTBot project into source code folder.
4. Invoke test generation for org.utbot.examples.exceptions.ExceptionExamples file with specific Framework - all test frameworks suggested to be installed.
5. Generate tests - tests are generated without error with proper test Framework.
6. Run tests - tests run without error.
7. Invoke test generation again for org.utbot.examples.controlflow.Conditions, all except installed framework suggested to be installed.
8. Generate and run tests with the same framework - tests are generated and executed without errors.

Project	JDK	Framework	Status	Comment
Gradle	1.8	JUnit 5	✅
Maven	1.8	TestNG	✅	7.5 added
Gradle	11	JUnit 5	✅
Maven	11	TestNG	✅	7.6 added
Maven	17	JUnit 5	✅
Gradle	17	TestNG	✅	7.6 added

JDK <-> Framework <-> Parametrization

All possible combinations of Parametrization and Framework need to be tested. But actually all cases with Parametrization OFF were tested in previous section, so we select only case Parametrization is ON.
Additionally Parametrization is possible only with JUnit5 and TestNG frameworks, so we leave only with 6 test cases:

JDK	Framework	Parametrization	Status	Comment
1.8	JUnit 4	On	➖	Not applicable
1.8	JUnit 5	On	✅
1.8	TestNG	On	➖	Not applicable
11	JUnit 4	On	➖	Not applicable
11	JUnit 5	On	✅
11	TestNG	On	✅
17	JUnit 4	On	➖	Not applicable
17	JUnit 5	On	✅
17	TestNG	On	✅

JDK	Fuzzing/Symbolic	Symbolic 100%	Fuzzing 100%
1.8	✅	✅	✅
11	✅	✅	✅
17	✅	✅	✅

Test Generation Timeout	Fuzzing/Symbolic	Symbolic 100%	Fuzzing 100%
60	✅	✅	✅
10	✅	✅	✅
1	✅	✅	✅

• timeout limits only test generation, building and reading classes phases are not included.

[tyuldashev]

Entered issues:

• Occasionally duplication of the error in Problems View #1867
• @utbot.invokes {@link soot.dummy.InvokeDynamic in JavaDoc for test method #1869
• No assert in test for method which returns Exception object #1870
• 'View generated test' cannot locate parametrized test #1876
• UTBot generates several empty test classes when run for a pack of classes #1879
• Concrete execution fails during symbolic test generation for ListAlgorithms #1886

Is seems that #1879 is a regression problem, all others look like pre-existing.