Manual testing of Release 2023.3 Part 2

[alisevych]

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

• Choose appropriate build
  22/03 New build taken
• Download plugin
• Check downloaded zip-file size < 100 MB
  It is 123 MB
• Open IntelliJ IDE 2022.3 Ultimate
• Remove previously installed UTBot plugin
• Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• Open the project in the IDE
• Install the downloaded plugin

JDK <-> Framework Pair:

All possible combinations of JDK and Framework are selected.
Project type is iterated independently.
Parametrization is OFF.

Scenario:

1. Create new project with specified project type, JDK.
2. Remove default test dependency from Maven or Gradle configuration and refresh project structure.
3. Copy paste ... and ... from UTBot project into source code folder.
4. Invoke test generation for ... file with specific Framework - all test frameworks suggested to be installed.
5. Generate tests - tests are generated without error with proper test Framework.
6. Run tests - tests run without error.
7. Invoke test generation again for ..., all except installed framework suggested to be installed.
8. Generate and run tests with the same framework - tests are generated and executed without errors.

Project	JDK	Framework	Sample
Intellij	1.8	JUnit 4	arrays/ArrayOfObjects
Intellij	11	JUnit 4	lambda
Intellij	17	JUnit 4	enums/ClassWithEnum

Settings to check in pairs
add sample name

Mocks/Mode pairs
tested with Project, JDK, Framework

Mocks/Mode	Fuzzing + Symbolic	Symbolic 100%	Fuzzing 100%
outside package + static	stream/BaseStreamExample	stdlib	stdlib
outside class + static	arrays/ArrayOfObjects	algorithms/CorrectBracketSequences	controlflow/Switch
outside package + no static	mixed/LoggerExample	collections/CustomerExamples	collections/CustomerExamples
outside class + no static		algorithms/CorrectBracketSequences
no mocks		algorithms/CorrectBracketSequences	stdlib

Mocks/Project pairs
tested with Project, JDK, Framework

Mocks/Project	Gradle	Maven	IntelliJ
outside package + static	types/Generics	mixed/LoggerExample	stream/BaseStreamExample
outside class + no static	algorithms/CorrectBracketSequences	mixed/LoggerExample	collections/CustomerExamples
no mocks	algorithms/CorrectBracketSequences	mixed/LoggerExample	collections/CustomerExamples

[alisevych]

Known issues:

• Unneeded variable creation for final Enum state #2020
• Compare returning expected and actual Stream #1046
• Functional interfaces verification #1061

New bugs:

• Project is not built automatically before UnitTestBot reads classes #2021
• Missing errors in SARIF report and on Problems view #2023
• UtUtils methods using setAccessible doesn't work in Java 17 #2026
• NPE from Fuzzing providers for static code block with File operations #2001
  actual for all package-private classes (was not cherry-picked)