Manual testing of Release 2022.10.2

[alisevych]

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

• Build from the unit-test-bot/rc10192022 branch
• Download plugin
• Check downloaded zip-file size < 100 MB
• Open IntelliJ IDEA 2022.2.3 (last released)
• Remove previously installed UTBot plugin
• Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• Open the project in the IDE
• Install the downloaded plugin

Go through manual scenarios

Manual scenario #1

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• Generate tests for the class
• Remove results
• Generate and Run test for a method

Manual scenario #2

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• Generate tests with different Mocking options combinations

Verify fixes

• Test generation timeout is exceeded 15 sec - 1 min for every run #1028
• Other class is not mocked as required #747
• Cannot save tests generation report: error occurred 'List is empty., reason: java.util.NoSuchElementException #1098
• Many similar sandboxed tests on sockets generated by Fuzzing and not minimized #1129
• IndexNotReadyException thrown in IDEA with installed UnitTestBot plugin #273
• Only one of generated tests is run with Generate and Run #1149
• Number of generated tests is not properly counted in some cases #983
• Plugin Action stays locked even when previous generation is finished #1173
• Functional interfaces defined as class members - InaccessibleObjectException on test generation #448
• Concrete execution fails for a static method using a static field #711
• IntelliJ Plugin memory leakage #1054

Regression checks

Collect compulsory reflection in tests

Add a class / package - generate / generate and run tests for class / method / class - check tests

• Installed on 2021.2.4 - Gradle project
• Installed on 2022.1.4 - Gradle project
• Installed on 2022.2.3 - Gradle project
• Installed on 2021.2.4 - Maven project
• Installed on 2022.1.4 - Maven project
• Installed on 2022.2.3 - Maven project
• Installed on 2021.2.4 - IntelliJ project
• Installed on 2022.1.4 - IntelliJ project
• Installed on 2022.2.3 - IntelliJ project

Add a class / package - generate tests for class / method / class

• Gradle - existing test sources root - No mocks
• Maven - existing test sources root - Mocks outside package
• Maven - new test sources root - Mocks outside class
• IntelliJ - new test sources root - Mocks outside class - Mock static methods

Add a class / package - generate tests for class / method / class

• JDK 8 - JUnit 5 - Parametrized
• JDK 11 - JUnit 5 - Parametrized off
• JDK 11 - JUnit 4 - Parametrized off
• JDK 11 - TestNG - Parametrized
• JDK 17 - JUnit 5 - Parametrized
• JDK 17 - JUnit 4 - Parametrized off
• JDK 17 - TestNG - Parametrized off

UI checks - in a project with JDK 17

• timeout changes - 1/10/60 sec - check with stopwatch
• cancellation - check on different steps
• regions: Symbolic first, Fuzzing, error suite (NPE), explicitly thrown exceptions, caught exceptions (successful)
• exceptions - check regions for exceptions
• exceptions - check Problems tab
• context menu / shortcut
• call from method / class / package - check Members list

Check Settings - in a project with JDK 17

• Symbolic 100%
• Symbolic 95%
• Symbolic 5%
• Symbolic 0% - Fuzzing 100%
• Summaries with custom tags - check all tags
• Summaries with plain text - check for Fuzzing
• Summaries turned off
• Tests with exceptions: Fail
• Tests with exceptions: Pass
• Overflow : Ignore overflows
• Overflow: Treat overflows as errors

Other features testing

• Sandboxing - check default behavior
• Sandboxing - add some permission
• Strings

[alisevych]

new bugs:

• Run action cannot be performed for several classes #1195
• Progress bar shows the progress stepwise and incorrectly #1196

[alisevych]

reproducing

• Incorrect links in Java docs for classes without packages #1174

[alisevych]

Bug is not reproducing, but there are concrete executor errors on the same code:

• Many similar sandboxed tests on sockets generated by Fuzzing and not minimized #1129 (comment)
  As investigated by @Markoutte , it is a common case - constructor throws exception if input is not correct.
  Need to think of a common solution for such cases.
  Should such cases be shown as error test?

[alisevych]

new bug - has not reproduced yet:

• com.intellij.util.IncorrectOperationException: Sorry but parent: ClientId(value=Host) (class com.intellij.openapi.client.ClientProjectSessionImpl) has already been disposed (see the cause for stacktrace) so the child: Action Update (class com.intellij.openapi.util.Disposer$2) will never be disposed

[alisevych]

RC branch reconstructed: unit-test-bot/rc10202022 branch

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

• Build from the unit-test-bot/rc10202022 branch
• Download plugin
• Check downloaded zip-file size < 100 MB
• Open IntelliJ IDEA 2021.2.4 (first supported)
• Remove previously installed UTBot plugin
• Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• Open the project in the IDE
• Install the downloaded plugin

Go through manual scenarios

Manual scenario #1

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• Generate tests for the class
• Remove results
• Generate and Run test for a method

Manual scenario #2

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• Generate tests with different Mocking options combinations

Recheck fixes

• Test generation timeout is exceeded 15 sec - 1 min for every run #1028
• Cannot save tests generation report: error occurred 'List is empty., reason: java.util.NoSuchElementException #1098
• IndexNotReadyException thrown in IDEA with installed UnitTestBot plugin #273
• Only one of generated tests is run with Generate and Run #1149
• Plugin Action stays locked even when previous generation is finished #1173
• Functional interfaces defined as class members - InaccessibleObjectException on test generation #448
• Concrete execution fails for a static method using a static field #711
• IntelliJ Plugin memory leakage #1054

Regression checks

Collect compulsory reflection in tests

Add a class / package - generate / generate and run tests for class / method / class - check tests

• Installed on 2021.2.4 - Gradle project
• Installed on 2022.1.4 - Gradle project
• Installed on 2022.2.3 - Gradle project
• Installed on 2021.2.4 - IntelliJ project
• Installed on 2022.2.3 - IntelliJ project

Add a class / package - generate tests for class / method / class

• Gradle - existing test sources root - No mocks
• Maven - new test sources root - Mocks outside class
• IntelliJ - new test sources root - Mocks outside class - Mock static methods

Add a class / package - generate tests for class / method / class

• JDK 8 - JUnit 5 - Parametrized
• JDK 11 - JUnit 4 - Parametrized off
• JDK 17 - JUnit 5 - Parametrized
• JDK 17 - TestNG - Parametrized

UI checks - in a project with JDK 17

• regions: Symbolic first, Fuzzing, error suite (NPE), explicitly thrown exceptions, caught exceptions (successful)
• exceptions - check regions for exceptions
• exceptions - check Problems tab
• context menu / shortcut
• call from method / class / package - check Members list

Check Settings - in a project with JDK 17

• Symbolic 100%
• Symbolic 5%
• Symbolic 0% - Fuzzing 100%
• Summaries with custom tags - check all tags
• Summaries with plain text - check for Fuzzing
• Summaries turned off
• Tests with exceptions: Fail
• Tests with exceptions: Pass
• Overflow : Ignore overflows
• Overflow: Treat overflows as errors

Other features testing

• Sandboxing - check default behavior
• Sandboxing - add some permission
• Strings

[alisevych]

new bug:

• Empty region names and missing javadoc for Sort.arrayCopy #1204

[alisevych]

new bug:

• Test generation is started for several classes. Progress bar shows that generation for 2nd / 3rd class has been already started. Cancel is clicked. No test classes are produced. - expected: Tests for already finished classes should be saved.

[alisevych]

new blocker bug - IDEA is not responding during/after test generation:

• IDEA hangs during test generation #1207

[alisevych]

Testing was blocked