Manual testing of release 2022.12

[tyuldashev]

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

• Build from the unit-test-bot/rc16122022 branch
• Download plugin
• Check downloaded zip-file size < 100 MB
• Open IntelliJ IDEA 2022.2.4 (last minor release of 2022.2, 2022.3 not supported yet - Plugin is not compatible with IntelliJ IDE 2022.3 #1466)
• Remove previously installed UTBot plugin
• Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• Open the project in the IDE
• Install the downloaded plugin

Go through manual scenarios

Manual scenario #1

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• Generate tests for the class
• Remove results
• Generate and Run test for a method

Manual scenario #2

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• Generate tests with different Mocking options combinations

Regression checks

Collect too complex reflection in tests

Add a class / package - generate / generate and run tests for class / method / class - check tests

• Installed on 2022.1.4 - Gradle project
• Installed on 2022.2.4 - Gradle project
• Installed on 2022.1.4 - Maven project
• Installed on 2022.2.4 - Maven project
• Installed on 2022.1.4 - IntelliJ project
• Installed on 2022.2.4 - IntelliJ project

Add a class / package - generate tests for class / method / class

• Gradle - existing test sources root - No mocks
• Maven - existing test sources root - Mocks outside package
• Maven - new test sources root - Mocks outside class
• IntelliJ - new test sources root - Mocks outside class - Mock static methods

Add a class / package - generate tests for class / method / class

• JDK 8 - JUnit 5 - Parametrized
• JDK 8 - JUnit 4 - Parametrized off
• JDK 8 - TestNG - Parametrized off
• JDK 11 - JUnit 5 - Parametrized
• JDK 11 - JUnit 4 - Parametrized off
• JDK 11 - TestNG - Parametrized
• JDK 17 - JUnit 5 - Parametrized (@alisevych)
• JDK 17 - JUnit 4 - Parametrized off (@alisevych)
• JDK 17 - TestNG - Parametrized off (@alisevych)

UI checks - in a project with JDK 11

• timeout changes - 1/10/60 sec - check with stopwatch
• cancellation - check on different steps
• regions: Symbolic first, Fuzzing,
• regions: error suite (NPE), explicitly thrown exceptions, caught exceptions (successful)
• exceptions - check Problems tab
• context menu / shortcut
• call from method / class / package - check Members list
• call for several classes - Generate and Run - check all test classes are run

Check Settings - in a project with JDK 17

• Symbolic 100%
• Symbolic 95%
• Symbolic 5%
• Fuzzing 100%
• Summaries with custom tags - check all tags
• Summaries with plain text - check for Fuzzing
• Summaries turned off
• Tests with exceptions: Fail
• Tests with exceptions: Pass
• Overflow : Ignore overflows
• Overflow: Treat overflows as errors

Other features testing

• Strings
• Mocks
• Collections
• IDEA Community 2022.2.4 smoke test

[tyuldashev]

Bugs found:

• Identical @DisplayName for tests on enum methods #1588
• Problems view is not opened after generation Kotlin tests #1589
• No JS tests are generated when project path has spaces #1586
• UtBot mocks interface even when Do not mock is selected #1560