Testing of UI checklist 1.0

[alisevych]

UI plugin checklist - version 1.0

Install plugin

• Choose appropriate workflow from the next list (by default, use the latest one) https://github.com/UnitTestBot/UTBotJava/actions/workflows/publish-plugin-and-cli.yml
• Open IntelliJ IDE
• Uninstall previously installed UTBot plugin (if present)
• Install the downloaded plugin
• Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• Open the project in the IDE

Make the following checks for IntelliJ, Maven, Gradle projects with different JDK 8/11

Open dialog

• Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• From Project view. Right-click on one class, select Create tests with UTBot... => Dialog is opened. All methods from class are listed and selected
• From code. Open context menu, select Generate..., select Create tests with UTBot... => Dialog is opened
• Shortcut. Cursor on the class and outside all methods, press Alt+Shift+U => all class methods are selected
• Shortcut. Cursor on/inside one method, press Alt+Shift+U => correct method is selected
• Shortcut. Not working from build.gradle file, Java class file in test sources root, resources, etc.
• Context menu item is disabled for --||--

Dialog. Members

• From Project view. Right-click on a package with several classes, select Create tests with UTBot... => Dialog is opened. All classes from package are listed and selected
• From Project view. Select several classes, right-click, select Create tests with UTBot... => Dialog is opened. Selected classes are listed and selected
• From Project view. Right-click on one class, select Create tests with UTBot... => Dialog is opened. All methods of the class are listed and selected (except private)
• Create a public class with 4 different methods: private, without access modifier, protected, public. Open UTBot test generation for the class => all methods except private are in list.
• Make the class package-private (remove "public") => check the same

Dialog. Settings

Verify tests are generated with all these options

• Dialog. Test source root is defaulted (if test source root is set in project)
• Dialog. Test source root (if no test sources root set in project) => suggests to create a new one (red) - create => verify created
• Dialog. Code generation language. Default is Java // change request:
  UI. Code generation language. Set default to the analyzed class language #500
• Dialog. Code generation language => Java, Kotlin
• Dialog. Test framework => is defaulted to JUnit5 // change request:
  UI. Test framework. Set default to the existing one in project #499
• Dialog. Test framework (for project with JDK 8) => JUnit4, JUnit5 only // change request:
  Install appropriate version of TestNg in user project depending on JDK version #309
• Dialog. Test framework (for project with JDK 11) => JUnit4, JUnit5, TestNG (not existing in project dependencies must have red text "(will be installed)")
• Dialog. Mock strategy => 1) No mocks, 2) Other packages: Mockito, 3) Other classes: Mockito
• Dialog. Mock strategy "No mocks" is selected => Mock static is set to "No static mocking" and disabled
• Dialog. Mock strategy - select 2) and 3) options => Mock static options: 1) No static mocking, 2) Mockito static mocking.
• Dialog. Mock strategy (for a project without Mockito in dependencies) => 2), 3) options have red text "(will be installed)"
• Dialog. Timeout for class => default to 60 sec, only natural numbers can be saved, up and down arrows work
• Dialog. Timeout for class => minimize, check on ArrayQuickSort or some other code with conditions and loops
• Dialog. Parametrized test => 2 options: Not parametrized / Parametrized
• Dialog. Parametrized test - select "Parametrized" => "No mocks" and "No static mocking" are chosen and both disabled
• Dialog. Destination package. Check not present // change request:
  UI. IDEA plugin dialog. Hide UI possibility to change destination package #488

SARIF report

• Dialog. Set "Not parametrized" , generate tests => SARIF report is generated into resources folder
• do this for a single class => check place, package, file
• do this for several classes in different packages
• Dialog. Set "Parametrized" , generate tests => no SARIF report is generated

File -> Settings... -> UTBot

• Force static mocking. Add a class...
• Code analysis. Select Simplest, Deepest, 1st mark from left, 1st mark from right, somewhere in the middle...

[amandelpie]

How do you think @alisevych : could we use automation framework to check some scenarios with code generation like for web application with Selenium?

[alisevych]

@amandelpie I'm pretty sure that it is possible. This checklist is the base for the future UI test automation.
There are several possible UI TA tools. Investigation is needed to verify which of them suits the project fine. That depends not only on how they interact with IDEA UI. But also on how these tests are to be integrated into CI/CD process.
That's a pleasant thing to do but that's not the first priority for now. There are core functional cases to analyze manually and cover them first.