System Tests: Investigate which manual system tests can be automated

[NikolaRoev]

As a developer I want to automate as many of the manual system tests as possible. There is a fair number of tests that can be automated. This will reduce the developer time spent on manually testing each release, and any issues will be caught a lot faster.

Acceptance Criteria

• A list of manual system tests that can be automated is created.
• An umbrella ticket is created to implement the automation.

Extra Information

https://github.com/ISISComputingGroup/ibex_developers_manual/wiki/Manual-system-tests

[NikolaRoev]

1-1: NO - Somewhat collaterally tested. Not really a UI test.
1-2: NO - Somewhat collaterally tested. Not really a UI test.
1-3: NO - Somewhat collaterally tested. Not really a UI test.
1-4: NO - Somewhat collaterally tested. Not really a UI test.
1-5: NO - Somewhat collaterally tested. Not really a UI test.
1-6: NO - Somewhat collaterally tested. Not really a UI test.
1-7: NO - Somewhat collaterally tested. Not really a UI test.
1-8: NO - Not a UI test.
1-9: NO - Not a UI test.
1-10: NO
2-1: NO - Not a UI test.
2-2: NO - Not a UI test.
2-3: DONE - tst_pydev_completer_works
2-4: YES - Will need to add genie_python_dae.py to Scripts. We can check if the output of the script is as expected.
2-5: NO - Not a UI test.
2-6: YES - Will need to add genie_python_blocks.py to Scripts. We can check if the output of the script is as expected.
2-7: NO - Not a UI test.
2-8: NO - Can test parts of this, but not if the graph is displayed correctly.
2-9: YES
2-10: YES
3-1: DONE - #6737
3-2: NO
3-3: DONE - Somewhat collaterally tested.
3-4: YES - Using dragItemBy, https://doc.qt.io/squish/java-convenience-api.html#java-dragitemby-function.
4-1: DONE - Functionality tested in other tests.
4-2: NO
4-3: YES
5-1: YES - Can check the background property of the perspective button.
5-2: YES - Can check with known alarms.
6-1: NO - Can't really verify that the graph is correct in Squish.
6-2: YES
6-3: YES
7-1: YES - Will need to combine 7-1, 7-2, 7-3, and 7-4.
7-2: YES - See 7-1.
7-3: YES - See 7-1.
7-4: YES - See 7-1.
7-5: DONE - Functionality tested in other tests.
7-6: DONE - Functionality tested in other tests.
7-7: YES - Will need to generate dummy log messages.
7-8: YES - Merge 7-8, 7-9, 7-10, 7-11, and 7-12.
7-9: YES - See 7-8.
7-10: YES - See 7-8. Refactor tst_can_set_wiring_tables.
7-11: YES - See 7-8.
7-12: YES - See 7-8.
7-13: YES - Using the title property of the Axis object.
7-14: YES - Merge with 7-15.
7-15: YES - See 7-14.
7-16: YES - Merge 7-16, 7-17, 7-18, 7-19, and 7-20. Will need to modify the relevant folder. Using init() and cleanup() will make this easier.
7-17: YES - See 7-16.
7-18: YES - See 7-16.
7-19: YES - See 7-16.
7-20: YES - See 7-16.
7-21: NO
7-22: YES
8-1: YES - Merge 8-1 and 8-2.
8-2: YES - See 8-1.
8-3: DONE - tst_rb_number_search_works_correctly
8-4: YES
8-5: DONE - Will be covered by other tests.
8-6: YES - Merge 8-6, 8-7, and 8-8. Using two AUTs and setApplicationContext. Refactor tst_utf8_characters_in_user_name and tst_user_names_can_be_set.
8-7: YES - See 8-6.
8-8: YES - See 8-6.
9-1: YES - Merge 9-1, 9-2, 9-3, and 9-4. Using the log message generator.
9-2: YES - See 9-1.
9-3: YES - See 9-1.
9-4: YES - See 9-1.
9-5: NO
9-6: YES - Using simple commands and checking if they execute can constitute as "working".
10-1: YES
10-2: DONE - tst_new_plot_from_block
11-1: NO - Will need a reflectometry configuration on build machine.
12-1: YES - Merge 12-1 and 12-2.
12-2: YES - See 12-1.
12-3: YES - Merge 12-3 and 12-4. Can test this as we don't care if the other instrument is online or not.
12-4: YES - See 12-3.
13-1: DONE - Other tests import and use GeniePython.
13-2: YES - Merge 13-2, 13-3, 13-4, 13-5, and 13-6 into a single test.
13-3: YES - See 13-2.
13-4: YES - See 13-2.
13-5: YES - See 13-2.
13-6: YES - See 13-2.
13-7: YES - Merge 13-7, 13-8, 13-9, and 13-10 into a single test.
13-8: YES - See 13-7.
13-9: YES - See 13-7.
13-10: YES - See 13-7.
14-1: NO - We can check if all the elements exist, but not really if they are rendered and positioned correctly without making this really complicated.
14-2: YES
15-1: YES - Merge 15-1 and 15-2. Can check if url property called from requests returns OK.
15-2: YES - See 15-1.
16-1: NO
16-2: YES - Limit this to just testing incorrect names as creating a configuration is covered by other tests.
16-3: YES
16-4: DONE - Functionality tested in other tests.
16-5: YES - Merge 16-5 and 16-7.
16-6: YES
16-7: YES - See 16-5.
16-8: YES
16-9: YES
16-10: YES - In the <INSTRUMENT_NAME><RUN_NUMBER>_ICPputlog.txt log file check <INSTRUMENT_PREFIX>:CS:SB:<BLOCK_NAME>:RC:ENABLE.
16-11: YES - In the <INSTRUMENT_NAME><RUN_NUMBER>_ICPputlog.txt log file check <INSTRUMENT_PREFIX>:CS:SB:<BLOCK_NAME>:RC:SOI.
16-12: YES - Merge with 16-10.
16-13: YES - Merge with 16-11.
16-14: DONE - Functionality tested in other tests.
16-15: DONE - tst_can_add_block_to_a_group
16-16: YES
16-17: YES
16-18: DONE - Will be covered in other tests.
16-19: YES
16-20: DONE - Functionality tested in other tests.
16-21: DONE - Covered in tst_confirm_before_deleting_component_or_config.
16-22: YES
16-23: DONE - Covered in tst_confirm_before_deleting_component_or_config.
16-24: DONE - Covered in tst_confirm_before_deleting_component_or_config.
16-25: DONE - Will be covered in other tests.
16-26: YES - Merge 16-26 and 16-29.
16-27: DONE - Will be covered in other tests.
16-28: DONE - Will be covered in other tests.
16-29: YES - See 16-26.
16-30: DONE - Functionality tested in other tests.
16-31: DONE - Functionality will be tested in other tests.
16-32: DONE - Covered in tst_confirm_before_deleting_component_or_config.
16-33: YES
16-34: YES
16-35: NO
16-36: YES
16-37: YES
16-38: YES
16-39: YES - Merge 16-39 and 16-40.
16-40: YES - See 16-39.
16-41: YES
16-42: NO
17-1: YES - Merge 17-1, 17-2, 17-3, 17-4, 17-5, 17-6, and 17-7.
17-2: YES - See 17-1.
17-3: YES - See 17-1.
17-4: YES - See 17-1.
17-5: YES - See 17-1.
17-6: YES - See 17-1.
17-7: YES - See 17-1.
17-8: DONE
18-1: DONE
18-2: YES
18-3: YES
18-4: YES
18-5: YES - Merge 18-5 and 18-6.
18-6: YES - See 18-5.
18-7: YES - Merge 18-7 and 18-8.
18-8: YES - See 18-7.
18-9: DONE
18-10: YES
18-11: YES
18-12: YES
18-13: NO
18-14: NO - Could not find the "Beam" object in Squish.
18-15: NO - Could not find the "Beam" object in Squish.
18-16: YES
19-1: DONE
19-2: DONE
19-3: YES - Merge 19-3, 19-4, and 19-5.
19-4: YES - See 19-3.
19-5: YES - See 19-3.
20-1: DONE - Covered in other run-control related tests.
20-2: DONE - Covered in more focused run-control tests.
20-3: YES
21-1: YES - Possibly merge 21-1, 21-2, 21-3, 21-4, and 21-5.
21-2: YES - See 21-1.
21-3: YES - See 21-1.
21-4: YES - See 21-1.
21-5: YES - See 21-1.
21-6: NO
21-7: YES
22-1: NO - Not really UI tests.
22-2: NO - Not really UI tests.
22-3: NO - Not really UI tests.
22-4: NO - Not really UI tests.
22-5: NO - Not really UI tests.
22-6: NO - Not really UI tests.
22-7: NO - Not really UI tests.
23-1: YES - Merge these tests into a block display test.
23-2: YES - See 23-1.
23-3: YES - See 23-1.
23-4: YES - See 23-1.
23-5: YES - See 23-1.
23-6: YES - See 23-1.
23-7: YES - See 23-1.
23-8: YES - See 23-1.
23-9: YES - See 23-1.
23-10: YES - See 23-1.
23-11: YES - See 23-1.
23-12: YES - See 23-1.
23-13: YES - See 23-1.
23-14: YES - See 23-1.
23-15: YES - See 23-1.
23-16: YES - See 23-1.
23-17: YES - See 23-1.
23-18: YES - See 23-1.
23-19: YES - See 23-1.
23-20: YES - See 23-1.
23-21: YES - See 23-1.
23-22: YES - See 23-1.
23-23: YES - See 23-1.
23-24: YES - See 23-1.
23-25: DONE - Will be covered in other tests.
23-26: DONE - tst_can_edit_block_host_config
23-27: YES
23-28: YES
23-29: YES
23-30: NO
24-1: YES
24-2: NO
24-3: DONE - Covered in 24-1.
24-4: DONE - tst_switch_to_known
24-5: YES - Do not need instrument to be online to test.
24-6: YES
24-7: YES - Can test with current instrument.
25-1: DONE - tst_only_remote_device_screens_are_preserved_between_clients
25-2: YES - Will need to test Edit and Delete.
25-3: DONE - tst_only_remote_device_screens_are_preserved_between_clients
25-4: YES
26-1: YES - Merge 26-1, 26-5, and 26-6. Will need to probably add manager password as an environment variable.
26-2: YES - Merge 26-2, 26-3, and 26-4. Don't need instrument to be online to test.
26-3: YES - See 26-2.
26-4: YES - See 26-2.
26-5: YES - See 26-1.
26-6: YES - See 26-1.
26-7: YES - Merge 26-7 and 26-8.
26-8: YES - See 26-7.
27-1: YES - Merge 27-1 and 27-2.
27-2: YES - See 27-1.
27-3: YES - Can check the last updated time.
28-1: YES - Can be done by executing a script with an expected output.
28-2: NO - Should probably be a separate Help button and test removed.
29-1: NO - Not really a UI test.

Notes:

Manual system tests spreadsheet used for above list: manual_system_tests_template_v2 (1).xlsx

[Tom-Willemsen]

A few nitpicks with the list above:

3-4: YES - Using dragItemBy, https://doc.qt.io/squish/java-convenience-api.html#java-dragitemby-function.

I think we should have a "difficult" category for ones like this - both the dragging by pixel offsets and the detecting that layout is back to standard are likely to be quite brittle.

6-2: YES
6-3: YES

Again maybe put these in a "difficult" category as we have to be a little bit careful with the assertions we use here to handle in cycle/out of cycle/accelerator maintenance periods when these tests may legitimately fail.

7-13: YES - Using the title property of the Axis object.

I'd be cautious about this one, when squish testing OPIs any minor change to this OPI will likely cause this test to fail. Maybe put in a "difficult" category to be done after the lower-hanging fruit.

16-1: NO
16-42: NO

We might actually be able to do this using JMX commands if we wanted to. Especially if we're going to be automating all the 16-xx tests in between these two.

24-6: NO - Needs a dedicated machine to be consistent.

I don't think this actually needs a machine that's up, so could it just switch to FAKEINST or NOTEXIST?

---

Once you've addressed the above points I think I'm mostly happy with the list, so can you create the umbrella ticket to implement this, so that you've satisfied the acceptance criteria of this ticket?

[NikolaRoev]

The umbrella ticket is already created and referenced above my initial comment.

The more difficult or tricky tests will be expanded on and I will add helpful comments when I create each sub ticket.

16-1 and 16-42: While I do agree that this is technically possible and that section of the manual tests will need to be reworked eventually, this feels a little too complex to implement using Squish. I also think that to effectively catch any memory issues the test would take a lot of time to run, as it will need to do all the various stuff a number of times. But, I can definitely add another sub-ticket if something like that is okay.

24-6: The workflow mentioned as an example is covered in 24-7 which is marked as YES. I think to 100% cover 24-6 we would need a running block server, so we can validate that connecting works correctly. These two tests are very similar anyways.

[Tom-Willemsen]

Ah sorry I didn't spot the umbrella reference, it got a bit lost in the noise of labels being added/removed etc.

The more difficult or tricky tests will be expanded on and I will add helpful comments when I create each sub ticket.

Ok, can you do this then so that I can review the "expansion & helpful comments"? Even just some bullet-point notes on the umbrella ticket would be sufficient for the moment if you don't want to create the sub tickets. Otherwise when we come to discuss the subtickets in planning meetings we will be missing information about the complexity and will have to waste time as a group discussing it.

16-1 and 16-42: While I do agree that this is technically possible and that section of the manual tests will need to be reworked eventually, this feels a little too complex to implement using Squish. I also think that to effectively catch any memory issues the test would take a lot of time to run, as it will need to do all the various stuff a number of times. But, I can definitely add another sub-ticket if something like that is okay.

I think my worry is that we've said we can automate many of the other 16-xx tests, and 16-42 actually becomes kind of useless if we automate most of the other tests in between. Let's add a subticket for this as you suggest, with details that we should be aiming to use a proper JMX api and not just e.g. psutil.

24-6: The workflow mentioned as an example is covered in 24-7 which is marked as YES. I think to 100% cover 24-6 we would need a running block server, so we can validate that connecting works correctly. These two tests are very similar anyways.

I'm not sure about this - to me it sounds like the test in 24-6 could be covered by switching to FAKEINST with pv prefix IN:FAKE: and checking that in help->about the pv prefix is IN:FAKE and the dae status says FAKEINST is UNKNOWN, while the test in 24-7 could be covered by switching to name LOCAL and then explicitly providing the pv prefix used by the squish machine - after which the DAE status for example should say LOCAL is SETUP.

---

In any case I feel some of these implementation details for various tests need to be written somewhere - either on the umbrella ticket or the subtickets as you prefer - otherwise there's a bunch of complexity/implementation ideas which are just in your head and so can't be reviewed.