[Enhancement] Enable flow-based test scenarios

[kdmukai]

Depends on #347 and #348.

Goal:

Run full end-to-end user flow test scenarios that verify the routing and internal Controller state.

Akin to Selenium website testing where the test case progresses across a series of UI clicks that lead to further web pages through a scripted test flow.

Approach Overview

A View normally instantiates a specific Screen and waits for the user's interactions with the Screen to be completed, at which point the Screen returns the user's selection/action/info/etc back to the View. The test suite will now mock out the Screen class (thereby avoiding hardware-specific dependencies that can only run on a Raspi) and will return the "user's" input for the given test scenario.

e.g. instead of instantiating and rendering a ButtonListScreen, the test suite's mock will simply report back to the View that the user selected option 2.

unittest patches (side_effect and return_value) are used to override the normal behavior, but done so in a way that preserves the normal Controller main loop. The overrided values are provided as a sequence list that makes the test flow scenarios easy to read.

Some light changes are needed in the Controller, Views, and Destination classes to support these test suite operations.

Test Suite Additions

• The new tests/base.py contains base classes for tests (BaseTest) and a variant with specific support for testing flows (FlowTest).
  • Simplifies the Controller setup/reset.
  • Mocks out the modules that can only be imported on a physical Raspi device. All tests currently running successfully on a Macbook.
• Initial tests across important flows. For example, the Load Seed flow from MainMenuView, through ScanView reading a SeedQR, completing SeedFinalizeView, and ultimately landing on SeedOptionsView.
• Also tests flow re-routing when Controller.resume_main_flow is active.
• Tests are grouped into test classes to take advantage of setup_method which runs before each test in the test class is run.

Necessary Changes

• Adds hooks in the Controller that are only meant to be used by the test suite. They support:
  • Starting a test flow at an arbitrary view (initial_destination input var)
  • Stopping the Controller at the end of a test flow (StopFlowBasedTest)
  • Adds explicit except handing for a base Letting the test suite handle its own specific exceptions: FlowTestUnexpectedViewException, FlowTestInvalidButtonDataSelectionException, and FlowTestRunScreenNotExecutedException`.
• Adds View.run_screen(Screen_cls: Type[BaseScreen], **kwargs) which instantiates the Screen and calls its display() method. The resulting return value when the Screen exits is returned to the View.

This is a replacement for the current approach where an explicit Screen class is instantiated by the View, which then calls Screen.display() to start the user interaction.

With this in place, we can now just mock out the View.run_screen() calls and deliver whatever return value our test needs. The test suite code does not need to care about all the different Screen implementation classes; they simply never get instantiated in these tests.

• Breaks Destination.run() into two parts (instantiate the View, run the View) so that the test suite can alter the View after it is instantiated but before it is run. This is necessary for certain Views that require specific workarounds to work with the test suite.

• Modifies Views to make its button_data options available to external callers (FlowTests).

• Removes the recursive run() call in SettingsEntryUpdateSelectionView.

[jdlcdl]

I assume this removes a future deprecation warning.

[kdmukai]

Yes, I forgot to note that in the description. The deprecation warning only appears (afaict) when running the test suite, but since I was trying to get everything in there neatly tied up, it was too glaring not to fix, too!

[jdlcdl]

Concept ACK at the least.

I really like the idea of being able to test from dev environment and not being forced to test on the RPi. I'll try to remain skeptical about "What am I missing out on by not testing on the RPi?" if testing primarily on dev computer becomes the norm. It's nice to be able to test View flows... but what are we missing out on by completely mocking the screens out of the picture?

I've taken a few passes reading only the code changes here, while I try to wrap my head around exactly how this is working. I left some comments on code where I wasn't sure if imports were needed? or maybe if they were temporary/obsoleted adds during your prototyping/dev process? or maybe they are put there because you know you need them to merge with another dev branch that is on deck?

I look forward to better understanding this, so please invite me in on the explainer chat sessions. Once this proof-of-concept is fine-tuned, and once I've understood it, I'd likely be game for the monotonous tasks of exhaustively testing our other flows... if that would be useful and could free you up for other evolutions that you're spiking into existence.

Nice work!

[kdmukai]

I look forward to better understanding this, so please invite me in on the explainer chat sessions.

@jdlcdl the biggest comprehension hurdle is to understand mock/patch and the complexity of targeting exactly what you want to patch. It's trivial to patch a single class or function:

with patch('seedsigner.gui.screens.SomeExactScreen') as mock_screen:
    mock_screen().display.return_value = 2   # mock screen interaction as if user selected button_data[2]

    # The patch will now override the normal `SomeExactScreen` when called in the View:
    next_destination = my_view.run()

But the challenge here is to be able to patch out any kind of Screen without knowing ahead of time which Screen a given View might be instantiating (it would be cumbersome to have to tell patch every single Screen we need to mock out; it would work, but it would be ugly. Plus would create maintenance nightmares as the code evolves).

That's where the new View.run_screen() call comes in. Rather than trying to figure out which individual Screen classes we need to patch out for each scenario, we can just patch out run_screen() itself. If the real version of run_screen() never runs, then the Screen classes are never instantiated; we just cut out that whole mechanism entirely.

So now our patched run_screen() is just given a return_value to simulate the user's selection.

[kdmukai]

'll try to remain skeptical about "What am I missing out on by not testing on the RPi?" if testing primarily on dev computer becomes the norm.

@jdlcdl It probably makes sense to have a separate set of tests that ONLY run on the Raspi hardware. There's already a basic logical separation anyway: "business logic"/command and control logic is pure python with no hardware dependencies vs user interaction, camera input, screen output is all hardware-dependent w/no bearing on the biz logic layer.

Controller, Views, Destination, BackStack, etc.

vs

seedsigner.hardware and, to some extent, seedsigner.gui

---

Longer term, we'll want to make that divide explicit as the MicroPython/esp32 codebase nears the finish line. We'll want that biz/routing layer completely identical across the two codebases (e.g. refactor so we can do something like git pull seedsigner-core) while the hardware, display, and gui parts remain custom to each platform (e.g. rendering onscreen fonts and shapes via PIL for python, LVGL for MicroPython).

[kdmukai]

Fairly major structural update. Now managing the flows with FlowStep sequences. More complete flow tests. One tough workaround for the recursive multiselect handling in SettingsEntryUpdateSelectionView.

[jdlcdl]

This is a flowtest-todo pattern that I'm just noticing for the first time. Perhaps include this in an eventual flow-test writeup.

[jdlcdl]

todo: delete this comment once understood why this has changed so much.

[kdmukai]

@jdlcdl The recursive self.run() made it impossible for the flow-based tests to track the click-to-click progression through the settings update. So instead we just re-call the same View with the same params to achieve the same end result (re-display the associated Screen and await next user input).

[jdlcdl]

This morning, I've finally made it thru a good-faith top-to-bottom review of this pr; I have no concerns that were not expressed in per-line code comments above.

I still have a pr that is unmerged on this pr at Keith's repo. In that pr, there are proposed changes that are not yet reflected here (seed views and seed tests), so I didn't leave additional comments here.

Keith, if you have not reviewed my pr yet, then I'd be willing to just add the changes that I've implied in my comments above (mostly unnecessary imports and the StopFlowBasedTest() pattern that is not needed anymore) so that when you do review it, these comments would already be resolved. But I don't want to push another commit if you're in the middle of reviewing my changes from yesterday.

Keith, assuming that you're not in the middle of reviewing my commit from yesterday, and regarding the per-line code comments that I left above, I have implemented the "I'm sure these are fine" ones into a single commit with a comment like "pr_339 post-review comments implemented" and pushed them to my repo, so that you'll see them in the pr I've issued to yours.

[kdmukai]

Additional test flows (plus other edits) merged into this PR by @newtonick and @jdlcdl.

This PR is ready for any final review and merge.

[jdlcdl]

ACK tested 4ce3fba

...all tests pass on my Ubuntu pc x86_64, and pi2 armv7l

[newtonick]

ACK. I'm merging this with a known issue. #395 causes the tests/test_controller.py to fail because the rotation changed since the test was created.