End to end integration testing

[kushaldas]

We currently have unit test cases for the different parts of the workstation
project, and the securedrop-client also has some functional tests for the
client itself.

But, we have too many different moving sub-projects under workstation, having
an end to end integration test is really important for us. This will help us
to automate many of the steps/checks which take a lot of time for us (humans)
to do.

What is the plan?

The rough idea is to have one (at least) dedicated QubesOS system with stable
updates channel (we can later have another one with Qubes testing repositories
enabled) and securedrop-workstation installed. Then based on the decided
time (nightly or a few times daily) we have our own test framework (in dom0)
to click on the securedrop-client desktop icon, and login (or the update
story)and do all QA steps one by one. It can store the results in a local
directory, and also will keep taking screenshots on the way. It will copy the
over the results to a dedicated vm, and that vm will push and publish the
result (details we have to figure out).

The - points

It will take a lot of development time and new hardware cost to have this.

The + points

We can save enormous QA time if we let the computer do this testing. We are
not going to test in QubesOS works, except the features we are dependent on,
thus we can find any bug which will hit us in production in the coming days.

We will also have regular screenshots taken, which can be used in the
documentation and other demos.

For me one of the major reason to write this proposal is that I am a lazy
Python programmer, and I would love to get the computer do this work (which it
can do better than humans) and we can focus on other interesting problems in
the SecureDrop world.

Now, any proposal is too theoretical unless we have a PoC. So, here is one
super quick demo that we can click on the desktop icon, and login into the
client.

Oh, you have to trust me that I did not type or clicked in that video, it is
my code :D

[redshiftzero]

thanks for filing this @kushaldas - would this be using OpenQA?

linking #16 which is related to this

[kushaldas]

thanks for filing this @kushaldas - would this be using OpenQA?

My idea is not to use OpenQA and skipping all the issues with nested-virt and performance. Instead, having our own testing framework.

[eloquence]

Keeping this in the near-term backlog because it's so important to discuss, but I think the next step here is to figure out a realistic scope and timeline, before we can add this to a future sprint.

[zenmonkeykstop]

Leaving open - work is ongoing on automating provisioning of Qubes setups for test purposes, which aligns with the approach in this issue.

[deeplow]

As I understand, the setting up a CI groundwork is already done and what we're missing now actually creating the integration tests.

[deeplow]

I am planning on taking a peek at integrations tests with my Qubes-updater integration PR.

What I am thinking is expanding tests to create some extra VMs as needed and tearing them down at the end, likely under tests/integration. We could piggy-back on some of the stuff already put in place by the Qubes team.

[legoktm]

Just some notes/thoughts from the hacking we did on this last week:

• otter provides framework for doing GUI testing by communicating with Qubes over the serial console and then taking screenshots of parts of the screen and OCRing them
  • this requires access to VMWare, it should run in a VM inside VMWare, which could either be a separate VM or just the bastion itself
  • I think extending the bastion to support this wouldn't be too hard; it would be much nicer if we were doing this in GHA though (Switch to GitHub Actions securedrop-workstation-ci#41)
• openQA - what Qubes uses
  • I haven't explored what this looks like, but I understand that it's similar, conceptually, just has a lot of other things we might want.
  • some opportunity to leverage Qubes infra resources/setup, need to discuss what that would look like

I think we should make an initial decision regarding framework soon-ish, so we can try writing basic tests and see what it looks like, and then make a more final decision on which direction to go and then write even more tests.

[deeplow]

We've been playing around with OpenQA and we've almost got a full SecureDrop Workstation installation test (respective PR here). Thanks to the Qubes team for letting up use their infrastructure.

Here's a picture of a successful test run up to the launcher / updater:

Next Steps

These are some notes on potential next steps that @legoktm and I converged on.

Which tests to create?

• going through clients test and figure out which ones should actually be GUI tests
• Basic workstation workflow: click icon, enter credentials, etc.
• Going through requirements and creating tests for GUI-automatable stuff. May help with not to overlaping test - but we’re in the beginning so we can figure out this along the way

Which SecureDrop server do we use

We want a fresh dev server at the beginning of the test run. Easiest way is make dev-tor in the work:

• Start make dev-tor, copy config.json to dom0
• Run SDW provisioning, SD dev container builds in background
• Reboot post-provision
• Re-run make dev-tor

Note: The Tor network may add some flakiness to the tests. At some point we may want to have some local server (without Tor) and then if the server connection fails on the test, we softfail it and use a localserver a a workaround. This way it can be clear in the test interface that not everything went alright, but we still get to see if the remaining of the test runs. Otherwise we won't have CI working for a while due to bad Tor weather

Note2: We may also want to consider having different make dev-tor variants for different kinds of initial server setups. For example in some tests we want to test downloading various file types. Easily spinning up a server like this would be nice.

Running integrations tests in OpenQA

The Qubes team also does run integration tests in OpenQA and uploads the test results in a way that's easily visible in the user interface (using the JunitXML format), unlike our current solution where we need to visually parse the log.

We agreed that this is a nice thing to have but not critical yet since we have a working alternative, which already has the "tigger on PR" part figured out.

How are tests going to be triggered?

Running nightlies for now is a good start. Later look into how we can make these triggered on PRs.

[marmarek]

Which SecureDrop server do we use

FYI openQA supports multi-machine tests, with one use case being an extra "support server" for the test. It's documented at https://open.qa/docs/#mm-tests. I have never used this feature, so I don't know if that's a good fit here (and whether the extra complexity is worth it).

[deeplow]

That is pretty interesting. Thanks for bringing it up. We should definitely explore that.