Add SecureDrop installation test #25

deeplow · 2024-09-11T13:11:42Z

First attempt at adding a test for SecureDrop.

marmarek · 2024-09-11T14:04:00Z

tests/securedrop/install.pm

+    assert_and_click("menu-vm-xterm");
+
+
+    assert_script_run('gpg --keyserver hkps://keys.openpgp.org --recv-key "2359 E653 8C06 13E6 5295 5E6C 188E DD3B 7B22 E6A3"');


assert_script_run depends on seeing serial console output - serial console from "work" VM isn't directly connected to the one of the host; for this to work you either need to run something like tail -F /var/log/xen/console/guest-work.log >> /dev/hvc0 in dom0 (we do that here), or do all that from dom0's terminal via qvm-run

I see. Would type_string and then "ret" work as well? I'm trying not to deviate to much from the original instructions so it's easy to update.

Yes, that would work, but your test wouldn't detect if any of those command fails (other than possible some later step dom0 in dom0 failing).

Fair point 😔. I'll just go ahead an use qvm-run, then.

tests/securedrop/install.pm

marmarek · 2024-09-12T11:03:12Z

Hint: add send_key('alt-f10') to see more output at once in xterm. Not relevant much when everything goes right, but helps quite a bit when debugging.

deeplow · 2024-09-12T12:56:00Z

Hint: add send_key('alt-f10') to see more output at once in xterm. Not relevant much when everything goes right, but helps quite a bit when debugging.

Thanks for the tip. I had seen that in some places and was wondering about its purpose. I'll add it in the next round.

deeplow · 2024-09-12T18:25:18Z

tests/securedrop/install.pm

+    assert_script_run('curl https://raw.githubusercontent.com/freedomofpress/securedrop/d91dc67/securedrop/tests/files/test_journalist_key.sec.no_passphrase | sudo tee /usr/share/securedrop-workstation-dom0-config/sd-journalist.sec');
+    assert_script_run('sdw-admin --validate');
+
+    assert_script_run('xfce4-power-manager -q'); # disable screen blanking during long command


@marmarek there's a command which takes quite a while and in the meantime the screen blanks. I don't think it's xscreensaver because I think that's killed at the beginning of the test. Then I tried to disable XFCE's power management, but didn't help.

Have you encountered this before?

My notes have this line:

x11_start_program('env xset s off', valid => 0);

but I'm not sure if that was enough either.

Thanks! I had to combine it with env xset -dpms for this to fully work.

And FYI I noticed that just with env xset s off it still blanked for a lot of the slow command (sdw-admin --apply), but oddly enough the screen showed up just the logs upload command (video). No idea what went on there.

It unblanked on the key press.

Oh! I totally forgot that it was literally typing each letter. That's why, then.

I recall that the above options were still not working perfectly (the screeen was still bllanking at some point). What seems to have solved it is enabling presentation mode. I haven't look at what it's doing under the hood. But it seems to work. And because the setting is persistent, I think it shouldn't need anymore all the xscreensaver exits.

marmarek · 2024-09-12T19:43:13Z

Anyway:

# Test died: command 'sdw-admin --apply' timed out at /usr/lib/os-autoinst/autotest.pm line 412.

So, longer timeout? This is running virtualized, so runs slower than native.

And also, I recommend collecting and uploading logs. For example wrap it with script, or use tee (see https://github.com/QubesOS/openqa-tests-qubesos/blob/main/tests/update2.pm#L110-L111 for example). You can also do post-fail hook to collect extra info on failure too: https://github.com/QubesOS/openqa-tests-qubesos/blob/main/tests/update2.pm#L168-L174

deeplow · 2024-09-16T14:03:32Z

So, longer timeout? This is running virtualized, so runs slower than native.

Fair point. I have added some timeout.

Now I am running into another issue. I have created a needle through the web interface added for this step an assert_and_click. However, when it runs, it's not even listing the needle. Do I need to add the needle's PNG and respective JSON to the commit?

marmarek · 2024-09-16T14:21:35Z

Have you restarted the test after adding the needle? Or did you added it via developer mode?

deeplow · 2024-09-16T15:07:36Z

I thought I had restarted it afterwards. But will try again. It for sure wasn't via developer mode. Let's see if it now finds the needle.

marmarek · 2024-09-16T15:48:14Z

I see the issue: you haven't added the securedrop-launcher tag, it only has desktop tag (which shouldn't be there I think). I guess you added it by clicking on an earlier screenshot (you can do that too, but then you need to adjust tags manually, as the default will be about that other screenshot).

deeplow · 2024-09-16T15:52:06Z

OK. Makes sense. I was afraid to create new tags. Where can I edit the needle? Or should I create a new one?

marmarek · 2024-09-16T15:57:58Z

For this one I just edited it manually.
But generally create new one, and don't be afraid about adding tags. In fact, do add more of them :) for SD-specific needles add ENV-securedrop tag (in addition to any others).

marmarek · 2024-11-19T15:29:21Z

tests/securedrop/upload_packages.pm

+        next unless /Template/;
+        s/\|.*//;
+        $fname = $self->save_and_upload_log("qvm-run --no-gui -ap $_ 'rpm -qa; dpkg -l; pacman -Q; true'",
+                "template-$_-packages.txt", ('timeout' => 90, 'failok' => 1));


I'm not a perl expert by any means, but in other places I see a syntax like this:

Suggested change

"template-$_-packages.txt", ('timeout' => 90, 'failok' => 1));

"template-$_-packages.txt", timeout => 90, failok => 1);

So, without extra quotes or parenthesis.

Thanks! I'll try this tomorrow. If you look on the other save_and_upload_log calls (the ones which have the timeout), it does call with {timeout => 90}. The thing here is that save_and_upload_log does not have explicit parameters like timeout and failok but instead a kind of "python dict". I'm really tempted to just converting the function to have explicit args just to make things easier and then I think will work as you suggest.

lib/installedtest.pm

marmarek · 2024-11-19T20:26:52Z

Test died: Can't open file "ulogs/upload_packages-dom0-packages.txt": No such file or directory at /var/lib/openqa/pool/4/openqa-tests-qubesos/tests/securedrop/upload_packages.pm line 32.

Upload failed after all...

Make it reusable, deduplicate the code (it's already in two places). While at it, add support for failok argument and log info if it actually failed to upload.

Diff at openqa webui doesn't show context at all. While it's possible to guess from where a given package is based in the version format, lets make it easier by making it explicit.

marmarek · 2024-11-19T22:59:41Z

Ok, I think I got it working: #27, feel free to merge/rebase/whatever

marmarek · 2024-11-20T12:40:29Z

FYI now that the install job worked, you can click restart on just the second job and not wait for re-install (assuming that install was okay and you don't want to change something at that stage).

marmarek · 2024-11-20T13:40:37Z

The failing assert_screen x11 is part of switching back to the graphical console - it expects no windows open... If SD client autostart is expected, then I suggest making a needle with x11 tag (and maybe also some "securedrop-login" or such, even if unused for now?).

deeplow · 2024-11-20T14:34:09Z

FYI now that the install job worked, you can click restart on just the second job and not wait for re-install (assuming that install was okay and you don't want to change something at that stage).

Thanks! I've been waiting for quite a bit now for the main test to actually pass, so that I can iterate on the second test. That's why it's also failing. I think it's worth dealing with the X11 needle situation in a separate PR, since the most important part is to get this first test across the finish line.

As the order of operations, I would suggest the following:

merging Upload packages #27
then I rebase this PR on top of main
lastly your final review / merge of this branch

Whatever is left to polish can be done in a subsequent PR as you suggested

marmarek · 2024-11-20T14:48:59Z

then I rebase this PR on top of main

you already did ;)

As for merging this PR, I'd prefer it to be green (even if the second job - the actual test - is more or less empty). For example I see the SECUREDROP_TEST variable was set to wrong value (I fixed it in settings now). If I'm correct about the X11 needle situation, it's just a matter of adding it via needle editor.

deeplow · 2024-11-20T16:53:30Z

That was it - it's now passing. I added the x11 tag to a previous needle with the SecureDrop client. I don't know exactly how the needle matching works, but hopefully this doesn't break any other tests 🤞

So I'd say this is ready for merging.

deeplow · 2024-11-20T16:55:20Z

then I rebase this PR on top of main

you already did ;)

I was assuming the break I was proposing would go into #27, but it it can be added on this one, I'm fine as well. Then there's no need to rebase anything.

marmarek · 2025-01-09T15:00:33Z

@deeplow unrelated to this PR, but still about openqa tests:
https://openqa.qubes-os.org/tests/125341#downloads guest-sd-large-bookworm-template.log:

[2025-01-09 09:19:39] [    0.796063] Run /init as init process
[2025-01-09 09:19:39] Loading, please wait...
[2025-01-09 09:19:39] Starting systemd-udevd version 252.31-1~deb12u1

[2025-01-09 09:19:39] [    1.075869] Invalid max_queues (4), will use default max: 2.
[2025-01-09 09:19:40] [    1.157320] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2025-01-09 09:19:40] [    1.180986]  xvda: xvda1 xvda2 xvda3
[2025-01-09 09:19:40] [    1.195462] blkfront: xvdb: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2025-01-09 09:19:40] [    1.211144] blkfront: xvdc: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2025-01-09 09:19:40] Begin: Loading essential drivers ... [    1.497039] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
[2025-01-09 09:19:40] [    1.497183] device-mapper: uevent: version 1.0.3
[2025-01-09 09:19:40] [    1.497935] device-mapper: ioctl: 4.48.0-ioctl (2023-03-01) initialised: dm-devel@redhat.com
[2025-01-09 09:19:40] done.
[2025-01-09 09:19:40] Begin: Running /scripts/init-premount ... done.
[2025-01-09 09:19:40] Begin: Mounting root file system ... Begin: Running /scripts/local-top ... Begin: Waiting for /dev/xvda* devices... ... [2025-01-09 09:19:40] done.
[2025-01-09 09:19:40] Begin: Qubes: Doing R/W setup for TemplateVM... ... [2025-01-09 09:19:40] [    1.903477]  xvdc: xvdc1 xvdc3
[2025-01-09 09:19:40] Setting up swapspace version 1, size = 1073737728 bytes
[2025-01-09 09:19:40] UUID=038db4ec-7460-494c-91b9-1eb34836714b
[2025-01-09 09:19:40] [    1.973225] Adding 1048572k swap on /dev/xvdc1.  Priority:-2 extents:1 across:1048572k SS
[2025-01-09 09:19:41] done.
[2025-01-09 09:19:41] done.
[2025-01-09 09:19:41] Begin: Running /scripts/local-premount ... done.
[2025-01-09 09:19:41] Begin: Waiting for root file system ... [2025-01-09 09:19:42] Begin: Running /scripts/local-block ... done.
[2025-01-09 09:19:43] Begin: Running /scripts/local-block ... done.
[2025-01-09 09:19:44] Begin: Running /scripts/local-block ... done.
[2025-01-09 09:19:45] Begin: Running /scripts/local-block ... done.
[2025-01-09 09:19:46] Begin: Running /scripts/local-block ... done.
[2025-01-09 09:19:47] Begin: Running /scripts/local-block ... done.
...
[2025-01-09 09:20:11] Gave up waiting for root file system device.  Common problems:

Is it something you know about, or maybe need help with? This part is normally quite reliable...

deeplow · 2025-01-10T19:06:51Z

Is it something you know about, or maybe need help with? This part is normally quite reliable...

Thanks for the prompt response, as always. I ended up not having time yesterday, but I was focusing more on first getting the dependent test right. But this issue is something we may need help with if it happens to be too unreliable and we can't figure it out. But I'll let you know.

FYI we also had sys-net failing in a previous run, but there there the xen/console/ logs provide less useful from what I could find.

If any case, I'll let you know if one of these issues is adding too much unreliability. Unless you think it's unwise to take a look at these failures later, of course.

Thanks again.

marmarek · 2025-01-28T12:55:25Z

@deeplow I see you are hitting hard job timeout. Not sure why (it seems that install simply takes that long now? I guess Tor is having bad days...), but you can increase the timeout by setting MAX_JOB_TIME (in seconds).

deeplow · 2025-01-28T13:03:31Z

Thanks! Just started a new one with a longer timeout. I'm going to eventually have to parallelize at least the server installation part. Otherwise it just takes too long.

marmarek · 2025-02-01T21:34:25Z

FYI the fedora-41 update issue you've hit recently is QubesOS/qubes-issues#9744, it's fixed now.

deeplow · 2025-02-03T11:45:55Z

Thank you.

marmarek reviewed Sep 11, 2024

View reviewed changes

tests/securedrop/install.pm Outdated Show resolved Hide resolved

deeplow force-pushed the add-securedrop-test branch 2 times, most recently from 3502fd7 to 2008e3f Compare September 12, 2024 10:54

deeplow force-pushed the add-securedrop-test branch 6 times, most recently from 5c8b79c to 3a2149a Compare September 12, 2024 12:39

deeplow force-pushed the add-securedrop-test branch from 3a2149a to deebce7 Compare September 12, 2024 17:29

deeplow commented Sep 12, 2024

View reviewed changes

deeplow force-pushed the add-securedrop-test branch 8 times, most recently from bf7f90e to fb294a2 Compare September 13, 2024 16:48

deeplow force-pushed the add-securedrop-test branch from fb294a2 to ff78699 Compare September 16, 2024 18:35

deeplow force-pushed the add-securedrop-test branch 3 times, most recently from ce75b93 to 502e47b Compare November 19, 2024 15:20

marmarek reviewed Nov 19, 2024

View reviewed changes

lib/installedtest.pm Show resolved Hide resolved

deeplow force-pushed the add-securedrop-test branch 4 times, most recently from 64e7e92 to bd03839 Compare November 19, 2024 19:16

marmarek added 2 commits November 19, 2024 21:56

Move uploading packages list to a base class

fe89cdf

Make it reusable, deduplicate the code (it's already in two places). While at it, add support for failok argument and log info if it actually failed to upload.

Adjust sut_packages.txt format to prefix each line with VM name

871d08c

Diff at openqa webui doesn't show context at all. While it's possible to guess from where a given package is based in the version format, lets make it easier by making it explicit.

deeplow force-pushed the add-securedrop-test branch from bd03839 to 2b22546 Compare November 20, 2024 10:26

Add SecureDrop installation test

ef7fbce

deeplow force-pushed the add-securedrop-test branch from 2b22546 to ef7fbce Compare November 20, 2024 12:10

marmarek merged commit ef7fbce into QubesOS:main Nov 20, 2024
1 check passed

deeplow deleted the add-securedrop-test branch November 21, 2024 09:53

		assert_and_click("menu-vm-xterm");


		assert_script_run('gpg --keyserver hkps://keys.openpgp.org --recv-key "2359 E653 8C06 13E6 5295 5E6C 188E DD3B 7B22 E6A3"');

	"template-$_-packages.txt", ('timeout' => 90, 'failok' => 1));
	"template-$_-packages.txt", timeout => 90, failok => 1);

Add SecureDrop installation test #25

Add SecureDrop installation test #25

Conversation

deeplow commented Sep 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marmarek commented Sep 12, 2024

deeplow commented Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deeplow Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

marmarek commented Sep 12, 2024

deeplow commented Sep 16, 2024

marmarek commented Sep 16, 2024

deeplow commented Sep 16, 2024

marmarek commented Sep 16, 2024

deeplow commented Sep 16, 2024

marmarek commented Sep 16, 2024 • edited Loading

marmarek Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marmarek commented Nov 19, 2024 • edited Loading

marmarek commented Nov 19, 2024

marmarek commented Nov 20, 2024

marmarek commented Nov 20, 2024

deeplow commented Nov 20, 2024

marmarek commented Nov 20, 2024

deeplow commented Nov 20, 2024

deeplow commented Nov 20, 2024

marmarek commented Jan 9, 2025

deeplow commented Jan 10, 2025

marmarek commented Jan 28, 2025

deeplow commented Jan 28, 2025

marmarek commented Feb 1, 2025

deeplow commented Feb 3, 2025

deeplow commented Sep 12, 2024 •

edited

Loading

deeplow Oct 11, 2024 •

edited

Loading

marmarek commented Sep 16, 2024 •

edited

Loading

marmarek Nov 19, 2024 •

edited

Loading

marmarek commented Nov 19, 2024 •

edited

Loading