[xenial] Ensures config test coverage for Xenial #4099

conorsch · 2019-02-05T01:37:45Z

Status

Ready for review.

Description of Changes

Fixes #3964.

Changes proposed in this pull request:

Ensures testinfra coverage for staging environment against both Trusty & Xenial
Updates the testinfra syntax throughout to use the host pragma, due to deprecation warnings
Small tweak to dev docs on Xenial support

Testing

make build-debs passes without error (caveat: a single failing test for unapplied updates is OK)
make build-debs-xenial passes without error (caveat: a single failing test for unapplied updates is OK)
make staging && molecule verify -s libvirt-staging passes without error
make staging-xenial && molecule verify -s libvirt-staging-xenial passes without error

There's still a bit of branching logic for Trusty/Xenial support, which can't be avoided, but tried to keep it clear and maintainable. Most of the diff is the new-ish host syntax for testinfra, which was easy enough to slot in here given that I was reading through all the files anyway.

The divergence for unconfined AppArmor processes isn't as clearly documented as it should be—see #4098 for tracking follow-up.

Deployment

No, these changes are dev-env & CI only.

Checklist

If you made changes to the server application code:

Linting (make ci-lint) and tests (make -C securedrop test) pass in the development container

If you made changes to `securedrop-admin`:

Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

Configuration tests pass

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

If you made changes to documentation:

Doc linting (make docs-lint) passed locally

kushaldas

make build-debs passes without error (caveat: a single failing test for unapplied updates is OK)
make build-debs-xenial passes without error (caveat: a single failing test for unapplied updates is OK)
make staging && molecule verify -s libvirt-staging passes without error 🔥

The third test is giving the following error.

    -- generated xml file: /home/kdas/code/securedrop/junit/testinfra-results.xml --
    =================================== FAILURES ===================================
    ________________ test_mon_iptables_rules[ansible://mon-staging] ________________
    [gw3] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    
    host = <testinfra.host.Host object at 0x7f15d41b1450>
    
        def test_mon_iptables_rules(host):
        
            # Build a dict of variables to pass to jinja for iptables comparison
            kwargs = dict(
                app_ip=os.environ.get('APP_IP', securedrop_test_vars.app_ip),
    >           default_interface=Command.check_output(
                    "ip r | head -n 1 | awk '{ print $5 }'"),
                tor_user_id=Command.check_output("id -u debian-tor"),
                ssh_group_gid=Command.check_output("getent group ssh | cut -d: -f3"),
                postfix_user_id=Command.check_output("id -u postfix"),
                dns_server=securedrop_test_vars.dns_server)
    E       NameError: global name 'Command' is not defined
    
    ../testinfra/staging/mon/test_mon_network.py:17: NameError
    ________________ test_app_iptables_rules[ansible://app-staging] ________________
    [gw3] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    
    host = <testinfra.host.Host object at 0x7f15d6da6f90>
    
        def test_app_iptables_rules(host):
        
            # Build a dict of variables to pass to jinja for iptables comparison
            kwargs = dict(
                mon_ip=os.environ.get('MON_IP', securedrop_test_vars.mon_ip),
    >           default_interface=Command.check_output("ip r | head -n 1 | "
                                                       "awk '{ print $5 }'"),
                tor_user_id=Command.check_output("id -u debian-tor"),
                securedrop_user_id=Command.check_output("id -u www-data"),
                ssh_group_gid=Command.check_output("getent group ssh | cut -d: -f3"),
                dns_server=securedrop_test_vars.dns_server)
    E       NameError: global name 'Command' is not defined
    
    ../testinfra/staging/app/test_app_network.py:17: NameError
    === 2 failed, 444 passed, 13 skipped, 8 xfailed, 2 xpassed in 227.74 seconds ===

conorsch · 2019-02-06T17:38:32Z

@kushaldas Changes addressed, please have another look!

kushaldas · 2019-02-07T09:25:52Z

I suddenly got the following for libvirt-staging-xenial, redoing all the steps once more.

    
    =================================== FAILURES ===================================
    __________ test_pax_flags[ansible://app-staging-/usr/sbin/grub-probe] __________
    [gw2] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://app-staging-/usr/sbin/grub-mkdevicemap] _______
    [gw2] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://app-staging-/usr/bin/grub-script-check] _______
    [gw2] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://mon-staging-/usr/sbin/grub-mkdevicemap] _______
    [gw2] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    __________ test_pax_flags[ansible://mon-staging-/usr/sbin/grub-probe] __________
    [gw0] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://mon-staging-/usr/bin/grub-script-check] _______
    [gw1] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    === 6 failed, 446 passed, 13 skipped, 2 xfailed, 2 xpassed in 264.63 seconds ===

kushaldas

make build-debs passes without error (caveat: a single failing test for unapplied updates is OK)
make build-debs-xenial passes without error (caveat: a single failing test for unapplied updates is OK)
make staging && molecule verify -s libvirt-staging passes without error
🔥 🔥 🔥 🔥 🔥 make staging-xenial && molecule verify -s libvirt-staging-xenial fails with the following errors:

    =================================== FAILURES ===================================
    __________ test_pax_flags[ansible://app-staging-/usr/sbin/grub-probe] __________
    [gw1] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://app-staging-/usr/sbin/grub-mkdevicemap] _______
    [gw1] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://app-staging-/usr/bin/grub-script-check] _______
    [gw1] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    __________ test_pax_flags[ansible://mon-staging-/usr/sbin/grub-probe] __________
    [gw3] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://mon-staging-/usr/sbin/grub-mkdevicemap] _______
    [gw1] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://mon-staging-/usr/bin/grub-script-check] _______
    [gw2] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    === 6 failed, 446 passed, 13 skipped, 2 xfailed, 2 xpassed in 328.45 seconds ===

conorsch · 2019-02-08T01:12:22Z

Had some trouble reproducing @kushaldas's findings, but finally narrowed it down to base box version. In short, newer base boxes show the problem, whereas older don't. That kind of variation isn't useful in the config tests, so I've removed the strict=True flag on the PaX tests to resolve. More detail can be found in #3916.

redshiftzero · 2019-02-08T18:58:59Z

Looks like there is a linting failure - @conorsch want to fix and repush when you get a chance? then @kushaldas can a final spin and approve if all else looks good

Using wrapper functions to call discrete checks depending on platform, which is always either Trusty or Xenial. Most of the branching logic was already present, just cleaning it up a bit in order to make the functions more readable. Added a systemd-style check for xvfb under Xenial, since we weren't inspecting that config on disk. Accordingly, updates the dev docs to state that we've handled the config tests.

The old sytanx of automatic module imports as part of the test function args has been deprecated since testinfra v1.6.0 (2017-04-21). It's still working, but planned to be dropped as of testinfra v2.0. Took the opportunity to convert our checks throughout.

The "strict=true" pragma on pytest xfail means that the config test suite will exit non-zero if the tests pass. We added strict=True in order to understand better the problem of PaX flags not matching, and it appears we've confirmed that the age of installed packages is sufficient determinant. That is, if apt such as grub are upgraded as part of the install, then PaX flags set by the grsecurity role will be clobbered very quickly.

As I reported the issue, there is now an upstream PR opened python/typeshed#2787 We can remove these two lines after a new typeshed release.

kushaldas · 2019-02-11T11:43:50Z

Just now pushed a commit in this branch to fix the mypy issue. There is already an upstream PR python/typeshed#2787 opened to fix the issue. I will approve the PR after the CI passes.

kushaldas · 2019-02-11T12:31:17Z

Kicking the updater-gui-tests again.

kushaldas

make build-debs passes without error (caveat: a single failing test for unapplied updates is OK)
make build-debs-xenial passes without error (caveat: a single failing test for unapplied updates is OK)
make staging && molecule verify -s libvirt-staging passes without error
make staging-xenial && molecule verify -s libvirt-staging-xenial without error

This is now good. Approved. 👍

conorsch requested review from kushaldas, zenmonkeykstop and emkll February 5, 2019 01:37

conorsch requested review from msheiny and redshiftzero as code owners February 5, 2019 01:37

conorsch force-pushed the 3964-testinfra-for-xenial branch from f94c462 to 66ac605 Compare February 5, 2019 17:59

kushaldas suggested changes Feb 6, 2019

View reviewed changes

kushaldas suggested changes Feb 7, 2019

View reviewed changes

conorsch mentioned this pull request Feb 8, 2019

PaX flags are unset during install time #3916

Closed

Conor Schaefer added 3 commits February 8, 2019 11:33

conorsch force-pushed the 3964-testinfra-for-xenial branch from cbd810d to 948d7e9 Compare February 8, 2019 19:34

Fixes mypy errors with type: ignore

9d06704

As I reported the issue, there is now an upstream PR opened python/typeshed#2787 We can remove these two lines after a new typeshed release.

conorsch requested a review from heartsucker as a code owner February 11, 2019 11:42

kushaldas approved these changes Feb 11, 2019

View reviewed changes

kushaldas merged commit 8339e32 into develop Feb 11, 2019

kushaldas deleted the 3964-testinfra-for-xenial branch February 11, 2019 13:34

eloquence changed the title ~~Ensures config test coverage for Xenial~~ [xenial] Ensures config test coverage for Xenial Feb 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[xenial] Ensures config test coverage for Xenial #4099

[xenial] Ensures config test coverage for Xenial #4099

conorsch commented Feb 5, 2019

kushaldas left a comment

conorsch commented Feb 6, 2019

kushaldas commented Feb 7, 2019

kushaldas left a comment

conorsch commented Feb 8, 2019

redshiftzero commented Feb 8, 2019

kushaldas commented Feb 11, 2019

kushaldas commented Feb 11, 2019

kushaldas left a comment

[xenial] Ensures config test coverage for Xenial #4099

[xenial] Ensures config test coverage for Xenial #4099

Conversation

conorsch commented Feb 5, 2019

Status

Description of Changes

Testing

Deployment

Checklist

If you made changes to the server application code:

If you made changes to securedrop-admin:

If you made changes to the system configuration:

If you made non-trivial code changes:

If you made changes to documentation:

kushaldas left a comment

Choose a reason for hiding this comment

conorsch commented Feb 6, 2019

kushaldas commented Feb 7, 2019

kushaldas left a comment

Choose a reason for hiding this comment

conorsch commented Feb 8, 2019

redshiftzero commented Feb 8, 2019

kushaldas commented Feb 11, 2019

kushaldas commented Feb 11, 2019

kushaldas left a comment

Choose a reason for hiding this comment

If you made changes to `securedrop-admin`: