[xenial] Perform timeboxed install attempt of SecureDrop against Ubuntu 16.04

[eloquence]

As part of the #3204 (xenial epic), once we have Xenial staging VMs (#3206), we should be able to attempt an install of SecureDrop against 16.04 VMs. This task is to perform a strictly timeboxed install attempt and to record some first observations.

[ghost]

https://github.com/dachary/securedrop/tree/wip-dachary-xenial are the changes I did to get SecureDrop to run staging using Xenial app/mon base servers instead of Trusty. It was a while back and I only remember that I got it running. There may have been undocumented manual tweaks as well. I remember spending less than a day working on that, most of it dealing with firefox / selenium. I was then able to run the tests successfully.

[eloquence]

For the purpose of the 5/30-6/13 sprint, we've agreed to decouple this from the other tasks in #3204; that is, the objective is to perform a from-scratch install with Xenial similar to what @dachary describes above, without this being necessarily easily repeatable. This is timeboxed to approximately one working day.

We want to gather information that will inform the architecture meeting on June 14 about the operating system transition. What level of effort is likely to be involved in fixing Xenial-specific issues and upgrading existing SecureDrop installs? Based on the answer to that question, we can decide if a transition to an altogether different base OS (e.g., Atomic, Ubuntu Core) could be a reasonable alternative.

[msheiny]

Install attempt notes

Still in progress... will also post a block of code tweaks I had to make in ansible and friends at the end.

Ansible logic

• Problem with reboot task it just bombs out in a really strange way.. havent dug deep on it yet fixed.

Apt

• Date field is now required in apt repositories and if not found will throw
  warnings.

  • https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1649086
  • https://wiki.debian.org/DebianRepository/Format?action=show&redirect=RepositoryFormat

• Our FPF repos are currently locked to the trusty distro target which shouldn't break upgrades but will break on new installs since the distro target is determined dynamically from ansible vars.

• Following packages need to be explicitly installed in order to allow our ansible logic to run without without breakage: aptitude, ntpdate, python-pip

apache2

• apache2-mpm-worker no longer in xenial. Depedency of securedrop-app pkg. https://askubuntu.com/questions/808556/apache2-mpm-worker-missing-in-ubuntu-16-04

• Can not enable both mpm_worker and mpm_event modules at same time. Ended up ditching mpm_worker in favor of mpm_event. See comments in Xenial branch

• apache2 needs to be explicitly installed

Firewall

• Need to add _apt user to list of users whom can perform DNS lookups and outbound TCP on web ports (thanks @conorsch ;) )

Apparmor

• explicit rules needed for mpm_worker to even start up apache
• upon source submit get a denial for link (l) flag for gpg2 at /var/lib/securedrop/keys/*

Other logs to investigate

• ntp errors:

Jun  1 19:15:02 app-staging ntpd[5255]: error resolving pool 2.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Jun  1 19:15:03 app-staging ntpd[5255]: error resolving pool 0.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Jun  1 19:15:04 app-staging ntpd[5255]: error resolving pool 1.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Jun  1 19:15:06 app-staging ntpd[5255]: error resolving pool 3.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Jun  1 19:15:09 app-staging ntpd[5255]: error resolving pool ntp.ubuntu.com: Temporary failure in name resolution (-3)
Jun  1 19:16:08 app-staging ntpd[5255]: error resolving pool 2.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Jun  1 19:16:10 app-staging ntpd[5255]: error resolving pool 1.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Jun  1 19:16:10 app-staging ntpd[5255]: error resolving pool 0.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Jun  1 19:16:12 app-staging ntpd[5255]: error resolving pool 3.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Jun  1 19:16:15 app-staging ntpd[5255]: error resolving pool ntp.ubuntu.com: Temporary failure in name resolution (-3)

• The png logos werent showing up under Chromium (on the source and journalist interface) ... i was perplexed and didnt see any apparmor errors :| Didn't drill deep into it yet

[msheiny]

See also this branch for a few tweaks that I needed to make to get the playbooks to complete on Xenial. Note that I stripped out the test dependencies completely and did not attempt to try running the application tests.

[conorsch]

ntp errors

Try updating the firewall rules to permit _apt instead of root on the apt calls. That will at least allow sudo apt-get update to resolve DNS, which was how I tested it in #3491.

[msheiny]

im timeboxing the passing of the tests but its mostly firefox/selenium related and some integration stuff. im sure im doing something dumb. attached are the failing 31 tests.

test-failures.txt

[eloquence]

Since this was a timeboxed research task, if there are no objections, I suggest adding any final notes and closing this task before the beginning of the next sprint.

[msheiny]

The install worked. There were some failing tests, some dependency changes but overall not too bad.