Moves securedrop-admin into Python 3 #4867

kushaldas · 2019-09-25T16:40:19Z

Status

Work in progress

Description of Changes

Fixes #3489

This now needs an update of prompt_toolkit so that we can use it in Python3.5.

Testing

In both Tails3, and Tails4, execute the following scenarios with prod vms/hardware:

Deployment

Any special considerations for deployment? Consider both:

Upgrading existing production instances.
New installs.

Checklist

If you made changes to the server application code:

Linting (make lint) and tests (make test) pass in the development container

If you made changes to `securedrop-admin`:

Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

Configuration tests pass

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

If you made changes to documentation:

Doc linting (make docs-lint) passed locally

redshiftzero · 2019-09-26T11:36:12Z

rebase issue?

kushaldas · 2019-09-26T13:31:52Z

There can be two different ways to migrate the virtualenv for the admins.

After checking out 1.1.0, if the admins run ./securedrop-admin install, it will fail and ask to rerun securedrop-admin setup first. This is the (current implementation) and seems to be simpler. We can decide what kind of message to tell the admins about why should they run the setup command again.
The other option is when ./securedrop-admin install will see that there is a Python2 venv exists, and it will automatically do the setup work for Python3 venv.

I am going to test the first item in a vm now.

kushaldas · 2019-09-26T17:49:12Z

_./securedrop-admin install fails with following:


TASK [validate : Confirm host OS is Tails.] ************************************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'ansible_lsb.major_release >= 9' failed. The error was: Unexpected templating type error occurred on ({% if ansible_lsb.major_release >= 9 %} True {% else %} False {% endif %}): unorderable types: AnsibleUnsafeText() >= int()"}

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************
	to retry, use: --limit @/home/amnesia/Persistent/securedrop/install_files/ansible-base/securedrop-prod.retry

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1

Probable reason: Old version of Ansible using Python3 in the controller (Tails)

conorsch · 2019-09-26T23:33:24Z

@kushaldas I've appended a commit that resolves a few of the Ansible-related Python 3 issues. Tried most (but not all) of the test scenarios you outlined, and they've all been successful with the changes in place in Tails. For the record, I tested on Tails 3.16, with Python 3.5. Noticed some "weakref" stderr output in a few places, which is a significant cosmetic issue, but should not be a factor on Tails 4 / Python 3.7.

Not approving, but I believe you should be unblocked for more vigorous testing now. Take another lap through it and see if you can identify any lingering Python 2 references that we haven't caught yet. When you're happy with it, I'd suggest we have @rmol and/or @zenmonkeykstop dig in for comprehensive review.

kushaldas · 2019-09-27T06:37:44Z

@rmol @zenmonkeykstop please go ahead with full scale review. This worked in my local testing.

conorsch · 2019-09-27T20:40:42Z

#4865 was just merged, introducing conflicts here. We choose to merge #4865 first to minimize the manual conflict resolution. @rmol / @zenmonkeykstop please give a rebase here, holler in gitter with any blockers!

zenmonkeykstop · 2019-09-27T21:21:22Z

On Tails 4.0~beta2 VM, checked out this branch and verified:

./securedrop-admin setup
./securedrop-admin sdconfig
./securedrop-admin install _(all good except for unmerged fixes in Enables Source and Journalist desktop icons in Tails 4 #4872)
./securedrop-admin tailsconfig
./securedrop-admin backup
./securedrop-admin restore
./securedrop-admin update
./securedrop-admin logs
./securedrop-admin reset_admin_access

rmol · 2019-09-27T22:11:20Z

Tested on Tails 3.16 and prod VMs:

./securedrop-admin sdconfig
./securedrop-admin install
./securedrop-admin tailsconfig
./securedrop-admin backup
./securedrop-admin restore
./securedrop-admin update
./securedrop-admin logs
./securedrop-admin reset_admin_access

This will inform the admins to rerun ./securedrop-admin setup as this will create .venv3 dirctory for Python3 dependencies.

As Tor4 has Python 3.7.3, we should just mention without any particular version.

A smattering of the familiar Python 2 -> 3 changes, mostly around string handling, filepaths to interpreters and virtualenvs, and syntax updates. Specifically: * coerces major release number to int * converts iteritems() -> items() in dynamic inventory * updates string handling in dynamic inventory * ensures securedrop_init runs under python3 Tested under Tails 3.16 (based on Stretch), running Python 3.5. Worked well, after these fixes landed, although I've observed a handful of the "weakref" stderr output. The weakref output doesn't break anything, but it may be alarming to Admins who haven't updated to Tails 4 & Python 3.7 yet.

conorsch · 2019-09-27T22:23:46Z

Rebased to resolve conflicts introduced via merge of #4865. The sole conflict was:

diff --cc admin/requirements.in
index 3e7fda7e0,2eebee385..000000000
--- a/admin/requirements.in
+++ b/admin/requirements.in
@@@ -1,2 -1,2 +1,7 @@@
++<<<<<<< HEAD
 +prompt_toolkit
 +pyyaml>=5.1.2
++=======
+ prompt_toolkit==2.0.9
+ pyyaml
++>>>>>>> 3a15f0e13... Moves securedrop-admin to Python3

which was manually solved as:

$ cat admin/requirements.in
prompt_toolkit==2.0.9
pyyaml>=5.1.2

zenmonkeykstop · 2019-09-27T22:59:40Z

On Tails 3.15 VM, checked out the rebased branch and verified:

LGTM!

zenmonkeykstop

LGTM based on test plan.

emkll · 2019-10-07T18:29:59Z

While testing #4877 ran into an issue that may be worth flagging: we should ensure the 1.0.0 -> 1.1.0 docs have admins run ./securedrop-admin setup as part of the upgrade process to 1.1.0, as the SecureDrop GUI updater will not run if the checked out branch is python3 . This will affect the version after SecureDrop 1.1.0, if a user manually checks out a 1.1.0 + branch and does not run ./securedrop-admin setup

redshiftzero · 2019-10-08T14:18:13Z

admin/requirements.txt

-    --hash=sha256:1df952620eccb399c53ebb359cc7d9a8d3a9538cb34c5a1344bdbeb29fbcc381 \
-    --hash=sha256:3f473ae040ddaa52b52f97f6b4a493cfa9f5920c255a12dc56a7d34397a398a4 \
-    --hash=sha256:858588f1983ca497f1cf4ffde01d978a3ea02b01c8a26a8bbc5cd2e66d816917
+prompt_toolkit==2.0.9 \


was diff review performed here?

Diff review has been performed (timeboxed to ~4 hr): https://github.com/freedomofpress/securedrop-debian-packaging/wiki/prompt_toolkit-1.0.15--2.0.9

kushaldas force-pushed the admins_love_python3 branch from 459f7f6 to 3a15c7a Compare September 26, 2019 13:28

kushaldas force-pushed the admins_love_python3 branch 2 times, most recently from c2f930e to 3a15f0e Compare September 26, 2019 15:54

kushaldas marked this pull request as ready for review September 26, 2019 17:01

kushaldas requested review from emkll, heartsucker, redshiftzero and zenmonkeykstop as code owners September 26, 2019 17:01

kushaldas requested a review from conorsch as a code owner September 26, 2019 23:29

kushaldas and others added 3 commits September 27, 2019 15:21

Moves securedrop-admin to Python3

30940ab

This will inform the admins to rerun ./securedrop-admin setup as this will create .venv3 dirctory for Python3 dependencies.

Works for both Tor4 and Tor3

2f1b44e

As Tor4 has Python 3.7.3, we should just mention without any particular version.

conorsch force-pushed the admins_love_python3 branch from 6266b56 to bc898e3 Compare September 27, 2019 22:22

zenmonkeykstop approved these changes Sep 28, 2019

View reviewed changes

zenmonkeykstop merged commit 6cf415e into develop Oct 7, 2019

emkll deleted the admins_love_python3 branch October 7, 2019 18:30

redshiftzero reviewed Oct 8, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Moves securedrop-admin into Python 3 #4867

Moves securedrop-admin into Python 3 #4867

kushaldas commented Sep 25, 2019 •

edited

Loading

redshiftzero commented Sep 26, 2019

kushaldas commented Sep 26, 2019 •

edited

Loading

kushaldas commented Sep 26, 2019 •

edited

Loading

conorsch commented Sep 26, 2019

kushaldas commented Sep 27, 2019

conorsch commented Sep 27, 2019

zenmonkeykstop commented Sep 27, 2019

rmol commented Sep 27, 2019 •

edited

Loading

conorsch commented Sep 27, 2019

zenmonkeykstop commented Sep 27, 2019

zenmonkeykstop left a comment

emkll commented Oct 7, 2019 •

edited

Loading

redshiftzero Oct 8, 2019

redshiftzero Oct 9, 2019

Moves securedrop-admin into Python 3 #4867

Moves securedrop-admin into Python 3 #4867

Conversation

kushaldas commented Sep 25, 2019 • edited Loading

Status

Description of Changes

Testing

Deployment

Checklist

If you made changes to the server application code:

If you made changes to securedrop-admin:

If you made changes to the system configuration:

If you made non-trivial code changes:

If you made changes to documentation:

redshiftzero commented Sep 26, 2019

kushaldas commented Sep 26, 2019 • edited Loading

kushaldas commented Sep 26, 2019 • edited Loading

conorsch commented Sep 26, 2019

kushaldas commented Sep 27, 2019

conorsch commented Sep 27, 2019

zenmonkeykstop commented Sep 27, 2019

rmol commented Sep 27, 2019 • edited Loading

conorsch commented Sep 27, 2019

zenmonkeykstop commented Sep 27, 2019

zenmonkeykstop left a comment

Choose a reason for hiding this comment

emkll commented Oct 7, 2019 • edited Loading

redshiftzero Oct 8, 2019

Choose a reason for hiding this comment

redshiftzero Oct 9, 2019

Choose a reason for hiding this comment

kushaldas commented Sep 25, 2019 •

edited

Loading

If you made changes to `securedrop-admin`:

kushaldas commented Sep 26, 2019 •

edited

Loading

kushaldas commented Sep 26, 2019 •

edited

Loading

rmol commented Sep 27, 2019 •

edited

Loading

emkll commented Oct 7, 2019 •

edited

Loading