Implement versioning for migrations

[eloquence]

The preflight updater currently has a very limited mechanism for triggering a full sdw-admin --apply run (i.e. enforcing all Salt states and not just the ones specified for dom0 via https://github.com/freedomofpress/securedrop-workstation/blob/main/dom0/sd-workstation.top#L5-L21).

That mechanism works as follows:

• The check-migration script performs feature tests to drop migration flag in /tmp/sdw-migrations
• The preflight updater checks for that flag and calls the run_full_install method if it is found.

In the case of #661, the drawbacks of that approach became apparent:

• Because we don't want to run feature detection code within an AppVM every time the updater runs, we ended up dropping the flag in postinst, but
• That means the flag will be lost (because it's in /tmp) if the user reboots after a failed update that successfully upgraded the RPM and the state of the system is not necessarily assured;
• Any fresh install from that RPM version will have an unnecessary migration flag present immediately after the RPM install;
• If users skip this RPM version, they'll never get the migration at all.

In short, feature detection alone is insufficient to handle all migration types, and postinst-type hacks are hackish.

As we do with Alembic-managed database migrations, we need a way to version migrations and the state of the system, so we can apply the migrations that are required. The scope of this issue is:

1. Propose a lightweight implementation approach for versioning migrations (i.e. upgrades that require an sdw-admin --apply run);
2. If there is consensus, implement said approach.

[eaon]

The extraction of the updater from the workstation repository turns out to require a couple of small migrations to ensure we don't leave old files laying around. We used this as an opportunity to start thinking about how the goals described above can be implemented in sensible way.

The approach we've settled on so far is described as follows:

• Use as much standard packaging tooling as possible to allow us to be independent of the existing updater migration code
• Ship Python/Shell scripts or salt states that have a similar approach to SQLAlchemy's Alembic, except use version numbers of the respective package to decide which files to execute.
  • Example: we're updating from 0.7.0 to 0.8.0, so we'll run migrations for 0.7.1, 0.7.2 and finally 0.8.0
• Hence, use %pre, %post (and possibly %triggerin) scriptlets as follows:
  • Use %pre exclusively to help %post with knowing which version of package we're upgrading from by creating a temporary file with the version number to be read later by %post
  • %post reads the version file left by %pre, checks which migrations are available and figures out the delta (see example from above) and runs the respective scripts/states

This will be implemented as a kind of test-balloon for securedrop-updater (to be reviewed / doula'd by @cfm) and if this approach works well, is intended to be subsequently ported to securedrop-workstation.

[zenmonkeykstop]

This is still an open issue - for now the updater has been merged back to securedrop-workstation without the initial versioned migration code, but we can pull it back in if necessary.

[cfm]

At this year's Qubes OS Summit, @marmarek suggested this as a specific application of #1178/#1053. With a version or generation pillar attribute per VM and dom0, a scripted qubesctl invocation could perform upgrade/downgrade operations via Salt states parameterized on that attribute.

The philosophical question still worth considering is how much we gain by mixing imperative and declarative approaches like this, especially as we get more interested in enforcing and verifying system state, e.g. in #951.