Add database migration

[onur]

Closes: #303

This patch adds a database migration and cratesfyi database migrate command. It will allow us to upgrade or downgrade our database to a newer or older version.

By default cratesfyi database migrate command will upgrade database to newest version. It is also possible to use cratesfyi database migrate <VERSION>. Specifying a VERSION will bring database to specified VERSION (upgrade or downgrade if necessary).

Adding a new migration is easy and can be done by adding a new migration to migrations vector with migration macro:

migration!(100,
           "Create test table",
           "CREATE TABLE test ( id SERIAL);",
           "DROP TABLE test;");

I checked bunch of migration libraries and decided to use schemamama. It is super lightweight, extensible and have no dependency, allowing us to use our own database connection and defining our own migration types if we need them in future.

[QuietMisdreavus]

I'm worried whether this will attempt to recreate all the tables if this is used with an existing database (that uses the config table to track its version). Should we include a check to look for a database_version entry in the config table before assuming an empty database?

[QuietMisdreavus]

Thanks so much for writing this up! I really like this. If we can make sure that the cratesfyi database migrate command won't report an error with a pre-schemamama database, i'd love to merge this.

[onur]

Yes I forgot to mention that. schemamama is using it's own table (database_versions) to keep track of applied database migrations. So deploying this will require one last manual action; creating that table by hand and inserting version 1 to table. This will ensure our database is at version 1. Otherwise it will just fail to migrate first version since tables are already exist.

There is one more thing. cratesfyi database migrate <VERSION>, if you try to use version 0, it will bring database to a fresh state (it will remove everything). This is actually useful for development I don't think we can accidentally use version 0 in with this command. So we should be fine.

We can also split database queries and store in another location (but I don't think it's necessary). Also instead of using incremental values for database version we can use epoch (I don't think this is necessary or useful either).

[QuietMisdreavus]

So deploying this will require one last manual action; creating that table by hand and inserting version 1 to table.

Hmm, i'm nervous about this, but i guess it would be alright if we had a script for it. Something like this?

-- these queries taken from `schemamama_postgres`'s implementation
CREATE TABLE IF NOT EXISTS database_versions (version BIGINT PRIMARY KEY);
INSERT INTO database_versions (version) VALUES (1);

-- clean up the old version value
DELETE FROM config WHERE name = 'database_version';

We can also split database queries and store in another location (but I don't think it's necessary). Also instead of using incremental values for database version we can use epoch (I don't think this is necessary or useful either).

I agree, i don't think there's much value to these. Having the queries next to their version number and description is helpful, and i feel like separating them out would just create more line count with nothing to gain for it. As for the version numbers, i'm fine with just having an incremental identifier - it plays nice with the library we're using, and doing anything on top of that feels like it would complicate things.

[onur]

Something like this?

Exactly like that!

[QuietMisdreavus]

Coming back to this PR, there are still some problems:

• There's now a merge conflict in the lock file.
• We should update the Vagrantfile with the new database command.
• I'm still hesitant to merge this without some means of detection for existing databases without the database_versions table. If there's a three-line script to run that will initialize the table properly, why don't we include that in our code? We can add an extra read of the config table if we don't detect a current version, to see whether we're using an existing database, and assume an empty database if the database_version entry (or the table) doesn't exist. That way, we don't have to add the manual migration step.

[onur]

Fixed merge conflict, Vagrantfile was already updated.

I'm still hesitant to merge this without some means of detection for existing databases without the database_versions table. If there's a three-line script to run that will initialize the table properly, why don't we include that in our code? We can add an extra read of the config table if we don't detect a current version, to see whether we're using an existing database, and assume an empty database if the database_version entry (or the table) doesn't exist. That way, we don't have to add the manual migration step.

Schemama is already doing this, schemama is assuming there is no migration installed in database if there is no database_versions and it's creating this table once we migrate our first version. We have been upgrading our database manually until now and we will only need to run one last query manually in order to entirely automatize this process. I am not sure if I understand the problem or your concern here.

[QuietMisdreavus]

We have been upgrading our database manually until now and we will only need to run one last query manually in order to entirely automatize this process.

I guess looking at it like this, it's fine. One last breaking change before stability, so to speak, like when a Rust feature is stabilized. In that case, I'd like to make sure you're the one to deploy this, since you're more comfortable with making changes to the database. Otherwise, this PR is good to go.

[onur]

@QuietMisdreavus successfully deployed (created database_versions table and added our first version: 1, and tested migrate command). I also added cratesfyi database migrate command to our build script (update-docs.rs), so from now on, we only need to add new migrations to migrate.rs whenever we need to change our database schema.