Use flask-migrate for database migrations (#2432)

With this flask-migrate is used on application setup to automatically
migrate the configured database to the latest revision shipped with
MSColab. This will make manual migrations in production environments
unnecessary in the future.

Also included is an upgrade path from databases which were previously
manually managed. This entails generating a new database and copying
over all data.

docs/development.rst

@@ -399,6 +399,26 @@ like e.g. a running MSColab server or a QApplication instance for GUI tests,
 are collected in :mod:`tests.fixtures` in the form of pytest fixtures that can be requested as needed in tests.
 
 
+Changing the database model
+---------------------------
+
+Changing the database model requires adding a corresponding migration script to MSS,
+so that existing databases can be migrated automatically.
+
+To generate such a migration script you can run::
+
+  flask --app mslib.mscolab.app db migrate -d mslib/mscolab/migrations -m "To version <next-major-version>"
+
+Depending on the complexity of the changes that were made,
+the generated migration script might need some tweaking.
+
+If there is already a migration script for the next release,
+then please incorporate the generated migration script into this existing one,
+instead of adding a new one.
+You can still generate a script with the above command first
+to get a starting point for the changes.
+
+
 Pushing your changes
 --------------------
 

docs/mscolab.rst

@@ -150,75 +150,24 @@ by `pg_dump <https://www.postgresql.org/docs/current/app-pgdump.html>`_ using a
     pg_dump -d mscolab -f "/home/mscolab/dump/$timestamp.sql"
 
 
-Data Base Migration from version 8
-..................................
-
-.. important::
-   This manual migration on the server side by a user is deprecated and will become removed with version 10.0.0.
-   With version 10.0.0, the initialization of the database will be refactored and migrations will be performed automatically when mscolab is started
-
-For an easy way to update the database scheme we implemented  `flask migrate <https://flask-migrate.readthedocs.io/en/latest/>`_.
-
-You have to create based on your configuration a migration script and call that afterwards. ::
-
-    mamba activate instance
-    cd ~/INSTANCE/config
-    export PYTHONPATH=`pwd`
-    cd ~/INSTANCE/wsgi
-    flask --app mscolab.py db init
-    flask --app mscolab.py db migrate -m "To version 9.0.0"
-    flask --app mscolab.py db upgrade
-
-The migration script builder does the base but the created script needs first allow nullable so that we afterwards set the default for the existing data.
-Use this as an example for your script ::
-
-    """To version 9.0.0
-
-    Revision ID: e62d08ce88a4
-    Revises: 27a026b0daec
-    Create Date: 2024-06-07 16:53:43.314338
-
-    """
-    from alembic import op
-    import sqlalchemy as sa
-
-
-    # revision identifiers, used by Alembic.
-    revision = 'e62d08ce88a4'
-    down_revision = '27a026b0daec'
-    branch_labels = None
-    depends_on = None
-
-
-    def upgrade():
-        # ### commands auto generated by Alembic - please adjust! ###
-        with op.batch_alter_table('users', schema=None) as batch_op:
-            batch_op.add_column(sa.Column('authentication_backend', sa.String(length=255), nullable=True))
-        op.execute('UPDATE users SET authentication_backend = \'local\' WHERE authentication_backend IS NULL')
-
-        with op.batch_alter_table('users', schema=None) as batch_op:
-            batch_op.alter_column('authentication_backend', existing_type=sa.String(length=255), nullable=False)
-            batch_op.drop_constraint('users_password_key', type_='unique')
-
-        # ### end Alembic commands ###
-
-
-    def downgrade():
-        # ### commands auto generated by Alembic - please adjust! ###
-        with op.batch_alter_table('users', schema=None) as batch_op:
-            batch_op.create_unique_constraint('users_password_key', ['password'])
-            batch_op.drop_column('authentication_backend')
-
-        # ### end Alembic commands ###
-
-
-The output looks like ::
-
-    ~/INSTANCE/wsgi$ flask --app mscolab.py db upgrade
-    INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
-    INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
-    INFO  [alembic.runtime.migration] Running upgrade  -> e62d08ce88a, To version 9.0.0
-
+Database Migration from Version 8 or 9
+.................................
+
+From v10 onwards MSColab uses `Flask-Migrate <https://flask-migrate.readthedocs.io/en/latest/>` to automatically deal with database migrations.
+To upgrade from v8 or v9 a recreation of the database and subsequent copy of existing data is necessary.
+To do this follow these steps:
+
+#. Stop MSColab completely, no process interacting with the MSColab database should remain running
+#. **Make a backup of your existing database**
+#. Set ``SQLALCHEMY_DB_URI_TO_MIGRATE_FROM`` to your existing database
+#. Set ``SQLALCHEMY_DB_URI`` to a new database
+#. If you are not using SQLite: create the new database
+#. Start MSColab
+#. Check that everything was migrated successfully
+#. Unset ``SQLALCHEMY_DB_URI_TO_MIGRATE_FROM``
+
+If you want to keep using your old database URI you can first rename your existing database so that it has a different URI
+and just set ``SQLALCHEMY_DB_URI_TO_MIGRATE_FROM`` to that.
 
 
 Steps to use the MSColab UI features

mslib/mscolab/app/__init__.py

@@ -25,6 +25,7 @@
 """
 
 import os
+import sqlalchemy
 
 from flask_migrate import Migrate
 
@@ -62,8 +63,23 @@
 APP.config['MAIL_USE_TLS'] = getattr(mscolab_settings, "MAIL_USE_TLS", None)
 APP.config['MAIL_USE_SSL'] = getattr(mscolab_settings, "MAIL_USE_SSL", None)
 
-db = SQLAlchemy(APP)
-migrate = Migrate(APP, db, render_as_batch=True)
+db = SQLAlchemy(
+    metadata=sqlalchemy.MetaData(
+        naming_convention={
+            # For reference: https://alembic.sqlalchemy.org/en/latest/naming.html#the-importance-of-naming-constraints
+            "ix": "ix_%(column_0_label)s",
+            "uq": "uq_%(table_name)s_%(column_0_name)s",
+            "ck": "ck_%(table_name)s_`%(constraint_name)s`",
+            "fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
+            "pk": "pk_%(table_name)s",
+        },
+    ),
+)
+db.init_app(APP)
+import mslib.mscolab.models
+
+migrate = Migrate(render_as_batch=True, user_module_prefix="cu.")
+migrate.init_app(APP, db)
 
 
 def get_topmenu():

mslib/mscolab/conf.py

@@ -63,6 +63,9 @@ class default_mscolab_settings:
     # MYSQL CONNECTION STRING: "mysql+pymysql://<username>:<password>@<host>:<port>/<db_name>?charset=utf8mb4"
     SQLALCHEMY_DB_URI = 'sqlite:///' + os.path.join(DATA_DIR, 'mscolab.db')
 
+    # SQLAlchemy connection string to migrate data from, if set
+    SQLALCHEMY_DB_URI_TO_MIGRATE_FROM = None
+
     # Set to True for testing and False for production
     SQLALCHEMY_ECHO = False
 

mslib/mscolab/custom_migration_types.py

@@ -0,0 +1,27 @@
+# -*- coding: utf-8 -*-
+"""
+
+    mslib.mscolab.custom_migration_types
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    Consolidated module of all custom/non-standard database types used in MSColab.
+
+    This file is part of MSS.
+
+    :copyright: Copyright 2024 Matthias Riße
+    :copyright: Copyright 2024 by the MSS team, see AUTHORS.
+    :license: APACHE-2.0, see LICENSE for details.
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+"""
+from mslib.mscolab.models import AwareDateTime  # noqa: F401

mslib/mscolab/migrations/env.py

@@ -45,7 +45,7 @@
 # target_metadata = mymodel.Base.metadata
 config.set_main_option(
     'sqlalchemy.url',
-    str(current_app.extensions['migrate'].db.get_engine().url).replace(
+    str(current_app.extensions['migrate'].db.engine.url).replace(
         '%', '%%'))
 target_metadata = current_app.extensions['migrate'].db.metadata
 
@@ -94,7 +94,7 @@ def process_revision_directives(context, revision, directives):
                 directives[:] = []
                 logger.info('No changes in schema detected.')
 
-    connectable = current_app.extensions['migrate'].db.get_engine()
+    connectable = current_app.extensions['migrate'].db.engine
 
     with connectable.connect() as connection:
         context.configure(

mslib/mscolab/migrations/script.py.mako

@@ -7,6 +7,7 @@ Create Date: ${create_date}
 """
 from alembic import op
 import sqlalchemy as sa
+import mslib.mscolab.custom_migration_types as cu
 ${imports if imports else ""}
 
 # revision identifiers, used by Alembic.

mslib/mscolab/migrations/versions/922e4d9c94e2_to_version_10_0_0.py

@@ -0,0 +1,33 @@
+"""To version 10.0.0
+
+Revision ID: 922e4d9c94e2
+Revises: c171019fe3ee
+Create Date: 2024-07-24 15:28:42.009581
+
+"""
+from alembic import op
+import sqlalchemy as sa
+import mslib.mscolab.custom_migration_types as cu
+
+
+# revision identifiers, used by Alembic.
+revision = '922e4d9c94e2'
+down_revision = 'c171019fe3ee'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('users', schema=None) as batch_op:
+        batch_op.add_column(sa.Column('profile_image_path', sa.String(length=255), nullable=True))
+
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('users', schema=None) as batch_op:
+        batch_op.drop_column('profile_image_path')
+
+    # ### end Alembic commands ###