Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reset database migrations for v3.0 release #6471

Closed
jeremystretch opened this issue May 21, 2021 · 1 comment
Closed

Reset database migrations for v3.0 release #6471

jeremystretch opened this issue May 21, 2021 · 1 comment
Assignees
Labels
status: accepted This issue has been accepted for implementation type: housekeeping Changes to the application which do not directly impact the end user
Milestone

Comments

@jeremystretch
Copy link
Member

Proposed Changes

We should take the upcoming v3.0 release as an opportunity to reset the roughly five years' worth of database migrations we've accumulated since the initial release. I'm not sure yet exactly what the process will entail.

Justification

  • Simplifies the deployment process for new users
  • Reduces risk of hidden technical debt in older migrations
  • Reduces CI testing time
@jeremystretch jeremystretch added type: housekeeping Changes to the application which do not directly impact the end user status: under review Further discussion is needed to determine this issue's scope and/or implementation labels May 21, 2021
@jeremystretch jeremystretch added this to the v3.0 milestone Jun 11, 2021
@jeremystretch jeremystretch added status: accepted This issue has been accepted for implementation and removed status: under review Further discussion is needed to determine this issue's scope and/or implementation labels Jul 9, 2021
@jeremystretch
Copy link
Member Author

@DanSheps and I have been working on this, and I'm confident in the approach we've arrived at. What we've done is remove and rebuild all database migrations created prior to v2.11.0. Individual migrations from v2.11.0 and later releases have been retained. This means that users will be required to upgrade to a v2.11.x release before upgrading to a v3.0 release.

To test the integrity of the squashed migrations, I compared both the resulting table schemas and the end state of the migrations table. When comparing new installations using the original and squashed migrations, the only differences found in the resulting database schema were the names of some sequences, a cosmetic difference. Further, the only differences found in the resulting migration tables were the addition of the squashed migrations.

I have also tested upgrading a v2.11.0 installation using the squashed migrations with similar results.

As a result of squashing the migrations, we have reduced the time needed to run a complete migration on my local machine from ~84 seconds to ~21 seconds. 🎉 This should help significantly reduce the time needed to run our CI tests.

Testing Methodology

Migration timing:

time ./manage.py migrate

Schema export:

sudo -u postgres psql netbox -c "SELECT table_name, column_name, data_type, column_default, is_nullable, character_maximum_length, numeric_precision FROM information_schema.columns WHERE table_schema = 'public' ORDER BY table_name ASC, column_name ASC"

(Note that we're not using pg_dump for this as manner in which it orders table columns makes the output unsuitable for generating a diff.)

Migrations table export:

sudo -u postgres psql netbox -c "select app, name from django_migrations order by app, name"

jeremystretch added a commit that referenced this issue Jul 12, 2021
Closes #6471: Squash database migrations up to v2.11.0
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 11, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status: accepted This issue has been accepted for implementation type: housekeeping Changes to the application which do not directly impact the end user
Projects
None yet
Development

No branches or pull requests

2 participants