Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the migration script to work with new psql db #47

Merged
merged 5 commits into from
Oct 5, 2020

Conversation

john-dupuy
Copy link
Contributor

@john-dupuy john-dupuy commented Sep 28, 2020

Leaving as a draft until we figure out the indexes we need.

Some things I considered:

  • Only migrating run, result, and artifact files from the most recent 3 months (determined by the newest record in the MongoDB)
    • This will keep our initial DB size small and also vastly reduce the time to run the migration script
    • If we were to migrate everything it would take ~20 hrs I believe
  • Adding a upload_date field to the artifact model
  • Only migrating artifacts, we don't have many/any recent reports, so I thought it best to just migrate artifacts
  • Not migrating any imports

I got many of the column indexes, but there are still the JSON indexes to consider.

@rsnyman I'd figure I make this part of ibutsu-server so that we can use the models' from_dict method that we have defined in the db models.

The pre-commit update is because I hit PyCQA/pycodestyle#373

@john-dupuy john-dupuy marked this pull request as draft September 28, 2020 19:31
@john-dupuy john-dupuy force-pushed the add-migration-script branch 2 times, most recently from 5d9caff to 2f0d248 Compare September 29, 2020 20:09
@john-dupuy
Copy link
Contributor Author

The foreign key constraint is causing some problems with bad data in my local database, I don't think we'll hit the same issues on prod (especially if we're ignoring the old stuff) but I just pulled the latest DB dump to test out tomorrow.

Copy link
Contributor

@rsnyman rsnyman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a couple suggestions :-)

@john-dupuy john-dupuy force-pushed the add-migration-script branch 2 times, most recently from faca18b to 05dee12 Compare October 2, 2020 18:15
@john-dupuy john-dupuy marked this pull request as ready for review October 2, 2020 18:16
@john-dupuy john-dupuy force-pushed the add-migration-script branch 2 times, most recently from 47b1d7a to 8fdc189 Compare October 2, 2020 18:21
@john-dupuy
Copy link
Contributor Author

OK @rsnyman I think this is finally in the "final" state. I applied a similar logic for the artifact files that is done for the results so that we can avoid any foreign key constraint errors (i.e. getting a list of migrated results and searching for artifacts within that result list).

@rsnyman rsnyman merged commit 1247b46 into ibutsu:postgres-migration Oct 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants