Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database Migrations for CoreAccount after migration to Generic #3977

Merged
merged 24 commits into from
Aug 5, 2024

Conversation

dgarros
Copy link
Collaborator

@dgarros dgarros commented Jul 31, 2024

Fixes #3915

This PR implements the migrations to update the schema and the data after the conversion of CoreAccount to use the new CoreGenericAccount (#3807)

It was an interesting exercice because it was probably our most complicated migration to date and it gave me the opportunity to do some cleanup in the code and to implement some helpers that will make it easier next time.

These migrations needs to be applied both to the schema and the data (existing nodes) and they are applied with 2 commands

  • infrahub db migrate
  • infrahub db update-core-schema

The first command is very low level and can run any queries to modify either the schema or the data. the main constraint for this command is that we don't have access to the registry or the existing schema so the query we execute must have very few dependencies.

The second command infrahub db update-core-schema is doing a standard schema update by calculating the diff between the current core schema and the one currently defined in the code.
A significant part of the schema updates can automatically managed by this command but some preparation works can be required if the change is not entirely supported.

For this migration, it was required to :

  • Delete all attributes & relationships on CoreAccount that have been moved to the new generic from the schema before applying the new schema
  • Reset some attributes on CoreAccount that are referencing the attributes and relationships that have been migrated.

the second change was required because infrahub db update-core-schema expect a valid schema so it was raising an error because fields like order_by or display_labels were referencing non existing attributes. see below for more information

For the migration of the existing nodes, it was required to

  • Rename the attribute type to account_type
  • Rename all the relationships to & from CoreAccount
  • Add the label CoreGenericAccount to CoreAccount

In order to avoid duplicating some complicated queries, I've converted some existing SchemaMigration into more generic queries that can be used within the graph migrations and outside of the context of schema migrations.

  • AttributeRenameQuery
  • NodeDuplicateQuery

And I've introduced a few new ones

  • SchemaAttributeUpdateQuery
  • RelationshipDuplicateQuery
  • DeleteElementInSchemaQuery

The fix for #4004 is also required in order to successfully complete the upgrade because some relationships on CoreThread and CoreComment will be updated during the migration.

Disabling some schema validations when running infrahub db update-core-schema

One of the main issue to address was all the validations enforced when we are loading the existing schema at the start beginning of infrahub db update-core-schema and during the update process.
For example today it's not supported to update the value of inherit_from (to add the new generic). it was not possible to update the value manually without also creating the entire generic in the schema so instead I implemented a flag to disable the error related to update_support when we are loading the latest schema during infrahub db update-core-schema

At some point I also explore the option to disable more validators, especially when we are processing the new schema after loading it from the database. Ultimately I decided to take another route but I left the flag in place because it feels like we'll need it at some point.

@github-actions github-actions bot added the group/backend Issue related to the backend (API Server, Git Agent) label Jul 31, 2024
@dgarros dgarros force-pushed the dga-20240731-db-migrations branch from e792e56 to 32d3585 Compare August 2, 2024 08:40
@dgarros dgarros force-pushed the dga-20240731-db-migrations branch from 32d3585 to 6b8188c Compare August 4, 2024 09:28
@dgarros dgarros requested a review from a team August 4, 2024 17:33
@dgarros
Copy link
Collaborator Author

dgarros commented Aug 4, 2024

I still need to make the queries compatible with memgraph but appart from that the PR is ready for review

@dgarros dgarros added the type/feature New feature or request label Aug 4, 2024
@dgarros dgarros marked this pull request as ready for review August 5, 2024 06:07
Copy link
Contributor

@gmazoyer gmazoyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR, learnt a lot while reviewing it 👍🏻

name="Account",
namespace="Core",
branch_support=BranchSupportType.AGNOSTIC.value,
labels=["CoreAccount", InfrahubKind.GENERICACCOUNT, InfrahubKind.LINEAGEOWNER, InfrahubKind.LINEAGESOURCE],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace by constant InfrahubKind.ACCOUNT instead of "CoreAccount"

@dgarros dgarros merged commit d77670d into develop Aug 5, 2024
45 checks passed
@dgarros dgarros deleted the dga-20240731-db-migrations branch August 5, 2024 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
group/backend Issue related to the backend (API Server, Git Agent) type/feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

task: Add migration to convert account to generic after IFC-29
2 participants