Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix migration validation issue #6154

Merged
merged 4 commits into from
Sep 16, 2021
Merged

Fix migration validation issue #6154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bec8c4b
Make attempts schema for forward compatible with latest change
tuliren Sep 16, 2021
899d346
Revert attempts schema
tuliren Sep 16, 2021
92cfd46
Make attempts forward compatible
tuliren Sep 16, 2021
081e7e6
Make migration idempotent for extra safety
tuliren Sep 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ...yte/db/instance/jobs/migrations/V0_29_15_001__Add_temporalWorkflowId_col_to_Attempts.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ public void migrate(Context context) throws Exception {
// As database schema changes, the generated jOOQ code can be deprecated. So
// old migration may not compile if there is any generated code.
DSLContext ctx = DSL.using(context.getConnection());
ctx.alterTable("attempts").addColumn(DSL.field("temporal_workflow_id", SQLDataType.VARCHAR(256).nullable(true)))
ctx.alterTable("attempts")
.addColumnIfNotExists(DSL.field("temporal_workflow_id", SQLDataType.VARCHAR(256).nullable(true)))
Copy link
Contributor Author

@tuliren tuliren Sep 16, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not necessary. But I feel that we should be extra cautious now. This change makes the migration idempotent. Even if the user's database is in a weird state (e.g. this column is manually added), it will work.

.execute();
}

Expand Down
4 changes: 1 addition & 3 deletions airbyte-db/lib/src/main/resources/jobs_database/Attempts.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ required:
- status
- created_at
- updated_at
additionalProperties: false
additionalProperties: true
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the attempts schema forward compatible before the file-based migration system is removed.

properties:
id:
type: number
Expand All @@ -25,8 +25,6 @@ properties:
type: ["null", object]
status:
type: string
temporal_workflow_id:
type: ["null", string]
Copy link
Contributor Author

@tuliren tuliren Sep 16, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This schema is used for validation in the file-based migration system, which does not know the existence of this property created by Flyway.

created_at:
# todo should be datetime.
type: string
Expand Down
2 changes: 1 addition & 1 deletion airbyte-migration/src/main/resources/migrations/migrationV0_14_0/airbyte_db/Attempts.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ required:
- status
- created_at
- updated_at
additionalProperties: false
additionalProperties: true
properties:
id:
type: number
Expand Down