Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure database initialization in test container #17697

Merged
merged 13 commits into from
Oct 13, 2022
Merged

Ensure database initialization in test container #17697

merged 13 commits into from
Oct 13, 2022

Conversation

jdpgrailsdev
Copy link
Contributor

What

  • Ensure that dump schema task executes without error

How

  • Initialize the configs and jobs database in the test container before running the schema dump

Recommended reading order

  1. MigrationDevCenter.java

Tests

  • Successfully ran ./gradlew :airbyte-db:db-lib:dumpConfigsSchema
  • Successfully ran ./gradlew :airbyte-db:db-lib:dumpJobsSchema

@jdpgrailsdev jdpgrailsdev temporarily deployed to more-secrets October 6, 2022 20:29 Inactive
@jdpgrailsdev jdpgrailsdev temporarily deployed to more-secrets October 6, 2022 21:24 Inactive
@terencecho
Copy link
Contributor

looks like this PR helps correctly dump the schema! I got ./gradlew :airbyte-db:db-lib:dumpConfigsSchema to work after rebuilding with SUB_BUILD=PLATFORM ./gradlew build -x test.

Before merging this PR, we should rebuild and push some changes (some formatting/removing unused imports) as well as running the schema dump, since it adds jobs db schema, not that we're adding that db

@terencecho
Copy link
Contributor

I pushed the changes I mentioned in my previous comment. feel free to revert or edit the commits if you think something else should have been done

@terencecho terencecho temporarily deployed to more-secrets October 6, 2022 22:14 Inactive
@jdpgrailsdev jdpgrailsdev temporarily deployed to more-secrets October 7, 2022 12:43 Inactive
@jdpgrailsdev jdpgrailsdev temporarily deployed to more-secrets October 7, 2022 13:36 Inactive
@jdpgrailsdev jdpgrailsdev temporarily deployed to more-secrets October 7, 2022 14:45 Inactive
@jdpgrailsdev
Copy link
Contributor Author

@alovew FYI. I think that you ran into this being broken as well. This PR will finally fix the ability to dump the schema using the Gradle tasks.

@jdpgrailsdev
Copy link
Contributor Author

@terencecho I don't understand what this check is doing: https://github.com/airbytehq/airbyte/actions/runs/3205588088/jobs/5238324179. Is this actually an issue or is that we have been out of sync for so long that we should just merge to reset what the base schema dump file contains?

@jdpgrailsdev jdpgrailsdev temporarily deployed to more-secrets October 11, 2022 14:17 Inactive
@jdpgrailsdev jdpgrailsdev temporarily deployed to more-secrets October 11, 2022 16:31 Inactive
@jdpgrailsdev jdpgrailsdev temporarily deployed to more-secrets October 11, 2022 18:40 Inactive
@jdpgrailsdev jdpgrailsdev temporarily deployed to more-secrets October 12, 2022 13:02 Inactive
@terencecho
Copy link
Contributor

since that test is just running

      - name: Test if Seed spec is updated
        run: SUB_BUILD=PLATFORM ./gradlew format && git --no-pager diff && test -z "$(git --no-pager diff)"

it's only failing because there is output in git diff. So i think if we merge this, it'll fail the build, but subsequent builds will pass.

@jdpgrailsdev jdpgrailsdev temporarily deployed to more-secrets October 13, 2022 13:45 Inactive
@jdpgrailsdev jdpgrailsdev merged commit 27e089e into master Oct 13, 2022
@jdpgrailsdev jdpgrailsdev deleted the jonathan/ensure-db-init-migration-tasks branch October 13, 2022 16:17
This was referenced Oct 13, 2022
Closed
Merged
letiescanciano added a commit that referenced this pull request Oct 14, 2022
@letiescanciano
Merge branch 'master' into leti/experiment-speedy-connection-for-acti…
e0b19de 
…vation

* master: (98 commits)
  🐛 Source Bing Ads - Fix Campaigns stream misses Audience and Shopping (#17873)
  Source S3 - fix schema inference (#17991)
  🎉 JDBC sources: store cursor record count in db state (#15535)
  Introduce webhook configs into workspace api and persistence (#17950)
  ci: upload test results to github for analysis (#17953)
  Trigger the connectors build if there are worker changes. (#17976)
  Add additional sync timing information (#17643)
  Use page_token_option instead of page_token (#17892)
  capture metrics around json messages size (#17973)
  🐛 Correct kube annotations variable as per the docs. (#17972)
  🪟 🎉 Add /connector-builder page with embedded YAML editor (#17482)
  fix `est_num_metrics_emitted_by_reporter` not being emitted (#17929)
  Update schema dumps (#17960)
  Remove the bump in the value.yml (#17959)
  Ensure database initialization in test container (#17697)
  Remove typo line from incremental reads docs (#17920)
  DocS: Update authentication.md (#17931)
  Use MessageMigration for Source Connection Check. (#17656)
  fixed links (#17949)
  remove usages of YamlSeedConfigPersistence (#17895)
  ...
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
@jdpgrailsdev @terencecho @jhammarstedt
Ensure database initialization in test container (airbytehq#17697)
2534345 
* Ensure database initialization in test container

* Remove unused import

* Update schema dump

* Add schema dump

Co-authored-by: terencecho <terence@airbyte.io>
tuliren
tuliren reviewed Nov 4, 2022
View reviewed changes
airbyte-db/db-lib/src/main/java/io/airbyte/db/instance/development/MigrationDevCenter.java
private static void initializeDatabase(final PostgreSQLContainer container) {
final var containerDelegate = new JdbcDatabaseDelegate(container, "");
ScriptUtils.runInitScript(containerDelegate, "configs_database/schema.sql");
ScriptUtils.runInitScript(containerDelegate, "jobs_database/schema.sql");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdpgrailsdev, I think this is incorrect. Each migration dev center should run specific its own init script, i.e. the config migration dev center should only run the configs initial script.

Currently when I run schema dump for one database, it will include the schemas for both configs and jobs database.

The schema dump is still correct in production because it is updated by ConfigsDatabaseMigratorTest and JobsDatabaseMigratorTest, which only does not have this behavior.

@tuliren
Copy link
Contributor

tuliren commented Nov 4, 2022

@terencecho I don't understand what this check is doing: https://github.com/airbytehq/airbyte/actions/runs/3205588088/jobs/5238324179. Is this actually an issue or is that we have been out of sync for so long that we should just merge to reset what the base schema dump file contains?

I think this diff is the result of running initialization for both databases all the time. That's why the jobs db schema ends up in the configs db dump, and vice versa.

@tuliren tuliren mentioned this pull request Nov 4, 2022
Merged
@tuliren
Copy link
Contributor

tuliren commented Nov 4, 2022

Added a fix here: #18984

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tuliren tuliren tuliren left review comments

@gosusnp gosusnp gosusnp approved these changes

@terencecho terencecho Awaiting requested review from terencecho

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jdpgrailsdev @terencecho @tuliren @gosusnp