Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

db data catalog jobs #16543

Merged
merged 1 commit into from
Oct 9, 2022
Merged

db data catalog jobs #16543

merged 1 commit into from
Oct 9, 2022

Conversation

cgardens
Copy link
Contributor

@cgardens cgardens commented Sep 9, 2022 •
edited
Loading

What

To do

  • get feedback from subject matter experts to confirm accuracy.

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Sep 9, 2022
jdpgrailsdev
jdpgrailsdev approved these changes Sep 9, 2022
View reviewed changes
Copy link
Contributor

@jdpgrailsdev jdpgrailsdev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@gosusnp
Copy link
Contributor

gosusnp commented Sep 9, 2022

Is the configs/jobs db split still accurate?
Looking at the current data model, we write all the tables in airbyte db. The only remnant of that split seems to be the two migrations sets we have in flyway.

@cgardens
Copy link
Contributor Author

cgardens commented Sep 9, 2022

Is the configs/jobs db split still accurate?
Looking at the current data model, we write all the tables in airbyte db. The only remnant of that split seems to be the two migrations sets we have in flyway.

Yeah. I think we should consider these data stores to be supporting 2 separate services. It was a mistake that they ended up in the same db instance. I would advocate for us to just move this over in a future maintenance window, because everyone who joins asks this question.

cc @davinchia because I know we have generally had different points of view on this one.
cc @supertopher would this be a project we could ask GL to do? (not high priority, just trying to understand if it would be a good fit).

@supertopher
Copy link
Contributor

@cgardens I think this seems like work the GL team has experience in. I made a story to track this here

Please make sure it sounds like the solution you are looking for since I'm getting this information a bit second hand

@davinchia
Copy link
Contributor

@cgardens less of a different pov and more of a sequencing bit. I didn't think it was worth the hassle of managing 2 databases back then given we weren't sure how Cloud would evolve. I do think now is a good time to shift this over, as we scales with the goal of making it easier to manage the different scaling requirements.

@supertopher I think we should try this on a dev env first. There will be some env vars we likely need to change to make sure the applications are reading from the right database.

Base automatically changed from cgardens/db-data-catalog to master October 9, 2022 04:11
@cgardens cgardens force-pushed the cgardens/db-data-catalog-jobs branch from 529c245 to 8cb2626 Compare October 9, 2022 04:14
@cgardens
Copy link
Contributor Author

cgardens commented Oct 9, 2022

Thank you. Agreed with all of your follow ups. @supertopher @davinchia

@cgardens cgardens merged commit 675e153 into master Oct 9, 2022
@cgardens cgardens deleted the cgardens/db-data-catalog-jobs branch October 9, 2022 04:16
letiescanciano added a commit that referenced this pull request Oct 10, 2022
@letiescanciano
Merge branch 'master' into leti/experiment-speedy-connection-for-acti…
c16214f 
…vation

* master: (22 commits)
  Update full-refresh-append.md (#17784)
  Update full-refresh-overwrite.md (#17783)
  Update incremental-append.md (#17785)
  Update incremental-deduped-history.md (#17786)
  Update cdc.md (#17787)
  🪟 🔧 Ignore classnames during jest snapshot comparison (#17773)
  feat: replace openjdk with amazoncorretto:17.0.4 on connectors for seсurity compliance (#17511)
  Start testing buildpulse. (#17712)
  Add missing types to the registry (#17763)
  jobs db descriptions (#16543)
  config db data catalog (#16427)
  Update lowcode docs (#17752)
  db migrations to support new webhook operations (#17671)
  Bump Airbyte version from 0.40.13 to 0.40.14 (#17762)
  September Release Notes (#17754)
  Revert "Use java-datadog-tracer-base image (#17625)" (#17759)
  Add connection migrations for schema changes (#17651)
  Connection Form Refactor - Part Two (#16748)
  Improve E2E testing around the Connection Form (#17577)
  Bump strict encrypt version (#17747)
  ...
This was referenced Oct 13, 2022
Closed
Merged
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
@cgardens @jhammarstedt
jobs db descriptions (airbytehq#16543)
8ef2b2a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@davinchia davinchia davinchia approved these changes

@jdpgrailsdev jdpgrailsdev jdpgrailsdev approved these changes

@gosusnp gosusnp Awaiting requested review from gosusnp

@pmossman pmossman Awaiting requested review from pmossman

@salima-airbyte salima-airbyte Awaiting requested review from salima-airbyte

Assignees
No one assigned
Labels
area/documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@cgardens @gosusnp @supertopher @davinchia @jdpgrailsdev