Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add write discover fetch event API #21506

Merged
merged 10 commits into from
Jan 23, 2023
Merged

Add write discover fetch event API #21506

merged 10 commits into from
Jan 23, 2023

Conversation

xiaohansong
Copy link
Contributor

What

#21279

To add a new internal API endpoint that workers will call.

How

Implement a new Api and hook it up with the existing configRepository logic. Have to implement a converter helper to translate from API model to protocol model.

Recommended reading order

  1. x.java
  2. y.python

🚨 User Impact 🚨

This function will not be used at this moment because workers are still calling DB directly. Thus no user impact.

@octavia-squidington-iv octavia-squidington-iv added area/api Related to the api area/documentation Improvements or additions to documentation area/platform issues related to the platform area/server labels Jan 18, 2023
@xiaohansong xiaohansong temporarily deployed to more-secrets January 18, 2023 00:27 — with GitHub Actions Inactive
@xiaohansong xiaohansong temporarily deployed to more-secrets January 18, 2023 00:27 — with GitHub Actions Inactive
@xiaohansong xiaohansong temporarily deployed to more-secrets January 18, 2023 00:30 — with GitHub Actions Inactive
@xiaohansong xiaohansong temporarily deployed to more-secrets January 18, 2023 00:30 — with GitHub Actions Inactive
@github-actions
Copy link
Contributor

github-actions bot commented Jan 18, 2023

Airbyte Code Coverage

File Coverage [86.88%] 🍏
SchedulerHandler.java 94.33% 🍏
CatalogConverter.java 91.85% 🍏
SourceHandler.java 90.83% 🍏
WebBackendConnectionsHandler.java 90.23% 🍏
ConnectionsHandler.java 82.02% 🍏
SourceApiController.java 0%
Total Project Coverage 26.68% 🍏

@xiaohansong xiaohansong marked this pull request as ready for review January 18, 2023 00:47
@xiaohansong xiaohansong temporarily deployed to more-secrets January 18, 2023 01:28 — with GitHub Actions Inactive
@xiaohansong xiaohansong temporarily deployed to more-secrets January 18, 2023 01:28 — with GitHub Actions Inactive
Copy link
Contributor

@pmossman pmossman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xiaohansong I forgot to submit this review yesterday, sorry about that! Left a few comments and suggestions, let me know what you think of them. I can do another pass when needed, just let me know

airbyte-api/src/main/openapi/config.yaml Outdated Show resolved Hide resolved
airbyte-api/src/main/openapi/config.yaml Outdated Show resolved Hide resolved
@@ -181,6 +181,30 @@ public static io.airbyte.protocol.models.ConfiguredAirbyteCatalog toProtocolKeep
return toProtocol(clone);
}

public static io.airbyte.protocol.models.AirbyteCatalog toAirbyteCatalogProtocol(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the naming of this method is a little confusing compared to the method below it, toProtocol. Perhaps we should rename the method below this to toProtocolConfigured and then this method can be called toProtocol?

At the very least, I would add some javadoc to this method and maybe try to include some guidance on when this method should be used versus the below method which returns a configured catalog

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah agreed. Mike was mentioning we should work with move team to consolidate these two states, but that was a bit out of scope for this PR/tickets.

@xiaohansong xiaohansong requested a review from a team as a code owner January 19, 2023 23:30
@xiaohansong xiaohansong requested a review from pmossman January 19, 2023 23:31
@xiaohansong xiaohansong temporarily deployed to more-secrets January 19, 2023 23:32 — with GitHub Actions Inactive
@xiaohansong xiaohansong temporarily deployed to more-secrets January 19, 2023 23:32 — with GitHub Actions Inactive
@xiaohansong
Copy link
Contributor Author

@pmossman Ready for you to take another pass!

@xiaohansong xiaohansong temporarily deployed to more-secrets January 20, 2023 00:29 — with GitHub Actions Inactive
@xiaohansong xiaohansong temporarily deployed to more-secrets January 20, 2023 00:29 — with GitHub Actions Inactive
Copy link
Contributor

@pmossman pmossman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, I think @mfsiega-airbyte should take a quick pass as well to make sure I didn't miss anything as this isn't my most familiar area of code

@xiaohansong xiaohansong temporarily deployed to more-secrets January 23, 2023 18:12 — with GitHub Actions Inactive
@xiaohansong xiaohansong temporarily deployed to more-secrets January 23, 2023 18:12 — with GitHub Actions Inactive
Copy link
Contributor

@mfsiega-airbyte mfsiega-airbyte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@xiaohansong xiaohansong merged commit 0837781 into master Jan 23, 2023
@xiaohansong xiaohansong deleted the xiaohan/discover branch January 23, 2023 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/api Related to the api area/documentation Improvements or additions to documentation area/platform issues related to the platform area/server
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants