Use MessageMigration for Source Connection Check. #17656
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gosusnp
requested review from
colesnodgrass,
jdpgrailsdev,
benmoriceau and
alovew
github-actions
bot
added
area/platform
jdpgrailsdev
approved these changes
Oct 6, 2022
LGTM...looks like there is an unrelated, non-test related failure in the Kube acceptance tests:
This was referenced Oct 13, 2022
letiescanciano
added a commit
that referenced
this pull request
Oct 14, 2022
…vation * master: (98 commits) 🐛 Source Bing Ads - Fix Campaigns stream misses Audience and Shopping (#17873) Source S3 - fix schema inference (#17991) 🎉 JDBC sources: store cursor record count in db state (#15535) Introduce webhook configs into workspace api and persistence (#17950) ci: upload test results to github for analysis (#17953) Trigger the connectors build if there are worker changes. (#17976) Add additional sync timing information (#17643) Use page_token_option instead of page_token (#17892) capture metrics around json messages size (#17973) 🐛 Correct kube annotations variable as per the docs. (#17972) 🪟 🎉 Add /connector-builder page with embedded YAML editor (#17482) fix `est_num_metrics_emitted_by_reporter` not being emitted (#17929) Update schema dumps (#17960) Remove the bump in the value.yml (#17959) Ensure database initialization in test container (#17697) Remove typo line from incremental reads docs (#17920) DocS: Update authentication.md (#17931) Use MessageMigration for Source Connection Check. (#17656) fixed links (#17949) remove usages of YamlSeedConfigPersistence (#17895) ...
jhammarstedt
pushed a commit
to jhammarstedt/airbyte
that referenced
this pull request
Oct 31, 2022
* More AirbyteVersion references fix * Propagate protocol version from sourceDef to SchedulerClient * Propagate protocol version to LauncherConfig * Add VersionedMigratorFactory * Update VersionedAirbyteStreamFactory * Fix Version Json serialization/deserialization * Plug message migration in CheckConnection for Sources
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Use MessageMigration for Source Connection Check.
Also contains some remaining API fixes for Versioning.
Relates to #16038 and #15991
How
The approach is to pass the protocol version of the connector into the
CheckConnectionActivityImplbecause it is currently where we create theAirbyteStreamFactorywe need to override to handle versioning.In terms of passing the information around, it is added to
IntegrationLauncherConfig. Ideally we should probably pass more connector specific information, however, it is currently not done and probably not worth refactoring for now.IntegrationLauncherConfiglooked like a reasonable option because eventually, we may want to update theIntegrationLauncher, to directly provide theVersionedStream. Currently, it has an uneven abstraction depending if it is for replication or the other operations. For replication, we provide a more advanced abstraction that exposesAirbyteMessagedirectly where the other expose aStreamand let the worker convert messages intoAirbyteMessageDetails of the changes
AirbyteVersioninstances that should beVersioninsteadVersionto be able to use it directly in our internal objects.DefaultCheckConnectionWorkerRecommended reading order
🚨 User Impact 🚨
There should be no visible impact, but we will start using the new migration layer for Source Connection Checks.