-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Read the information from both the definition file and the spec and add logs if have difference #19799
Conversation
…DBT and normalizationImage fields. Added to the GenerateInputActivityImpl and TemporalClient classes code parts for read destination_definition.yaml and get suportDBT and normalizationImage fields. Added logging and comparing normalization images from NormalizationRunnerFactory and destination_definition.yaml
Affected Connector ReportNOTE
|
Connector | Version | Changelog | Publish |
---|
- See "Actionable Items" below for how to resolve warnings and errors.
❌ Destinations (47)
Connector | Version | Changelog | Publish |
---|---|---|---|
destination-aws-datalake |
0.1.1 |
✅ | ✅ |
destination-azure-blob-storage |
0.1.6 |
⚠ (doc not found) |
✅ |
destination-bigquery |
1.2.8 |
✅ | ✅ |
destination-bigquery-denormalized |
1.2.8 |
⚠ (doc not found) |
✅ |
destination-cassandra |
0.1.4 |
❌ (changelog missing) |
✅ |
destination-clickhouse |
0.2.0 |
✅ | ✅ |
destination-clickhouse-strict-encrypt |
0.2.0 |
✅ | ⚠ (not in seed) |
destination-csv |
0.2.10 |
⚠ (doc not found) |
✅ |
destination-databricks |
0.3.1 |
✅ | ✅ |
destination-dev-null |
0.2.7 |
⚠ (doc not found) |
⚠ (not in seed) |
destination-doris |
0.1.0 |
❌ (changelog missing) |
✅ |
destination-dynamodb |
0.1.7 |
✅ | ✅ |
destination-e2e-test |
0.2.4 |
✅ | ✅ |
destination-elasticsearch |
0.1.6 |
✅ | ✅ |
destination-elasticsearch-strict-encrypt |
0.1.6 |
✅ | ⚠ (not in seed) |
destination-gcs |
0.2.12 |
✅ | ✅ |
destination-iceberg |
0.1.0 |
❌ (changelog missing) |
✅ |
destination-jdbc |
0.3.14 |
⚠ (doc not found) |
⚠ (not in seed) |
destination-kafka |
0.1.10 |
❌ (changelog missing) |
✅ |
destination-keen |
0.2.4 |
❌ (changelog missing) |
✅ |
destination-kinesis |
0.1.5 |
❌ (changelog missing) |
✅ |
destination-local-json |
0.2.11 |
✅ | ✅ |
destination-mariadb-columnstore |
0.1.7 |
✅ | ✅ |
destination-mongodb |
0.1.9 |
✅ | ✅ |
destination-mongodb-strict-encrypt |
0.1.9 |
✅ | ⚠ (not in seed) |
destination-mqtt |
0.1.3 |
✅ | ✅ |
destination-mssql |
0.1.22 |
✅ | ✅ |
destination-mssql-strict-encrypt |
0.1.22 |
✅ | ⚠ (not in seed) |
destination-mysql |
0.1.20 |
✅ | ✅ |
destination-mysql-strict-encrypt |
❌ 0.1.21 (mismatch: 0.1.20 ) |
✅ | ⚠ (not in seed) |
destination-oracle |
0.1.19 |
✅ | ✅ |
destination-oracle-strict-encrypt |
0.1.19 |
✅ | ⚠ (not in seed) |
destination-postgres |
0.3.26 |
✅ | ✅ |
destination-postgres-strict-encrypt |
0.3.26 |
✅ | ⚠ (not in seed) |
destination-pubsub |
0.1.6 |
❌ (changelog missing) |
✅ |
destination-pulsar |
0.1.3 |
❌ (changelog missing) |
✅ |
destination-r2 |
0.1.0 |
✅ | ✅ |
destination-redis |
0.1.4 |
✅ | ✅ |
destination-redpanda |
0.1.0 |
❌ (changelog missing) |
✅ |
destination-redshift |
0.3.51 |
✅ | ✅ |
destination-rockset |
0.1.4 |
❌ (changelog missing) |
✅ |
destination-s3 |
0.3.17 |
✅ | ✅ |
destination-s3-glue |
0.1.0 |
❌ (changelog missing) |
✅ |
destination-scylla |
0.1.3 |
✅ | ✅ |
destination-snowflake |
0.4.40 |
✅ | ✅ |
destination-tidb |
0.1.0 |
✅ | ✅ |
destination-yugabytedb |
0.1.0 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
Category | Status | Actionable Item |
---|---|---|
Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml ), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks andrii! Didn't look too deeply at the logic, but my main concern at the moment is in using local definitions providers instead of reading the data directly from the database. More info in comments!
LocalDefinitionsProvider provider = new LocalDefinitionsProvider(LocalDefinitionsProvider.DEFAULT_SEED_DEFINITION_RESOURCE_CLASS); | ||
List<StandardDestinationDefinition> destinationDefinitionList = provider.getDestinationDefinitions(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We shouldn't be reading from a definitions provider or direct yamls here- they're mainly used to seed/update the connector definitions in the database (actor_definitions table) and then any other operations within the platform should use the data in the db. For example, cloud doesn't use local definitions and instead loads remote definitions from a service.
I think by this point we should have the normalization tag / repo / anything else in that database table, specially since I see a migration adding those columns to the db... is this true?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes you're right. This is a very good point. Thank you for it. I changed this piece of code and now I get the data I need from the database. Also, I removed this logic from TemporalClient.class - as far as I understand, that part working on the synchronization process for which we may not retrieve this information.
...ommons-worker/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java
Outdated
Show resolved
Hide resolved
airbyte-worker-models/src/main/resources/workers_models/IntegrationLauncherConfig.yaml
Outdated
Show resolved
Hide resolved
airbyte-worker-models/src/main/resources/workers_models/IntegrationLauncherConfig.yaml
Outdated
Show resolved
Hide resolved
...c/main/java/io/airbyte/workers/temporal/scheduling/activities/GenerateInputActivityImpl.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mainly some questions!
Also, it would be great to see some tests added to make sure that we're retrieving the normalization image/tag as expected from the db
private static DestinationType getDestinationTypeByNormalizationImageNamePart(final String normalizationImageName) { | ||
return EnumSet.allOf(DestinationType.class).stream() | ||
.filter(destinationType -> normalizationImageName.contains(destinationType.name().toLowerCase())) | ||
.findFirst() | ||
.orElseGet(null); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we planning on / is there a way to remove this DestinationType enum? What is the destination type used for? I think leaving these in the platform means we still have some coupling and would require a platform change for adding normalization to a new connector, if I'm reading this correctly. Ideally we don't have anything connector-specific in the platform so they can evolve independently.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently we have an enum with specific destinations:
public enum DestinationType {
BIGQUERY,
MSSQL,
MYSQL,
ORACLE,
POSTGRES,
REDSHIFT,
SNOWFLAKE,
CLICKHOUSE,
TIDB
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Everything you wrote about is true. I plan to remove the DestinationType enum in the next pull request, where the NormalizationRunnerFactory will also be removed. You must have been misled by the getDestinationTypeByNormalizationImageNamePart method. I will delete it.
...ommons-worker/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java
Outdated
Show resolved
Hide resolved
...ommons-worker/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java
Show resolved
Hide resolved
...ntainer-orchestrator/src/main/java/io/airbyte/container_orchestrator/DbtJobOrchestrator.java
Show resolved
Hide resolved
List<StandardDestinationDefinition> destinationDefinitionList = configRepository.listStandardDestinationDefinitions(true); | ||
Optional<StandardDestinationDefinition> optionalDestinationDefinition = destinationDefinitionList.stream() | ||
.filter(destinationDefinition -> config.getDestinationDockerImage() | ||
.equalsIgnoreCase(destinationDefinition.getDockerRepository() + ":" + destinationDefinition.getDockerImageTag())) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
note: there's a utility function DockerUtils.getTaggedImageName
that deals with combining the repo and tag. Not the most complicated thing to do here but might as well use it if it's available.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have not yet come across the use of this extra class. Thanks for sharing it)
...c/main/java/io/airbyte/workers/temporal/scheduling/activities/GenerateInputActivityImpl.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should standardize to supportsDbt
(with an s
) but no need for another review from me for this change.
Please ensure that the tests in CI are passing before merging, seems like the branch is currently red.
Note: after this pr is merged we will still be using the platform-provided normalization image/versions but will start to read the definitions and log if there's a mismatch. A follow-up PR will actually remove the paltform-provided normalization images/versions and just use the ones from the definitions so I'm not expecting this to introduce a user-impacting change just yet.
log.error( | ||
"The normalization image name or tag in the definition file is different from the normalization image or tag in the NormalizationRunnerFactory!"); | ||
log.error( | ||
"the definition file value - {}, the NormalizationRunnerFactory value - {}", normalizationImage, factoryNormalizationImage); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there any action to take if we get these logs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
currently - no, just to see that different versions are used. In general, this is necessary for testing and making sure that we are getting the correct data from the actor_definition table
supportDbt: | ||
type: boolean | ||
default: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
supportDbt: | |
type: boolean | |
default: false | |
supportsDbt: | |
type: boolean | |
default: false |
nit to keep consistent
supportDbt: | ||
type: boolean | ||
default: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
supportDbt: | |
type: boolean | |
default: false | |
supportsDbt: | |
type: boolean | |
default: false |
supportDbt: | ||
type: boolean | ||
default: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
supportDbt: | |
type: boolean | |
default: false | |
supportsDbt: | |
type: boolean | |
default: false |
log.error("destination DBT -> {}", destinationLauncherConfig.getSupportDbt()); | ||
log.error("destination normalization -> {}", destinationLauncherConfig.getNormalizationDockerImage()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are these error logs intentional?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I added them on purpose, but I'd probably rather remove them. They are needed more for development than for the end user
normalizationRepository: | ||
type: string | ||
normalizationTag: | ||
type: string | ||
supportDbt: | ||
type: boolean | ||
default: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this api is called when adding custom destinations in OSS... do we need to add these to the form in the frontend?
What happens if a destination doesn't have these fields set? Are we properly handling this as a destination that doesn't run normalization?
What
Added the ability to read the name and tag for the normalization container docker from the definition file, and compare this value with the value obtained from the NormalizationRunnerFactory. In case of difference, the message is added to the logs.
How
Describe the solution
Recommended reading order
NormalizationRunnerFactory.java
NormalizationActivityImpl.java
DefaultNormalizationRunner.java
🚨 User Impact 🚨
No impact
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereUpdating a connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changesTests
Unit
Put your unit tests output here.
Integration
Put your integration tests output here.
Acceptance
Put your acceptance tests output here.