Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove supportsDBT and supportsNormalization spec flags #20854

Closed
wants to merge 62 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
fadad7f
updated IntegrationLauncherConfig.yaml and added to this class suport…
andriikorotkov Nov 24, 2022
7caa00c
updated minor remarks
andriikorotkov Nov 25, 2022
8c46fd3
updated minor remarks
andriikorotkov Nov 25, 2022
8c7f464
fixed minor remarks
andriikorotkov Nov 28, 2022
1d90e5d
added normalization data to the tests
andriikorotkov Nov 29, 2022
7f97179
fixed minor remarks
andriikorotkov Nov 29, 2022
980459c
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Nov 29, 2022
6360b55
removed NormalizationRunnerFactory
andriikorotkov Nov 29, 2022
e04c848
fixed remarks
andriikorotkov Nov 29, 2022
6a4bebb
Merge branch 'akorotkov/18232' of github.com:airbytehq/airbyte into a…
andriikorotkov Nov 29, 2022
a035df6
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Nov 29, 2022
cd65ebb
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Nov 29, 2022
7f72805
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Nov 30, 2022
f77d768
Merge branch 'akorotkov/18232' of github.com:airbytehq/airbyte into a…
andriikorotkov Nov 30, 2022
f1cf84d
fixed remarks
andriikorotkov Nov 30, 2022
0b0cfce
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Nov 30, 2022
d972133
fixed remarks
andriikorotkov Dec 1, 2022
f97da1d
updated acceptance tests
andriikorotkov Dec 1, 2022
ee82e54
updated acceptance tests
andriikorotkov Dec 2, 2022
d6203ba
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 4, 2022
77fe436
updated check_images_exist.sh script
andriikorotkov Dec 4, 2022
3afde21
updated method for get normalization image name for destination accep…
andriikorotkov Dec 4, 2022
f91c1b1
fixed code style
andriikorotkov Dec 4, 2022
002d253
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 5, 2022
48077f7
fixed code style and removed tests data
andriikorotkov Dec 5, 2022
ccc42c7
updated JobErrorReporterTest.java
andriikorotkov Dec 5, 2022
57f5087
updated JobErrorReporterTest.java
andriikorotkov Dec 5, 2022
413e91b
fixed remarks
andriikorotkov Dec 6, 2022
d2ac3ae
added integration type field to the dectination_definition file and a…
andriikorotkov Dec 8, 2022
afc8afa
fixed tests
andriikorotkov Dec 8, 2022
c60b268
Merge branch 'master' into akorotkov/remove_factory_
andriikorotkov Dec 8, 2022
213290a
fixed tests
andriikorotkov Dec 8, 2022
040f2a2
fixed minor changes after pulling master changes
andriikorotkov Dec 9, 2022
5ea2e14
fixed minor changes after pulling master changes
andriikorotkov Dec 9, 2022
534ea58
renamed integrationType to normalizationIntegrationType/ fixed minor …
andriikorotkov Dec 12, 2022
85ce1f5
renamed extra dependencies
andriikorotkov Dec 12, 2022
1dff835
updated docs
andriikorotkov Dec 12, 2022
8f035c2
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 12, 2022
53f9640
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 12, 2022
16c364d
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 12, 2022
8aa5eff
updated docs
andriikorotkov Dec 13, 2022
fb92225
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 13, 2022
bfe5c9d
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 13, 2022
f196bea
fixed minor remarks
andriikorotkov Dec 14, 2022
1df7802
added NormalizationDestinationDefinitionConfig.yaml for StandardDesti…
andriikorotkov Dec 14, 2022
33fba46
updated normalization tag
andriikorotkov Dec 14, 2022
b6fb7f0
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 14, 2022
3e0a6b3
updated DestinationAcceptanceTest.java
andriikorotkov Dec 14, 2022
811e855
updated DestinationAcceptanceTest.java
andriikorotkov Dec 14, 2022
5c85c66
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 15, 2022
465a7a8
removed normalization and dbt flags from all specs
andriikorotkov Dec 21, 2022
fd4635d
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 21, 2022
c8cea11
removed normalization and dbt flags from all specs
andriikorotkov Dec 22, 2022
3b8ffb8
removed normalization and dbt flags from all specs
andriikorotkov Dec 22, 2022
e551f04
update ConnectionFormService and formConfig
dizel852 Dec 22, 2022
c086e88
update ConnectionTransformationTab
dizel852 Dec 22, 2022
855e3a2
update useConnectionFormService usage
dizel852 Dec 22, 2022
7a2ac10
fix tests after updates in destDef and destDefSpec
dizel852 Dec 22, 2022
76e9a07
- rename "mockDestination" mock file to "mockDestinationDefinitionSpe…
dizel852 Dec 22, 2022
b9a6d71
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 23, 2022
0c04b12
fixed imports
andriikorotkov Dec 23, 2022
09e8b5d
added default values to dbt and normalization marks
andriikorotkov Dec 23, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions airbyte-api/src/main/openapi/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2858,6 +2858,12 @@ components:
description: The date when this connector was first released, in yyyy-mm-dd format.
type: string
format: date
supportsDbt:
default: false
type: boolean
supportsNormalization:
default: false
type: boolean
resourceRequirements:
$ref: "#/components/schemas/ActorDefinitionResourceRequirements"
DestinationDefinitionReadList:
Expand Down Expand Up @@ -2934,10 +2940,6 @@ components:
type: array
items:
$ref: "#/components/schemas/DestinationSyncMode"
supportsDbt:
type: boolean
supportsNormalization:
type: boolean
# DESTINATION
DestinationId:
type: string
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -280,20 +280,6 @@ protected boolean implementsOverwrite() throws WorkerException {
}
}

/**
* Override to return true to if the destination implements basic normalization and it should be
* tested here.
*
* @return - a boolean.
*/
protected boolean supportsNormalization() {
return false;
}

protected boolean supportsDBT() {
return false;
}

/**
* Override to return true if a destination implements size limits on record size (then destination
* should redefine getMaxRecordValueLimit() too)
Expand Down Expand Up @@ -566,7 +552,6 @@ public void testLineBreakCharacters() throws Exception {
@Test
public void specNormalizationValueShouldBeCorrect() throws Exception {
final boolean normalizationFromDefinition = normalizationFromDefinition();
assertEquals(normalizationFromDefinition, supportsNormalization());
if (normalizationFromDefinition) {
boolean normalizationRunnerFactorySupportsDestinationImage;
try {
Expand All @@ -582,11 +567,6 @@ public void specNormalizationValueShouldBeCorrect() throws Exception {
}
}

@Test
public void specDBTValueShouldBeCorrect() {
assertEquals(dbtFromDefinition(), supportsDBT());
}

/**
* Verify that the integration successfully writes records incrementally. The second run should
* append records to the datastore instead of overwriting the previous run.
Expand Down Expand Up @@ -1706,7 +1686,7 @@ public void testSyncIntegerInfinityDataType() throws Exception {
}

private void runAndCheck(AirbyteCatalog catalog, ConfiguredAirbyteCatalog configuredCatalog, List<AirbyteMessage> messages) throws Exception {
if (supportsNormalization()) {
if (normalizationFromDefinition()) {
LOGGER.info("Normalization is supported! Run test with normalization.");
runAndCheckWithNormalization(messages, configuredCatalog, catalog);
} else {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@
"TODO, available options are: 'overwrite', 'append', and 'append_dedup'"
],
"supportsIncremental": true,
"supportsDBT": false,
"supportsNormalization": false,
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Destination {{capitalCase name}}",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/amazon-sqs",
"supported_destination_sync_modes": ["append"],
"supportsIncremental": true,
"supportsDBT": false,
"supportsNormalization": false,
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Destination Amazon Sqs",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/aws-datalake",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/azureblobstorage",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/bigquery",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": true,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,11 +81,6 @@ protected JsonNode getFailCheckConfig() {
return config;
}

@Override
protected boolean supportsDBT() {
return true;
}

@Override
protected boolean implementsNamespaces() {
return true;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/bigquery",
"supportsIncremental": true,
"supportsNormalization": true,
"supportsDBT": true,
"supported_destination_sync_modes": ["overwrite", "append", "append_dedup"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,16 +80,6 @@ protected JsonNode getFailCheckConfig() {
return config;
}

@Override
protected boolean supportsNormalization() {
return true;
}

@Override
protected boolean supportsDBT() {
return true;
}

@Override
protected boolean implementsNamespaces() {
return true;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/cassandra",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,16 +66,6 @@ protected String getImageName() {
return "airbyte/destination-clickhouse-strict-encrypt:dev";
}

@Override
protected boolean supportsNormalization() {
return true;
}

@Override
protected boolean supportsDBT() {
return false;
}

@Override
protected boolean implementsNamespaces() {
return true;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/clickhouse",
"supportsIncremental": true,
"supportsNormalization": true,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append", "append_dedup"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/clickhouse",
"supportsIncremental": true,
"supportsNormalization": true,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append", "append_dedup"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,16 +43,6 @@ protected String getImageName() {
return "airbyte/destination-clickhouse:dev";
}

@Override
protected boolean supportsNormalization() {
return true;
}

@Override
protected boolean supportsDBT() {
return false;
}

@Override
protected boolean implementsNamespaces() {
return true;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,16 +48,6 @@ protected String getImageName() {
return "airbyte/destination-clickhouse:dev";
}

@Override
protected boolean supportsNormalization() {
return true;
}

@Override
protected boolean supportsDBT() {
return false;
}

@Override
protected boolean implementsNamespaces() {
return true;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/local-csv",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/databricks",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/e2e-test",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.io/integrations/destinations/doris",
"supportsIncremental": false,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["append", "overwrite"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/dynamodb",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/e2e-test",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"protocol_version": "0.2.1",
"connectionSpecification": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,6 @@ protected boolean implementsNamespaces() {
return true;
}

@Override
protected boolean supportsNormalization() {
return false;
}

@Override
protected boolean supportBasicDataTypeTest() {
return true;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/elasticsearch",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsNamespaces": true,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/elasticsearch",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsNamespaces": true,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,6 @@ protected boolean implementsNamespaces() {
return true;
}

@Override
protected boolean supportsNormalization() {
return false;
}

@Override
protected boolean supportBasicDataTypeTest() {
return true;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/firebolt",
"supported_destination_sync_modes": ["overwrite", "append"],
"supportsIncremental": true,
"supportsDBT": true,
"supportsNormalization": false,
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Firebolt Spec",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/firestore",
"supported_destination_sync_modes": ["append", "overwrite"],
"supportsIncremental": true,
"supportsDBT": false,
"supportsNormalization": false,
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Destination Google Firestore",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
"$schema": "http://json-schema.org/draft-07/schema#",
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/gcs",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/google-sheets",
"supported_destination_sync_modes": ["overwrite", "append", "append_dedup"],
"supportsIncremental": true,
"supportsDBT": false,
"supportsNormalization": false,
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Destination Google Sheets",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/iceberg",
"supportsNormalization": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/postgres",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/destinations/kafka",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": false,
"supported_destination_sync_modes": ["append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Loading