Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#23949 Remove check() from read() method for sources Cockroach DB and Postgres #24000

Merged
merged 16 commits into from
Mar 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@
- name: Cockroachdb
sourceDefinitionId: 9fa5862c-da7c-11eb-8d19-0242ac130003
dockerRepository: airbyte/source-cockroachdb
dockerImageTag: 0.1.20
dockerImageTag: 0.1.21
documentationUrl: https://docs.airbyte.com/integrations/sources/cockroachdb
icon: cockroachdb.svg
sourceType: database
Expand Down Expand Up @@ -1536,7 +1536,7 @@
- name: Postgres
sourceDefinitionId: decd338e-5647-4c0b-adf4-da0e75f5a750
dockerRepository: airbyte/source-postgres
dockerImageTag: 2.0.2
dockerImageTag: 2.0.3
documentationUrl: https://docs.airbyte.com/integrations/sources/postgres
icon: postgresql.svg
sourceType: database
Expand Down
4 changes: 2 additions & 2 deletions airbyte-config/init/src/main/resources/seed/source_specs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2616,7 +2616,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-cockroachdb:0.1.20"
- dockerImage: "airbyte/source-cockroachdb:0.1.21"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/cockroachdb"
connectionSpecification:
Expand Down Expand Up @@ -11763,7 +11763,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-postgres:2.0.2"
- dockerImage: "airbyte/source-postgres:2.0.3"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/postgres"
connectionSpecification:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-cockroachdb-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.20
LABEL io.airbyte.version=0.1.21
LABEL io.airbyte.name=airbyte/source-cockroachdb-strict-encrypt
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-cockroachdb

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.20
LABEL io.airbyte.version=0.1.21
LABEL io.airbyte.name=airbyte/source-cockroachdb
Original file line number Diff line number Diff line change
Expand Up @@ -94,20 +94,6 @@ public Set<String> getExcludedInternalNameSpaces() {
"crdb_internal");
}

@Override
public AutoCloseableIterator<AirbyteMessage> read(final JsonNode config,
final ConfiguredAirbyteCatalog catalog,
final JsonNode state)
throws Exception {
final AirbyteConnectionStatus check = check(config);

if (check.getStatus().equals(AirbyteConnectionStatus.Status.FAILED)) {
throw new RuntimeException("Unable establish a connection: " + check.getMessage());
}

return super.read(config, catalog, state);
}

@Override
public Set<JdbcPrivilegeDto> getPrivilegesTableForCurrentUser(final JdbcDatabase database, final String schema) throws SQLException {
try (final Stream<JsonNode> stream = database.unsafeQuery(getPrivileges(database), sourceOperations::rowToJson)) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=2.0.2
LABEL io.airbyte.version=2.0.3
LABEL io.airbyte.name=airbyte/source-postgres-strict-encrypt
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=2.0.2
LABEL io.airbyte.version=2.0.3
LABEL io.airbyte.name=airbyte/source-postgres
Original file line number Diff line number Diff line change
Expand Up @@ -335,22 +335,6 @@ public List<CheckedConsumer<JdbcDatabase, Exception>> getCheckOperations(final J
return checkOperations;
}

@Override
public AutoCloseableIterator<AirbyteMessage> read(final JsonNode config,
final ConfiguredAirbyteCatalog catalog,
final JsonNode state)
throws Exception {
// this check is used to ensure that have the pgoutput slot available so Debezium won't attempt to
// create it.
final AirbyteConnectionStatus check = check(config);

if (check.getStatus().equals(Status.FAILED)) {
throw new RuntimeException("Unable establish a connection: " + check.getMessage());
}

return super.read(config, catalog, state);
}

@Override
public List<AutoCloseableIterator<AirbyteMessage>> getIncrementalIterators(final JdbcDatabase database,
final ConfiguredAirbyteCatalog catalog,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -224,25 +224,6 @@ void testCheckWithoutReplicationSlot() throws Exception {
assertEquals(status.getStatus(), AirbyteConnectionStatus.Status.FAILED);
}

@Test
void testReadWithoutPublication() throws SQLException {
database.query(ctx -> ctx.execute("DROP PUBLICATION " + PUBLICATION + ";"));

assertThrows(Exception.class, () -> {
source.read(config, CONFIGURED_CATALOG, null);
});
}

@Test
void testReadWithoutReplicationSlot() throws SQLException {
final String fullReplicationSlot = SLOT_NAME_BASE + "_" + dbName;
database.query(ctx -> ctx.execute("SELECT pg_drop_replication_slot('" + fullReplicationSlot + "');"));

assertThrows(Exception.class, () -> {
source.read(config, CONFIGURED_CATALOG, null);
});
}

@Override
protected void assertExpectedStateMessages(final List<AirbyteStateMessage> stateMessages) {
assertEquals(1, stateMessages.size());
Expand Down
4 changes: 2 additions & 2 deletions connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
| **ClickUp** | <img alt="ClickUp icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/clickup.svg" height="30" height="30"/> | Source | airbyte/source-clickup-api:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/click-up) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-clickup-api) | <small>`311a7a27-3fb5-4f7e-8265-5e4afe258b66`</small> |
| **Clockify** | <img alt="Clockify icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/clockify.svg" height="30" height="30"/> | Source | airbyte/source-clockify:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/clockify) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-clockify) | <small>`e71aae8a-5143-11ed-bdc3-0242ac120002`</small> |
| **Close.com** | <img alt="Close.com icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/close.svg" height="30" height="30"/> | Source | airbyte/source-close-com:0.2.1 | beta | [link](https://docs.airbyte.com/integrations/sources/close-com) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-close-com) | <small>`dfffecb7-9a13-43e9-acdc-b92af7997ca9`</small> |
| **Cockroachdb** | <img alt="Cockroachdb icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/cockroachdb.svg" height="30" height="30"/> | Source | airbyte/source-cockroachdb:0.1.20 | alpha | [link](https://docs.airbyte.com/integrations/sources/cockroachdb) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-cockroachdb) | <small>`9fa5862c-da7c-11eb-8d19-0242ac130003`</small> |
| **Cockroachdb** | <img alt="Cockroachdb icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/cockroachdb.svg" height="30" height="30"/> | Source | airbyte/source-cockroachdb:0.1.21 | alpha | [link](https://docs.airbyte.com/integrations/sources/cockroachdb) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-cockroachdb) | <small>`9fa5862c-da7c-11eb-8d19-0242ac130003`</small> |
| **Coda** | <img alt="Coda icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/coda.svg" height="30" height="30"/> | Source | airbyte/source-coda:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/coda) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-coda) | <small>`27f910fd-f832-4b2e-bcfd-6ab342e434d8`</small> |
| **Coin API** | <img alt="Coin API icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/coinapi.svg" height="30" height="30"/> | Source | airbyte/source-coin-api:0.1.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/coin-api) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-coin-api) | <small>`919984ef-53a2-479b-8ffe-9c1ddb9fc3f3`</small> |
| **CoinGecko Coins** | <img alt="CoinGecko Coins icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/coingeckocoins.svg" height="30" height="30"/> | Source | airbyte/source-coingecko-coins:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/coingecko-coins) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-coingecko-coins) | <small>`9cdd4183-d0ba-40c3-aad3-6f46d4103974`</small> |
Expand Down Expand Up @@ -175,7 +175,7 @@
| **PokeAPI** | <img alt="PokeAPI icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/pokeapi.svg" height="30" height="30"/> | Source | airbyte/source-pokeapi:0.1.5 | alpha | [link](https://docs.airbyte.com/integrations/sources/pokeapi) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-pokeapi) | <small>`6371b14b-bc68-4236-bfbd-468e8df8e968`</small> |
| **Polygon Stock API** | <img alt="Polygon Stock API icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/polygon.svg" height="30" height="30"/> | Source | airbyte/source-polygon-stock-api:0.1.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/polygon-stock-api) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-polygon-stock-api) | <small>`5807d72f-0abc-49f9-8fa5-ae820007032b`</small> |
| **PostHog** | <img alt="PostHog icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/posthog.svg" height="30" height="30"/> | Source | airbyte/source-posthog:0.1.8 | beta | [link](https://docs.airbyte.com/integrations/sources/posthog) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-posthog) | <small>`af6d50ee-dddf-4126-a8ee-7faee990774f`</small> |
| **Postgres** | <img alt="Postgres icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/postgresql.svg" height="30" height="30"/> | Source | airbyte/source-postgres:2.0.2 | generally_available | [link](https://docs.airbyte.com/integrations/sources/postgres) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-postgres) | <small>`decd338e-5647-4c0b-adf4-da0e75f5a750`</small> |
| **Postgres** | <img alt="Postgres icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/postgresql.svg" height="30" height="30"/> | Source | airbyte/source-postgres:2.0.3 | generally_available | [link](https://docs.airbyte.com/integrations/sources/postgres) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-postgres) | <small>`decd338e-5647-4c0b-adf4-da0e75f5a750`</small> |
| **Postmark App** | <img alt="Postmark App icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/postmark.svg" height="30" height="30"/> | Source | airbyte/source-postmarkapp:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/postmarkapp) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-postmarkapp) | <small>`cde75ca1-1e28-4a0f-85bb-90c546de9f1f`</small> |
| **PrestaShop** | <img alt="PrestaShop icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/prestashop.svg" height="30" height="30"/> | Source | airbyte/source-prestashop:0.3.1 | beta | [link](https://docs.airbyte.com/integrations/sources/prestashop) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-prestashop) | <small>`d60a46d4-709f-4092-a6b7-2457f7d455f5`</small> |
| **Primetric** | <img alt="Primetric icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/primetric.svg" height="30" height="30"/> | Source | airbyte/source-primetric:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/primetric) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-primetric) | <small>`f636c3c6-4077-45ac-b109-19fc62a283c1`</small> |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/cockroachdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ Your database user should now be ready for use with Airbyte.

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :--- | :--- |
| 0.1.21 | 2023-03-14 | [24000](https://github.com/airbytehq/airbyte/pull/24000) | Removed check method call on read. |
| 0.1.20 | 2023-03-06 | [23455](https://github.com/airbytehq/airbyte/pull/23455) | For network isolation, source connector accepts a list of hosts it is allowed to connect |
| 0.1.19 | 2022-12-14 | [20436](https://github.com/airbytehq/airbyte/pull/20346) | Consolidate date/time values mapping for JDBC sources |
| | 2022-10-13 | [15535](https://github.com/airbytehq/airbyte/pull/16238) | Update incremental query to avoid data missing when new data is inserted at the same time as a sync starts under non-CDC incremental mode |
Expand Down
Loading