Skip to content

Commit

Permalink
#23949 Remove check() from read() method for sources Cockroach DB and…
Browse files Browse the repository at this point in the history
… Postgres (#24000)

* Removed extraneous read on the check() method for CockroachDB and Postgres
* Bumped dockerfile version for relevant conenctors
* Bumped versions for strick-encrypt tests
* Removed withoutPublication and withoutReplicationSlot tests for postgres cdc
* Automated Change
* Updated sources docs
* Update incorrect year in change logs doc for Cockroach and Postgres
* Removed read override method from cockroachDB
* Removed read method override for source-postgres
* auto-bump connector version

---------

Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
  • Loading branch information
3 people authored and erohmensing committed Mar 22, 2023
1 parent 9b7c972 commit 14389bb
Show file tree
Hide file tree
Showing 12 changed files with 26 additions and 73 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@
- name: Cockroachdb
sourceDefinitionId: 9fa5862c-da7c-11eb-8d19-0242ac130003
dockerRepository: airbyte/source-cockroachdb
dockerImageTag: 0.1.20
dockerImageTag: 0.1.21
documentationUrl: https://docs.airbyte.com/integrations/sources/cockroachdb
icon: cockroachdb.svg
sourceType: database
Expand Down Expand Up @@ -1536,7 +1536,7 @@
- name: Postgres
sourceDefinitionId: decd338e-5647-4c0b-adf4-da0e75f5a750
dockerRepository: airbyte/source-postgres
dockerImageTag: 2.0.2
dockerImageTag: 2.0.3
documentationUrl: https://docs.airbyte.com/integrations/sources/postgres
icon: postgresql.svg
sourceType: database
Expand Down
4 changes: 2 additions & 2 deletions airbyte-config/init/src/main/resources/seed/source_specs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2616,7 +2616,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-cockroachdb:0.1.20"
- dockerImage: "airbyte/source-cockroachdb:0.1.21"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/cockroachdb"
connectionSpecification:
Expand Down Expand Up @@ -11763,7 +11763,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-postgres:2.0.2"
- dockerImage: "airbyte/source-postgres:2.0.3"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/postgres"
connectionSpecification:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-cockroachdb-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.20
LABEL io.airbyte.version=0.1.21
LABEL io.airbyte.name=airbyte/source-cockroachdb-strict-encrypt
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-cockroachdb

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.20
LABEL io.airbyte.version=0.1.21
LABEL io.airbyte.name=airbyte/source-cockroachdb
Original file line number Diff line number Diff line change
Expand Up @@ -94,20 +94,6 @@ public Set<String> getExcludedInternalNameSpaces() {
"crdb_internal");
}

@Override
public AutoCloseableIterator<AirbyteMessage> read(final JsonNode config,
final ConfiguredAirbyteCatalog catalog,
final JsonNode state)
throws Exception {
final AirbyteConnectionStatus check = check(config);

if (check.getStatus().equals(AirbyteConnectionStatus.Status.FAILED)) {
throw new RuntimeException("Unable establish a connection: " + check.getMessage());
}

return super.read(config, catalog, state);
}

@Override
public Set<JdbcPrivilegeDto> getPrivilegesTableForCurrentUser(final JdbcDatabase database, final String schema) throws SQLException {
try (final Stream<JsonNode> stream = database.unsafeQuery(getPrivileges(database), sourceOperations::rowToJson)) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=2.0.2
LABEL io.airbyte.version=2.0.3
LABEL io.airbyte.name=airbyte/source-postgres-strict-encrypt
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-postgres/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=2.0.2
LABEL io.airbyte.version=2.0.3
LABEL io.airbyte.name=airbyte/source-postgres
Original file line number Diff line number Diff line change
Expand Up @@ -335,22 +335,6 @@ public List<CheckedConsumer<JdbcDatabase, Exception>> getCheckOperations(final J
return checkOperations;
}

@Override
public AutoCloseableIterator<AirbyteMessage> read(final JsonNode config,
final ConfiguredAirbyteCatalog catalog,
final JsonNode state)
throws Exception {
// this check is used to ensure that have the pgoutput slot available so Debezium won't attempt to
// create it.
final AirbyteConnectionStatus check = check(config);

if (check.getStatus().equals(Status.FAILED)) {
throw new RuntimeException("Unable establish a connection: " + check.getMessage());
}

return super.read(config, catalog, state);
}

@Override
public List<AutoCloseableIterator<AirbyteMessage>> getIncrementalIterators(final JdbcDatabase database,
final ConfiguredAirbyteCatalog catalog,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -224,25 +224,6 @@ void testCheckWithoutReplicationSlot() throws Exception {
assertEquals(status.getStatus(), AirbyteConnectionStatus.Status.FAILED);
}

@Test
void testReadWithoutPublication() throws SQLException {
database.query(ctx -> ctx.execute("DROP PUBLICATION " + PUBLICATION + ";"));

assertThrows(Exception.class, () -> {
source.read(config, CONFIGURED_CATALOG, null);
});
}

@Test
void testReadWithoutReplicationSlot() throws SQLException {
final String fullReplicationSlot = SLOT_NAME_BASE + "_" + dbName;
database.query(ctx -> ctx.execute("SELECT pg_drop_replication_slot('" + fullReplicationSlot + "');"));

assertThrows(Exception.class, () -> {
source.read(config, CONFIGURED_CATALOG, null);
});
}

@Override
protected void assertExpectedStateMessages(final List<AirbyteStateMessage> stateMessages) {
assertEquals(1, stateMessages.size());
Expand Down
4 changes: 2 additions & 2 deletions connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
| **ClickUp** | <img alt="ClickUp icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/clickup.svg" height="30" height="30"/> | Source | airbyte/source-clickup-api:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/click-up) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-clickup-api) | <small>`311a7a27-3fb5-4f7e-8265-5e4afe258b66`</small> |
| **Clockify** | <img alt="Clockify icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/clockify.svg" height="30" height="30"/> | Source | airbyte/source-clockify:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/clockify) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-clockify) | <small>`e71aae8a-5143-11ed-bdc3-0242ac120002`</small> |
| **Close.com** | <img alt="Close.com icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/close.svg" height="30" height="30"/> | Source | airbyte/source-close-com:0.2.1 | beta | [link](https://docs.airbyte.com/integrations/sources/close-com) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-close-com) | <small>`dfffecb7-9a13-43e9-acdc-b92af7997ca9`</small> |
| **Cockroachdb** | <img alt="Cockroachdb icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/cockroachdb.svg" height="30" height="30"/> | Source | airbyte/source-cockroachdb:0.1.20 | alpha | [link](https://docs.airbyte.com/integrations/sources/cockroachdb) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-cockroachdb) | <small>`9fa5862c-da7c-11eb-8d19-0242ac130003`</small> |
| **Cockroachdb** | <img alt="Cockroachdb icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/cockroachdb.svg" height="30" height="30"/> | Source | airbyte/source-cockroachdb:0.1.21 | alpha | [link](https://docs.airbyte.com/integrations/sources/cockroachdb) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-cockroachdb) | <small>`9fa5862c-da7c-11eb-8d19-0242ac130003`</small> |
| **Coda** | <img alt="Coda icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/coda.svg" height="30" height="30"/> | Source | airbyte/source-coda:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/coda) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-coda) | <small>`27f910fd-f832-4b2e-bcfd-6ab342e434d8`</small> |
| **Coin API** | <img alt="Coin API icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/coinapi.svg" height="30" height="30"/> | Source | airbyte/source-coin-api:0.1.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/coin-api) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-coin-api) | <small>`919984ef-53a2-479b-8ffe-9c1ddb9fc3f3`</small> |
| **CoinGecko Coins** | <img alt="CoinGecko Coins icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/coingeckocoins.svg" height="30" height="30"/> | Source | airbyte/source-coingecko-coins:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/coingecko-coins) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-coingecko-coins) | <small>`9cdd4183-d0ba-40c3-aad3-6f46d4103974`</small> |
Expand Down Expand Up @@ -175,7 +175,7 @@
| **PokeAPI** | <img alt="PokeAPI icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/pokeapi.svg" height="30" height="30"/> | Source | airbyte/source-pokeapi:0.1.5 | alpha | [link](https://docs.airbyte.com/integrations/sources/pokeapi) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-pokeapi) | <small>`6371b14b-bc68-4236-bfbd-468e8df8e968`</small> |
| **Polygon Stock API** | <img alt="Polygon Stock API icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/polygon.svg" height="30" height="30"/> | Source | airbyte/source-polygon-stock-api:0.1.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/polygon-stock-api) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-polygon-stock-api) | <small>`5807d72f-0abc-49f9-8fa5-ae820007032b`</small> |
| **PostHog** | <img alt="PostHog icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/posthog.svg" height="30" height="30"/> | Source | airbyte/source-posthog:0.1.8 | beta | [link](https://docs.airbyte.com/integrations/sources/posthog) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-posthog) | <small>`af6d50ee-dddf-4126-a8ee-7faee990774f`</small> |
| **Postgres** | <img alt="Postgres icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/postgresql.svg" height="30" height="30"/> | Source | airbyte/source-postgres:2.0.2 | generally_available | [link](https://docs.airbyte.com/integrations/sources/postgres) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-postgres) | <small>`decd338e-5647-4c0b-adf4-da0e75f5a750`</small> |
| **Postgres** | <img alt="Postgres icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/postgresql.svg" height="30" height="30"/> | Source | airbyte/source-postgres:2.0.3 | generally_available | [link](https://docs.airbyte.com/integrations/sources/postgres) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-postgres) | <small>`decd338e-5647-4c0b-adf4-da0e75f5a750`</small> |
| **Postmark App** | <img alt="Postmark App icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/postmark.svg" height="30" height="30"/> | Source | airbyte/source-postmarkapp:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/postmarkapp) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-postmarkapp) | <small>`cde75ca1-1e28-4a0f-85bb-90c546de9f1f`</small> |
| **PrestaShop** | <img alt="PrestaShop icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/prestashop.svg" height="30" height="30"/> | Source | airbyte/source-prestashop:0.3.1 | beta | [link](https://docs.airbyte.com/integrations/sources/prestashop) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-prestashop) | <small>`d60a46d4-709f-4092-a6b7-2457f7d455f5`</small> |
| **Primetric** | <img alt="Primetric icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/primetric.svg" height="30" height="30"/> | Source | airbyte/source-primetric:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/primetric) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-primetric) | <small>`f636c3c6-4077-45ac-b109-19fc62a283c1`</small> |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/cockroachdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ Your database user should now be ready for use with Airbyte.

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :--- | :--- |
| 0.1.21 | 2023-03-14 | [24000](https://github.com/airbytehq/airbyte/pull/24000) | Removed check method call on read. |
| 0.1.20 | 2023-03-06 | [23455](https://github.com/airbytehq/airbyte/pull/23455) | For network isolation, source connector accepts a list of hosts it is allowed to connect |
| 0.1.19 | 2022-12-14 | [20436](https://github.com/airbytehq/airbyte/pull/20346) | Consolidate date/time values mapping for JDBC sources |
| | 2022-10-13 | [15535](https://github.com/airbytehq/airbyte/pull/16238) | Update incremental query to avoid data missing when new data is inserted at the same time as a sync starts under non-CDC incremental mode |
Expand Down
Loading

0 comments on commit 14389bb

Please sign in to comment.