Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#23949 Remove check() from read() method for sources Cockroach DB and Postgres #24000

Merged
merged 16 commits into from
Mar 15, 2023
Merged

#23949 Remove check() from read() method for sources Cockroach DB and Postgres #24000

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
ff5bfee
Removed extraneous read on the check() method for CockroachDB and Pos…
Mar 13, 2023
b92d154
Bumped dockerfile version for relevant conenctors
Mar 13, 2023
c1c2ebb
Bumped versions for strick-encrypt tests
Mar 13, 2023
e92d933
Removed withoutPublication and withoutReplicationSlot tests for postg…
Mar 13, 2023
a59116b
Merge branch 'master' into duy/remove-check-postgres-cockroach
nguyenaiden Mar 14, 2023
4f132ea
Automated Change
nguyenaiden Mar 14, 2023
0f212cf
UPdated sources docs
Mar 14, 2023
a94ffdb
Update incorrect year in change logs doc for Cockroach and Postgres
Mar 14, 2023
99d4264
Merge branch 'master' into duy/remove-check-postgres-cockroach
nguyenaiden Mar 14, 2023
2140268
Merge branch 'master' into duy/remove-check-postgres-cockroach
nguyenaiden Mar 15, 2023
b6e7ed3
Removed read override method from cockroachDB
Mar 15, 2023
336b785
Removed read method override for source-postgres
Mar 15, 2023
8c4c70d
Merge branch 'master' into duy/remove-check-postgres-cockroach
nguyenaiden Mar 15, 2023
a6b14d2
auto-bump connector version
octavia-squidington-iii Mar 15, 2023
882965a
auto-bump connector version
octavia-squidington-iii Mar 15, 2023
a53c252
Merge branch 'master' into duy/remove-check-postgres-cockroach
nguyenaiden Mar 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-cockroachdb-strict-encrypt/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-cockroachdb-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.20
LABEL io.airbyte.version=0.1.21
LABEL io.airbyte.name=airbyte/source-cockroachdb-strict-encrypt
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-cockroachdb/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-cockroachdb

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.20
LABEL io.airbyte.version=0.1.21
LABEL io.airbyte.name=airbyte/source-cockroachdb
14 changes: 0 additions & 14 deletions ...ckroachdb/src/main/java/io/airbyte/integrations/source/cockroachdb/CockroachDbSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,20 +94,6 @@ public Set<String> getExcludedInternalNameSpaces() {
"crdb_internal");
}

@Override
public AutoCloseableIterator<AirbyteMessage> read(final JsonNode config,
final ConfiguredAirbyteCatalog catalog,
final JsonNode state)
throws Exception {
final AirbyteConnectionStatus check = check(config);

if (check.getStatus().equals(AirbyteConnectionStatus.Status.FAILED)) {
throw new RuntimeException("Unable establish a connection: " + check.getMessage());
}

return super.read(config, catalog, state);
}

@Override
public Set<JdbcPrivilegeDto> getPrivilegesTableForCurrentUser(final JdbcDatabase database, final String schema) throws SQLException {
try (final Stream<JsonNode> stream = database.unsafeQuery(getPrivileges(database), sourceOperations::rowToJson)) {
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=2.0.2
LABEL io.airbyte.version=2.0.3
LABEL io.airbyte.name=airbyte/source-postgres-strict-encrypt
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-postgres/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=2.0.2
LABEL io.airbyte.version=2.0.3
LABEL io.airbyte.name=airbyte/source-postgres
16 changes: 0 additions & 16 deletions ...source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -335,22 +335,6 @@ public List<CheckedConsumer<JdbcDatabase, Exception>> getCheckOperations(final J
return checkOperations;
}

@Override
public AutoCloseableIterator<AirbyteMessage> read(final JsonNode config,
final ConfiguredAirbyteCatalog catalog,
final JsonNode state)
throws Exception {
// this check is used to ensure that have the pgoutput slot available so Debezium won't attempt to
// create it.
final AirbyteConnectionStatus check = check(config);

if (check.getStatus().equals(Status.FAILED)) {
throw new RuntimeException("Unable establish a connection: " + check.getMessage());
}

return super.read(config, catalog, state);
}

@Override
public List<AutoCloseableIterator<AirbyteMessage>> getIncrementalIterators(final JdbcDatabase database,
final ConfiguredAirbyteCatalog catalog,
Expand Down
19 changes: 0 additions & 19 deletions ...postgres/src/test/java/io/airbyte/integrations/source/postgres/CdcPostgresSourceTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,25 +224,6 @@ void testCheckWithoutReplicationSlot() throws Exception {
assertEquals(status.getStatus(), AirbyteConnectionStatus.Status.FAILED);
}

@Test
void testReadWithoutPublication() throws SQLException {
database.query(ctx -> ctx.execute("DROP PUBLICATION " + PUBLICATION + ";"));

assertThrows(Exception.class, () -> {
source.read(config, CONFIGURED_CATALOG, null);
});
}

@Test
void testReadWithoutReplicationSlot() throws SQLException {
final String fullReplicationSlot = SLOT_NAME_BASE + "_" + dbName;
database.query(ctx -> ctx.execute("SELECT pg_drop_replication_slot('" + fullReplicationSlot + "');"));

assertThrows(Exception.class, () -> {
source.read(config, CONFIGURED_CATALOG, null);
});
}

@Override
protected void assertExpectedStateMessages(final List<AirbyteStateMessage> stateMessages) {
assertEquals(1, stateMessages.size());
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/cockroachdb.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ Your database user should now be ready for use with Airbyte.

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :--- | :--- |
| 0.1.21 | 2023-03-14 | [24000](https://github.com/airbytehq/airbyte/pull/24000) | Removed check method call on read. |
| 0.1.20 | 2023-03-06 | [23455](https://github.com/airbytehq/airbyte/pull/23455) | For network isolation, source connector accepts a list of hosts it is allowed to connect |
| 0.1.19 | 2022-12-14 | [20436](https://github.com/airbytehq/airbyte/pull/20346) | Consolidate date/time values mapping for JDBC sources |
| | 2022-10-13 | [15535](https://github.com/airbytehq/airbyte/pull/16238) | Update incremental query to avoid data missing when new data is inserted at the same time as a sync starts under non-CDC incremental mode |
Expand Down
29 changes: 15 additions & 14 deletions docs/integrations/sources/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -396,20 +396,21 @@ The root causes is that the WALs needed for the incremental sync has been remove

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.0.2 | 2022-03-13 | [23112](https://github.com/airbytehq/airbyte/pull/21727) | Add state checkpointing for CDC sync. |
| 2.0.0 | 2022-03-06 | [23112](https://github.com/airbytehq/airbyte/pull/23112) | Upgrade Debezium version to 2.1.2 |
| 1.0.51 | 2022-03-02 | [23642](https://github.com/airbytehq/airbyte/pull/23642) | Revert : Support JSONB datatype for Standard sync mode |
| 1.0.50 | 2022-02-27 | [21695](https://github.com/airbytehq/airbyte/pull/21695) | Support JSONB datatype for Standard sync mode |
| 1.0.49 | 2022-02-24 | [23383](https://github.com/airbytehq/airbyte/pull/23383) | Fixed bug with non readable double-quoted values within a database name or column name |
| 1.0.48 | 2022-02-23 | [22623](https://github.com/airbytehq/airbyte/pull/22623) | Increase max fetch size of JDBC streaming mode |
| 1.0.47 | 2022-02-22 | [22221](https://github.com/airbytehq/airbyte/pull/23138) | Fix previous versions which doesn't verify privileges correctly, preventing CDC syncs to run. |
| 1.0.46 | 2022-02-21 | [23105](https://github.com/airbytehq/airbyte/pull/23105) | Include log levels and location information (class, method and line number) with source connector logs published to Airbyte Platform. |
| 1.0.45 | 2022-02-09 | [22221](https://github.com/airbytehq/airbyte/pull/22371) | Ensures that user has required privileges for CDC syncs. |
| | 2022-02-15 | [23028](https://github.com/airbytehq/airbyte/pull/23028) | |
| 1.0.44 | 2022-02-06 | [22221](https://github.com/airbytehq/airbyte/pull/22221) | Exclude new set of system tables when using `pg_stat_statements` extension. |
| 1.0.43 | 2022-02-06 | [21634](https://github.com/airbytehq/airbyte/pull/21634) | Improve Standard sync performance by caching objects. |
| 1.0.42 | 2022-01-23 | [21523](https://github.com/airbytehq/airbyte/pull/21523) | Check for null in cursor values before replacing. |
| 1.0.41 | 2022-01-25 | [20939](https://github.com/airbytehq/airbyte/pull/20939) | Adjust batch selection memory limits databases. |
| 2.0.3 | 2023-03-14 | [24000](https://github.com/airbytehq/airbyte/pull/24000) | Removed check method call on read. |
| 2.0.2 | 2023-03-13 | [23112](https://github.com/airbytehq/airbyte/pull/21727) | Add state checkpointing for CDC sync. |
| 2.0.0 | 2023-03-06 | [23112](https://github.com/airbytehq/airbyte/pull/23112) | Upgrade Debezium version to 2.1.2 |
| 1.0.51 | 2023-03-02 | [23642](https://github.com/airbytehq/airbyte/pull/23642) | Revert : Support JSONB datatype for Standard sync mode |
| 1.0.50 | 2023-02-27 | [21695](https://github.com/airbytehq/airbyte/pull/21695) | Support JSONB datatype for Standard sync mode |
| 1.0.49 | 2023-02-24 | [23383](https://github.com/airbytehq/airbyte/pull/23383) | Fixed bug with non readable double-quoted values within a database name or column name |
| 1.0.48 | 2023-02-23 | [22623](https://github.com/airbytehq/airbyte/pull/22623) | Increase max fetch size of JDBC streaming mode |
| 1.0.47 | 2023-02-22 | [22221](https://github.com/airbytehq/airbyte/pull/23138) | Fix previous versions which doesn't verify privileges correctly, preventing CDC syncs to run. |
| 1.0.46 | 2023-02-21 | [23105](https://github.com/airbytehq/airbyte/pull/23105) | Include log levels and location information (class, method and line number) with source connector logs published to Airbyte Platform. |
| 1.0.45 | 2023-02-09 | [22221](https://github.com/airbytehq/airbyte/pull/22371) | Ensures that user has required privileges for CDC syncs. |
| | 2023-02-15 | [23028](https://github.com/airbytehq/airbyte/pull/23028) | |
| 1.0.44 | 2023-02-06 | [22221](https://github.com/airbytehq/airbyte/pull/22221) | Exclude new set of system tables when using `pg_stat_statements` extension. |
| 1.0.43 | 2023-02-06 | [21634](https://github.com/airbytehq/airbyte/pull/21634) | Improve Standard sync performance by caching objects. |
| 1.0.42 | 2023-01-23 | [21523](https://github.com/airbytehq/airbyte/pull/21523) | Check for null in cursor values before replacing. |
| 1.0.41 | 2023-01-25 | [20939](https://github.com/airbytehq/airbyte/pull/20939) | Adjust batch selection memory limits databases. |
| 1.0.40 | 2023-01-24 | [21825](https://github.com/airbytehq/airbyte/pull/21825) | Put back the original change that will cause an incremental sync to error if table contains a NULL value in cursor column. |
| 1.0.39 | 2023-01-20 | [21683](https://github.com/airbytehq/airbyte/pull/21683) | Speed up esmtimates for trace messages in non-CDC mode. |
| 1.0.38 | 2023-01-17 | [20436](https://github.com/airbytehq/airbyte/pull/20346) | Consolidate date/time values mapping for JDBC sources |
Expand Down