Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#23949 Remove check() from read() method for sources Cockroach DB and Postgres #24000

Merged
merged 16 commits into from
Mar 15, 2023
Merged

#23949 Remove check() from read() method for sources Cockroach DB and Postgres #24000

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
ff5bfee
Removed extraneous read on the check() method for CockroachDB and Pos…
Mar 13, 2023
b92d154
Bumped dockerfile version for relevant conenctors
Mar 13, 2023
c1c2ebb
Bumped versions for strick-encrypt tests
Mar 13, 2023
e92d933
Removed withoutPublication and withoutReplicationSlot tests for postg…
Mar 13, 2023
a59116b
Merge branch 'master' into duy/remove-check-postgres-cockroach
nguyenaiden Mar 14, 2023
4f132ea
Automated Change
nguyenaiden Mar 14, 2023
0f212cf
UPdated sources docs
Mar 14, 2023
a94ffdb
Update incorrect year in change logs doc for Cockroach and Postgres
Mar 14, 2023
99d4264
Merge branch 'master' into duy/remove-check-postgres-cockroach
nguyenaiden Mar 14, 2023
2140268
Merge branch 'master' into duy/remove-check-postgres-cockroach
nguyenaiden Mar 15, 2023
b6e7ed3
Removed read override method from cockroachDB
Mar 15, 2023
336b785
Removed read method override for source-postgres
Mar 15, 2023
8c4c70d
Merge branch 'master' into duy/remove-check-postgres-cockroach
nguyenaiden Mar 15, 2023
a6b14d2
auto-bump connector version
octavia-squidington-iii Mar 15, 2023
882965a
auto-bump connector version
octavia-squidington-iii Mar 15, 2023
a53c252
Merge branch 'master' into duy/remove-check-postgres-cockroach
nguyenaiden Mar 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-cockroachdb-strict-encrypt/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-cockroachdb-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.20
LABEL io.airbyte.version=0.1.21
LABEL io.airbyte.name=airbyte/source-cockroachdb-strict-encrypt
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-cockroachdb/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-cockroachdb

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.20
LABEL io.airbyte.version=0.1.21
LABEL io.airbyte.name=airbyte/source-cockroachdb
5 changes: 0 additions & 5 deletions ...ckroachdb/src/main/java/io/airbyte/integrations/source/cockroachdb/CockroachDbSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,11 +99,6 @@ public AutoCloseableIterator<AirbyteMessage> read(final JsonNode config,
final ConfiguredAirbyteCatalog catalog,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just rid of the read() method override here and below in general, since all it's doing is delegating to the parent class

final JsonNode state)
throws Exception {
final AirbyteConnectionStatus check = check(config);

if (check.getStatus().equals(AirbyteConnectionStatus.Status.FAILED)) {
throw new RuntimeException("Unable establish a connection: " + check.getMessage());
}

return super.read(config, catalog, state);
}
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=2.0.2
LABEL io.airbyte.version=2.0.3
LABEL io.airbyte.name=airbyte/source-postgres-strict-encrypt
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-postgres/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=2.0.2
LABEL io.airbyte.version=2.0.3
LABEL io.airbyte.name=airbyte/source-postgres
7 changes: 0 additions & 7 deletions ...source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,13 +340,6 @@ public AutoCloseableIterator<AirbyteMessage> read(final JsonNode config,
final ConfiguredAirbyteCatalog catalog,
final JsonNode state)
throws Exception {
// this check is used to ensure that have the pgoutput slot available so Debezium won't attempt to
// create it.
final AirbyteConnectionStatus check = check(config);

if (check.getStatus().equals(Status.FAILED)) {
throw new RuntimeException("Unable establish a connection: " + check.getMessage());
}

return super.read(config, catalog, state);
}
Expand Down
19 changes: 0 additions & 19 deletions ...postgres/src/test/java/io/airbyte/integrations/source/postgres/CdcPostgresSourceTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,25 +224,6 @@ void testCheckWithoutReplicationSlot() throws Exception {
assertEquals(status.getStatus(), AirbyteConnectionStatus.Status.FAILED);
}

@Test
void testReadWithoutPublication() throws SQLException {
database.query(ctx -> ctx.execute("DROP PUBLICATION " + PUBLICATION + ";"));

assertThrows(Exception.class, () -> {
source.read(config, CONFIGURED_CATALOG, null);
});
}

@Test
void testReadWithoutReplicationSlot() throws SQLException {
final String fullReplicationSlot = SLOT_NAME_BASE + "_" + dbName;
database.query(ctx -> ctx.execute("SELECT pg_drop_replication_slot('" + fullReplicationSlot + "');"));

assertThrows(Exception.class, () -> {
source.read(config, CONFIGURED_CATALOG, null);
});
}

@Override
protected void assertExpectedStateMessages(final List<AirbyteStateMessage> stateMessages) {
assertEquals(1, stateMessages.size());
Expand Down
1 change: 1 addition & 0 deletions connectors.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@
| **Freshdesk** | <img alt="Freshdesk icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/freshdesk.svg" height="30" height="30"/> | Source | airbyte/source-freshdesk:3.0.2 | generally_available | [link](https://docs.airbyte.com/integrations/sources/freshdesk) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-freshdesk) | <small>`ec4b9503-13cb-48ab-a4ab-6ade4be46567`</small> |
| **Freshsales** | <img alt="Freshsales icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/freshsales.svg" height="30" height="30"/> | Source | airbyte/source-freshsales:0.1.2 | alpha | [link](https://docs.airbyte.com/integrations/sources/freshsales) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-freshsales) | <small>`eca08d79-7b92-4065-b7f3-79c14836ebe7`</small> |
| **Freshservice** | <img alt="Freshservice icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/freshservice.svg" height="30" height="30"/> | Source | airbyte/source-freshservice:0.1.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/freshservice) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-freshservice) | <small>`9bb85338-ea95-4c93-b267-6be89125b267`</small> |
| **GCS** | <img alt="GCS icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/gcs.svg" height="30" height="30"/> | Source | airbyte/source-gcs:0.1.0 | unknown | [link](https://docs.airbyte.com/integrations/sources/gcs) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-gcs) | <small>`2a8c41ae-8c23-4be0-a73f-2ab10ca1a820`</small> |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GCS was unaffected by this PR, yet it shows up here. I believe this happens when the feature branch is out of date of master. Once you merge this section should be removed. I'd double check before merging the PR in.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep! Merging in master fixed it.

| **GNews** | <img alt="GNews icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/gnews.svg" height="30" height="30"/> | Source | airbyte/source-gnews:0.1.3 | alpha | [link](https://docs.airbyte.com/integrations/sources/gnews) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-gnews) | <small>`ce38aec4-5a77-439a-be29-9ca44fd4e811`</small> |
| **Genesys** | <img alt="Genesys icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/genesys.svg" height="30" height="30"/> | Source | airbyte/source-genesys:0.1.0 | unknown | [link](https://docs.airbyte.com/integrations/sources/genesys) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-genesys) | <small>`5ea4459a-8f1a-452a-830f-a65c38cc438d`</small> |
| **GetLago** | <img alt="GetLago icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/getlago.svg" height="30" height="30"/> | Source | airbyte/source-getlago:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/getlago) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-getlago) | <small>`e1a3866b-d3b2-43b6-b6d7-8c1ee4d7f53f`</small> |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/cockroachdb.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ Your database user should now be ready for use with Airbyte.

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :--- | :--- |
| 0.1.21 | 2022-03-14 | [24000](https://github.com/airbytehq/airbyte/pull/24000) | Removed check method call on read. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix year in date :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh good call! On it now.

| 0.1.20 | 2023-03-06 | [23455](https://github.com/airbytehq/airbyte/pull/23455) | For network isolation, source connector accepts a list of hosts it is allowed to connect |
| 0.1.19 | 2022-12-14 | [20436](https://github.com/airbytehq/airbyte/pull/20346) | Consolidate date/time values mapping for JDBC sources |
| | 2022-10-13 | [15535](https://github.com/airbytehq/airbyte/pull/16238) | Update incremental query to avoid data missing when new data is inserted at the same time as a sync starts under non-CDC incremental mode |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,7 @@ The root causes is that the WALs needed for the incremental sync has been remove

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.0.3 | 2022-03-14 | [24000](https://github.com/airbytehq/airbyte/pull/24000) | Removed check method call on read. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix year, i suggest correcting all the ones showing up as 2022 since the start of this year.

| 2.0.2 | 2022-03-13 | [23112](https://github.com/airbytehq/airbyte/pull/21727) | Add state checkpointing for CDC sync. |
| 2.0.0 | 2022-03-06 | [23112](https://github.com/airbytehq/airbyte/pull/23112) | Upgrade Debezium version to 2.1.2 |
| 1.0.51 | 2022-03-02 | [23642](https://github.com/airbytehq/airbyte/pull/23642) | Revert : Support JSONB datatype for Standard sync mode |
Expand Down