Source Postgres: Fix not being able to configure wal2json plugin

[edgao]

https://github.com/airbytehq/airbyte/pull/18779/files#diff-144a4a0053518cdd47b47a03ce984697ca4081a3d49e8d70505915ab7650dadaL266-R250 accidentally switched the plugin config option to const: pgoutput. Switch it back to default, assuming it still passes SATs.

[github-actions]

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

• Run integration tests
• Bump connector or module version
• Add changelog
• Publish the new version

⚠ Sources (3)

Connector	Version	Changelog	Publish
source-alloydb	1.0.17	✅	✅
source-alloydb-strict-encrypt	1.0.17	✅	⚠ (not in seed)
source-postgres-strict-encrypt	1.0.30	✅	⚠ (not in seed)

• See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

• See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version.

[smartpierre]

Hi, is this why I get these errors on my Postgresql syncs ? If yes, how can I fix ?

2022-12-01 18:31:04 INFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed. 
errors: $.method: must be a constant value Standard
2022-12-01 18:31:04 INFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed. 
errors: $.plugin: must be a constant value pgoutput
2022-12-01 18:31:04 INFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed. 
errors: $.method: does not have a value in the enumeration [Standard]

Here the source configuration in Airbyte database :

{
    "host": "redacted",
    "port": 5432,
    "database": "redacted",
    "password": {
        "_secret": "redacted"
    },
    "username": "airbyte",
    "replication_method": {
        "method": "CDC",
        "plugin": "wal2json",
        "publication": "airbyte_publication_dataupload",
        "replication_slot": "airbyte_replication_slot_dataupload"
    }
}

Best regards, Pierre.

[edgao]

@smartpierre yeah, that sounds related. As a workaround, I think you can downgrade source-postgres to 1.0.23. But also I'm hoping to get this PR merged in the next couple days, if you're ok with waiting til then.

[edgao]

/test connector=connectors/source-jdbc

🕑 connectors/source-jdbc https://github.com/airbytehq/airbyte/actions/runs/3597791206
✅ connectors/source-jdbc https://github.com/airbytehq/airbyte/actions/runs/3597791206
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     199     93    53%   35, 41-43, 48, 53, 59, 65, 71, 77-79, 98, 103-105, 111-113, 119-120, 125-126, 131, 137, 146-155, 161-166, 181, 205, 236, 242, 250-255, 263-268, 276-289, 294-300, 307-318, 325-341
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1569    350    78%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 20.74s ==================

[edgao]

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3597793540
✅ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3597793540
No Python unittests run

Build Passed

Test summary info:

All Passed

[edgao]

/test connector=connectors/source-postgres-strict-encrypt

🕑 connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3597795469
✅ connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3597795469
No Python unittests run

Build Passed

Test summary info:

All Passed

[edgao]

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3597798535
✅ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3597798535
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     199     93    53%   35, 41-43, 48, 53, 59, 65, 71, 77-79, 98, 103-105, 111-113, 119-120, 125-126, 131, 137, 146-155, 161-166, 181, 205, 236, 242, 250-255, 263-268, 276-289, 294-300, 307-318, 325-341
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1569    350    78%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 22.35s ==================

[edgao]

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3623810271
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3623810271
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     199     93    53%   35, 41-43, 48, 53, 59, 65, 71, 77-79, 98, 103-105, 111-113, 119-120, 125-126, 131, 137, 146-155, 161-166, 181, 205, 236, 242, 250-255, 263-268, 276-289, 294-300, 307-318, 325-341
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1569    350    78%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 35.95s ==================

[edgao]

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3623809387

[edgao]

merging b/c this only affects source-postgres production code. source-mysql has broken tests but they're slightly less broken after this PR. @rodireich will publish this as part of another PR.

[edgao]

@smartpierre - source-postgres:1.0.31 has also been released, so you can upgrade to the latest version and it will include this fix.

[smartpierre]

Hello @edgao we updated airbyte and our connectors, however, now it doesn't block the execution, but I can still see the errors.
Here's the output from one of our connections :

2022-12-19 14:26:17 INFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed. 
errors: $.method: must be a constant value Standard
2022-12-19 14:26:17 INFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed. 
errors: $.access_token: is missing but it is required, $.refresh_token: is missing but it is required
2022-12-19 14:26:17 INFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed. 
errors: $.private_key: is missing but it is required
2022-12-19 14:26:17 INFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed. 
errors: $.password: object found, string expected
2022-12-19 14:26:17 INFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed. 
errors: $.method: does not have a value in the enumeration [Standard]

Airbyte version : 0.40.26
Postgresql source : 1.0.34
Snowflake destination : 0.4.40

Hope that helps, thanks for the fix 😄

[edgao]

Is this causing sync failures? (and can you post the full logs?) I think this is expected (just a misleading log message)

[smartpierre]

No, the syncs are running properly !

But I was wondering why the log messages are still here. Marking validation errors as INFO are kinda misleading I guess.

[edgao]

yeah... maybe makes sense to shift this to debug level

airbyte/airbyte-json-validation/src/main/java/io/airbyte/validation/json/JsonSchemaValidator.java

Lines 126 to 134 in 1a6a1c6

	public boolean test(final JsonNode schemaJson, final JsonNode objectJson) {
	final Set<ValidationMessage> validationMessages = validateInternal(schemaJson, objectJson);
	if (!validationMessages.isEmpty()) {
	LOGGER.info("JSON schema validation failed. \nerrors: {}", Strings.join(validationMessages, ", "));
	}
	return validationMessages.isEmpty();
	}

it's generally safe to ignore though - there are a number of cases where schema validation is actually expected to fail (e.g. after a connector spec change)