Destination BigQuery: using application default GCP credentials broken

[rob-pomelo]

Connector Name

destination-bigquery

Connector Version

2.3.14

What step the error happened?

During the sync

Relevant information

https://github.com/airbytehq/airbyte/pull/32190/files#diff-71f64097f537f4492650bf539bdc98817ebf73178d7b7054a08b339abdd952f9 broke using application default credentials because it tries to parse the creds when they might be empty.

Broken since 2.3.7

Relevant log output

2023-11-13 23:31:24 destination > java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.isTextual()" because "serviceAccountKey" is null
2023-11-13 23:31:24 destination >         at io.airbyte.integrations.destination.bigquery.BigQueryDestination.getSerializedMessageConsumer(BigQueryDestination.java:243) ~[io.airbyte.airbyte-integrations.connectors-destination-bigquery-0.50.33.jar:?]
2023-11-13 23:31:24 destination >         at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:186) ~[airbyte-cdk-core-0.4.5.jar:?]
2023-11-13 23:31:24 destination >         at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) ~[airbyte-cdk-core-0.4.5.jar:?]
2023-11-13 23:31:24 destination >         at io.airbyte.integrations.destination.bigquery.BigQueryDestination.main(BigQueryDestination.java:493) ~[io.airbyte.airbyte-integrations.connectors-destination-bigquery-0.50.33.jar:?]
2023-11-13 23:32:27 INFO i.a.w.p.KubePodProcess(cleanup):732 - (pod: airbyte / source-typeform-read-2055-3-piwdq) - Destroying Kube process.

Contribute

• Yes, I want to contribute

[stevenmurphy12]

I'm seeing the same problem with v2.3.25, having upgraded from v2.2.0. I'm using application default credentials.

Connection test works fine, presumably as it hits this section:

airbyte/airbyte-integrations/connectors/destination-bigquery/src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryDestination.java

Lines 193 to 197 in f1099f5

	if (!BigQueryUtils.isUsingJsonCredentials(config)) {
	LOGGER.info("No service account key json is provided. It is required if you are using Airbyte cloud.");
	LOGGER.info("Using the default service account credential from environment.");
	return GoogleCredentials.getApplicationDefault();
	}

Whereas I get the same error as rob above in this section:

airbyte/airbyte-integrations/connectors/destination-bigquery/src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryDestination.java

Lines 238 to 239 in f1099f5

	final JsonNode serviceAccountKey = config.get(BigQueryConsts.CONFIG_CREDS);
	if (serviceAccountKey.isTextual()) {

Any thoughts on this @edgao ? (saw your commit on this code block )

[edgao]

ah, nice catch. For now - are you able to downgrade to an older version of destination-bigquery as a workaround?

[stevenmurphy12]

ah, nice catch. For now - are you able to downgrade to an older version of destination-bigquery as a workaround?

Yeah I can downgrade for now

[shikanime]

I'd like to further investigate this hypothesis and submit a pull request to fix this issue if no one else has done so. As I see it, the solution might be to add a guard before the JSON parsing via !config.has(BigQueryConsts.CONFIG_CREDS)) and remove the silent failure when parsing JSON fails in the isUsingJsonCredentials function.

[evantahler]

Thanks for taking a look at this @shikanime! Let me know when your PR is ready for review

[shikanime]

@evantahler I think the PR should be ok for a review

[evantahler]

Thanks! I opened a new PR (#34073) to address some metadata stuff. I'll close this PR and merge in #34073 soon!