🐛 Destination BigQuery-denormalized: Added conversion from JSON Datetime to BigQuery format #7413

alexandertsukanov · 2021-10-27T13:47:13Z

What

The bug when DATETIME comes in the wrong format. Next error occurs:
JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details

How

Added the logic to handle DATETIME from JSON and adapt it for BigQuery format.
See Loading Data cloud storage JSON

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

alexandertsukanov · 2021-10-28T07:52:41Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1393622435
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1393622435
🐛 https://gradle.com/s/22vqudg3n6vby

alexandertsukanov · 2021-10-29T07:54:21Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1398037534
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1398037534
🐛 https://gradle.com/s/nzufdgh4jnnou

alexandertsukanov · 2021-10-29T08:29:53Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1398153331
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1398153331
No Python unittests run

alexandertsukanov · 2021-10-29T08:43:14Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1398197562
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1398197562
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1062    403    62%

docs/integrations/destinations/bigquery.md

...query/src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryRecordConsumer.java

etsybaev · 2021-10-30T16:22:05Z

One more point, please do not forget to go through the checklist. For example, nor now I see that title doesn't match the requirement in the checklist. Thanks

sherifnada · 2021-11-02T16:31:16Z

...ation-bigquery/src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryUtils.java

+   * @return The list of fields with datetime format.
+   *
+   */
+  public static List<String> getDateTimeFieldsFromSchema(Schema schema) {


does this handle nested fields? could the unit test also address it?

Thanks, I have added handling of that. Updated an integration test.

alexandertsukanov · 2021-11-03T08:03:59Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1415763266
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1415763266
No Python unittests run

alexandr-shegeda

LGTM.

alexandertsukanov · 2021-11-03T11:30:56Z

/publish connector=connectors/destination-bigquery-denormalized

Error: No ref found for: otsukanov/6638_destination_bigquery_denormalized_datetime_fix

alexandertsukanov · 2021-11-03T11:31:42Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1416504095
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1416504095
No Python unittests run

alexandertsukanov · 2021-11-03T11:32:51Z

/publish connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1416508154
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1416508154

…ime to BigQuery format (#7413)

…ime to BigQuery format (airbytehq#7413)

BUG-6638: Added conversion for DATETIME format.

745ea53

alexandertsukanov requested review from irynakruk and alexandr-shegeda October 27, 2021 13:47

github-actions bot added the area/connectors Connector related issues label Oct 27, 2021

alexandertsukanov linked an issue Oct 27, 2021 that may be closed by this pull request

Destination BigQuery Desnormalized: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details #6638

Closed

BUG-6638: Added conversion for DATETIME format.

74f3a98

alexandertsukanov temporarily deployed to more-secrets October 27, 2021 15:07 Inactive

alexandertsukanov added the gl label Oct 28, 2021

jrhizor temporarily deployed to more-secrets October 28, 2021 07:55 Inactive

BUG-6638: Added conversion for DATETIME format.

9802b04

alexandertsukanov temporarily deployed to more-secrets October 29, 2021 07:56 Inactive

jrhizor temporarily deployed to more-secrets October 29, 2021 07:56 Inactive

BUG-6638: Added conversion for DATETIME format.

e89b587

alexandertsukanov temporarily deployed to more-secrets October 29, 2021 08:30 Inactive

jrhizor temporarily deployed to more-secrets October 29, 2021 08:31 Inactive

jrhizor temporarily deployed to more-secrets October 29, 2021 08:45 Inactive

BUG-6638: Bumped versions and docs.

12de917

github-actions bot added the area/documentation Improvements or additions to documentation label Oct 29, 2021

alexandertsukanov temporarily deployed to more-secrets October 29, 2021 11:38 Inactive

irynakruk reviewed Oct 29, 2021

View reviewed changes

docs/integrations/destinations/bigquery.md Outdated Show resolved Hide resolved

irynakruk approved these changes Oct 29, 2021

View reviewed changes

BUG-6638: bigquery.md

53dcd5d

alexandertsukanov temporarily deployed to more-secrets October 29, 2021 13:27 Inactive

etsybaev reviewed Oct 30, 2021

View reviewed changes

...query/src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryRecordConsumer.java Outdated Show resolved Hide resolved

alexandertsukanov changed the title ~~🐛 BUG-6638: Added conversion from JSON Datetime to BigQuery format~~ 🐛 Destination BigQuery and BigQuery-denormalized: Added conversion from JSON Datetime to BigQuery format Nov 1, 2021

alexandertsukanov requested review from Phlair and removed request for sherifnada November 1, 2021 13:17

alexandertsukanov changed the title ~~🐛 Destination BigQuery and BigQuery-denormalized: Added conversion from JSON Datetime to BigQuery format~~ 🐛 Destination BigQuery-denormalized: Added conversion from JSON Datetime to BigQuery format Nov 2, 2021

sherifnada reviewed Nov 2, 2021

View reviewed changes

BUG-6638: Added handling of nested datetime.

49ae963

alexandertsukanov temporarily deployed to more-secrets November 3, 2021 08:03 Inactive

jrhizor temporarily deployed to more-secrets November 3, 2021 08:06 Inactive

BUG-6638: renaming of argument list.

a0b017f

alexandertsukanov temporarily deployed to more-secrets November 3, 2021 08:20 Inactive

alexandertsukanov requested a review from sherifnada November 3, 2021 09:07

alexandr-shegeda approved these changes Nov 3, 2021

View reviewed changes

alexandertsukanov merged commit a915034 into master Nov 3, 2021

alexandertsukanov deleted the otsukanov/6638_destination_bigquery_denormalized_datetime_fix branch November 3, 2021 11:29

alexandertsukanov restored the otsukanov/6638_destination_bigquery_denormalized_datetime_fix branch November 3, 2021 11:31

alexandertsukanov temporarily deployed to more-secrets November 3, 2021 11:33 Inactive

jrhizor temporarily deployed to more-secrets November 3, 2021 11:33 Inactive

jrhizor temporarily deployed to more-secrets November 3, 2021 11:34 Inactive

jrhizor mentioned this pull request Nov 3, 2021

Bump Airbyte version from 0.30.25-alpha to 0.30.26-alpha #7599

Merged

lmossman pushed a commit that referenced this pull request Nov 3, 2021

🐛 Destination BigQuery-denormalized: Added conversion from JSON Datet…

90e808e

…ime to BigQuery format (#7413)

alexandertsukanov deleted the otsukanov/6638_destination_bigquery_denormalized_datetime_fix branch November 8, 2021 12:17

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

🐛 Destination BigQuery-denormalized: Added conversion from JSON Datet…

9d8e73b

…ime to BigQuery format (airbytehq#7413)

karinakuz added connectors/destination/bigquery connectors/destination/bigquery-denormalized connectors/destinations-api connectors/destinations-warehouse and removed connectors/destinations-api labels Jan 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Destination BigQuery-denormalized: Added conversion from JSON Datetime to BigQuery format #7413

🐛 Destination BigQuery-denormalized: Added conversion from JSON Datetime to BigQuery format #7413

alexandertsukanov commented Oct 27, 2021 •

edited

Loading

alexandertsukanov commented Oct 28, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Oct 29, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Oct 29, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Oct 29, 2021 •

edited by github-actions bot

Loading

etsybaev commented Oct 30, 2021

sherifnada Nov 2, 2021

alexandertsukanov Nov 3, 2021

alexandertsukanov commented Nov 3, 2021 •

edited by github-actions bot

Loading

alexandr-shegeda left a comment

alexandertsukanov commented Nov 3, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Nov 3, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Nov 3, 2021 •

edited by github-actions bot

Loading

🐛 Destination BigQuery-denormalized: Added conversion from JSON Datetime to BigQuery format #7413

🐛 Destination BigQuery-denormalized: Added conversion from JSON Datetime to BigQuery format #7413

Conversation

alexandertsukanov commented Oct 27, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

alexandertsukanov commented Oct 28, 2021 • edited by github-actions bot Loading

alexandertsukanov commented Oct 29, 2021 • edited by github-actions bot Loading

alexandertsukanov commented Oct 29, 2021 • edited by github-actions bot Loading

alexandertsukanov commented Oct 29, 2021 • edited by github-actions bot Loading

etsybaev commented Oct 30, 2021

sherifnada Nov 2, 2021

Choose a reason for hiding this comment

alexandertsukanov Nov 3, 2021

Choose a reason for hiding this comment

alexandertsukanov commented Nov 3, 2021 • edited by github-actions bot Loading

alexandr-shegeda left a comment

Choose a reason for hiding this comment

alexandertsukanov commented Nov 3, 2021 • edited by github-actions bot Loading

alexandertsukanov commented Nov 3, 2021 • edited by github-actions bot Loading

alexandertsukanov commented Nov 3, 2021 • edited by github-actions bot Loading

alexandertsukanov commented Oct 27, 2021 •

edited

Loading

alexandertsukanov commented Oct 28, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Oct 29, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Oct 29, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Oct 29, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Nov 3, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Nov 3, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Nov 3, 2021 •

edited by github-actions bot

Loading

alexandertsukanov commented Nov 3, 2021 •

edited by github-actions bot

Loading