-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BigQuery Denormalized : Cover arrays only if they are nested #14023
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
NOTE
|
…-denorm-no-array-sur
/test connector=connectors/destination-bigquery
Build PassedTest summary info:
|
/test connector=connectors/destination-bigquery-denormalized
Build PassedTest summary info:
|
NOTE
|
} | ||
|
||
/** | ||
* Compare field modes. Field can have on of three modes: NULLABLE, REQUIRED, REPEATED, null. Only |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo: "one of three modes"
also, "NULLABLE", "REQUIRED", and "REPEATED" are the modes, but then there is also "null" at the end of the sentence.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
NOTE
|
@grishick |
…hq#14023) * stop covering any array. cover only if we have array of arrays (restriction of BigQuery) * add test with nested arrays and update existing tests * [14058] fix datetime arrays * [11109] cover only array of arrays by object instead of any array * [14058] fix datetime format fail when we have an array of objects with datetime * enable Array and Array+Object DATs * reopen Issue airbytehq#11166 and disable functionality * Improve the tests by moving common part to Utils * Add tests to check `Array of arrays` cases * Increase version * Doc * format * review update: - update comment about reopen issue - added test case with multiply array sub values - fix nested arrays with datetime - add test case for nested arrays with datetime * fix date formatting * disable testAnyOf test and upd comments * remove some code duplication in the tests * [14668] cover by tests the BigQuery inheritance limitation * Make GCS implementation running same tests as standard impl * Make common format for returning date values to cover DateTime and Timestamp columns by one test * [15363] add backward compatibility for existing connections. * Populate stream config and messages by tablespace. Now it's required inside processing. * Compare only fields from the stream config * Rework BigQueryUploaderFactory and UploaderConfig to have possibility make a decision about array formmater before we create temporary table * Compare fields * remove extra logging * fix project:dataset format of the datasetId * missing import * remove debug logging * fix log messages * format * 4 > 3
…rbytehq#16494) * Increase version for BQ PR airbytehq#14023 * auto-bump connector version [ci skip] * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
…hq#14023) * stop covering any array. cover only if we have array of arrays (restriction of BigQuery) * add test with nested arrays and update existing tests * [14058] fix datetime arrays * [11109] cover only array of arrays by object instead of any array * [14058] fix datetime format fail when we have an array of objects with datetime * enable Array and Array+Object DATs * reopen Issue airbytehq#11166 and disable functionality * Improve the tests by moving common part to Utils * Add tests to check `Array of arrays` cases * Increase version * Doc * format * review update: - update comment about reopen issue - added test case with multiply array sub values - fix nested arrays with datetime - add test case for nested arrays with datetime * fix date formatting * disable testAnyOf test and upd comments * remove some code duplication in the tests * [14668] cover by tests the BigQuery inheritance limitation * Make GCS implementation running same tests as standard impl * Make common format for returning date values to cover DateTime and Timestamp columns by one test * [15363] add backward compatibility for existing connections. * Populate stream config and messages by tablespace. Now it's required inside processing. * Compare only fields from the stream config * Rework BigQueryUploaderFactory and UploaderConfig to have possibility make a decision about array formmater before we create temporary table * Compare fields * remove extra logging * fix project:dataset format of the datasetId * missing import * remove debug logging * fix log messages * format * 4 > 3
…rbytehq#16494) * Increase version for BQ PR airbytehq#14023 * auto-bump connector version [ci skip] * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
What
Recommended reading order
DefaultBigQueryDenormalizedRecordFormatter.java
BigQueryUtils.java
BigQueryDenormalizedDestinationTest.java
BigQueryDenormalizedDestinationAcceptanceTest.java