🐛 Destination BigQuery(Denormalized): Fix processing arrays of records

[DoNotPanicUA]

What

Fix issue #4487

Pre-merge Checklist

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions
• Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here

[DoNotPanicUA]

@sherifnada
Please be informed that the PR doesn't contain discussed test because in our test we use Java infrastructure which doesn't allow us to describe nested types at all.
So, tests for complex nested types will be added when we rework our java core for working with nested types.

[sherifnada]

@DoNotPanicUA we can add a catalog which contains nested records to BigqueryDestinationTest.java -- note that ConfiguredAirbyteCatalog supports nested schemas. They can be set in the property ConfiguredAirbyteStream.stream.json_schema. The non-nestable type io.airbyte.protocol.models.Field is just used as a helper in BigQueryDestinationTest.java but that's not strictly necessary, we can construct the configured stream directly and set it with a nested schema as needed.

We should ideally add a test for a nested schema to the destination acceptance tests, but those might require more effort to extend since they touch every destination. How about we add the catalog with the nested schema to bigquerydestinationtest.java specifically and create an issue to add it to the acceptance tests later?

[DoNotPanicUA]

@sherifnada
I've added an integration test for nested objects. The next step is to rework our sources for processing complex types and build a common base for running Sources and Connectors. After that, we could integration such testing into a common flow as it should be.

[DoNotPanicUA]

/test connector=destination-bigquery-denormalized

🕑 destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1113263813
✅ destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1113263813

[sherifnada]

great work!

[DoNotPanicUA]

/test connector=destination-bigquery-denormalized

🕑 destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1116580143
✅ destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1116580143

[DoNotPanicUA]

/publish connector=destination-bigquery-denormalized

❌ destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1116618147
❌ destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1116618147

[ChristopheDuong]

/publish connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1116651992
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1116651992