fix: properly convert nested types to BQ schema

[plaflamme]

This is related to #3184 but does not fix it. What it provides is a general function for converting a sqlglot schema to a bigquery schema. This new function is not in the codepath of altering schemas, but it should be possible to use it there to actually fix the issue.

[izeigerman]

Thanks a lot for this, @plaflamme . Can this logic be tested at all?

[Themiscodes]

Thank you @plaflamme! You're right when constructing the SchemaField we needed to traverse recursively to handle deeper levels of nesting correctly. And as you suggest changing:

sqlmesh/sqlmesh/core/engine_adapter/bigquery.py

Line 324 in 9833a61

bigquery.SchemaField(new_field[1], new_field[0], mode="NULLABLE")

to use your __get_bq_schemafield function:

self.__get_bq_schemafield(new_field[1], exp.DataType.build(new_field[0]))

does seem to resolve the issue when altering schemas, based on the examples I've tested so far.

[plaflamme]

@izeigerman I didn't add tests since this is a private method, but we could extract it and test it on different exp.DataTypes, yes. As for the codepath that uses this, probably under test in the bigquery integration tests, but I'm not sure.

@Themiscodes great! I wasn't confident applying this new function was going to be sufficient to fix the issue as there are other things going on in when altering a schema. Glad to hear it's working out.

[izeigerman]

Great fix, @plaflamme , thank you!