Metadata ingestion fails for Bigquery - Invalid column name found in table partition

[alangeorge]

Affected module
Ingestion Framework

Describe the bug
Metadata ingestion fails for Bigquery tables with case mismatch in partition column,

In the Bigquery console, If we check the table details,
On the schema tab, we have the partition column name in upper case, and on the details page it's in lower case.

Error Log:

[2024-07-08 06:22:14.206027+00:00] {process_utils.py:190} INFO - [2024-07-08 06:22:14] DEBUG {metadata.Utils:execution_time_tracker:169} - Sink._run executed in 0.03s
[2024-07-08 06:22:14.206282+00:00] {process_utils.py:190} INFO - [2024-07-08 06:22:14] WARNING {metadata.Ingestion:status:76} - Failed to ingest CreateTableRequest [table_name] due to api request failure: Invalid column name found in table partition
[2024-07-08 06:22:14.206510+00:00] {process_utils.py:190} INFO - [2024-07-08 06:22:14] DEBUG {metadata.Ingestion:status:77} - Traceback (most recent call last):
[2024-07-08 06:22:14.206735+00:00] {process_utils.py:190} INFO - File "/tmp/venvv5kg8zna/lib/python3.11/site-packages/metadata/ingestion/ometa/client.py", line 219, in _one_request
[2024-07-08 06:22:14.206787+00:00] {process_utils.py:190} INFO - resp.raise_for_status()
[2024-07-08 06:22:14.207037+00:00] {process_utils.py:190} INFO - File "/tmp/venvv5kg8zna/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
[2024-07-08 06:22:14.207074+00:00] {process_utils.py:190} INFO - raise HTTPError(http_error_msg, response=self)
[2024-07-08 06:22:14.207471+00:00] {process_utils.py:190} INFO - requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://host_name/api/v1/tables
[2024-07-08 06:22:14.207543+00:00] {process_utils.py:190} INFO -
[2024-07-08 06:22:14.207568+00:00] {process_utils.py:190} INFO - The above exception was the direct cause of the following exception:
[2024-07-08 06:22:14.207575+00:00] {process_utils.py:190} INFO -
[2024-07-08 06:22:14.207709+00:00] {process_utils.py:190} INFO - Traceback (most recent call last):
[2024-07-08 06:22:14.207906+00:00] {process_utils.py:190} INFO - File "/tmp/venvv5kg8zna/lib/python3.11/site-packages/metadata/ingestion/sink/metadata_rest.py", line 145, in _run
[2024-07-08 06:22:14.207974+00:00] {process_utils.py:190} INFO - return self._run_dispatch(record)
[2024-07-08 06:22:14.208155+00:00] {process_utils.py:190} INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-07-08 06:22:14.208357+00:00] {process_utils.py:190} INFO - File "/opt/python3.11/lib/python3.11/functools.py", line 946, in _method
[2024-07-08 06:22:14.208480+00:00] {process_utils.py:190} INFO - return method.get(obj, cls)(*args, **kwargs)
[2024-07-08 06:22:14.208621+00:00] {process_utils.py:190} INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-07-08 06:22:14.208853+00:00] {process_utils.py:190} INFO - File "/tmp/venvv5kg8zna/lib/python3.11/site-packages/metadata/ingestion/sink/metadata_rest.py", line 136, in _run_dispatch
[2024-07-08 06:22:14.208930+00:00] {process_utils.py:190} INFO - return self.write_create_request(record)
[2024-07-08 06:22:14.209070+00:00] {process_utils.py:190} INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-07-08 06:22:14.209299+00:00] {process_utils.py:190} INFO - File "/tmp/venvv5kg8zna/lib/python3.11/site-packages/metadata/ingestion/sink/metadata_rest.py", line 166, in write_create_request
[2024-07-08 06:22:14.209443+00:00] {process_utils.py:190} INFO - created = self.metadata.create_or_update(entity_request)
[2024-07-08 06:22:14.209561+00:00] {process_utils.py:190} INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-07-08 06:22:14.209803+00:00] {process_utils.py:190} INFO - File "/tmp/venvv5kg8zna/lib/python3.11/site-packages/metadata/ingestion/ometa/ometa_api.py", line 276, in create_or_update
[2024-07-08 06:22:14.209906+00:00] {process_utils.py:190} INFO - return self._create(data=data, method="put")
[2024-07-08 06:22:14.210030+00:00] {process_utils.py:190} INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-07-08 06:22:14.210236+00:00] {process_utils.py:190} INFO - File "/tmp/venvv5kg8zna/lib/python3.11/site-packages/metadata/ingestion/ometa/ometa_api.py", line 267, in _create
[2024-07-08 06:22:14.210358+00:00] {process_utils.py:190} INFO - resp = fn(self.get_suffix(entity), data=data.json(encoder=show_secrets_encoder))
[2024-07-08 06:22:14.210555+00:00] {process_utils.py:190} INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-07-08 06:22:14.210753+00:00] {process_utils.py:190} INFO - File "/tmp/venvv5kg8zna/lib/python3.11/site-packages/metadata/utils/execution_time_tracker.py", line 195, in inner
[2024-07-08 06:22:14.210791+00:00] {process_utils.py:190} INFO - result = func(*args, **kwargs)
[2024-07-08 06:22:14.210972+00:00] {process_utils.py:190} INFO - ^^^^^^^^^^^^^^^^^^^^^
[2024-07-08 06:22:14.211059+00:00] {process_utils.py:190} INFO - File "/tmp/venvv5kg8zna/lib/python3.11/site-packages/metadata/ingestion/ometa/client.py", line 298, in put
[2024-07-08 06:22:14.211160+00:00] {process_utils.py:190} INFO - return self._request("PUT", path, data)
[2024-07-08 06:22:14.211230+00:00] {process_utils.py:190} INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-07-08 06:22:14.211434+00:00] {process_utils.py:190} INFO - File "/tmp/venvv5kg8zna/lib/python3.11/site-packages/metadata/ingestion/ometa/client.py", line 193, in _request
[2024-07-08 06:22:14.211518+00:00] {process_utils.py:190} INFO - return self._one_request(method, url, opts, retry)
[2024-07-08 06:22:14.211692+00:00] {process_utils.py:190} INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-07-08 06:22:14.211850+00:00] {process_utils.py:190} INFO - File "/tmp/venvv5kg8zna/lib/python3.11/site-packages/metadata/ingestion/ometa/client.py", line 237, in _one_request
[2024-07-08 06:22:14.211908+00:00] {process_utils.py:190} INFO - raise APIError(error, http_error) from http_error
[2024-07-08 06:22:14.212096+00:00] {process_utils.py:190} INFO - metadata.ingestion.ometa.client.APIError: Invalid column name found in table partition
[2024-07-08 06:22:14.212128+00:00] {process_utils.py:190} INFO -
[2024-07-08 06:22:14.212331+00:00] {process_utils.py:190} INFO - [2024-07-08 06:22:14] DEBUG {metadata.Utils:execution_time_tracker:169} - Sink executed in 0.03s
[2024-07-08 06:22:14.212460+00:00] {process_utils.py:190} INFO - [2024-07-08 06:22:14] DEBUG {metadata.Utils:execution_time_tracker:169} - Source executed in 0.0s

To Reproduce

Try to create a table using the python script attached below and do the ingestion

from google.cloud import bigquery

client = bigquery.Client()

dataset_id = 'test_openmetadata'
table_id = 'table_2'

schema = [
    bigquery.SchemaField('id', 'STRING', mode='REQUIRED'),
    bigquery.SchemaField('name', 'STRING', mode='REQUIRED'),
    bigquery.SchemaField('TRGT_DATE', 'DATE', mode='REQUIRED'),
    bigquery.SchemaField('value', 'FLOAT', mode='NULLABLE'),
]

table = bigquery.Table(f'{client.project}.{dataset_id}.{table_id}', schema=schema)
table.time_partitioning = bigquery.TimePartitioning(
    type_=bigquery.TimePartitioningType.DAY,
    field='trgt_date',  # The field used for partitioning
)

table = client.create_table(table)

print(f'Table {table.table_id} created with partitioning on {table.time_partitioning.field}')

Expected behavior
Able to ingest the tables with correct partition column name case present in the schema tab

Version:

• Python version: 3.10
• OpenMetadata version: 1.4.3
• OpenMetadata Ingestion package version: 1.4.3