You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Hive tables with complex struct types are not correctly loaded.
E.g. Ingestion of following table:
`test_table`(
`a` struct<b:decimal(20,0)>
)
throws an exception:
File "/home/airflow/.local/lib/python3.9/site-packages/metadata/ingestion/source/database/common_db_source.py", line 361, in yield_table
) = self.get_columns_and_constraints(
File "/home/airflow/.local/lib/python3.9/site-packages/metadata/ingestion/source/database/sql_column_handler.py", line 236, in get_columns_and_constraints
columns = inspector.get_columns(table_name, schema_name, db_name=db_name)
File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 497, in get_columns
col_defs = self.dialect.get_columns(
File "/home/airflow/.local/lib/python3.9/site-packages/metadata/ingestion/source/database/hive/utils.py", line 65, in get_columns
args = (int(charlen),)
ValueError: invalid literal for int() with base 10: '20,0'
The source code in metadata/ingestion/source/database/hive/utils.py expect any types containing number in round brackets to be int or decimal and cannot process complex struct types.
The text was updated successfully, but these errors were encountered:
Affected module
Ingestion Framework
Describe the bug
Hive tables with complex struct types are not correctly loaded.
E.g. Ingestion of following table:
throws an exception:
The source code in
metadata/ingestion/source/database/hive/utils.py
expect any types containing number in round brackets to be int or decimal and cannot process complex struct types.The text was updated successfully, but these errors were encountered: