Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hive ingestion fails on complex struct types #13429

Closed
lunive opened this issue Oct 4, 2023 · 0 comments · Fixed by #14785
Closed

Hive ingestion fails on complex struct types #13429

lunive opened this issue Oct 4, 2023 · 0 comments · Fixed by #14785
Assignees
Labels
bug Something isn't working collaboration Ingestion

Comments

@lunive
Copy link

lunive commented Oct 4, 2023

Affected module
Ingestion Framework

Describe the bug
Hive tables with complex struct types are not correctly loaded.

E.g. Ingestion of following table:

`test_table`(
  `a` struct<b:decimal(20,0)>
)

throws an exception:

 File "/home/airflow/.local/lib/python3.9/site-packages/metadata/ingestion/source/database/common_db_source.py", line 361, in yield_table
    ) = self.get_columns_and_constraints(
  File "/home/airflow/.local/lib/python3.9/site-packages/metadata/ingestion/source/database/sql_column_handler.py", line 236, in get_columns_and_constraints
    columns = inspector.get_columns(table_name, schema_name, db_name=db_name)
  File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 497, in get_columns
    col_defs = self.dialect.get_columns(
  File "/home/airflow/.local/lib/python3.9/site-packages/metadata/ingestion/source/database/hive/utils.py", line 65, in get_columns
    args = (int(charlen),)
ValueError: invalid literal for int() with base 10: '20,0'

The source code in metadata/ingestion/source/database/hive/utils.py expect any types containing number in round brackets to be int or decimal and cannot process complex struct types.

@pmbrull pmbrull added bug Something isn't working Ingestion labels Oct 27, 2023
@harshach harshach moved this to Ingestion - Bugs & Minor Features in Release 1.3.0 Nov 30, 2023
@ayush-shah ayush-shah moved this from Ingestion - Bugs & Minor Features to Backend - Bugs & Minor Features in Release 1.3.0 Jan 19, 2024
@ayush-shah ayush-shah moved this from Backend - Bugs & Minor Features to Done in Release 1.3.0 Jan 19, 2024
Abhishek332 pushed a commit to Abhishek332/OpenMetadata that referenced this issue Jan 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working collaboration Ingestion
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants