Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When ingesting a sql_database with a UUID column, getting a flood of warnings #1862

Closed
neuromantik33 opened this issue Sep 23, 2024 · 1 comment · Fixed by #1914
Closed
Assignees
Labels
bug Something isn't working

Comments

@neuromantik33
Copy link
Contributor

dlt version

1.0.0

Describe the problem

Exactly what the title says.

Expected behavior

No annoying warnings log messages.

Steps to reproduce

  • Create a table in any sql database with a UUID column type
  • run any pipeline with any destination

Operating system

Linux

Runtime environment

Local

Python version

3.11

dlt data source

built-in sql_database

dlt destination

Google BigQuery

Other deployment details

No response

Additional information

The bug can be easily corrected by changing https://github.com/dlt-hub/dlt/blob/devel/dlt/sources/sql_database/schema_types.py#L89

Before

    if hasattr(sqltypes, "Uuid") and isinstance(sql_t, sqltypes.Uuid):
        # we represent UUID as text by default, see default_table_adapter
        col["data_type"] = "text"
    if isinstance(sql_t, sqltypes.Numeric):
        # check for Numeric type first and integer later, some numeric types (ie. Oracle)
        # derive from both
        # all Numeric types that are returned as floats will assume "double" type
        # and returned as decimals will assume "decimal" type
        if sql_t.asdecimal is False:
            col["data_type"] = "double"

After

    if hasattr(sqltypes, "Uuid") and isinstance(sql_t, sqltypes.Uuid):
        # we represent UUID as text by default, see default_table_adapter
        col["data_type"] = "text"
    elif isinstance(sql_t, sqltypes.Numeric):
        # check for Numeric type first and integer later, some numeric types (ie. Oracle)
        # derive from both
        # all Numeric types that are returned as floats will assume "double" type
        # and returned as decimals will assume "decimal" type
        if sql_t.asdecimal is False:
            col["data_type"] = "double"
@rudolfix
Copy link
Collaborator

rudolfix commented Oct 1, 2024

this is actually a bug! do you maybe want to push this fix as PR? we'll release it early next week. thanks!

@rudolfix rudolfix added the bug Something isn't working label Oct 1, 2024
@rudolfix rudolfix moved this from Todo to In Progress in dlt core library Oct 1, 2024
@rudolfix rudolfix self-assigned this Oct 1, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in dlt core library Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants