Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mock test environment doesn't test Trino/Iceberg limitations #30

Open
MichaelTiemannOSC opened this issue May 7, 2022 · 2 comments
Open
Assignees
Labels
bug Something isn't working

Comments

@MichaelTiemannOSC
Copy link
Contributor

 # This passes Mock test, but fails when used in Trino/Iceberg environment

def test_trino_pandas_insert():
    import pandas as pd
    # mock up an sqlalchemy table
    tbl = mock.MagicMock()
    tbl.name = "test_pandas"
    # mock up an sqlalchemy Connnection
    cxn = mock.MagicMock()
    df = pd.DataFrame(
        {"A": [4.5], "B'C": [math.nan], None: [math.inf], "D": [-math.inf], "E": [datetime(2022, 1, 1)], ":F": [1.0]}
    ).convert_dtypes()
    assert (df.dtypes == ['Float64', 'Int64', 'Float64', 'Float64', 'datetime64[ns]', 'Int64']).all()
    df.to_sql(
        tbl.name,
        con=cxn,
        schema="test",
        if_exists="append",
        index=False,
        method=TrinoBatchInsert(batch_size=5, verbose=True),
    )

The failure is that datetime64[ns] is translated to timestamp(3) by Pandas, but Iceberg wants timestamp(6). I have tried to figure out the correct dtype argument to pass to to_sql, but everything I've tried passing so far resolves to timestamp(3) inside of Pandas. I need to figure out how to pass it something that will resolve to timestamp(6).

@MichaelTiemannOSC MichaelTiemannOSC added the bug Something isn't working label May 7, 2022
@MichaelTiemannOSC
Copy link
Contributor Author

Some hacking required: trinodb/trino-python-client#107

@erikerlandson
Copy link
Contributor

Will trinodb/trino-python-client#181 resolve?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: No status
Development

No branches or pull requests

2 participants