Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LargeList and List type coercion not working in CASE WHEN #12370

Closed
ion-elgreco opened this issue Sep 7, 2024 · 1 comment · Fixed by #12490
Closed

LargeList and List type coercion not working in CASE WHEN #12370

ion-elgreco opened this issue Sep 7, 2024 · 1 comment · Fixed by #12490
Assignees
Labels
bug Something isn't working

Comments

@ion-elgreco
Copy link

ion-elgreco commented Sep 7, 2024

Describe the bug

Recently we made a change in delta-rs to make the schema's agnostic of large/normal/view types. However datafusion can't properly coerce List and LargeList yet

To Reproduce

import polars as pl
tmp_path = "test_table"

pl.DataFrame({"a": ["test"], "b": [[1]]}).write_delta(tmp_path)
pl.DataFrame({"a": ["test1"], "b": [[1]]}).write_delta(
    tmp_path,
    mode="merge",
    delta_merge_options={
        "predicate": "s.a = t.a",
        "source_alias": "s",
        "target_alias": "t",
    },
).when_not_matched_insert_all().execute()
DeltaError: Generic DeltaTable error: type_coercion
caused by
Error during planning: Failed to coerce then ([LargeList(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })]) and else (None) to common types in CASE WHEN expression

Expected behavior

Be able to coerce any list type combination, whether its large, view or normal

Additional context

delta-io/delta-rs#2851

@ion-elgreco ion-elgreco added the bug Something isn't working label Sep 7, 2024
@goldmedal
Copy link
Contributor

take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants