Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid data type for Delta Lake: Dictionary(Int32, Utf8) #1546

Closed
yefetBenTili opened this issue Jul 19, 2023 · 2 comments
Closed

Invalid data type for Delta Lake: Dictionary(Int32, Utf8) #1546

yefetBenTili opened this issue Jul 19, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@yefetBenTili
Copy link

yefetBenTili commented Jul 19, 2023

Environment

Delta-rs version: 0.10.0

Binding:

Environment:

  • Cloud provider: AWS
  • OS: macOs
  • Other:

**Exception: Schema error: **:

I am trying to write to an already existing deltalake destination in s3 using delta

import pyarrow.parquet as pq
from deltalake.writer import write_deltalake

df = pq.read_table('data')

storage_options = {
    “AWS_DEFAULT_REGION”: “eu-central-1",
    “AWS_ACCESS_KEY_ID”: os.environ[“AWS_ACCESS_KEY_ID”],
    “AWS_SECRET_ACCESS_KEY”: os.environ[“AWS_SECRET_ACCESS_KEY”],
    “AWS_S3_ALLOW_UNSAFE_RENAME”: “true”,
}

destination =  "s3://some_s3_location"
write_deltalake(destination,
    df,
    mode=“append”,
    storage_options=storage_options,
    partition_by=[“titles”, "train_title", "date"]
)

The data gets written to s3 but somehow I get this exception
Invalid data type for Delta Lake: Dictionary(Int32, Utf8)

Any clue why this is happening?
I played around it little bit and seem there is something wrong with data type of the partition keys folders

@yefetBenTili yefetBenTili added the bug Something isn't working label Jul 19, 2023
@cmackenzie1
Copy link
Contributor

I believe this is the same issue as #1445, which will be fixed in #1481

@ion-elgreco
Copy link
Collaborator

@yefetBenTili are you still having this bug?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants