Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support parquet files with nanoseconds timestamps when converting a Parquet table to a Delta table #1721

Open
junjunjd opened this issue Oct 13, 2023 · 3 comments
Assignees
Labels
bug Something isn't working enhancement New feature or request

Comments

@junjunjd
Copy link
Contributor

junjunjd commented Oct 13, 2023

Description

When converting from ArrowSchema to deltalake::Schema, the try_from method will return an error if the arrow schema has a Timestamp(nanosecond). This causes the convert_to_delta function to fail. It makes sense to support parquet files with nanoseconds timestamps when converting a Parquet table to a Delta table.

Use Case

Convert parquet files in rust/tests/data/delta-1.2.1-only-struct-stats to a Delta table.

Related Issue(s)
#1041, #1682, #1686

@junjunjd junjunjd added the enhancement New feature or request label Oct 13, 2023
@rtyler rtyler added the bug Something isn't working label Oct 26, 2023
@rtyler rtyler added this to the Correct timestamp handling milestone Oct 26, 2023
@hntd187
Copy link
Collaborator

hntd187 commented Oct 27, 2023

Take

@MrPowers
Copy link
Collaborator

Sweet, assigned it to you @hntd187

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants