Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iceberg column rename handling #6118

Open
devinrsmith opened this issue Sep 24, 2024 · 2 comments
Open

Iceberg column rename handling #6118

devinrsmith opened this issue Sep 24, 2024 · 2 comments
Assignees
Labels
bug Something isn't working core Core development tasks iceberg parquet Related to the Parquet integration
Milestone

Comments

@devinrsmith
Copy link
Member

Deephaven does not currently handle Iceberg column renames.

Here is an example to produce a set of data that illustrates the issue
https://gist.github.com/devinrsmith/a6d537e5dce3ff0b25ba4b9c62d80ada which will first create the table, add some data, rename a column, and add some more data.

When read into Deephaven, only the latest data files have their data represented:

Screenshot from 2024-09-24 10-02-30

@devinrsmith devinrsmith added bug Something isn't working triage parquet Related to the Parquet integration iceberg labels Sep 24, 2024
@devinrsmith devinrsmith added this to the 0.37.0 milestone Sep 24, 2024
@devinrsmith devinrsmith self-assigned this Sep 24, 2024
@devinrsmith
Copy link
Member Author

https://github.com/apache/iceberg/blob/main/format/spec.md#schema-evolution should be consulted in general for issues about how schema evolution and tracking is supposed to happen.

https://www.mail-archive.com/dev@iceberg.apache.org/msg02236.html also talks a bit about Iceberg field-id and Parquet field_id.

@rcaudy rcaudy added core Core development tasks and removed triage labels Sep 25, 2024
@devinrsmith
Copy link
Member Author

Potentially related to #6124

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working core Core development tasks iceberg parquet Related to the Parquet integration
Projects
None yet
Development

No branches or pull requests

2 participants