Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parquet page level skipping (page index pruning) doesn't work with evolved schemas #5104

Closed
alamb opened this issue Jan 29, 2023 · 3 comments · Fixed by #5268
Closed

parquet page level skipping (page index pruning) doesn't work with evolved schemas #5104

alamb opened this issue Jan 29, 2023 · 3 comments · Fixed by #5268
Assignees
Labels
bug Something isn't working

Comments

@alamb
Copy link
Contributor

alamb commented Jan 29, 2023

Describe the bug
parquet page level skipping (page index pruning) panic's with evolved schemas

To Reproduce

Enable setting like

        pub enable_page_index: bool, default = true

And then run test:

cargo test -p datafusion --lib -- evolved_schema_disjoint_schema_filter

Which fails

test physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter ... FAILED

failures:

---- physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter' panicked at 'index out of bounds: the len is 1 but the index is 1', /Users/alamb/.cargo/registry/src/github.com-1ecc6299db9ec823/parquet-31.0.0/src/file/metadata.rs:251:10

Expected behavior
Test should pass

Additional context
I found this while working to enable this feature by default
#5099

@alamb alamb added the bug Something isn't working label Jan 29, 2023
@alamb alamb self-assigned this Jan 29, 2023
@alamb
Copy link
Contributor Author

alamb commented Jan 29, 2023

FYI @Ted-Jiang -- I plan to fix this

@alamb
Copy link
Contributor Author

alamb commented Jan 29, 2023

I am pretty sure the issue is that the page pruning is trying to use a column index into the merged schema to find the relevant statistics, but each file schema can be slightly different. I think the fix will be to look up statistics based on name rather than index

@alamb
Copy link
Contributor Author

alamb commented Feb 12, 2023

The tests that fail are:


failures:
    physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter
    physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_with_filter_pushdown
    physical_plan::file_format::parquet::tests::evolved_schema_intersection_filter
    physical_plan::file_format::parquet::tests::evolved_schema_intersection_filter_with_filter_pushdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant