Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do not write ColumnIndex for null columns when not writing page statistics #6011

Merged
merged 4 commits into from
Jul 16, 2024
Merged

Do not write ColumnIndex for null columns when not writing page statistics #6011

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1704483
disable column_index_builder if no page stats are collected
etseidl Jul 5, 2024
8f059e9
add test
etseidl Jul 5, 2024
9b34950
no need to clone descr
etseidl Jul 5, 2024
735b73e
Merge remote-tracking branch 'apache/master' into gh-6010-colidx
alamb Jul 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion parquet/src/column/writer/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,12 @@ impl<'a, E: ColumnValueEncoder> GenericColumnWriter<'a, E> {
// Used for level information
encodings.insert(Encoding::RLE);

// Disable column_index_builder if not collecting page statistics.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the reason this works currently for columns without null is that the ColumnIndex builder is marked as invalid if there are no page statistics at the end of the page. However, when the column has no data this code is not run

arrow-rs/parquet/src/column/writer/mod.rs

Line 661 in 9b34950

self.column_index_builder.to_invalid();

So TLDR is I think this change looks good to me

let mut column_index_builder = ColumnIndexBuilder::new();
if statistics_enabled != EnabledStatistics::Page {
column_index_builder.to_invalid()
}

Self {
descr,
props,
Expand Down Expand Up @@ -289,7 +295,7 @@ impl<'a, E: ColumnValueEncoder> GenericColumnWriter<'a, E> {
num_column_nulls: 0,
column_distinct_count: None,
},
column_index_builder: ColumnIndexBuilder::new(),
column_index_builder,
offset_index_builder: OffsetIndexBuilder::new(),
encodings,
data_page_boundary_ascending: true,
Expand Down Expand Up @@ -3020,6 +3026,30 @@ mod tests {
assert!(incremented.is_none())
}

#[test]
fn test_no_column_index_when_stats_disabled() {
// https://github.com/apache/arrow-rs/issues/6010
// Test that column index is not created/written for all-nulls column when page
// statistics are disabled.
let descr = Arc::new(get_test_column_descr::<Int32Type>(1, 0));
let props = Arc::new(
WriterProperties::builder()
.set_statistics_enabled(EnabledStatistics::None)
.build(),
);
let column_writer = get_column_writer(descr, props, get_test_page_writer());
let mut writer = get_typed_column_writer::<Int32Type>(column_writer);

let data = Vec::new();
let def_levels = vec![0; 10];
writer.write_batch(&data, Some(&def_levels), None).unwrap();
writer.flush_data_pages().unwrap();

let column_close_result = writer.close().unwrap();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified that this test covers the code by removing the change and verifying the test fails like

assertion failed: column_close_result.column_index.is_none()
thread 'column::writer::tests::test_no_column_index_when_stats_disabled' panicked at parquet/src/column/writer/mod.rs:3044:9:
assertion failed: column_close_result.column_index.is_none()
stack backtrace:
   0: rust_begin_unwind
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:652:5
   1: core::panicking::panic_fmt
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panicking.rs:146:5
   3: parquet::column::writer::tests::test_no_column_index_when_stats_disabled
             at ./src/column/writer/mod.rs:3044:9
   4: parquet::column::writer::tests::test_no_column_index_when_stats_disabled::{{closure}}
             at ./src/column/writer/mod.rs:3024:50
   5: core::ops::function::FnOnce::call_once
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/ops/function.rs:250:5
   6: core::ops::function::FnOnce::call_once
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

assert!(column_close_result.offset_index.is_some());
assert!(column_close_result.column_index.is_none());
}

#[test]
fn test_boundary_order() -> Result<()> {
let descr = Arc::new(get_test_column_descr::<Int32Type>(1, 0));
Expand Down
Loading