Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move more parquet functionality behind experimental feature flag (#1032) #1134

Merged
merged 2 commits into from
Jan 10, 2022

Conversation

tustvold
Copy link
Contributor

@tustvold tustvold commented Jan 5, 2022

Draft as builds on #1133

Which issue does this PR close?

Closes #1032

Rationale for this change

See ticket

What changes are included in this PR?

This hides a number of modules behind an "experimental" feature flag. These largely seem to contain implementation details, and so making them private will allow for faster iteration without having to worry about breaking clients that haven't explicitly opted in to instability.

Are there any user-facing changes?

Yes, this moves a number of APIs to be crate-local. I am skeptical people are using these APIs, but it is possible.

@github-actions github-actions bot added the parquet Changes to the parquet crate label Jan 5, 2022
//! assert_eq!(output, data);
//! ```

#[cfg_attr(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is necessary because without the experimental feature this will fail to compile complaining about compression being private.

FWIW this module is never documented now, but I opted to preserve the test.

You can see it running with:

$ cargo test --all-features --doc
    Finished test [unoptimized + debuginfo] target(s) in 0.04s
   Doc-tests parquet

running 19 tests
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 1092) ... ignored
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 1098) ... ignored
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 1103) ... ignored
test src/file/mod.rs - file (line 64) - compile ... ok
test src/compression.rs - compression::CodecType (line 27) - compile ... ok
test src/file/mod.rs - file (line 29) - compile ... ok
test src/file/mod.rs - file (line 81) - compile ... ok
test src/record/api.rs - record::api::Row::get_column_iter (line 62) - compile ... ok
test src/column/mod.rs - column (line 38) - compile ... ok
test src/file/properties.rs - file::properties (line 22) ... ok
test src/schema/types.rs - schema::types::ColumnPath::append (line 674) ... ok
test src/arrow/mod.rs - arrow (line 58) ... ok
test src/schema/types.rs - schema::types::ColumnPath::string (line 663) ... ok
test src/schema/mod.rs - schema (line 22) ... ok
test src/schema/printer.rs - schema::printer (line 23) ... ok
test src/schema/parser.rs - schema::parser (line 24) ... ok
test src/file/statistics.rs - file::statistics (line 23) ... ok
test src/arrow/mod.rs - arrow (line 25) ... ok
test src/arrow/mod.rs - arrow (line 71) ... ok

test result: ok. 16 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 3.12s

@alamb
Copy link
Contributor

alamb commented Jan 5, 2022

Rebased on top of master

@tustvold tustvold marked this pull request as ready for review January 10, 2022 10:45
@tustvold tustvold marked this pull request as draft January 10, 2022 10:45
@tustvold
Copy link
Contributor Author

Looking into test failures

@tustvold tustvold marked this pull request as ready for review January 10, 2022 11:07
@alamb alamb added the api-change Changes to the arrow API label Jan 10, 2022
@alamb alamb merged commit 5302b92 into apache:master Jan 10, 2022
asayers added a commit to asayers/sqlite2parquet that referenced this pull request Jan 28, 2022
This requires adding the "experimental" flag because of
apache/arrow-rs#1134
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api-change Changes to the arrow API parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reduce Public Parquet API
2 participants