Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Bumped parquet and aligned API to fit into it #795

Merged
merged 4 commits into from
Feb 2, 2022
Merged

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Jan 30, 2022

In essence, the API to write parquet files and (sync and async) was not very flexible. This PR addresses this with a simpler design based on a struct.

How to use the new API:

  • FileWriter::start, FileWriter::write and FileWriter::end
  • FileStreamer::start, FileStreamer::write and FileStreamer::end

the sync and async variants respectively. This allows more control over the lifetime of the writer.

@codecov
Copy link

codecov bot commented Jan 30, 2022

Codecov Report

Merging #795 (1c1dd10) into main (601fa08) will increase coverage by 0.40%.
The diff coverage is 53.84%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #795      +/-   ##
==========================================
+ Coverage   71.08%   71.48%   +0.40%     
==========================================
  Files         319      320       +1     
  Lines       16672    16757      +85     
==========================================
+ Hits        11851    11979     +128     
+ Misses       4821     4778      -43     
Impacted Files Coverage Δ
src/io/parquet/write/mod.rs 61.08% <ø> (-1.24%) ⬇️
src/io/parquet/write/stream.rs 0.00% <0.00%> (ø)
src/io/parquet/write/file.rs 68.00% <68.00%> (ø)
src/io/parquet/write/row_group.rs 83.33% <81.81%> (ø)
src/bitmap/utils/slice_iterator.rs 91.04% <0.00%> (-1.50%) ⬇️
src/compute/arithmetics/time.rs 26.60% <0.00%> (+0.91%) ⬆️
src/array/binary/mutable.rs 78.57% <0.00%> (+1.29%) ⬆️
src/io/avro/write/schema.rs 66.66% <0.00%> (+3.03%) ⬆️
src/io/avro/read/deserialize.rs 74.79% <0.00%> (+3.25%) ⬆️
src/io/avro/read/schema.rs 65.92% <0.00%> (+11.08%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 601fa08...1c1dd10. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit 777f375 into main Feb 2, 2022
@jorgecarleitao jorgecarleitao deleted the bump_parquet branch February 2, 2022 20:00
@jorgecarleitao jorgecarleitao changed the title Bump parquet and aligned API to fit into it Bumped parquet and aligned API to fit into it Mar 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant