Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Fixed error in writing compressed IPC arrow #855

Merged
merged 1 commit into from
Feb 18, 2022

Conversation

jorgecarleitao
Copy link
Owner

There is some ambiguity in the format over how to count buffer lengths, but I think this handles it.

Will work on apache/arrow to clear up the ambiguity and add an integration test for this.

@jorgecarleitao jorgecarleitao added the bug Something isn't working label Feb 18, 2022
@codecov
Copy link

codecov bot commented Feb 18, 2022

Codecov Report

Merging #855 (7124086) into main (6ccb154) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #855      +/-   ##
==========================================
- Coverage   71.49%   71.48%   -0.01%     
==========================================
  Files         335      335              
  Lines       17910    17902       -8     
==========================================
- Hits        12805    12798       -7     
+ Misses       5105     5104       -1     
Impacted Files Coverage Δ
src/io/ipc/write/serialize.rs 65.18% <100.00%> (-0.29%) ⬇️
src/io/ipc/write/common_sync.rs 96.66% <0.00%> (-3.34%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ccb154...7124086. Read the comment docs.

@jorgecarleitao
Copy link
Owner Author

cc @ghuls (related to pola-rs/polars#2639)

@ghuls
Copy link
Contributor

ghuls commented Feb 18, 2022

Thanks, I will test it on a big file to see if PyArrow likes it now.

@ghuls
Copy link
Contributor

ghuls commented Feb 18, 2022

Pyarrow seems to be able to read lz4 and zstd feather files created by arrow2 now (for the full big file).
Thanks a lot.

@jorgecarleitao jorgecarleitao merged commit c012c9d into main Feb 18, 2022
@jorgecarleitao jorgecarleitao deleted the fix_write_compressed branch February 18, 2022 21:34
sydduckworth pushed a commit to mindx/arrow2 that referenced this pull request Mar 2, 2022
sydduckworth pushed a commit to mindx/arrow2 that referenced this pull request Mar 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants