Handle len of -1 in "compressed" buffers from other languages #442
Conversation
It's unclear why other language implementations will have a compression set for arrow data, then indicate that the length is -1, as a sentinel value that the data is actually _not_ compressed. But since they do, we can handle that case pretty easily. I'm basically just adding a test here from @DrChainsaw's original PR.
quinnj
changed the title
Codecov Report
@@ Coverage Diff @@
## main #442 +/- ##
==========================================
+ Coverage 87.06% 87.09% +0.03%
==========================================
Files 26 26
Lines 3279 3279
==========================================
+ Hits 2855 2856 +1
+ Misses 424 423 -1
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
Thanks alot for this! In case it is helpful, here is the java code which determines whether to set the -1 flag or not: https://github.com/apache/arrow/blob/fbe5f641d327ee81db00ce5f056940a69f4d8603/java/vector/src/main/java/org/apache/arrow/vector/compression/AbstractCompressionCodec.java#L42-L53 The tl;dr is that they check whether the size after compression is larger than the uncompressed data. Since this can be different for different columns you can end up with a table with a mixture of compressed and non-compressed columns. I suppose this is an optimization that the Julia writer could implement as well given that it seems like it is out there. I have no idea what the potential gains are though. I have searched the "Specification and Protocols" section of the Arrow docs for rules on how to set the length when applying compression but I could not find anything. If you happen to know where it is specified I would be happy to take a look since it might help with the other issue I have encountered when reading files generated by the java implementation. |
Wow, what a terrible idea! If it ends up larger compressed, it's usually by a very small amount and you're usually dealing w/ small amount of data anyway, so the complication of mixing/matching compression w/ length sentinels just seems way overboard. |
It's unclear why other language implementations will have a compression set for arrow data, then indicate that the length is -1, as a sentinel value that the data is actually not compressed. But since they do, we can handle that case pretty easily. I'm basically just adding a test here from @DrChainsaw's original PR (#436 ).