Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Write 64bit aligned IPC files #1201

Merged
merged 1 commit into from
Aug 3, 2022
Merged

Write 64bit aligned IPC files #1201

merged 1 commit into from
Aug 3, 2022

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Aug 1, 2022

This is a follow-up to https://lists.apache.org/thread/3wh51ml8zt00131fgmxj1lqg92dlgj41 and related to #1197

The gist is that, in my opinion, to read mmapped Arrow IPC files soundly, the buffers need to be written with a bit alignment consistent with their types. Currently the spec does not to support this. As such, i128 and i256 can't be soundly read via mmap with 64bit alignment, and no type other than i8 and u8 can be soundly read in 8bit aligned files.

We currently write files whose every buffer is 8bit aligned, which restricts zero-copy reading every type other than i8 and u8. This PR changes the write alignment to 64 bits. Note that this has virtually no impact to performance or file size, but is a requirement to mmap IPC files as per #1197 (and in general).

@jorgecarleitao jorgecarleitao added the feature A new feature label Aug 1, 2022
@codecov
Copy link

codecov bot commented Aug 1, 2022

Codecov Report

Merging #1201 (435e9b4) into main (3f3febf) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1201   +/-   ##
=======================================
  Coverage   83.44%   83.44%           
=======================================
  Files         366      366           
  Lines       36697    36697           
=======================================
+ Hits        30621    30623    +2     
+ Misses       6076     6074    -2     
Impacted Files Coverage Δ
src/io/ipc/write/common.rs 80.67% <100.00%> (ø)
src/io/ipc/write/common_async.rs 95.83% <100.00%> (ø)
src/io/ipc/write/common_sync.rs 94.73% <100.00%> (ø)
src/io/ipc/write/serialize.rs 96.15% <100.00%> (ø)
src/io/ipc/read/reader.rs 96.32% <0.00%> (-0.34%) ⬇️
src/array/binary/mod.rs 90.12% <0.00%> (+1.23%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3f3febf...435e9b4. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit 3db00fc into main Aug 3, 2022
@jorgecarleitao jorgecarleitao deleted the write64bit branch August 3, 2022 03:30
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant