Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The new code is not templated, as we only ever encode uint32_t values currently.
It is much cleaner now, and the encoder itself is probably faster, but this does not really make
write_parquet()
faster, at least not for flights.In fact, it is slightly slower for flights x 200, probably because we pack a lot of sections that are shorter than 32 values, so we need to copy them into a buffer first. I guess I can use the old bit packer for these shorter sections.
Bit-packing directly into the output buffer is also problematic because the bit packer is writing out unaligned uint32_t values. It seems to work on aarch64 and x86_64, but it might now work correctly everywhere.