This repository has been archived by the owner on Feb 18, 2024. It is now read-only.
Support loading Feather v2 (IPC) files with more than 1 million tables #231
Labels
enhancement
An improvement to an existing feature
As can be seen in pola-rs/polars#1023, loading of Feather v2 (IPC) files with more than 1 million tables does not work.
(py)arrow had the same bug: https://issues.apache.org/jira/projects/ARROW/issues/ARROW-10056
It boils down to the flatbuffer verification code, which has max_tables=1_000_000 by default.
Increasing this limit solves the problem.
In (py)arrow the max table value is determined per dataset based on the footer size, to prevent specially crafted IPC files to take an extraordinary amount of time to verify a very small input IPC file:
apache/arrow#9447
I think if your you replace gen::File::root_as_footer with gen::File::root_as_footer_with_opts, you can set the max_table option: At
arrow2/src/io/ipc/read/reader.rs
Line 98 in 16c089e
https://docs.rs/flatbuffers/2.0.0/src/flatbuffers/get_root.rs.html#39-49
The text was updated successfully, but these errors were encountered: