IPC projection that does not required to be ordered

[ritchie46]

Currently we require projection of IPC readers to be ordered. Is this for technical reasons?

Downstream code could be simpler if it does not have to bookkeep the column order.

[jorgecarleitao]

It is a technical requirement, but we could move the logic downstream to this crate; this is basically a middleware that receives the unordered projection, orders it and reads, and then re-orders the columns before returning the Chunk.

My feeling is that re-ordering of columns during a projection fits better to a query engine, but I do not have strong feelings here.

Other alternative is to change our indexing to Vec<bool> whose length must be equal to fields, to avoid confusion. Technically this is what happens: we need to know which columns to select in IPC, since an un-selected column must be correctly "skipped" from fields being read.

[ritchie46]

Yeah, I noticed that polars' csv parser also needed sorted projections, because that's the order we encounter the fields. Parquet does not seem to need it.

I made an abstraction downstream. I believe it makes sense to have this requirement based on the data format. I will close this as I don't think it matters much.

[ghuls]

IPC projection does not require to be ordered anymore: #961