parquet: parallelize over row groups ~3x

[ritchie46]

Can be ~3x faster if a parquet file contains many row groups.

Some timings:

[alexander-beedie]

Can be ~3x faster if a parquet file contains many row groups.

Awesome. On the other side of that, does polars automatically create row groups when writing to parquet now? There isn't an explicit param for it in write_ipc, but with these kinds of gains on the reading side, might be reasonable to expose one, so a round-trip through parquet could be optimized?

[ritchie46]

Awesome. On the other side of that, does polars automatically create row groups when writing to parquet now?

Not by default, but you can ensure polars does. DataFrame.write_parquet accepts a row_group_size argument that dictates the number of rows per rg.

Note that there is no free lunch, because polars likes contiguous data, so it can be that polars first needs to rechunk the data for some operations.

[alexander-beedie]

Not by default, but you can ensure polars does. DataFrame.write_parquet accepts a row_group_size argument that dictates the number of rows per rg.

Perfect; I have no idea why I went looking at write_ipc, hah... 🤦

[attila-lin]

Is the doc a typo?

[ritchie46]

Why?

[attila-lin]

seems Parallelize over the row groups should be the comment of RowGroups

[ritchie46]

Ah, now I see it. Jup, that's wrong.