fix: scan partitioned tables with datafusion

[roeap]

Description

This PR builds on #1293 and tries to address some of the issues we have seen with scanning
partitioned tables with datafusion. And while all our tests (mostly - more on that later) pass,
the fix involves some behaviours that we may or may not want to adopt. Specifically, Datafusion
appends partition columns at the end of the schema fields, while we have been reporting them
as leading columns.

In recent datafusion versions also changed the default for dictionary encoding partition columns
to be opt in. My thinking was that for the vast majority of tables keeping dictionary encoding for
partition columns would be the desired behaviour. (@wjones127, do you have an opinion on that?).
This was also a root cause or at least related to the second failing test.

I did have to comment out some caeses within out file pruning tests where we create expression with
nulls, as I have thus far not been able to create an expression that datafusion is happy with. I'll
keep trying, but have some work on expression parsing for handling user inputs planned as well.
There already is a draft PR open (#1267), which does not contain that yet, but where I plan to
address this.

cc @cmackenzie1

Related Issue(s)

• closes Unable to query partitioned tables #1291
• closes Datafusion: SQL projection returns wrong column for partitioned data #1292
• closes DataFusion integration incorrectly handles partition columns defined "first" in schema #1168

[rtyler]

All things considered I think this is fine to land. Are there some issues filed with Data Fusion that can be dropped in these comments for the missing kernels?

[wjones127]

Specifically, Datafusion appends partition columns at the end of the schema fields, while we have been reporting them as leading columns.

Eventually, I think our desired behavior is going to be put the partition columns where they are in the Delta table schema.

In recent datafusion versions also changed the default for dictionary encoding partition columns to be opt in. My thinking was that for the vast majority of tables keeping dictionary encoding for partition columns would be the desired behaviour.

Looking at the PR for that apache/datafusion#5545 (comment), they make a good case that dictionary encoding partition columns is really only helpful for string columns. So perhaps we should only use them for string / binary columns (any maybe dates too)? I think dictionary encoding most columns makes sense. The ones I would exclude are ones that are so small that dictionary arrays are a waste of space, such as boolean and i8. And if we wanted to get fancy we could choose the dictionary index bit width based on the known number of unique values (that is, i8 if 256 or fewer, i16 if there are much more). ¹

Footnotes

1. Eventually, I think we'd like them to use the new REE (run-end encoded) arrays. But that's far in the future. ↩

[wjones127]

Some minor suggestions for comments but otherwise seems good.

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits
specification for
release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.