[C++] Support specifying filters and projections with Substrait expressions

[ianmcook]

Describe the enhancement requested

In addition to representing full plans, Substrait can also be used to represent expressions (see substrait-io/substrait#405). It would be nice if Dataset and Acero could consume Substrait expressions and use them to specify filters and projections.

I would love to see us expose functions that:

• Receive a Boolean-valued Substrait scalar expression and use it to filter a Dataset
• Receive a list of Substrait scalar expressions and use them to Project a Dataset
• Receive a Boolean-valued Substrait scalar expression and use it to add a Filter node to the ExecPlan
• Receive a list of Substrait scalar expressions and use it to add a Project node to the ExecPlan

Component(s)

C++

[ianmcook]

This plus substrait-io/substrait-java#128 would allow users to specify filters and projections as SQL expressions and execute them with Dataset and Acero.

[westonpace]

The Arrow equivalent of Substrait's expression is arrow::compute::Expression. So I think an extended expression proto file would roughly translate to std::vector<std::pair<std::string, arrow::compute::Expression>> (not actually suggesting we use this API, just describing).

If we added an API for that then those compute expressions could be used when building Acero filters & projects.

Note: this sort of implies the user is not using Substrait to express their actual queries. This is fine, we have non-Substrait APIs for filter & project in pyarrow (e.g. Array.filter) and R (dplyr) so there is certainly room for it.

[ianmcook]

@westonpace to clarify: with the implementation you envision, could this also give us the ability to pass Substrait expressions to arrow::dataset::ScannerBuilder::Filter() and arrow::dataset::ScannerBuilder::Project(), correct? That would be very, very cool indeed.

[westonpace]

You would need to use this variant of Project() but otherwise yes.