[Java] Push-down filtering in Java

[noahfrn]

Push-down filtering (Java) questions

Hey all,

I'm exploring adding push-down filtering to the Java Datasets API, as it currently only supports push-down projection.
Just had a couple of questions before I start this work:

• Is this planned on any current roadmap? I'd like to avoid duplicating work that's already begun
• Does anyone have any smart ideas as to how to pass the filter predicate into ScanOptions? As far as I can tell, the only complication to make this happen would be figuring out some way to call into the Expression object from Java without creating future maintenance effort.

Thanks!

Component(s)

C++, Java

[lidavidm]

CC @lwhite1 @davisusanibar

An alternative might be to accept Substrait plans instead, binding to Acero as a whole instead of just Dataset. This would give you the full power of the query engine and avoid having to create bindings to individual C++ components. But of course this is more complex and Substrait is a bit of a moving target.

Another alternative is to track the discussion about a text format for expressions. See https://lists.apache.org/thread/7vch27t3gfz1hmv7d8w69n50gfc1nswf and apache/arrow#14287.

[noahfrn]

Thanks for your help @lidavidm - that expression parsing PR looks like exactly what I wanted!

[ianmcook]

There is a discussion here about passing Substrait expressions to the Dataset Project and Filter methods: apache/arrow#33985 (comment)

If this gets implemented in C++, can it be exposed to Java through the JNI bindings?

[lidavidm]

I think this is just about having a convenient user-facing API to the existing Dataset functionality, a Substrait API to Acero in Java would be a separate project

[ianmcook]

Ok, thanks. To clarify: my question is not about Substrait plans, it's about Substrait expressions which we now have a way to represent independent of plans. I opened a separate issue to request this feature: apache/arrow#34252

[lidavidm]

I think without a convenient API to build Substrait expressions in Java, it'd still not quite meet the goals here right?

[noahfrn]

Agreed @lidavidm - Something like substrait-io/substrait-java#128 would provide the necessary functionality here.

[ianmcook]

apache/arrow#34252 is now complete, and the Arrow Java Datasets API now supports pushdown projection and filtering using Substrait expressions.

More details here: https://github.com/apache/arrow/blob/main/docs/source/java/dataset.rst#projection-produce-new-columns-and-filters

Example here: https://github.com/apache/arrow/blob/main/docs/source/java/substrait.rst#executing-projections-and-filters-using-extended-expressions

However: this capability is not really ready for practical applications yet, because we do not yet have any user-friendly tools to create Substrait expressions in Java. I hope we can achieve that in substrait-io/substrait-java#128.