Predicate to filter conversion.

[fnothaft]

We need a way to convert Parquet predicates into Spark filters—this is needed for the adamRead methods for both read and variant data, as we ignore the predicate passed if reading BAM/SAM/VCF data.

[nealsid]

Just curious, why do we need to convert the predicate to a Spark filter rather than just not ignoring the parquet predicate in the BAM/SAM/VCF code path?

[nealsid]

Ah, please disregard. I thought the predicate was being ignored for BAM/SAM data inside ADAM files (the name adamLoad is confusing in this respect)

[arahuja]

I took an attempt at this here: https://github.com/hammerlab/adam/compare/predicate-bam?expand=1 Would love some feedback or alternative approaches - right now this only supports boolean equality filters.

[nealsid]

This looks good! Are you intentionally only starting with booleans for the
value to sketch out the broader approach, or am I missing something and
that will encompass all possible filters?

On Wed, Apr 16, 2014 at 2:54 PM, Arun Ahuja notifications@github.comwrote:

I took an attempt at this here:
https://github.com/hammerlab/adam/compare/predicate-bam?expand=1 Would
love some feedback or alternative approaches - right now this only supports
boolean equality filters.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-40637624
.

[arahuja]

Yea it was for simplicity, pushed an update now for more general equality support. I hadn't realized that Parquet only supported equality filters. @fnothaft I saw you were the last contributor to Parquet on that - is there support elsewhere for other types of ColumnFilters?

Also, I see what I did only supports AND's of conditions as well.

[nealsid]

Good stuff. I wonder if it would be worth it to generalize and let people
pass in arbitrary boolean functions rather than specific values to match
against. The function could take as input the Parquet value. Would that
work? Then you could express arbitrary computation as a condition for the
record to pass.
Neal

On Wed, Apr 16, 2014 at 7:32 PM, Arun Ahuja notifications@github.comwrote:

Yea it was for simplicity, pushed an update now for more general equality
support. I hadn't realized that Parquet only supported equality filters.
@fnothaft https://github.com/fnothaft I saw you were the last
contributor to Parquet on that - is there support elsewhere for other types
of ColumnFilters?

Also, I see what I did only supports AND's of conditions as well.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-40665152
.

[fnothaft]

Nice! Indeed, this does look cool, and like a nice approach.

@arahuja I've actually added code to Parquet that supports arbitrary predicate functions. These are the "applyFunctionTo*" predicates. If you've got any questions about the predicate functions, or would like any changes made to them, let me know. I'd be glad to help on this, or to work on anything that'd make the predicates easier to use.

[tdanford]

@arahuja @fnothaft Is this issue still live, or can it be closed?

[fnothaft]

Closed by #234.