Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Different numeric type may be able to compare #6357

Open
tisonkun opened this issue Sep 5, 2024 · 2 comments
Open

Different numeric type may be able to compare #6357

tisonkun opened this issue Sep 5, 2024 · 2 comments

Comments

@tisonkun
Copy link
Member

tisonkun commented Sep 5, 2024

} else if l_t != r_t {
return Err(ArrowError::InvalidArgumentError(format!(
"Invalid comparison operation: {l_t} {op} {r_t}"
)));
}

Crates like num-cmp shows that different numeric type (e.g., u64 and i64) is comparable.

Is it possible to extend arrow array comparison to support compare these two kind of datum?

@tustvold
Copy link
Contributor

tustvold commented Sep 5, 2024

The major difficulty here is combinatorial explosion of code generation that would result, something that is already a major issue. This not only bloats binaries but contributes to long compilation times.

Instead what most query engines, such as DataFusion do, is use the cast kernels to coerce the inputs to a common type before performing the comparison. This may not be as optimal as doing the comparison on the heterogenous types, but is likely good enough given such heterogeneous comparisons are comparatively rare.

@alamb
Copy link
Contributor

alamb commented Sep 9, 2024

It is also fairly straightforward to implement your own kernels for specific comparisons if you have special cases that are not handled by arrow-rs (the challenge is mananging the code complexity for all the possible type combinations, as alluded to above by @tustvold )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants