-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Fix empty aggregation function count() in Substrait #15345
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
||
let args = from_substrait_func_args(consumer, &f.arguments, input_schema).await?; | ||
|
||
// deal with situation that count(*) got no arguments |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you just explain in the comment why we need count()
to have arguments?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pretty much just rolled back whatever was removed in #14824, I would like some input first from @jayzhan211 about why was this removed and if it might be breaking something else that the tests are not catching
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I remove it because I don't why we need to convert count()
to count(1)
, is there any reason we need count(1)
in subtrait?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there is reason, it would be nice to document the reason besides the conversion
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The main reason is that DataFusion does not support aggregate functions with no arguments:
datafusion/datafusion/physical-expr/src/aggregate.rs
Lines 100 to 114 in 11838be
pub fn build(self) -> Result<AggregateFunctionExpr> { | |
let Self { | |
fun, | |
args, | |
alias, | |
human_display, | |
schema, | |
ordering_req, | |
ignore_nulls, | |
is_distinct, | |
is_reversed, | |
} = self; | |
if args.is_empty() { | |
return internal_err!("args should not be empty"); | |
} |
I imagine that this is the reason why it was there before.
If you do a similar operation and analyze DataFusion's logical plan (link), a similar Int64(1)
argument is injected.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll add a comment explaining that
2264291
to
cc45d06
Compare
cc45d06
to
18602af
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
THank you @gabotechs and @geoffreyclaude and @jayzhan211
|
||
let args = from_substrait_func_args(consumer, &f.arguments, input_schema).await?; | ||
|
||
// Datafusion does not support aggregate functions with no arguments, so |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
Thanks all! |
Which issue does this PR close?
Rationale for this change
Fixing empty
count()
aggregation functions provided through SubstraitWhat changes are included in this PR?
Adding back this check https://github.com/apache/datafusion/pull/14824/files#diff-474e53672159d74dae38992a914a74eba81b8350ebe161f11d755f06414ed7b4 and testing it
Are these changes tested?
yes, by a new test and by extending previously existing tests
Are there any user-facing changes?
yes, users will be able to feed empty
count()
functions through the Substrait interface