Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass return_type to AccumulatorFunctionImplementation for user defined aggregates #3428

Merged
merged 4 commits into from
Sep 12, 2022

Conversation

alamb
Copy link
Contributor

@alamb alamb commented Sep 9, 2022

Which issue does this PR close?

Closes #3427

Rationale for this change

See #3427

What changes are included in this PR?

  • Passreturn_type in AccumulatorFunctionImplementation
  • Improve some comments

Are there any user-facing changes?

Yes, if you use user defined aggregate functions -- one of the type signatures has changed

@alamb alamb added the api change Changes the API exposed to users of the crate label Sep 9, 2022
@github-actions github-actions bot added core Core DataFusion crate logical-expr Logical plan and expressions labels Sep 9, 2022
@alamb alamb changed the title Add return_type to accumulator factor signature Pass return_type to AccumulatorFunctionImplementation for user defined aggregates Sep 9, 2022
@@ -99,7 +100,7 @@ impl AggregateExpr for AggregateFunctionExpr {
}

fn create_accumulator(&self) -> Result<Box<dyn Accumulator>> {
(self.fun.accumulator)()
(self.fun.accumulator)(&self.data_type)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The is the key change -- pass the return type when instantiating the accumulator

pub type AccumulatorFunctionImplementation =
Arc<dyn Fn() -> Result<Box<dyn Accumulator>> + Send + Sync>;
Arc<dyn Fn(&DataType) -> Result<Box<dyn Accumulator>> + Send + Sync>;
Copy link
Contributor Author

@alamb alamb Sep 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the signature change

@alamb
Copy link
Contributor Author

alamb commented Sep 10, 2022

This will likely conflict with #3425 logically

@alamb alamb marked this pull request as ready for review September 10, 2022 10:29
Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me. Thanks @alamb

@codecov-commenter
Copy link

Codecov Report

Merging #3428 (91356af) into master (deea8c3) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3428      +/-   ##
==========================================
- Coverage   85.70%   85.68%   -0.02%     
==========================================
  Files         298      298              
  Lines       54612    54644      +32     
==========================================
+ Hits        46803    46824      +21     
- Misses       7809     7820      +11     
Impacted Files Coverage Δ
datafusion/core/tests/user_defined_aggregates.rs 100.00% <ø> (ø)
datafusion/expr/src/function.rs 97.51% <ø> (ø)
datafusion/core/src/execution/context.rs 79.06% <100.00%> (ø)
datafusion/core/src/physical_plan/udaf.rs 92.00% <100.00%> (ø)
datafusion/core/tests/sql/udf.rs 100.00% <100.00%> (ø)
datafusion/proto/src/lib.rs 94.12% <100.00%> (ø)
datafusion/expr/src/operator.rs 98.43% <0.00%> (-1.57%) ⬇️
datafusion/expr/src/logical_plan/plan.rs 77.19% <0.00%> (-0.17%) ⬇️
datafusion/physical-expr/src/expressions/binary.rs 97.63% <0.00%> (-0.08%) ⬇️
datafusion/sql/src/planner.rs 80.88% <0.00%> (-0.04%) ⬇️
... and 2 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@alamb alamb merged commit 447141c into apache:master Sep 12, 2022
@alamb alamb deleted the alamb/accumulator_state branch September 12, 2022 12:32
@alamb alamb restored the alamb/accumulator_state branch September 12, 2022 12:32
@ursabot
Copy link

ursabot commented Sep 12, 2022

Benchmark runs are scheduled for baseline = 1c8797d and contender = 447141c. 447141c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@alamb alamb deleted the alamb/accumulator_state branch August 8, 2023 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api change Changes the API exposed to users of the crate core Core DataFusion crate logical-expr Logical plan and expressions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pass return type to the accumulator state factory in aggregates
4 participants