Upgrade window UDF api #730
Labels
enhancement
New feature or request
Comments
Michael-J-Ward
added a commit
to Michael-J-Ward/datafusion-python
that referenced
this issue
Jun 14, 2024
|
@timsaucer could you take a look at this once #728 is merged? |
andygrove
pushed a commit
that referenced
this issue
Jun 14, 2024
* deps: update datafusion to 39.0.0, pyo3 to 0.21, and object_store to 0.10.1 `datafusion-common` also depends on `pyo3`, so they need to be upgraded together. * feat: remove GetIndexField datafusion replaced Expr::GetIndexField with a FieldAccessor trait. Ref apache/datafusion#10568 Ref apache/datafusion#10769 * feat: update ScalarFunction The field `func_name` was changed to `func` as part of removing `ScalarFunctionDefinition` upstream. Ref apache/datafusion#10325 * feat: incorporate upstream array_slice fixes Fixes #670 * update ExectionPlan::children impl for DatasetExec Ref apache/datafusion#10543 * update value_interval_daytime Ref apache/arrow-rs#5769 * update regexp_replace and regexp_match Fixes #677 * add gil-refs feature to pyo3 This silences pyo3's deprecation warnings for its new Bounds api. It's the 1st step of the migration, and should be removed before merge. Ref https://pyo3.rs/v0.21.0/migration#from-020-to-021 * fix signature for octet_length Ref apache/datafusion#10726 * update signature for covar_samp AggregateUDF expressions now have a builder API design, which removes arguments like filter and order_by Ref apache/datafusion#10545 Ref apache/datafusion#10492 * convert covar_pop to expr_fn api Ref: https://github.com/apache/datafusion/pull/10418/files * convert median to expr_fn api Ref apache/datafusion#10644 * convert variance sample to UDF Ref apache/datafusion#10667 * convert first_value and last_value to UDFs Ref apache/datafusion#10648 * checkpointing with a few todos to fix remaining compile errors * impl PyExpr::python_value for IntervalDayTime and IntervalMonthDayNano * convert sum aggregate function to UDF * remove unnecessary clone on double reference * apply cargo fmt * remove duplicate allow-dead-code annotation * update tpch examples for new pyarrow interval Fixes #665 * marked q11 tpch example as expected fail Ref #730 * add default stride of None back to array_slice
|
I’ll be happy to but on vacation so it will be a couple weeksOn Jun 14, 2024, at 1:49 PM, Michael J Ward ***@***.***> wrote:
@timsaucer could you take a look at this once #728 is merged?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Michael-J-Ward
added a commit
to Michael-J-Ward/datafusion-python
that referenced
this issue
Jun 25, 2024
Merged
Michael-J-Ward
added a commit
to Michael-J-Ward/datafusion-python
that referenced
this issue
Jun 25, 2024
andygrove
pushed a commit
that referenced
this issue
Jun 26, 2024
* provides workaround for half-migrated UDAF `sum` Ref #730 * provide compatibility for sqlparser::ast::NullTreatment This is now exposed as part of the API to `first_value` and `last_value` functions. If there's a more elegant way to achieve this, please let me know.
Merged
|
@Michael-J-Ward I know this issue was already closed but I have proposed a method to remove the work around. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
datafusionis in process of removing their built-in window-functions, which our currentwindowfunction relies on.sumhas already been removed which brakes theq11_important_stock_identification.pytpch example.Additional context
AggregateFunction::Sumenum variant is still defined, but can not be used.I suspect that the proper solution is to register the new UDAFs with the function registry.
As a workaround for releasing 39, I explicitly match on
name == "sum"and redirect to the UDAF.UDAF Migration Epic: apache/datafusion#8709
The text was updated successfully, but these errors were encountered: