Support arithmetic scalar operation with DictionaryArray #5151

viirya · 2023-02-01T19:34:29Z

Which issue does this PR close?

Closes #5150.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb · 2023-02-01T22:16:11Z

Thanks @viirya -- it is great to see someone else using Dictionary arrays ❤️

Looks like there are some compilation errors on this PR. Marking as draft until those are resolved.

error[E0425]: cannot find function `divide_decimal_scalar` in this scope
   --> datafusion/physical-expr/src/expressions/binary/kernels_arrow.rs:465:19
    |
[231](https://github.com/apache/arrow-datafusion/actions/runs/4068186587/jobs/7006440910#step:8:232) | / pub(crate) fn divide_decimal_dyn_scalar(
[232](https://github.com/apache/arrow-datafusion/actions/runs/4068186587/jobs/7006440910#step:8:233) | |     left: &dyn Array,
[233](https://github.com/apache/arrow-datafusion/actions/runs/4068186587/jobs/7006440910#step:8:234) | |     right: i128,
[234](https://github.com/apache/arrow-datafusion/actions/runs/4068186587/jobs/7006440910#step:8:235) | | ) -> Result<Decimal128Array> {
...   |
242 | |     Ok(decimal_array)
243 | | }
    | |_- similarly named function `divide_decimal_dyn_scalar` defined here
...
465 |           let err = divide_decimal_scalar(&left_decimal_array, 0).unwrap_err();
    |                     ^^^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `divide_decimal_dyn_scalar`

viirya · 2023-02-01T22:38:50Z

Thanks @alamb . Yea, I'm trying to fix them locally.

alamb

Looks good to me -- it would be great to eventually be able to write sql level tests for this feature, but there is no good way to make dictionary arrays yet from SQL (but I am hoping to get #5016 which would allow it).

Thank you for the contribution @viirya

cc @liukun4515 as this also impacts Decimals (though I don't expect any behavior change)

alamb · 2023-02-02T18:10:50Z

datafusion/expr/src/type_coercion/binary.rs

+        (DataType::Dictionary(_, value_type), _) => {
+            is_numeric(value_type) && is_numeric(rhs_type)
+        }
+        (_, DataType::Dictionary(_, value_type)) => {


I wonder if this need to handle the case when both lhs and rhs are Dictionaries 🤔

Actually I'm wondering what Dictionary for ScalaValue(rhs here) means although I have seen DataFusion has it.

It sounds reasonable to add the following:

(DataType::Dictionary(_, lhs_value_type), DataType::Dictionary(_, rhs_value_type2)) => { is_numeric(lhs_value_type) && is_numeric(rhs_value_type2) }

datafusion/physical-expr/src/expressions/binary.rs

alamb · 2023-02-02T18:18:29Z

datafusion/physical-expr/src/expressions/binary.rs

+
+        let mut dict_builder = PrimitiveDictionaryBuilder::<Int8Type, Int32Type>::new();
+
+        dict_builder.append(2)?;


viirya · 2023-02-02T19:15:11Z

datafusion/physical-expr/src/expressions/binary.rs

            }
            Operator::Modulo => {
+                // todo: change to binary_primitive_array_op_dyn_scalar! once modulo is implemented


modulo change: apache/arrow-rs#3649

we can remove binary_primitive_array_op_scalar once this is done?

yea, once modulo is ready to do.

viirya · 2023-02-02T19:34:14Z

Looks good to me -- it would be great to eventually be able to write sql level tests for this feature, but there is no good way to make dictionary arrays yet from SQL (but I am hoping to get #5016 which would allow it).

Thank you @alamb. Yea, we have internal e2e tests that use dictionary arrays but seems I don't find similar stuff on DataFusion so far. We can definitely expand test coverage for this in the future.

…e coercion pattern

viirya · 2023-02-02T19:55:09Z

BTW, I think array op array is also not supported yet on dictionary arrays. I will work on it after this.

datafusion/physical-expr/src/expressions/binary.rs

sunchao · 2023-02-02T23:53:14Z

datafusion/physical-expr/src/expressions/binary.rs

+macro_rules! binary_primitive_array_op_dyn_scalar {
+    ($LEFT:expr, $RIGHT:expr, $OP:ident) => {{
+        // unwrap underlying (non dictionary) value
+        let right = unwrap_dict_value($RIGHT);


It's interesting that we have "scalar dictionary":

/// Dictionary type: index type and value Dictionary(Box<DataType>, Box<ScalarValue>),

wonder how useful this is ...

sunchao · 2023-02-02T23:56:49Z

datafusion/physical-expr/src/expressions/binary.rs

            }
            Operator::Modulo => {
+                // todo: change to binary_primitive_array_op_dyn_scalar! once modulo is implemented


we can remove binary_primitive_array_op_scalar once this is done?

datafusion/physical-expr/src/expressions/binary.rs

liukun4515

LGTM

ursabot · 2023-02-03T08:02:40Z

Benchmark runs are scheduled for baseline = 224c682 and contender = c759865. c759865 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

github-actions bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates labels Feb 1, 2023

viirya force-pushed the arithmetic_dyn_scalar branch from 2a33923 to 8b7fcc7 Compare February 1, 2023 19:56

alamb marked this pull request as draft February 1, 2023 22:16

viirya force-pushed the arithmetic_dyn_scalar branch 2 times, most recently from e8669d4 to 699d7c2 Compare February 1, 2023 23:07

viirya marked this pull request as ready for review February 1, 2023 23:57

Support arithmetic dyn scalar

67cd4a7

viirya force-pushed the arithmetic_dyn_scalar branch from 3a0d313 to 67cd4a7 Compare February 2, 2023 00:42

liukun4515 self-requested a review February 2, 2023 04:19

alamb mentioned this pull request Feb 2, 2023

Add casting from arbitrary arrow types #5016

Closed

alamb approved these changes Feb 2, 2023

View reviewed changes

viirya commented Feb 2, 2023

View reviewed changes

For review: removing unnecessary macro parameter, adding one more typ…

841922c

…e coercion pattern

sunchao approved these changes Feb 3, 2023

View reviewed changes

For review: modify comment

3d2abb5

liukun4515 approved these changes Feb 3, 2023

View reviewed changes

liukun4515 merged commit c759865 into apache:master Feb 3, 2023

viirya mentioned this pull request Feb 4, 2023

Fix decimal scalar dyn kernels #5179

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support arithmetic scalar operation with DictionaryArray #5151

Support arithmetic scalar operation with DictionaryArray #5151

viirya commented Feb 1, 2023

alamb commented Feb 1, 2023

viirya commented Feb 1, 2023

alamb left a comment •

edited

Loading

alamb Feb 2, 2023

viirya Feb 2, 2023

alamb Feb 2, 2023

viirya Feb 2, 2023

sunchao Feb 2, 2023

viirya Feb 3, 2023

viirya commented Feb 2, 2023

viirya commented Feb 2, 2023

sunchao Feb 2, 2023

sunchao Feb 2, 2023

liukun4515 left a comment

ursabot commented Feb 3, 2023


		let mut dict_builder = PrimitiveDictionaryBuilder::<Int8Type, Int32Type>::new();

		dict_builder.append(2)?;

Support arithmetic scalar operation with DictionaryArray #5151

Support arithmetic scalar operation with DictionaryArray #5151

Conversation

viirya commented Feb 1, 2023

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb commented Feb 1, 2023

viirya commented Feb 1, 2023

alamb left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya commented Feb 2, 2023

viirya commented Feb 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liukun4515 left a comment

Choose a reason for hiding this comment

ursabot commented Feb 3, 2023

alamb left a comment •

edited

Loading