[C++][Python] Add multiply/divide kernels for duration and float/double

[randolf-scholz]

Describe the enhancement requested

numpy supports multiplying/dividing timedelta values with floats and performs automatic rounding to the current unit, e.g.

>>> import numpy as np
>>> x = np.array([0, 1, 2, 3, 4, 5], dtype="timedelta64[s]")
>>> x * 0.5
array([0, 0, 1, 1, 2, 2], dtype='timedelta64[s]')
>>> x * 1.5
array([0, 1, 3, 4, 6, 7], dtype='timedelta64[s]')

This is very useful for ML applications where one needs to transform timestamps first to durations and then to floats (and back after performing a forecast). It would be very nice if pyarrow supported this as well, so one can directly work with arrow dtypes.

Component(s)

C++

[rok]

Thanks for reporting @randolf-scholz !

Arrow/PyArrow should support multiplication, division, subtraction and addition via compute functions which are not mapped to the usual arithmetic symbols yet, making this a bit harder to find. (@raulcd @AlenkaF I think we wanted to map this at some point?)

So in pyarrow you can run:

import pyarrow as pa
import pyarrow.compute as pc
import numpy as np
x = np.array([0, 1, 2, 3, 4, 5], dtype="timedelta64[s]")
y = pa.array(x)
pc.divide(y, 3)

<pyarrow.lib.DurationArray object at 0x76c84b059960>
[
  0,
  0,
  0,
  1,
  1,
  1
]

[rok]

We also have more calendar aware features like temporal subtraction, component extraction and rounding. I suppose it's best you search for temporal in this docs page to discover functions that take temporal type as parameter.

[randolf-scholz]

@rok I am talking about multiply(duration, double) and divide(duration, double). These kernels do not exist (tested with version 22.0.0)

import pyarrow as pa

x = pa.array([1, 2, 3], type=pa.duration("s"))
y = pa.array([0.0, 1.5, 2.9], type=pa.float64())

pa.compute.multiply(x, y)
# pyarrow.lib.ArrowNotImplementedError: Function 'multiply' has no kernel matching input types (duration[s], double)

Because duration is internally an integer, implementing this operation requires automatic rounding as in numpy.

[randolf-scholz]

There are also some other kernels that would be useful for duration, like min and max, see my issue from ~ 2 years ago: #39233.

But the multiple/divide one is particularly useful because of the application I described. To apply a neural net, one needs to transform everything to float, and the usual route for timestamps is: timestamp - reference -> duration and then duration / unit -> double. But the reverse transform currently runs into the problem of the kernel not existing

import pyarrow as pa

deltas = pa.array([1, 2, 3], type=pa.duration("s"))
unit = pa.scalar(1, type=pa.duration("s"))
# forward transform to float:
x = pa.compute.divide(deltas, unit)  # gives double
# backward transform to duration
y = pa.compute.multiply(x, unit)  # ArrowNotImplementedError

[rok]

Thanks for compiling the list in #39233, it is useful to see!

I suppose the issue you're describing occurs because multiplying double with duration[xs] would need to return a duration[xs], which would requires an implicit rounding step (duration[xs] is physically an int64). We can discuss adding this as implicit logic, but we try to avoid implicit steps like this. You can work around this by adding a round step:

import pyarrow as pa

deltas = pa.array([1, 2, 3], type=pa.duration("s"))
unit = pa.scalar(1, type=pa.duration("s"))
# forward transform to float:
x = pa.compute.divide(deltas, unit)  # gives double
# added rounding step
x = pc.compute.round(x, unit.value).cast("int64") # This won't work if unit is not an integer
# backward transform to duration
y = pa.compute.multiply(x, unit)

But are you doing temporal rounding?

$$\frac{timestamp - reference}{unit} * unit + reference$$

If so you can try:

import pyarrow as pa
timestamps = pa.array([1, 2, 3], type=pa.timestamp("s"))
durations = pa.array([1, 2, 3], type=pa.duration("s"))
y = pa.compute.round_temporal(timestamps, unit="second", multiple=1) # We should have this work on durations too

Or go unitless:

import pyarrow as pa
durations = pa.array([1, 2, 3], type=pa.duration("s"))
y = pa.compute.round_to_multiple(durations.cast("int64"), 1)

[randolf-scholz]

Well, of course one can work around missing kernels by emulating them. But this is quite annoying to do, especially when working with libraries that use pyarrow as a backend like polars or pandas. For instance, existing code that just uses regular arithmetic operators floats = (time -time.iloc[0]) / unit; floats * unit will break when attempting to use pyarrow backend, as these engines try to look up a nonexisting kernel.

[AlenkaF]

I haven't worked on the compute module so I cannot really comment.
From PyArrow view, though, it does sound like a reasonable kernel to add.

I think we wanted to map this at some point?

Yes: #32007.

[rok]

@randolf-scholz I agree we should have this kernel. I'm just curious at this point if temporal rounding is a useful kernel. :)

I think we wanted to map this at some point?

Yes: #32007.

That's the one, thank you!

[rok]

To circle back:

@rok I am talking about multiply(duration, double) and divide(duration, double). These kernels do not exist (tested with version 22.0.0)

We have a request to add two kernels that could be expressed as composites of existing kernels:

• multiply(duration[xs], double) -> multiply(duration[xs], double).round_to_multiple(1).cast(duration[xs])
• divide(duration[xs], double) -> divide(duration[xs], double).round_to_multiple(1).cast(duration[xs])

This would be straightforward to do except for maybe the rounding parameters. However I am not sure we want to introduce composite kernels. Perhaps we could have alias kernels (either in c++ or python) that would run the series of kernels that would express the desired computation?

cc @pitrou @jorisvandenbossche what's you opinion on this?

[pitrou]

• multiply(duration[xs], double) -> multiply(duration[xs], double).round_to_multiple(1).cast(duration[xs])
• divide(duration[xs], double) -> divide(duration[xs], double).round_to_multiple(1).cast(duration[xs])

How does this make sense?

[rok]

Could you be a bit more specific? :)
The question is if we want kernels that can multiply duration with double and return duration.

[rok]

Could you be a bit more specific? :)
The question is if we want kernels that can multiply duration with double and return duration.

[pitrou]

Well, this seems a bit recursive unless I'm misreading:

• multiply(duration[xs], double) -> multiply(duration[xs], double).round_to_multiple(1).cast(duration[xs])