feature request: have simple way to create functions for use inside verbs that are unaware of groupedness

[ftobin]

Older versions of datar used pipda with a fairly easy to use way of creating vectorized functions:

@register_func(None, context=Context.EVAL)
def weighted_mean(
    x: NumericOrIter,
    w: NumericOrIter = None,
    na_rm: bool = False,
) 

From what I gather from the current methodology (see in the current weighted_mean()), there are multiple functions that needed to handle grouped/non-grouped versions of the dataframe.

What would be great if there if there an approach that allows me to be "dplyr-ish" and just have a vectorized function that is unaware of groupedness, similar to how functions used inside of dplyr functions can work over grouped and non-grouped tibbles.

[pwwang]

In the earlier versions (<v0.4), the implementation was easy. That's because each group is separated and applied with the defined functions so that we don't need to consider the groupedness.

Here is a rough illustration of the previous architecture:

Grouped data
                              +----+   Apply registered functions +----+
+----+                        | g1 |   -------------------------> | r1 |
| g1 |    Separated by datar  +----+                              +----+   Merged by datar  +--------+
+----+    ----------------->                                               -------------->  | result |
| g2 |                        +----+                              +----+                    +--------+
+----+                        | g2 |  --------------------------> | r2 |
                              +----+                              +----+

You can see that the vectorized function only needs to handle the data in each group, and datar handles the groupedness (using DataFrame.apply() under the hood). I believe this is exactly what you wanted (the unawareness of groupedness while registering a function).

However, this causes performance issues. See:

• group_by() very slow #33
• filter slows down on large datasets #37

That's because this way, we are unable to take the advantage of something like SeriesGroupBy.mean() since what is passed to mean() is already a separated group of data, we need to calculate the mean of each group and gather them into the final result. This slows it down, because SeriesGroupBy.mean() has been optimized significantly by pandas with some C code.

That's why now we have a new architecture:

Grouped data
                              
+----+                       
| g1 |    Pass the whole thing to registered functions   +--------+
+----+    ----------------------------- -------------->  | result |
| g2 |                                                   +--------+
+----+                      

This way, we can handle everything inside the function, which enables us to apply some strategies like SeriesGroupBy.mean() or GroupBy.agg() to get some performance improvements.

But it doesn't mean we have to handle every detail inside the function. We do have some helper functions to achieve the "unawareness of the groupedness". The helper functions are defined in core/factory.py. But there are some limitations. For example, we need to know the type of the result (e.g. whether the function is doing aggregation, transformation, or generally apply).

You can find the following example to define a function with "unawareness of the groupedness":

datar/datar/base/arithmetic.py

Lines 415 to 427 in cd2595b

	@func_factory("transform", {"x", "base"})
	def log(x: Series, base: Series = np.e) -> Series:
	"""Computes logarithms, by default natural logarithm
	Args:
	x: A numeric scalar or vector
	base: The base of the logarithm
	Returns:
	The value of the logarithm if x is scalar, otherwise element-wise
	logarithm of elements in x
	"""
	return np.log(x) / np.log(base)

base.log() could handle both grouped and ungrouped data, but you don't see any handling of groupedness inside the function. That's because with func_factory("transform", ...), datar knows how to separate the grouped data and merge the results (mimics the previous architecture).

We can do it with some simple and common functions. However, they are way more complicated cases, and the result is from none of the aggregation, transformation or apply. In such a case, we have to handle the groupedness by ourselves inside the function since datar doesn't know how we should separate the groups or merge the results.

It looks like weighted_mean() is doing a kind of aggregation. However, it has an extra argument w (weights). We are yet able to handle the groupedness with 2 columns with func_factory() (may be supported in the future). That's why we need dispatch or if...else... to handle ungrouped and grouped data by ourselves inside that function.

[ftobin]

Forgive my naivety, by could something like providing following decorator work together with func_factory to allow a user to easily create group-agnostic vectorized functions? I don't know enough about func_factory() to extend it to variable numbers of arguments.

def mydecor(func):
    @singledispatch
    def _worker(
        df: DataFrame,
    ) -> np.ndarray:
        return func(df["x"], df["x2"])
    
    @_worker.register(TibbleGrouped)
    def _(
        df: TibbleGrouped,
    ) -> Series:
        return df._datar["grouped"].apply(
            lambda subdf: _worker(subdf)
        )

    @func_factory(None, {"x", "x2"})
    def userland(
        x: Series, x2: Series, __args_raw=None, __args_frame=None,
    ) -> Series:
        return _worker(__args_frame)
    
    return userland

# example usage
@mydecor
def my_wm(n, n2):
    return sum_(n*n2)/sum_(n2)

# verifying the implementation
mtcars >> group_by(f.cyl) >> summarize(wm1 = my_wm(f.disp, f.mpg), wm2=weighted_mean(f.disp, f.mpg))

  | cyl | wm1 | wm2
<int64> | <float64> | <float64>
6 | 183.584660 | 183.584660
4 | 101.809274 | 101.809274
8 | 347.554494 | 347.554494

[pwwang]

I have actually thought about this. We can probably add one more kind (apply_df?) other than agg, transform, apply to func_factory to automatically have registered functions handle the whole dataframe, instead of individual columns.

[ftobin]

It does seem to make sense that given how pandas grouping works, passing around the whole dataframe helps. Would the function call still look something like this?

>> summarize(output = my_func(f.x, f.x2, f.x3, na_rm=True))

[pwwang]

I believe so.