Add external function

[TennyZhuang]

Signed-off-by: TennyZhuang zty0826@gmail.com

[CLAassistant]

All committers have signed the CLA.

[TennyZhuang]

@skyzh Can you ignore the HackMD bot in CLA checker?

[BowenXiao1999]

1. Can you also provide more context why the interface should be async?
2. How to understand cross-chunk concurrency? Do you mean if udf1 and udf2 both need chunk A, then they shared read this chunk?

[BugenZhao]

2. How to understand cross-chunk concurrency?

Take Project as an example, if the UDF is I/O bounded, then we may want to process the input chunks in a pipeline manner by buffering them to be concurrent.

[skyzh]

Do we allow external functions in places other than projection? (e.g., filter?)

[TennyZhuang]

Do we allow external functions in places other than projection? (e.g., filter?)

Yes, I hope everywhere.

[skyzh]

Before introducing external functions to the system, we should also consider deterministic problems.

Considering the following query:

select * from (select udf(x) as x from table) where x > 10;

Suppose we get the following events from table:

INSERT 1
DELETE 1

If udf is not deterministic, suppose it gets 1 -> 11 for the first message, and 1 -> 10 for the second message.

Many things will go wrong. The filter will emit:

INSERT 1

which ignores the second delete.

One feasible solution is to have a special kind of projection called MaterializeProjection, where we materialize data by their pk, so that UDF will only be computed once at the first time when the generated event goes into the system.

[skyzh]

That is to say, MaterializeProjection is a cache for UDF, whereas given one input we will always produce the same output for a given UDF.

[TennyZhuang]

Before introducing external functions to the system, we should also consider deterministic problems.

Considering the following query:

select * from (select udf(x) as x from table) where x > 10;

Suppose we get the following events from table:

INSERT 1
DELETE 1

If udf is not deterministic, suppose it gets 1 -> 11 for the first message, and 1 -> 10 for the second message.

Many things will go wrong. The filter will emit:

INSERT 1

which ignores the second delete.

One feasible solution is to have a special kind of projection called MaterializeProjection, where we materialize data by their pk, so that UDF will only be computed once at the first time when the generated event goes into the system.

Good catch! I have no idea how to resolve that, how about only allowing all UDFs on the append-only stream?

[TennyZhuang]

• Can you also provide more context why the interface should be async?

There may be RPC calls during the evaluation.

[skyzh]

Append-only stream should be fine. But this means that people can only do UDF before aggregation / TopN, which seems to be a common case in SQL. Not sure about the use case.

[TennyZhuang]

A basic idea:

Users must specify their UDFs are deterministic explicitly, otherwise, the UDFs can only be applied on an append-only stream or be materialized implicitly.

I will add this to the RFC.

[wangrunji0408]

Can we merge this RFC? as most of its proposals have been implemented now.

[TennyZhuang]

Can we merge this RFC? as most of its proposals have been implemented now.

You can give an approval.