feat(function): change retention return type from Variant to Array<T>

[fkuner]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Now the get method of Array is supported.

We can change the return type of retention function from Variant to Array.

Changelog

• New Feature

Related Issues

Fixes #5226

[mergify]

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
databend	⬜️ Ignored (Inspect)		May 15, 2022 at 8:10AM (UTC)

[BohuTANG]

Here is the ClickHouse performance of retention on my laptop, FYI:

SELECT version()

Query id: b0028bdf-53cc-4590-97b8-d5aed83844bd

┌─version()─┐
│ 22.2.2.1  │
└───────────┘


CREATE TABLE retention_test
(
    `date` DATE,
    `uid` INT
)
ENGINE = MergeTree
PARTITION BY date
ORDER BY uid


INSERT INTO retention_test SELECT '2018-08-06', number FROM numbers(10000000);
INSERT INTO retention_test SELECT '2018-08-07', number FROM numbers(9000000);
INSERT INTO retention_test SELECT '2018-08-08', number FROM numbers(1000000);

Performance:

SELECT
    sum(r[1]) AS r1,
    sum(r[2]) AS r2,
    sum(r[3]) AS r3
FROM
(
    SELECT
        uid,
        retention(date = '2018-08-06', date = '2018-08-07', date = '2018-08-08') AS r
    FROM retention_test
    GROUP BY uid
)

Query id: 7e90f2ca-62f4-419a-bf60-4aee4691e0f9

┌───────r1─┬──────r2─┬──────r3─┐
│ 10000000 │ 9000000 │ 1000000 │
└──────────┴─────────┴─────────┘

1 rows in set. Elapsed: 1.224 sec. Processed 20.00 million rows, 120.00 MB (16.34 million rows/s., 98.04 MB/s.)

[fkuner]

Here is the ClickHouse performance of retention on my laptop, FYI:

SELECT version()

Query id: b0028bdf-53cc-4590-97b8-d5aed83844bd

┌─version()─┐
│ 22.2.2.1  │
└───────────┘


CREATE TABLE retention_test
(
    `date` DATE,
    `uid` INT
)
ENGINE = MergeTree
PARTITION BY date
ORDER BY uid


INSERT INTO retention_test SELECT '2018-08-06', number FROM numbers(10000000);
INSERT INTO retention_test SELECT '2018-08-07', number FROM numbers(9000000);
INSERT INTO retention_test SELECT '2018-08-08', number FROM numbers(1000000);

Performance:

SELECT
    sum(r[1]) AS r1,
    sum(r[2]) AS r2,
    sum(r[3]) AS r3
FROM
(
    SELECT
        uid,
        retention(date = '2018-08-06', date = '2018-08-07', date = '2018-08-08') AS r
    FROM retention_test
    GROUP BY uid
)

Query id: 7e90f2ca-62f4-419a-bf60-4aee4691e0f9

┌───────r1─┬──────r2─┬──────r3─┐
│ 10000000 │ 9000000 │ 1000000 │
└──────────┴─────────┴─────────┘

1 rows in set. Elapsed: 1.224 sec. Processed 20.00 million rows, 120.00 MB (16.34 million rows/s., 98.04 MB/s.)

Got it. I will compare our performance with ck.

[BohuTANG]

Have tested this PR:

mysql> CREATE TABLE retention_test
    -> (
    ->     `date` DATE,
    ->     `uid` INT
    -> );
Query OK, 0 rows affected (0.01 sec)
Read 0 rows, 0.00 B in 0.008 sec., 0 rows/sec., 0.00 B/sec.

mysql> INSERT INTO retention_test SELECT '2018-08-06', number FROM numbers(10000000);
Query OK, 0 rows affected (0.42 sec)
Read 10000000 rows, 76.29 MiB in 0.412 sec., 24.25 million rows/sec., 185.00 MiB/sec.

mysql> INSERT INTO retention_test SELECT '2018-08-07', number FROM numbers(9000000);
Query OK, 0 rows affected (0.35 sec)
Read 9000000 rows, 68.66 MiB in 0.345 sec., 26.12 million rows/sec., 199.29 MiB/sec.

mysql> INSERT INTO retention_test SELECT '2018-08-08', number FROM numbers(1000000);


mysql> SELECT     sum(r[0]) AS r1,     sum(r[1]) AS r2,     sum(r[2]) AS r3 FROM (     SELECT         uid,         retention(date = '2018-08-06', date = '2018-08-07', date = '2018-08-08') AS r     FROM retention_test     GROUP BY uid );
+----------+---------+---------+
| r1       | r2      | r3      |
+----------+---------+---------+
| 10000000 | 9000000 | 1000000 |
+----------+---------+---------+
1 row in set (5.78 sec)
Read 20000000 rows, 152.59 MiB in 5.760 sec., 3.47 million rows/sec., 26.49 MiB/sec.

[fkuner]

I have optimized the implementation just now. The performance can decrease from 3 seconds to 2.4 seconds in my machine. But it still has a gap with ck. The reason should be that the cardinal number is too high, but we haven't use the parallel merge.

[BohuTANG]

With the new patch, the performance(5.760 to 4.226):

mysql> SELECT     sum(r[0]) AS r1,     sum(r[1]) AS r2,     sum(r[2]) AS r3 FROM (     SELECT         uid,         retention(date = '2018-08-06', date = '2018-08-07', date = '2018-08-08') AS r     FROM retention_test     GROUP BY uid );
+----------+---------+---------+
| r1       | r2      | r3      |
+----------+---------+---------+
| 10000000 | 9000000 | 1000000 |
+----------+---------+---------+
1 row in set (4.26 sec)
Read 20000000 rows, 152.59 MiB in 4.226 sec., 4.73 million rows/sec., 36.10 MiB/sec.