Roadmap 2023

[BohuTANG]

After a full year of research and development in 2022, the functionality and stability of Databend were significantly enhanced, and several users began using it in production. Databend has helped them greatly reduce costs and operational complexity issues.

This is Databend Roadmap in 2023 (discussion).

See also:

• Roadmap 2022: #3706
• Roadmap 2021: #746

Main tasks

v1.3

v1.2 (Prepare for release on May 15th)

v1.1 (Prepare for release on April 5th)

v1.0 (Prepare for release on March 5th)

Features

Task	Status	Comments
Update#9261	DONE	need optimized(release in v1.0)
Privileges	DONE
Alter table	DONE	high-priority(release in v1.0 )
Window function#6342	DONE
Lambda function and high-order functions	DONE
Materialized view	Aggregating index	DONE
Support SET_VAR hints#8833	DONE
Parquet reader	DONE
DataFrame	DONE
Data Sharing(community version)	DONE
Concurrent query enhance	IN PROGRESS
Distributed COPY#8594	DONE
Support Decimal data type#2931	DONE	high-priority(release in v1.0 )
Add Column-Level dynamic data masking support	PLAN

Improvements

Task	Status	Comments
New expression#9411	DONE
Error message	PLAN

Planner

Task	Status	Comments
Scalar expression normalization	DONE
Column constraint framework	DONE
Functional dependency framework#7438	DONE
Join reorder	DONE
CBO	DONE	high-priority(release in v1.0)
Support TPC-DS	DONE
Support optimization tracing	PLAN	Easy to debug/study.

Cache

Task	Status	Comments
Unified cache layer	DONE
Meta data cache	DONE
Index data cache	DONE
Block data cache	DONE	high-priority(release in v1.0 )

Data Storage

Task	Status	Comments
Fuse engine re-clustering	DONE	high-priority(release in v1.1)
Fuse engine orphan data cleanup	DONE	high-priority(release in v1.0)

Distributed Query Execution

Task	Status	Comments
Visualized profiling	IN PROGRESS
Aggregation spilling	DONE	high-priority(release in v1.1)

Resource Quota

Task	Status	Comments
Session-level quota control (CPU/Memory)	DONE

Schema-Less Search

Task	Status	Comments
JSON indexing	DONE	high-priority
Fulltext index#3915	IN PROGRESS	high-priority
Array functions#7931	DONE	high-priority
Faiss index#9699	PLAN

LakeHouse

Task	Status	Comments
Apache Hive	DONE
Apache Iceberg	DONE
Delta Lake	PLAN
Querying external storage(Parquet)	DONE

Integrations

Task	Status	Comments
Dbt integration	DONE
Airbyte integration	DONE
Datadog Vector integrate with Rust-driver	DONE
Datax integrate with Java-driver	DONE
CDC with Flink	DONE
CDC with Kafka	DONE

Meta

Task	Status	Comments
Jepsen test	DONE
Store membership in raft	DONE
Nonblocking snapshot building	DONE
Snapshot file format impl	DONE
Upgrade on-disk store format	DONE

Testing

Task	Status	Comments
SQLlogic Test	DONE	Supports more test cases
SQLancer Test	DONE	Supports more type and more cases
Fuzzer Test	IN PROGRESS

Releases

• v1.3#11868
• v1.2#11073
• v1.1#10334
• v1.0#9604
• v0.9#7052
• v0.8 #4591
• v0.7 #3428
• v0.6 #2525
• v0.5 #2257

[flaneur2020]

any plan about improving concurrency capabilities? so developers can depend on databend to make some data exploring platforms (like google analystics?) on the web.

[flaneur2020]

any plan about tuning the metasrv's memory usage? I've got a OOM last week, IMHO it can store most the data in the disk?

[BohuTANG]

any plan about improving concurrency capabilities? so developers can depend on databend to make some data exploring platforms (like google analystics?) on the web.

Added: Concurrent query enhance

[BohuTANG]

any plan about tuning the metasrv's memory usage? I've got a OOM last week, IMHO it can store most the data in the disk?

@drmingdrmer will fill the meta section, I think he will do it.

[yufan022]

Any plan to support decimal data type? This is essential If we want to use databend in financial related fields. Will we see it in the first half of the year?

[BohuTANG]

Any plan to support decimal data type? This is essential If we want to use databend in financial related fields. Will we see it in the first half of the year?

Added to the main task, thanks.

[flaneur2020]

will fault tolerance on query processing be planned in 2023?

likewise I have some spot instances, the cluster may handles a shutdowned instance gracefully and not affect the running queries.

[BohuTANG]

will fault tolerance on query processing be planned in 2023?

Will do but hard to do, so the priority is low.

likewise I have some spot instances, the cluster may handles a shutdowned instance gracefully and not affect the running queries.

Please file an issue for that.

[thatcort]

Is there a plan for when the vector index feature will be added? It is part of #10689 but doesn't seem to have an associated ticket.