Enhance to hybrid storage format

[jiacai2050]

Describe This Problem

Hybrid storage format with basic feature is already merged, but there are still things to be done to gain good performance.

Open an new issue to tracking those.

Proposal

Write

• Support variable-length type for ListArray
• Support table without tsid, only a row id is required
• Persist hybrid collapsible info

Read

• Support timestamp column filter, some extra columns may be needed
• Support variable-length type for ListArray
• Enable a total ordering, to support query with pagination

Misc

• Compaction support
• Ensure row group size is large enough, in case of list length within same row_id is to small
• Use dictionary array type to represent non-collapsible columns to reduce memory usage.
• Benchmark between two format

Additional Context

#77

[ShiKaiWi]

The poc of hybrid storage has finished.

[jiacai2050]

Lessons learnt from this feature.

• Arrow data layout of hybrid storage is same when primary key is tsid, timestamp
• The only advantage is less disk usage, which can be gained using dictionary compression.
• Also convert plain arrow data to nested type makes code hard to maintain, and some feature hard to implement.