exec: tracking issue for vectorized engine

[jordanlewis]

This issue is an "epic" to track work in the pkg/sql/exec vectorized engine.

Features:

P0

• serialization to Apache Arrow
  • encoder and decoder that takes coldata.Batch/Vec to Arrow data structures and back
  • (not absolutely necessary) transparent use of Apache Arrow data structures as coldata.Batch/Vec
  • decimal encoding exec: decimal encoding support for Arrow representation #37044
  • Use uint8 null bitmaps exec: change null representation to be uint8s instead of uint64s for arrow compat #37045
• distribution - distributed flows must be plannable. Needs serialization.
  • stream encoder and decoders that operate on columnar batches instead of encdatumrows.
  • unordered synchronizer - some async component to eagerly decode. Either reuse rowchannel or write columnar batch equivalents.
  • ordered synchronizer
  • outbox/inbox
  • hash router
    • 2 weeks: PR out (exec: add hash router #36173)
• metadata propagation. Make sure to not introduce metadata into operators. Metadata should forward to the end of flows. (sql: add metadata propagation through columnar operators #36039)

P1

• memory monitoring exec: add memory monitoring #37301
• disk spilling - computation that hits a memory limit must fall back to disk containers. Needs serialization.
  • spilling a queue - first in first out
  • spilling for sort computations - external sort
  • spilling for hash joins
    • ordinary hash lookup - how to make this fast? "grace" hash join?
    • marked rows for outer joins
  • spilling window computations

P2

• tracing / stats: understand quantity of rows/batches flowing through operators. selection vector selectivity, maybe?
• explain (shows flow of exec operators)

Relational operators:

• index / lookup join (accomplished with plan wrapping)
• offset
• finish hash aggregate
• sort chunks
• other unhandled operators - plan wrapping
• top k sort
• merge join
• unordered / partially ordered distinct
• add null support to ordered distinct
• add null support to sorter
• planning for projections which use comparison operators (SELECT x > 5 FROM ...)
• COUNT agg func (needed for distributed TPCH queries) exec: support COUNT aggregate function #38063
  Next:
• window functions

Scalar operators:

• LIKE, ILIKE, etc. (needed for several TPCH queries)
• min and max support
• random builtins like string matching (needed for TPCH queries)

[tim-o]

Zendesk ticket #3223 has been linked to this issue.

[RoachietheSupportRoach]

Zendesk ticket #3528 has been linked to this issue.

[yuzefovich]

Closing in favor of #34497