Improve performance of TPC-H q16

[andygrove]

What is the problem the feature request solves?

Comet is currently slower than Spark for TPC-H q16.

This query does not run fully native yet.

Issues

• BuildRight with LeftAnti is not supported (Comet doesn't support Spark BroadcastHashJoinExec if it is null-aware anti-join #457)
• BroadcastExchange is not supported (confusing error, needs improving)
• BroadcastHashJoin disabled because not all child plans are native
• SortMergeJoin disabled because not all child plans are native
• Disabling coalesce partitions (so that we can enable Comet shuffle) has quite a negative impact on performance - Enable Comet shuffle with AQE coalesce partitions #387

Describe the potential solution

No response

Additional context

No response

[andygrove]

As an experiment, I removed our check where we fall back to Spark for BuildRight with LeftAnti so that the query runs natively (possibly with incorrect results) and performance increased significantly and matched Spark (but was not faster). I no longer saw the "BroadcastExchange is not supported" error.

[andygrove]

At the end of the query we are performing a RowToColumnar to move to Comet just for a query stage that performs a sort and then perform a ColumnarToRow. This likely explains why we currently make the query slower than Spark.

[andygrove]

This issue is largely out of date, and current issues are noted in #391