[VL] Bad spill performance

[FelixYBW]

Description

It's a known issue that spill has bad performance using Velox backend. Here is some data:

It also confirmed that Gluten only uses 1/2 Spark's memory size. With the same partition Gluten has no spill but Spark has.

And with partition size increases, sort based shuffle has better and more stable performance than hash based shuffle.

[zhanglistar]

Hi, FelixYBW, how to reproduce this result? Thanks.

[FelixYBW]

Hi, FelixYBW, how to reproduce this result? Thanks.

It's a real customer query.

[FelixYBW]

partially fixed by facebookincubator/velox#11384