SQL tests for when sorting exceeded available memory and had to spill to disk

[alamb]

No description provided.

[yjshen]

Currently, #1596 has one SQL test on spill called test_sort_spill that spills during the test by controlling the total memory of the pool.

https://github.com/apache/arrow-datafusion/blob/6a458c67b5d11cf9dfa09281289c4d2da9484a2d/datafusion/src/physical_plan/sorts/sort.rs#L739-L765

However, since spill metrics are not exposed by sort yet, the current test cannot assert spill count > 0. Do you think it's appropriate if I add spill count as well as spill bytes into BaselineMetrics? I think these might be common metrics as we proceed with more memory consumers.

[alamb]

However, since spill metrics are not exposed by sort yet, the current test cannot assert spill count > 0. Do you think it's appropriate if I add spill count as well as spill bytes into BaselineMetrics? I think these might be common metrics as we proceed with more memory consumers.

I think adding those metrics to BaselineMetrics makes sense to me @yjshen

[yjshen]

@alamb, I think this one can be closed. Solved in #1641.

[alamb]

@yjshen I was thinking this ticket could track actually writing sql tests that spilled to disk (and verified both that they spilled as well as the correct answers).

Something like:

1. Make a table with 10,000 rows but set memory limits to only allow 1K rows, and then validate the answer comes out
2. A test harness that runs the same basic test, but checks various combinations of batch_size, and memory_limits to ensure the correct answer is produced

[yjshen]

@alamb Ah, that makes sense. I will do it.