Avoid allocating vector of indices in lexicographical_partition_ranges

[jhorstmann]

Which issue does this PR close?

Closes #997.

Draft for now since I still have to update the code comments and maybe add some more tests.

Are there any user-facing changes?

I haven't run any benchmarks yet but this should improve performance of window functions with high-cardinality partition-by in datafusion.

[codecov-commenter]

Codecov Report

Merging #998 (caa05c7) into master (e9be49d) will increase coverage by 0.00%.
The diff coverage is 97.36%.

@@           Coverage Diff            @@
##           master     #998    +/-   ##
========================================
  Coverage   82.31%   82.32%            
========================================
  Files         168      168            
  Lines       48763    49060   +297     
========================================
+ Hits        40139    40388   +249     
- Misses       8624     8672    +48     

Impacted Files	Coverage Δ
arrow/src/compute/kernels/partition.rs	97.45% <97.36%> (-0.21%)	⬇️
arrow/src/array/array_string.rs	97.08% <0.00%> (-0.83%)	⬇️
parquet/src/record/reader.rs	89.83% <0.00%> (-0.63%)	⬇️
parquet/src/schema/printer.rs	72.47% <0.00%> (-0.55%)	⬇️
arrow/src/datatypes/datatype.rs	65.95% <0.00%> (-0.43%)	⬇️
arrow/src/datatypes/field.rs	53.37% <0.00%> (-0.31%)	⬇️
arrow/src/datatypes/schema.rs	66.66% <0.00%> (-0.27%)	⬇️
parquet/src/arrow/arrow_array_reader.rs	77.87% <0.00%> (-0.15%)	⬇️
arrow/src/compute/kernels/take.rs	95.21% <0.00%> (-0.01%)	⬇️
arrow/src/buffer/immutable.rs	97.84% <0.00%> (ø)
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e9be49d...caa05c7. Read the comment docs.

[jhorstmann]

@jimexist I'd like to get your feedback on this change, especially whether the comment in LexicographicalPartitionIterator::next sounds correct.

[alamb]

I reviewed this code carefully and structurally it looks equivalent to me. However, I agree it would be great to get @jimexist 's take on it as well

[alamb]

Can you please document what start and end and bound are? (specifically how the relate to each other -- I think as written it seems like bound is some starting index and the search starts at start+bound and stops at end indexes?

[alamb]

I'll plan to merge this PR in later this week (arrow 6.5 would get created at the end of next week, so I think we have plenty of time to land this one). Perhaps @jimexist will have a chance to review by then