What is the main function of nvcomp LZ4 codec and what scene is it used for？ [QST]

[chenrui17]

What is your question?
#833 is mainly for what ? shuffle or all of the qeury stage in gpu ? Is there a benefit to current performance ？ where ？

[revans2]

This is all about shuffle. Spark, by default, compresses all shuffle data before it is written out to disk, although it is configurable.

When using the default shuffle implementation we still use the CPU compression algorithm. This is to be able to match that functionality when using the UCX based shuffle plugin, where we want to avoid going back to the CPU if possible.

There are two places where this can help us.

1. Reduces GPU memory usage. We compress the data before we cache it on the GPU. If the compression ratio is good then we can store more data in GPU memory and not have to spill it to host memory or disk.
2. The other is to use more computation to offset limited bandwidth. The amount of time it takes to move data from A to B is the (size_of_input_data / compression_rate) + (size_of_input_data / compression_ratio / transmission_rate) + (size_of_input_data / decompression_rate) This has a lot of variables in it, but essentially if we have a slow transmission_rate because of a spinning disk, over congested PCIe bus or network we can use some computation to offset it.

Because of the complexity of this it is hard to predict the impact of compression on a given query/hardware setup, but we have see improvements for queries using UCX that spill to disk a lot. We are still doing profiling and looking into more performance improvements, but the numbers are promising enough to try and put it into our next release. We are still evaluating if it will be off by default or not.

[jlowe]

Closing as answered.