How to use/test Comet?

[jinwenjie123]

Hi Team,

I am trying to evaluate the performance of using the comet plugin.
But I did not find any documentations about how to use the comet plugin after compiling it.
Specifically, how to use it in the cluster mode.

Thanks

[viirya]

I think we will provide more documents on this.

Currently, https://github.com/apache/arrow-datafusion-comet/blob/main/bin/comet-spark-shell contains the simply usage of how to come out necessary configuration in Spark side to use Comet.

You just need to build Comet, distribute the jar with Spark and set up necessary configs to trigger it.

Btw, we haven't open source some features which are performance related. So the performance number you run might be not really accurate.

[jinwenjie123]

Hi Team,

I also noticed that when comet detects unsupported features and fallback to Spark engine. Does it come with the cost of extra time to convert columnar data to row based data format ?

Thank you for your time.

[viirya]

Yes, that's correct. At the boundary between Comet operator and Spark operator, we need ColumnarToRow operator to convert from ColumnarBatch to Spark InternalRow which takes some time cost. Overall we expect that the gain obtained from native operators can be bigger than these cost. And with more native operator support, we can reduce such fallback and the cost of columnar to row conversion.

[jinwenjie123]

Hi Team,

I am wondering will we have a documentation about what kinda of data types are supported by Comet/Datafusion (Like, Decimal(16, 6), interger .... ) and will not fallback to vanilla spark and cause the regression.

Or where I can lookup to find related information. Since this is very important while we are trying to evaluate whether to use Comet or not.

Thank you so much !

[viirya]

Hello, I answered in #64.

[comphead]

Can it be closed via #125? @viirya

[viirya]

Okay

[viirya]

@jinwenjie123 Feel free to open new issues on Comet usage if you still have other questions. Thanks.