What's the Supported Input Data Type

[jinwenjie123]

What is the problem the feature request solves?

Hi Team,

I am wondering will we have a documentation about what kinda of data types are supported by Comet/Datafusion (Like, Decimal(16, 6), interger .... ) and will not fallback to vanilla spark and cause the regression.

Or where I can lookup to find related information. Since this is very important while we are trying to evaluate whether to use Comet or not.

Thank you so much !

Describe the potential solution

No response

Additional context

No response

[viirya]

We will add the document, maybe as part of user document in #57.

For a quick answer, primitive types should be supported now: ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, StringType, BinaryType, TimestampType, DecimalType, DateType, BooleanType, NullType. The support I mean is, Comet query planner will accept query plan using these types. But for certain expression or operator, the supported types might be various, for example, if the corresponding DataFusion expression or operator may not support some data types.

I think one important goal of Comet is to fulfill the gap and we are working on it. So welcome to report any missing datatypes when you test or evaluate Comet.

[jinwenjie123]

Thank you !

One more quick question. Is Decimal(26, 6) currently supported ?

[viirya]

DecimalType is supported.

[sunchao]

We should support all primitive data types as of now (if not, we should definitely fill the gap).

Complex types (e.g., list, map, struct) are not supported in the Parquet scan. We do have plan to support it soon.

[viirya]

We have better documentation for the issue: https://datafusion.apache.org/comet/user-guide/datatypes.html

[viirya]

Close it now. Please let me know if you still have other questions. Thanks.