-
Notifications
You must be signed in to change notification settings - Fork 285
Open
Labels
bugSomething isn't workingSomething isn't workingcrashNative engine crash/panic/segfaultNative engine crash/panic/segfault
Description
Description
When Spark's ConstantFolding optimizer rule is disabled, all-scalar (literal-only) expressions reach Comet's native engine instead of being folded at plan time. The native engine panics with messages like:
hour(scalar) should be fold in Spark JVM side.minute(scalar) should be fold in Spark JVM side.second(scalar) should be fold in Spark JVM side.unix_timestamp(scalar) should be fold in Spark JVM side.
How to Reproduce
Disable constant folding and run an all-literal datetime extraction query:
SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding;
SELECT hour(timestamp('2024-01-15 12:30:45'));Expected Behavior
The native engine should handle scalar inputs gracefully — either by computing the correct result or by falling back to Spark, not by panicking.
Affected Expressions
hour()minute()second()unix_timestamp()
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingcrashNative engine crash/panic/segfaultNative engine crash/panic/segfault