-
Notifications
You must be signed in to change notification settings - Fork 285
Open
Labels
bugSomething isn't workingSomething isn't workingnative_iceberg_compatSpecific to native_iceberg_compat scan typeSpecific to native_iceberg_compat scan typespark 4
Description
Describe the bug
When switching from native_comet to native_iceberg_compat in Spark 4.0.0, I see some tests fail in VariantShreddingSuite. It appears to be an issue with selecting fields by index from within structs.
[info] == Optimized Logical Plan ==
[info] Project [v#216.0 AS variant_get(v, $.a)#211, v#216.1 AS variant_get(v, $.b)#212L, v#216.2 AS variant_get(v, $.c)#213, v#216.3 AS variant_get(v, $.d)#214]
[info] +- Relation [v#216] parquet
[info]
[info] == Physical Plan ==
[info] *(1) CometColumnarToRow
[info] +- CometProject [variant_get(v, $.a)#211, variant_get(v, $.b)#212L, variant_get(v, $.c)#213, variant_get(v, $.d)#214], [v#216.0 AS variant_get(v, $.a)#211, v#216.1 AS variant_get(v, $.b)#212L, v#216.2 AS variant_get(v, $.c)#213, v#216.3 AS variant_get(v, $.d)#214]
[info] +- CometScan [native_iceberg_compat] parquet [v#216] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/home/andy/git/apache/apache-spark-4.0.0/target/tmp/spark-5897c3a..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<v:struct<0:int,1:bigint,2:double,3:decimal(9,4)>>
[info]
[info] == Results ==
[info]
[info] == Results ==
[info] !== Correct Answer - 3 == == Spark Answer - 3 ==
[info] !struct<> struct<variant_get(v, $.a):int,variant_get(v, $.b):bigint,variant_get(v, $.c):double,variant_get(v, $.d):decimal(9,4)>
[info] ![1,2,3.3,4.4] [null,null,null,null]
[info] ![5,-6,null,null] [null,null,null,null]
[info] [null,null,null,null] [null,null,null,null]
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingnative_iceberg_compatSpecific to native_iceberg_compat scan typeSpecific to native_iceberg_compat scan typespark 4