Skip to content

[native_iceberg_compat] VariantShreddingSuite test failures with Spark 4.0.0 #2209

@andygrove

Description

@andygrove

Describe the bug

When switching from native_comet to native_iceberg_compat in Spark 4.0.0, I see some tests fail in VariantShreddingSuite. It appears to be an issue with selecting fields by index from within structs.

[info]   == Optimized Logical Plan ==
[info]   Project [v#216.0 AS variant_get(v, $.a)#211, v#216.1 AS variant_get(v, $.b)#212L, v#216.2 AS variant_get(v, $.c)#213, v#216.3 AS variant_get(v, $.d)#214]
[info]   +- Relation [v#216] parquet
[info]   
[info]   == Physical Plan ==
[info]   *(1) CometColumnarToRow
[info]   +- CometProject [variant_get(v, $.a)#211, variant_get(v, $.b)#212L, variant_get(v, $.c)#213, variant_get(v, $.d)#214], [v#216.0 AS variant_get(v, $.a)#211, v#216.1 AS variant_get(v, $.b)#212L, v#216.2 AS variant_get(v, $.c)#213, v#216.3 AS variant_get(v, $.d)#214]
[info]      +- CometScan [native_iceberg_compat] parquet [v#216] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/home/andy/git/apache/apache-spark-4.0.0/target/tmp/spark-5897c3a..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<v:struct<0:int,1:bigint,2:double,3:decimal(9,4)>>
[info]   
[info]   == Results ==
[info]   
[info]   == Results ==
[info]   !== Correct Answer - 3 ==   == Spark Answer - 3 ==
[info]   !struct<>                   struct<variant_get(v, $.a):int,variant_get(v, $.b):bigint,variant_get(v, $.c):double,variant_get(v, $.d):decimal(9,4)>
[info]   ![1,2,3.3,4.4]              [null,null,null,null]
[info]   ![5,-6,null,null]           [null,null,null,null]
[info]    [null,null,null,null]      [null,null,null,null]

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingnative_iceberg_compatSpecific to native_iceberg_compat scan typespark 4

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions