Skip to content

[native_datafusion] [Spark SQL Tests] Missing files error handling differs from Spark #3314

@andygrove

Description

@andygrove

Summary

2 Spark SQL tests fail because native_datafusion produces different error behavior when files are missing or mutated.

Failing Tests

  • FileBasedDataSourceSuite: "Enabling/disabling ignoreMissingFiles using parquet" — native_datafusion throws "Object at location ... not found" instead of being silently ignored
  • SimpleSQLViewSuite: "alter temporary view should follow current storeAnalyzedPlanForView config" — file deleted after plan creation causes native error

Root Cause

nativeDataFusionScan already has a fallback check for ignoreMissingFiles, but the test scenario may not be triggering it (e.g., the config may be set after plan creation, or the fallback condition isn't matching). Native DataFusion's error message format also differs from Spark's expected error handling.

Related

Discovered in CI for #3307 (enable native_datafusion in auto scan mode).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions