-
Notifications
You must be signed in to change notification settings - Fork 285
Closed
Description
Summary
With the introduction of native_datafusion in auto scan mode (PR #3307), several test helpers that check the scan implementation config are broken when running in auto mode. The root cause is that helpers like usingDataSourceExec check if the config string is literally native_datafusion or native_iceberg_compat, but in auto mode the config reads as "auto" even though it resolves to native_datafusion at plan time.
Failing Tests (in auto mode)
- "schema evolution" (
ParquetReadSuite.scala:1256) — expectsSparkExceptionbutnative_datafusionhandles type widening gracefully - "row group skipping doesn't overflow when reading into larger type" (
ParquetReadSuite.scala:1523) — same issue
Proposed Fix
Since native_comet is deprecated and the default path is now DataSource-based (via auto), invert the check:
- Rename
usingDataSourceExec→usingLegacyNativeCometScanwhich returnstrueonly when config is explicitlynative_comet - Flip all ~40 call sites accordingly
- Update
usingDataSourceExecWithIncompatTypessimilarly - Fix the explicit
SCAN_NATIVE_DATAFUSIONcheck in the schema evolution test
This avoids needing to enumerate all non-legacy modes and is forward-compatible with future scan implementations.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels