[native_datafusion] Enable native_datafusion in auto scan mode — test failure tracking

[andygrove]

Summary

Tracking issue for all test failures discovered when enabling native_datafusion in auto scan mode (#3307).

Issues

Test infrastructure

• Invert usingDataSourceExec test helper to usingLegacyNativeCometScan #3309 — Invert usingDataSourceExec test helper to usingLegacyNativeCometScan

Spark SQL test failures by category

• [native_datafusion] [Spark SQL Tests] Schema incompatibility tests expect exceptions that native_datafusion handles gracefully #3311 — Schema incompatibility tests expect exceptions that native_datafusion handles gracefully (8+ tests)
• [native_datafusion] [Spark SQL Tests] input_file_name() returns empty string #3312 — input_file_name() returns empty string (3+ tests)
• [native_datafusion] [Spark SQL Tests] Dynamic Partition Pruning (DPP) not working correctly #3313 — Dynamic Partition Pruning (DPP) not working correctly (3+ tests)
• [native_datafusion] [Spark SQL Tests] Missing files error handling differs from Spark #3314 — Missing/corrupt files error handling differs from Spark (2+ tests)
• [native_datafusion] [Spark SQL Tests] Plan structure differences cause test failures #3315 — Plan structure differences cause test failures (4+ tests)
• [native_datafusion] [Spark SQL Tests] Parquet field ID matching not supported #3316 — Parquet field ID matching not supported (5+ tests)
• [native_datafusion] [Spark SQL Tests] _tmp_metadata_row_index not populated #3317 — _tmp_metadata_row_index / row index generation not populated (24+ tests)
• [native_datafusion] [Spark SQL Tests] Nested schema pruning tests fail — CometNativeScan not recognized as file source #3318 — Nested schema pruning tests fail — CometNativeScan not recognized as file source (~130 tests)
• [native_datafusion] [Spark SQL Tests] Bucketing not exposed by CometNativeScan #3319 — Bucketing not exposed by CometNativeScan (~7 tests)
• [native_datafusion] [Spark SQL Tests] CometNativeExec crash — executed without serialized plan #3320 — CometNativeExec crash — executed without serialized plan (4+ tests)

Total

~190+ test failures across 10 categories.