fix: prevent UnionExec panic with empty inputs #17449

EeshanBembi · 2025-09-05T20:27:20Z

Summary

This PR fixes a panic in UnionExec when constructed with empty inputs, replacing the crash with proper error handling and descriptive error messages.

Fixes: #17052

Problem

When UnionExec::new(vec![]) was called with an empty input vector, it would panic with:

thread '...' panicked at datafusion/physical-plan/src/union.rs:542:24:
index out of bounds: the len is 0 but the index is 0

This occurred because union_schema() directly accessed inputs[0] without checking if the array was empty.

Solution

Core Changes

Made UnionExec::new() return Result<Self>:
- Added validation: returns error if inputs.is_empty()
- Provides clear error message: "UnionExec requires at least one input"
Made union_schema() return Result<SchemaRef>:
- Added empty input validation before accessing inputs[0]
- Returns descriptive error: "Cannot create union schema from empty inputs"
Updated all call sites (7 files):
- physical_planner.rs - Core DataFusion integration
- repartition/mod.rs - Internal dependencies
- 4 test files - Updated to handle Result return type

Error Handling

Before: Index out of bounds panic (unhelpful)
After: Clear error messages that guide users

// Before: panic!
let union = UnionExec::new(vec![]); // PANIC!

// After: proper error handling
match UnionExec::new(vec![]) {
    Ok(_) => { /* use union */ }
    Err(e) => println!("Error: {}", e); // "UnionExec requires at least one input"
}

Testing

Added 4 comprehensive tests:

test_union_empty_inputs() - Verifies empty input validation
test_union_schema_empty_inputs() - Tests schema creation with empty inputs
test_union_single_input() - Ensures single input still works
test_union_multiple_inputs_still_works() - Verifies existing functionality unchanged

Test Results:

✅ All new tests pass
✅ All existing union tests pass (8/8)
✅ All physical planner integration tests pass

Backward Compatibility

Existing functionality unchanged for valid inputs (≥1 input)
Only adds error handling for previously crashing invalid inputs
API change: UnionExec::new() now returns Result<Self> instead of Self

This is a breaking change but justified because:

The previous behavior (panic) was incorrect
Empty inputs are invalid by design (no logical meaning)
Consistent with logical Union which requires ≥2 inputs
Better error handling improves user experience

Files Changed

datafusion/physical-plan/src/union.rs - Core fix + tests (main changes)
datafusion/core/src/physical_planner.rs - Handle Result return
datafusion/physical-plan/src/repartition/mod.rs - Update internal calls
4 test files - Update test utilities and test cases

The fix provides robust error handling while maintaining all existing functionality for valid use cases.

This commit fixes a panic in UnionExec when constructed with empty inputs. Previously, UnionExec::new(vec![]) would cause an index out of bounds panic at union.rs:542 when trying to access inputs[0]. Changes: - Made UnionExec::new() return Result<Self> with proper validation - Made union_schema() return Result<SchemaRef> with empty input checks - Added descriptive error messages for empty input cases - Updated all call sites to handle the new Result return type - Added comprehensive tests for edge cases Error messages: - "UnionExec requires at least one input" - "Cannot create union schema from empty inputs" The fix maintains backward compatibility for valid inputs while preventing crashes and providing clear error messages for invalid usage. Fixes apache#17052

alamb

Thanks @EeshanBembi

alamb · 2025-09-05T21:11:25Z

datafusion/physical-plan/src/union.rs

+    fn test_union_empty_inputs() {
+        // Test that UnionExec::new fails with empty inputs
+        let result = UnionExec::new(vec![]);
+        assert!(result.is_err());


I think the assertion check for is_err is redundant as unwrap_err will panic if result is not an err

alamb · 2025-09-05T21:12:26Z

datafusion/physical-plan/src/union.rs

    /// Create a new UnionExec
-    pub fn new(inputs: Vec<Arc<dyn ExecutionPlan>>) -> Self {
-        let schema = union_schema(&inputs);
+    pub fn new(inputs: Vec<Arc<dyn ExecutionPlan>>) -> Result<Self> {


this is technically an API change -- maybe to make it easier on others, we can make a new function called try_new that has the error checking, and deprecate the existing new function per https://datafusion.apache.org/contributor-guide/api-health.html#deprecation-guidelines

Good point on the API lifecycle. On separate note, can we make the new try_new method return Box<<dyn ExecutionPlan>>? This would allow it to return the only child in case input vector is a singleton. There is no point keeping UnionExec(a) in the plan.
Or maybe, the new method can simply require the input to have at least two elements?

findepi · 2025-09-10T07:29:25Z

datafusion/physical-plan/src/union.rs

    /// Create a new UnionExec
-    pub fn new(inputs: Vec<Arc<dyn ExecutionPlan>>) -> Self {
-        let schema = union_schema(&inputs);
+    pub fn new(inputs: Vec<Arc<dyn ExecutionPlan>>) -> Result<Self> {


Good point on the API lifecycle. On separate note, can we make the new try_new method return Box<<dyn ExecutionPlan>>? This would allow it to return the only child in case input vector is a singleton. There is no point keeping UnionExec(a) in the plan.
Or maybe, the new method can simply require the input to have at least two elements?

findepi · 2025-09-10T07:30:15Z

datafusion/physical-plan/src/union.rs

+    }
+
+    #[test]
+    fn test_union_multiple_inputs_still_works() -> Result<()> {


Suggested change

fn test_union_multiple_inputs_still_works() -> Result<()> {

fn test_union_schema_multiple_inputs() -> Result<()> {

xudong963 · 2025-09-10T08:41:46Z

This PR fixes a panic in UnionExec when constructed with empty inputs,

Not related to the PR, but it'll be better for df to have a optimizer rule to remove empty inputs from union

findepi · 2025-09-10T15:22:29Z

This PR fixes a panic in UnionExec when constructed with empty inputs,

Not related to the PR, but it'll be better for df to have a optimizer rule to remove empty inputs from union

@xudong963 , exactly

See also #17449 (comment)
This is, however, related to this PR -- we're adding a new API method and we can address the problem directly in this new code, leaving legacy code path intact.

- Add new try_new method that returns Result<Arc<dyn ExecutionPlan>> - Deprecate existing new method in favor of try_new - Optimize single-input case: try_new returns the input directly - Remove redundant assert!(result.is_err()) from tests - Rename test_union_multiple_inputs_still_works to test_union_schema_multiple_inputs - Update all call sites to use appropriate API (try_new for new code, deprecated new for tests) This maintains backward compatibility while providing better error handling and optimization for single-input cases.

findepi · 2025-09-13T16:53:26Z

datafusion/core/tests/physical_optimizer/enforce_distribution.rs

    );

    //  Union
+    #[allow(deprecated)]


Let's have an issue to clean these up and add a // TODO (issue link) resolve deprecation comment

Filed Update code to use new, non deprecated UnionExec API #17559

- Add proper feature gates for parquet_encryption in datasource-parquet - Format code to pass cargo fmt checks - All tests passing

…nputs-validation

alamb · 2025-09-15T10:39:30Z

I merged up from main and fixed a clippy lint

alamb

Thank you @EeshanBembi and @findepi

* fix: prevent UnionExec panic with empty inputs This commit fixes a panic in UnionExec when constructed with empty inputs. Previously, UnionExec::new(vec![]) would cause an index out of bounds panic at union.rs:542 when trying to access inputs[0]. Changes: - Made UnionExec::new() return Result<Self> with proper validation - Made union_schema() return Result<SchemaRef> with empty input checks - Added descriptive error messages for empty input cases - Updated all call sites to handle the new Result return type - Added comprehensive tests for edge cases Error messages: - "UnionExec requires at least one input" - "Cannot create union schema from empty inputs" The fix maintains backward compatibility for valid inputs while preventing crashes and providing clear error messages for invalid usage. Fixes apache#17052 * refactor: address PR review comments for UnionExec empty inputs fix - Add new try_new method that returns Result<Arc<dyn ExecutionPlan>> - Deprecate existing new method in favor of try_new - Optimize single-input case: try_new returns the input directly - Remove redundant assert!(result.is_err()) from tests - Rename test_union_multiple_inputs_still_works to test_union_schema_multiple_inputs - Update all call sites to use appropriate API (try_new for new code, deprecated new for tests) This maintains backward compatibility while providing better error handling and optimization for single-input cases. * Fix cargo fmt and clippy warnings - Add proper feature gates for parquet_encryption in datasource-parquet - Format code to pass cargo fmt checks - All tests passing * Fix clippy --------- Co-authored-by: Eeshan <eeshan@Eeshans-MacBook-Pro.local> Co-authored-by: ebembi-crdb <ebembi@cockroachlabs.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> (cherry picked from commit b122a16)

@asolimando

* Allow filter pushdown through AggregateExec (apache#18404) ## Which issue does this PR close? - Closes apache#18399 ## Rationale for this change Right now filters cannot pass through `AggregateExec` nodes, preventing filter pushdown optimization in queries with GROUP BY/DISTINCT operations. ## What changes are included in this PR? - Implemented `gather_filters_for_pushdown()` for `AggregateExec` that allows filters on grouping columns to pass through to children - Supports both Pre phase (static filters) and Post phase (dynamic filters from joins) Essentially, filter will pass through in the scenarios @asolimando mentioned [here](apache#18399 (comment)) ## Are these changes tested? Yes, added three tests: - `test_aggregate_filter_pushdown`: Positive case with aggregate functions - `test_no_pushdown_aggregate_filter_on_non_grouping_column`: Negative case ensuring filters on aggregate results are not pushed ## Are there any user-facing changes?   (cherry picked from commit 076b091) * physical-plan: push filters down to UnionExec children (apache#18054) Filters are safe to be pushed down, so we can override the default behavior here. Signed-off-by: Alfonso Subiotto Marques <alfonso.subiotto@polarsignals.com> (cherry picked from commit 0ecd59b) * fix: prevent UnionExec panic with empty inputs (apache#17449) * fix: prevent UnionExec panic with empty inputs This commit fixes a panic in UnionExec when constructed with empty inputs. Previously, UnionExec::new(vec![]) would cause an index out of bounds panic at union.rs:542 when trying to access inputs[0]. Changes: - Made UnionExec::new() return Result<Self> with proper validation - Made union_schema() return Result<SchemaRef> with empty input checks - Added descriptive error messages for empty input cases - Updated all call sites to handle the new Result return type - Added comprehensive tests for edge cases Error messages: - "UnionExec requires at least one input" - "Cannot create union schema from empty inputs" The fix maintains backward compatibility for valid inputs while preventing crashes and providing clear error messages for invalid usage. Fixes apache#17052 * refactor: address PR review comments for UnionExec empty inputs fix - Add new try_new method that returns Result<Arc<dyn ExecutionPlan>> - Deprecate existing new method in favor of try_new - Optimize single-input case: try_new returns the input directly - Remove redundant assert!(result.is_err()) from tests - Rename test_union_multiple_inputs_still_works to test_union_schema_multiple_inputs - Update all call sites to use appropriate API (try_new for new code, deprecated new for tests) This maintains backward compatibility while providing better error handling and optimization for single-input cases. * Fix cargo fmt and clippy warnings - Add proper feature gates for parquet_encryption in datasource-parquet - Format code to pass cargo fmt checks - All tests passing * Fix clippy --------- Co-authored-by: Eeshan <eeshan@Eeshans-MacBook-Pro.local> Co-authored-by: ebembi-crdb <ebembi@cockroachlabs.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> (cherry picked from commit b122a16) --------- Signed-off-by: Alfonso Subiotto Marques <alfonso.subiotto@polarsignals.com> Co-authored-by: Alfonso Subiotto Marqués <alfonso.subiotto@polarsignals.com> Co-authored-by: EeshanBembi <33062610+EeshanBembi@users.noreply.github.com> Co-authored-by: Eeshan <eeshan@Eeshans-MacBook-Pro.local> Co-authored-by: ebembi-crdb <ebembi@cockroachlabs.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

github-actions bot added core Core DataFusion crate physical-plan Changes to the physical-plan crate labels Sep 5, 2025

alamb reviewed Sep 5, 2025

View reviewed changes

findepi reviewed Sep 10, 2025

View reviewed changes

github-actions bot added the proto Related to proto crate label Sep 13, 2025

EeshanBembi requested review from alamb and findepi September 13, 2025 15:35

findepi approved these changes Sep 13, 2025

View reviewed changes

alamb mentioned this pull request Sep 14, 2025

Update code to use new, non deprecated UnionExec API #17559

Closed

Fix cargo fmt and clippy warnings

179beff

- Add proper feature gates for parquet_encryption in datasource-parquet - Format code to pass cargo fmt checks - All tests passing

github-actions bot added the datasource Changes to the datasource crate label Sep 14, 2025

xudong963 mentioned this pull request Sep 15, 2025

Add an optimizer rule to remove empty inputs from union #17568

Closed

alamb added 2 commits September 15, 2025 06:37

Merge remote-tracking branch 'apache/main' into fix/unionexec-empty-i…

53c341f

…nputs-validation

Fix clippy

0400ed5

alamb approved these changes Sep 15, 2025

View reviewed changes

findepi merged commit b122a16 into apache:main Sep 15, 2025
28 checks passed

alamb mentioned this pull request Sep 16, 2025

minor: fix compilation issue for extended tests due to missing parquet encryption flag #17579

Merged

Jefffrey mentioned this pull request Oct 2, 2025

feat: Add EliminateUnion rule #17610

Closed

	fn test_union_multiple_inputs_still_works() -> Result<()> {
	fn test_union_schema_multiple_inputs() -> Result<()> {

fix: prevent UnionExec panic with empty inputs #17449

fix: prevent UnionExec panic with empty inputs #17449

Uh oh!

Conversation

EeshanBembi commented Sep 5, 2025

Summary

Problem

Solution

Core Changes

Error Handling

Testing

Backward Compatibility

Files Changed

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

findepi Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

findepi Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

findepi Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

xudong963 commented Sep 10, 2025

Uh oh!

findepi commented Sep 10, 2025

Uh oh!

findepi Sep 13, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Sep 14, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Sep 15, 2025

Uh oh!

alamb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alamb left a comment •

edited

Loading