Error projecting statistics in DataSourceExec #14905
Description
Describe the bug
While working on upgrading Delta.rs to DataFusion 46 I am getting the following error
index out of bounds: the len is 2 but the index is 2
To Reproduce
I am still working on a minimal reproducer. I don't have one yet
Here is one from
Here is the error
/Users/andrewlamb/.cargo/bin/cargo test --color=always --lib operations::merge::tests::test_empty_table_with_schema_merge --profile test --no-fail-fast --config env.RUSTC_BOOTSTRAP=\"1\" --manifest-path /Users/andrewlamb/Software/delta-rs/crates/core/Cargo.toml -- --format=json --exact -Z unstable-options --show-output
Testing started at 9:59 AM ...
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package: /Users/andrewlamb/Software/delta-rs/python/Cargo.toml
workspace: /Users/andrewlamb/Software/delta-rs/Cargo.toml
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.23s
Running unittests src/lib.rs (target/debug/deps/deltalake_core-b6362ef613c3cec4)
index out of bounds: the len is 2 but the index is 2
thread 'operations::merge::tests::test_empty_table_with_schema_merge' panicked at /Users/andrewlamb/Software/datafusion2/datafusion/physical-plan/src/projection.rs:291:36:
index out of bounds: the len is 2 but the index is 2
stack backtrace:
0: rust_begin_unwind
at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/core/src/panicking.rs:76:14
2: core::panicking::panic_bounds_check
at /rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/core/src/panicking.rs:281:5
3: index<datafusion_common::stats::ColumnStatistics>
at /Users/andrewlamb/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/slice/index.rs:274:10
4: index<datafusion_common::stats::ColumnStatistics, usize>
at /Users/andrewlamb/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/slice/index.rs:16:9
5: index<datafusion_common::stats::ColumnStatistics, usize, alloc::alloc::Global>
at /Users/andrewlamb/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/mod.rs:3346:9
6: stats_projection<core::iter::adapters::map::Map<core::slice::iter::Iter<(alloc::sync::Arc<dyn datafusion_physical_expr_common::physical_expr::PhysicalExpr, alloc::alloc::Global>, alloc::string::String)>, datafusion_physical_plan::projection::{impl#2}::statistics::{closure_env#0}>>
at /Users/andrewlamb/Software/datafusion2/datafusion/physical-plan/src/projection.rs:291:36
7: statistics
at /Users/andrewlamb/Software/datafusion2/datafusion/physical-plan/src/projection.rs:234:12
8: should_swap_join_order
at /Users/andrewlamb/Software/datafusion2/datafusion/physical-optimizer/src/join_selection.rs:69:23
9: statistical_join_selection_subrule
at /Users/andrewlamb/Software/datafusion2/datafusion/physical-optimizer/src/join_selection.rs:329:28
10: {closure#1}
at /Users/andrewlamb/Software/datafusion2/datafusion/physical-optimizer/src/join_selection.rs:187:17
11: call_once<(alloc::sync::Arc<dyn datafusion_physical_plan::execution_plan::ExecutionPlan, alloc::alloc::Global>), datafusion_physical_optimizer::join_selection::{impl#1}::optimize::{closure_env#1}>
at /Users/andrewlamb/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:305:13
12: transform_parent<alloc::sync::Arc<dyn datafusion_physical_plan::execution_plan::ExecutionPlan, alloc::alloc::Global>, &mut datafusion_physical_optimizer::join_selection::{impl#1}::optimize::{closure_env#1}>
at /Users/andrewlamb/Software/datafusion2/datafusion/common/src/tree_node.rs:763:44
13: {closure#0}<alloc::sync::Arc<dyn datafusion_physical_plan::execution_plan::ExecutionPlan, alloc::alloc::Global>, datafusion_physical_optimizer::join_selection::{impl#1}::optimize::{closure_env#1}>
at /Users/andrewlamb/Software/datafusion2/datafusion/common/src/tree_node.rs:264:13
error: test failed, to rerun pass `--lib`
error: 1 target failed:
`--lib`
The full trace is here:
Expected behavior
Error should not happen
Additional context
- Related Release DataFusion
46.0.0#14123
Likely introduced as part of