Skip to content

Draft: Use upstream arrow coalesce kernel in DataFusion #16249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 19 commits into from
Closed

Draft: Use upstream arrow coalesce kernel in DataFusion #16249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
9a161d2
Pin to https://github.com/apache/arrow-rs/pull/7597
alamb Jun 4, 2025
083931d
Update pin
alamb Jun 4, 2025
e79454f
Use upstream BatchCoalescer
alamb Jun 4, 2025
4e8e1ce
Update the pin
alamb Jun 4, 2025
9e20973
Update tests
alamb Jun 4, 2025
8918b3c
Update rev
Dandandan Jun 7, 2025
49cb62e
Update rev
Dandandan Jun 7, 2025
140ee9c
New rev
Dandandan Jun 7, 2025
5d5683c
New rev
Dandandan Jun 7, 2025
f79dd09
New rev
Dandandan Jun 7, 2025
1c44c5c
cargo fmt
Dandandan Jun 7, 2025
ea8b700
update pin
alamb Jun 8, 2025
f2fc00b
Merge branch 'main' into alamb/test_upstream_coalesce
Dandandan Jun 13, 2025
a36065e
Merge branch 'main' into alamb/test_upstream_coalesce
Dandandan Jun 15, 2025
423137a
Merge remote-tracking branch 'apache/main' into alamb/test_upstream_c…
alamb Jun 17, 2025
1c61513
Temp pin to https://github.com/apache/arrow-rs/pull/7650
alamb Jun 17, 2025
ed31ce1
Update plans for smaller parquet files
alamb Jun 17, 2025
5349c73
Merge remote-tracking branch 'apache/main' into alamb/test_upstream_c…
alamb Jun 19, 2025
c5bb25e
update pin
alamb Jun 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 17 additions & 43 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

44 changes: 44 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,3 +219,47 @@ uninlined_format_args = "warn"
[workspace.lints.rust]
unexpected_cfgs = { level = "warn", check-cfg = ["cfg(tarpaulin)", "cfg(tarpaulin_include)"] }
unused_qualifications = "deny"

# pin to pre-release arrow
#patch.crates-io]
#arrow = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }
#arrow-array = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }
#arrow-buffer = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }
#arrow-cast = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }
#arrow-data = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }
#arrow-ipc = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }
#arrow-schema = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }
#arrow-select = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }
#arrow-string = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }
#arrow-ord = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }
#arrow-flight = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }
#parquet = { git = "https://github.com/apache/arrow-rs.git", rev = "52d8d568f4ddd9069b743fc0eab17ffe102da35b" }

# pin to https://github.com/apache/arrow-rs/pull/7650
[patch.crates-io]
arrow = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }
arrow-array = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }
arrow-buffer = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }
arrow-cast = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }
arrow-data = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }
arrow-ipc = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }
arrow-schema = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }
arrow-select = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }
arrow-string = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }
arrow-ord = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }
arrow-flight = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }
parquet = { git = "https://github.com/alamb/arrow-rs.git", rev = "99161728b473dc9d0ee1b8a1387ec94110cfc1cc" }


#arrow = { path= "/Users/andrewlamb/Software/arrow-rs/arrow" }
#arrow-array = { path= "/Users/andrewlamb/Software/arrow-rs/arrow-array" }
#arrow-buffer = { path= "/Users/andrewlamb/Software/arrow-rs/arrow-buffer" }
#arrow-cast = { path= "/Users/andrewlamb/Software/arrow-rs/arrow-cast" }
#arrow-data = { path= "/Users/andrewlamb/Software/arrow-rs/arrow-data" }
#arrow-ipc = { path= "/Users/andrewlamb/Software/arrow-rs/arrow-ipc" }
#arrow-schema = { path= "/Users/andrewlamb/Software/arrow-rs/arrow-schema" }
#arrow-select = { path= "/Users/andrewlamb/Software/arrow-rs/arrow-select" }
#arrow-string = { path= "/Users/andrewlamb/Software/arrow-rs/arrow-string" }
#arrow-ord = { path= "/Users/andrewlamb/Software/arrow-rs/arrow-ord" }
#arrow-flight = { path= "/Users/andrewlamb/Software/arrow-rs/arrow-flight" }
#parquet = { path= "/Users/andrewlamb/Software/arrow-rs/parquet" }
2 changes: 1 addition & 1 deletion datafusion/core/src/datasource/listing/table.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1880,7 +1880,7 @@ mod tests {
ParquetFormat::default().get_ext(),
FileCompressionType::UNCOMPRESSED,
Some(config_map),
2,
4,
)
.await?;
Ok(())
Expand Down
Loading
Loading