Skip to content

Commit cd6b2f8

Browse files
peaseephillipleblanckczimmsgrebnovJeadie
authored
feat: Update for Datafusion v50 (#109)
* fix: Ensure only tables or aliases that exist are projected (#52) fix: More dangling references (#54) UPSTREAM NOTE: This PR was attempted to be upstreamed in apache#13405 - but it was not accepted due to the complexity it brought. Phillip needs to figure out what a good solution that solves our problem and can be upstreamed is. * Support for metadata columns (`location`, `size`, `last_modified`) in ListingTableProvider (#74) UPSTREAM NOTE: This PR was attempted to be upstreamed but was not accepted. Needs to be applied manually apache#15181 * Infer placeholder datatype for `Expr::InSubquery` (#80) UPSTREAM NOTE: Upstream PR has been created but not merged yet. Should be available in DF49 apache#15980 * Infer placeholder datatype after `LIMIT` clause as `DataType::Int64` (#81) UPSTREAM NOTE: Upstream PR has been created but not merged yet. Should be available in DF49 apache#15980 * Do not double alias Exprs UPSTREAM NOTE: This was attempted to be fixed with apache#15008 but was closed This is the tracking issue on DataFusion: apache#14895 Do not double alias Exprs * Add prefix to location metadata column (#82) UPSTREAM NOTE: This will not be upstreamed as is. * Infer placeholder types for CASE expressions (#87) UPSTREAM NOTE: This has not been submitted upstream yet. * Expand `infer_placeholder_types` to infer all possible placeholder types based on their expression (#88) UPSTREAM NOTE: This has not been submitted upstream yet. * Fix `Expr::infer_placeholder_types` inference to not fail (#89) UPSTREAM NOTE: This has not been submitted upstream yet. * cherry-pick parquet patch (#94) * Fix array types coercion: preserve child element nullability for list types (#96) UPSTREAM NOTE: This was submitted upstream and should be available in DF50 apache#17306 * Expand `infer_placeholder_types` to infer all possible placeholder types based on their expression (#88) UPSTREAM NOTE: This has not been submitted upstream yet. * do not enforce type guarantees on all Expr traversed in infer_placeholder_types (#97) * Use UDTF function args in `LogicalPlan::TableScan` name (#98) * use UDTF function args in LogicalPlan::TableScan name * update test snapshots * Implement timestamp_cast_dtype for SqliteDialect (#99) * Use text for sqlite timestamp * Add test * Custom timestamp format for DuckDB (#102) * Revert "cherry-pick parquet patch (#94)" This reverts commit d780cc2. * Support ExprNamed arguments to Scalar UDFs (#104) * support ExprNamed until 17379 ships * add same exprnamed lifting to udtf * resolve projection against `ListingTable` table_schema incl. partition columns (#106) * fix: Ensure ListingTable partitions are pruned when filters are not used (#108) * fix: Prune partitions when no filters are defined * fix: Backport for DF49: * review: Address comments * FileScanConfig: Preserve schema metadata across serde boundary (#107) * FileScanConfig: preserve schema metadata across serde boundary * add test * Merge conflict fixes UPSTREAM NOTE: this should not be upstreamed. This contains conflict fixes from various cherry-picks and differences in v50. * update arrow-rs fork UPSTREAM NOTE: this should not be upstreamed --------- Co-authored-by: Phillip LeBlanc <phillip@leblanc.tech> Co-authored-by: Kevin Zimmerman <4733573+kczimm@users.noreply.github.com> Co-authored-by: sgrebnov <sergei.grebnov@gmail.com> Co-authored-by: jeadie <jack@spice.ai> Co-authored-by: Jack Eadie <jack.eadie0@gmail.com> Co-authored-by: Viktor Yershov <krinart@gmail.com> Co-authored-by: Viktor Yershov <viktor@spice.ai> Co-authored-by: David Stancu <david@spice.ai>
1 parent 7909909 commit cd6b2f8

File tree

43 files changed

+5099
-398
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+5099
-398
lines changed

‎Cargo.lock‎

Lines changed: 20 additions & 34 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,3 +232,12 @@ unexpected_cfgs = { level = "warn", check-cfg = [
232232
"cfg(tarpaulin_include)",
233233
] }
234234
unused_qualifications = "deny"
235+
236+
[patch.crates-io]
237+
arrow = { git = "https://github.com/spiceai/arrow-rs.git", rev = "9f9c372ff5744488226462e3ea7d94ff47909833" } # spiceai-56
238+
arrow-buffer = { git = "https://github.com/spiceai/arrow-rs.git", rev = "9f9c372ff5744488226462e3ea7d94ff47909833" } # spiceai-56
239+
arrow-flight = { git = "https://github.com/spiceai/arrow-rs.git", rev = "9f9c372ff5744488226462e3ea7d94ff47909833" } # spiceai-56
240+
arrow-ipc = { git = "https://github.com/spiceai/arrow-rs.git", rev = "9f9c372ff5744488226462e3ea7d94ff47909833" } # spiceai-56
241+
arrow-ord = { git = "https://github.com/spiceai/arrow-rs.git", rev = "9f9c372ff5744488226462e3ea7d94ff47909833" } # spiceai-56
242+
arrow-schema = { git = "https://github.com/spiceai/arrow-rs.git", rev = "9f9c372ff5744488226462e3ea7d94ff47909833" } # spiceai-56
243+
parquet = { git = "https://github.com/spiceai/arrow-rs.git", rev = "9f9c372ff5744488226462e3ea7d94ff47909833" } # spiceai-56

‎datafusion/catalog-listing/Cargo.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,8 @@ object_store = { workspace = true }
4848
tokio = { workspace = true }
4949

5050
[dev-dependencies]
51+
chrono = { workspace = true }
52+
tempfile = { workspace = true }
5153

5254
[lints]
5355
workspace = true

0 commit comments

Comments
 (0)