chore: bump substrait version to `0.60.0` to use substrait spec v0.75.0 #17866

benbellick · 2025-10-01T18:09:39Z

Which issue does this PR close?

Closes Update substrait to v0.75.0 for URI -> URN migration #17865.

What changes are included in this PR?

Bump the substrait version to v0.75.0 by bumping substrait-rs to v0.60.0.

This PR was originally dependent on this PR to update the versions of some common dependencies, but that PR is now merged in.

Are these changes tested?

There are no tests here, but there is no change to any logic within datafusion. It is simply a bump in a dependency. Technically the public API does change, but as noted in the issue description, there is no change to internal logic because uri / urn from substrait plans are not used.

Are there any user-facing changes?

Yes. Previously substrait plans of spec version v0.74.0 were accepted, and now v0.75.0 is accepted. However, this is a backwards compatible change. The only difference is the inclusion of additional urn-based fields in substrait plans. In a later PR, the old uri-based fields will be dropped, which will be a breaking change.

Jefffrey · 2025-10-02T02:23:44Z

Looks like substrait 0.60 bumps their msrv to 1.88: https://github.com/substrait-io/substrait-rs/releases/tag/v0.60.0

We'll need to wait until Rust 1.91 releases end of October so we can bump our msrv to 1.88 per our policy:

datafusion/docs/source/user-guide/introduction.md

Lines 198 to 210 in 297e537

    
           ## Rust Version Compatibility Policy 
        
           The Rust toolchain releases are tracked at [Rust Versions](https://releases.rs) and follow 
        
           [semantic versioning](https://semver.org/). A Rust toolchain release can be identified 
        
           by a version string like `1.80.0`, or more generally `major.minor.patch`. 
        
           DataFusion supports the last 4 stable Rust minor versions released and any such versions released within the last 4 months. 
        
           For example, given the releases `1.78.0`, `1.79.0`, `1.80.0`, `1.80.1` and `1.81.0` DataFusion will support 1.78.0, which is 3 minor versions prior to the most minor recent `1.81`. 
        
           Note: If a Rust hotfix is released for the current MSRV, the MSRV will be updated to the specific minor version that includes all applicable hotfixes preceding other policies. 
        
           DataFusion enforces MSRV policy using a [MSRV CI Check](https://github.com/search?q=repo%3Aapache%2Fdatafusion+rust-version+language%3ATOML+path%3A%2F%5ECargo.toml%2F&type=code)

comphead · 2025-10-06T17:21:06Z

Folks do you think #17933 failures are related to this PR?

alamb · 2025-10-07T19:46:24Z

Note that bumping substrait-rs resulted in a bump of prost and pbjson-types. These needed to be specified specifically in the datafusion-substrait crate because the currently released version of arrow-rs depends on different versions of prost and pbjson-types. Once the next version of arrow-rs is released and updated in datafusion, the special handling can be removed. The special override of the versions of those packages is noted in a comment in the Cargo.toml file.

I have a work in progress to upgrade here:

Upgrade DataFusion to arrow/parquet 57.0.0 #17888

I confirm the prost version was upgraded

benbellick · 2025-10-27T15:46:10Z

@Jefffrey I have (force-)pushed a new commit to make this PR much simpler. It now only updates substrait-rs to v0.60. This simpler change is due to @alamb's PR above. I am unsure if the msrv issue will have been fixed yet though. Thanks!

alamb

Thank you @benbellick

I took the liberty of reverting the datafusion-testing pin:

alamb · 2025-10-27T20:52:20Z

datafusion/substrait/src/extensions.rs


 impl From<Extensions> for Vec<SimpleExtensionDeclaration> {
+    // Silence deprecation warnings for `extension_uri_reference` during the uri -> urn migration
+    // See: https://github.com/substrait-io/substrait/issues/856


Do you plan a follow on PR to update DataFusion to use a non-deprecated API?

Yes I am leading the charge on the substrait side of things to do this migration from uris -> urns. You can find more info here.

It may take some time because it will require getting the rest of the ecosystem off of uris first.

(However, I am aware that datafusion doesn't actually use the extension uri / urn information)

alamb · 2025-10-27T20:57:53Z

FYI @gabotechs

gabotechs · 2025-10-28T08:13:06Z

Code LGTM, however the aggressive MSVR in substrait-rs seems like a problem. If we merge this, it will force anyone using DataFusion and a Rust version of 1.87 to upgrade to 1.88.

TBH the MSVR bump in substrait-rs from substrait-io/substrait-rs@3a7c0cc (Sep 14) seems like a bit too rushed, as 1.88 was release just a couple of months before that commit.

Options that I see is:

Contribute to https://github.com/substrait-io/substrait-rs a downgrade of the MSVR and benefit from that here.
Wait for the release of 1.91 (30 Oct https://releases.rs/docs/1.91.0/), bump DataFusion's MSVR, and rebase this PR with main

As 1.91 will just be released in a couple of days, I'd advocate for waiting until DataFusion's MSVR is set to 1.88

alamb · 2025-10-28T10:32:43Z

TBH the MSVR bump in substrait-rs from substrait-io/substrait-rs@3a7c0cc (Sep 14) seems like a bit too rushed, as 1.88 was release just a couple of months before that commit.

Maybe we can ask @mbrobbel if we could move the MSRV back in a subsequent version of substrait rs 🤔

mbrobbel · 2025-10-28T10:33:54Z

TBH the MSVR bump in substrait-rs from substrait-io/substrait-rs@3a7c0cc (Sep 14) seems like a bit too rushed, as 1.88 was release just a couple of months before that commit.

Maybe we can ask @mbrobbel if we could move the MSRV back in a subsequent version of substrait rs 🤔

Yes, we can revert to 1.85. It was bumped to get https://blog.rust-lang.org/2025/06/26/Rust-1.88.0/#let-chains.

mbrobbel · 2025-10-28T11:02:21Z

TBH the MSVR bump in substrait-rs from substrait-io/substrait-rs@3a7c0cc (Sep 14) seems like a bit too rushed, as 1.88 was release just a couple of months before that commit.

Maybe we can ask @mbrobbel if we could move the MSRV back in a subsequent version of substrait rs 🤔

Yes, we can revert to 1.85. It was bumped to get https://blog.rust-lang.org/2025/06/26/Rust-1.88.0/#let-chains.

https://github.com/substrait-io/substrait-rs/releases/tag/v0.62.0

benbellick · 2025-10-28T13:03:09Z

@gabotechs I just pushed a change to incorporate @mbrobbel's version bump, so the MSRV shouldn't be an issue anymore.

gabotechs · 2025-10-28T13:13:10Z

Cargo.lock

 name = "aws-lc-rs"
-version = "1.14.0"
+version = "1.14.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "94b8ff6c09cd57b16da53641caa860168b88c172a5ee163b0288d3d6eea12786"
+checksum = "879b6c89592deb404ba4dc0ae6b58ffd1795c78991cbb5b8bc441c48a070440d"
 dependencies = [


🤔 there seems to be several version upgrades for unrelated packages in the Cargo.lock file. The aws-lc-rs and other aws related packages should have nothing to do with the substrait version, so I'd expect them to remain as they were.

I would revert the changes in this Cargo.lock and run cargo build again just so that Cargo updates just whatever is strictly necessary.

This amounts to bumping substrait-rs to 0.60.

gabotechs

🚢 let's go!

alamb · 2025-10-29T18:42:24Z

@gabotechs perhaps you would like to merge this PR (when the CI passes) to make sure your github credentials are connected correctly to permit write access?

gabotechs · 2025-10-30T06:54:14Z

@gabotechs perhaps you would like to merge this PR (when the CI passes) to make sure your github credentials are connected correctly to permit write access?

I'm still pending on that, it seems like it might still take some days. If by the time you wake you still don't see this merged, feel free to hit that merge button.

…18310) ## Which issue does this PR close? ## Rationale for this change The `extended` tests rely on the checkout of datafusion-testing (that has the expected results for the sqlite sqllogictest suite) However, we don't currently run the extended tests when that pin is changed so we could potentially break CI on main if we don't catch changes in code review (this just happened to me in #17866 (review)) ## What changes are included in this PR? 1. Run extended CI tests on changes to datafusion-testing ## Are these changes tested? I tested this in PR - #18311 - #18312 ## Are there any user-facing changes? No

github-actions bot added the substrait Changes to the substrait crate label Oct 1, 2025

benbellick force-pushed the main branch from 59f8383 to 9dc3e11 Compare October 1, 2025 18:20

Jefffrey mentioned this pull request Oct 2, 2025

chore(deps): bump substrait from 0.58.0 to 0.60.1 #17841

Closed

benbellick force-pushed the main branch from 9dc3e11 to 894ea4e Compare October 27, 2025 15:00

alamb approved these changes Oct 27, 2025

View reviewed changes

alamb changed the title ~~chore: bump substrait version to use substrait spec v0.75.0~~ chore: bump substrait version to 0.60.0 to use substrait spec v0.75.0 Oct 27, 2025

alamb mentioned this pull request Oct 27, 2025

Run extended tests when there are changes to datafusion-testing pin #18310

Merged

mbrobbel mentioned this pull request Oct 28, 2025

Revert MSRV to 1.85.1 substrait-io/substrait-rs#414

Closed

gabotechs reviewed Oct 28, 2025

View reviewed changes

benbellick and others added 4 commits October 28, 2025 09:17

chore: bump substrait version to use substrait spec v0.75.0

6b7d12c

This amounts to bumping substrait-rs to 0.60.

revert datafusion-testing pin update

b5b53ca

chore: bump to substrait-rs 0.62.0

2032a1b

chore: revert cargo.lock to origin/main and rerun cargo build

ea3f434

benbellick force-pushed the main branch from a857d70 to ea3f434 Compare October 28, 2025 13:20

benbellick requested a review from gabotechs October 28, 2025 13:21

gabotechs approved these changes Oct 28, 2025

View reviewed changes

Jefffrey added this pull request to the merge queue Oct 31, 2025

Merged via the queue into apache:main with commit 69f0291 Oct 31, 2025
31 checks passed

chore: bump substrait version to 0.60.0 to use substrait spec v0.75.0 #17866

chore: bump substrait version to 0.60.0 to use substrait spec v0.75.0 #17866

Uh oh!

Conversation

benbellick commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Jefffrey commented Oct 2, 2025

Uh oh!

comphead commented Oct 6, 2025

Uh oh!

alamb commented Oct 7, 2025

Uh oh!

benbellick commented Oct 27, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

benbellick Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Oct 27, 2025

Uh oh!

gabotechs commented Oct 28, 2025

Uh oh!

alamb commented Oct 28, 2025

Uh oh!

mbrobbel commented Oct 28, 2025

Uh oh!

mbrobbel commented Oct 28, 2025

Uh oh!

benbellick commented Oct 28, 2025

Uh oh!

gabotechs Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

gabotechs left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Oct 29, 2025

Uh oh!

gabotechs commented Oct 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

chore: bump substrait version to `0.60.0` to use substrait spec v0.75.0 #17866

chore: bump substrait version to `0.60.0` to use substrait spec v0.75.0 #17866

benbellick commented Oct 1, 2025 •

edited

Loading