Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate dependency updates #37

Closed
alamb opened this issue Apr 23, 2021 · 8 comments
Closed

Automate dependency updates #37

alamb opened this issue Apr 23, 2021 · 8 comments
Labels
datafusion Changes in the datafusion crate enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@alamb
Copy link
Contributor

alamb commented Apr 23, 2021

As suggested by @returnString on #33 (comment)

rationale

Keep dependencies up to date, with as little manual intervention as possile

Proposed Changes

Add some sort of automation / bot that automatically updates dependencies. @returnString suggests that dependable is a good potential choice

https://docs.github.com/en/code-security/supply-chain-security/configuring-dependabot-security-updates

@alamb alamb added the datafusion Changes in the datafusion crate label Apr 23, 2021
@returnString
Copy link
Contributor

A few q's from my side:

Do we have a strict policy around dependency version bumps and their potential impact on dependent crates? For example, say Tokio releases a theoretical 2.0 which doesn't play nicely with the 1.x runtime (similar to the 0.2 => 1.0 transition), would we be ok upgrading to v2 and requiring that DataFusion users do the same? Would that require a major version bump for DataFusion?

Also, as a library crate, DataFusion doesn't include a lockfile - do we want to just raise the minimum supported version of dependencies where appropriate?

@returnString
Copy link
Contributor

returnString commented Apr 23, 2021

For science, I've just enabled this on my fork (enabling commit here) and it's only picked up one dep that needs updating, rustyline - here's the example PR, you can see how the bot interaction works too where I asked for a rebase: reservoirdb#1

edit: GHA builds are broken with macos provisioning failures, so ignore the CI failures

@alamb
Copy link
Contributor Author

alamb commented Apr 26, 2021

Do we have a strict policy around dependency version bumps and their potential impact on dependent crates?

I do not think we have any strict (or written) policy about this

For example, say Tokio releases a theoretical 2.0 which doesn't play nicely with the 1.x runtime (similar to the 0.2 => 1.0 transition), would we be ok upgrading to v2 and requiring that DataFusion users do the same?

This is what we did when tokio went to 1.0

Would that require a major version bump for DataFusion?

It seems like a good idea :)

Prior to 1 week ago the datafusion major release numbers were tied to the arrow major release numbers, so we didn't have the flexibility to implement such a policy.

Defining one seems like a good idea

@alamb
Copy link
Contributor Author

alamb commented Apr 26, 2021

@returnString reservoirdb#1 looks pretty cool to me

@houqp houqp added enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed labels Sep 15, 2021
@martinitus
Copy link

martinitus commented Apr 22, 2022

Just stumbled over this when I saw, that there is arrow-rs 12.x.y but datafusion still depends on arrow-rs 9.1.

I think general best practice (given you use semver) is that if you have a major increase in a dependency that appears in your public API, then you also need a major increase in your library version, as the stuff that appear in your public API may break backwards compatibility for the users of your API.

@xudong963
Copy link
Member

Just stumbled over this when I saw, that there is arrow-rs 12.x.y but datafusion still depends on arrow-rs 9.1.

I think datafusion always uses the newest arrow-rs, which is done by @alamb.

@xudong963
Copy link
Member

Btw, the issue had solved!

@alamb
Copy link
Contributor Author

alamb commented Apr 24, 2022

Just stumbled over this when I saw, that there is arrow-rs 12.x.y but datafusion still depends on arrow-rs 9.1.

this is true for the version of datafusion we released to crates.io https://crates.io/crates/datafusion/7.1.0/dependencies

I filed #2327 to track the idea of doing more frequent releases. thank you for raising it @martinitus

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datafusion Changes in the datafusion crate enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

5 participants