Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do away with strict version requirements for mainline? #653

Open
PGijsbers opened this issue Nov 25, 2024 · 0 comments
Open

Do away with strict version requirements for mainline? #653

PGijsbers opened this issue Nov 25, 2024 · 0 comments
Labels
Benchmark Design Discussion or problems with the design of the benchmark itself. enhancement New feature or request question Further information is requested

Comments

@PGijsbers
Copy link
Collaborator

Several issues arise from requiring fixed Python and dependency versions in mainline. Most commonly unexpected behavior when using newer versions of Python (often a failure to install, though this may be hidden in the subprocess which runs the experiment which makes it often hard to detect for users).

The original reason for this behavior was reproducibility. However, I don't think experiments should be done directly on main, rather they should be done on a stable release.
Another complication is that the same Python version is used for running the benchmark as installation (albeit in most cases either in Docker or a virtual environment). This is not technically necessary, and for the benchmark this is already somewhat configurable, but with all integrations sharing the same setting for this property, makes it hard to have integrations that rely on different Python versions.

I think we could consider instead to e.g., provided fixed dependencies for the stable releases instead and also make explicit which Python version integration(s) support. Prompting the user to install a missing version, if necessary.

@PGijsbers PGijsbers added enhancement New feature or request question Further information is requested Benchmark Design Discussion or problems with the design of the benchmark itself. labels Nov 25, 2024
@PGijsbers PGijsbers changed the title Do away with strict version requirements for mainline? [open discussion] Do away with strict version requirements for mainline? Dec 16, 2024
@PGijsbers PGijsbers changed the title [open discussion] Do away with strict version requirements for mainline? Do away with strict version requirements for mainline? Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Benchmark Design Discussion or problems with the design of the benchmark itself. enhancement New feature or request question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant