Do away with strict version requirements for mainline? #653
Labels
Benchmark Design
Discussion or problems with the design of the benchmark itself.
enhancement
New feature or request
question
Further information is requested
Several issues arise from requiring fixed Python and dependency versions in mainline. Most commonly unexpected behavior when using newer versions of Python (often a failure to install, though this may be hidden in the subprocess which runs the experiment which makes it often hard to detect for users).
The original reason for this behavior was reproducibility. However, I don't think experiments should be done directly on main, rather they should be done on a stable release.
Another complication is that the same Python version is used for running the benchmark as installation (albeit in most cases either in Docker or a virtual environment). This is not technically necessary, and for the benchmark this is already somewhat configurable, but with all integrations sharing the same setting for this property, makes it hard to have integrations that rely on different Python versions.
I think we could consider instead to e.g., provided fixed dependencies for the stable releases instead and also make explicit which Python version integration(s) support. Prompting the user to install a missing version, if necessary.
The text was updated successfully, but these errors were encountered: