Rethink core scientific package dependency support

[folmos-at-orange]

Description

Currently, our policy for supporting core python packages (scikit-learn, pandas, numpy [and we should add scipy]) is very lax.

Specifically, we take from our Python support policy the minimal python version supported, and then take the minimal version of these core packages that is supported for that python version.

Currently this gives (by looking at conda search results):

• Python >= 3.8 (14/10/2020)
• scikit-learn >= 0.23.2 (19/05/2020)
• scipy >= 1.5.2 (24/07/2020)
• numpy >= 1.19.2 (20/06/2020)
• pandas >= 1.1.3 (05/10/2020)

As we see, the support windows for these packages is of the same order as of the python support version. However these packages evolve faster than python, and posterior versions support a given Python version and so the trade-off between compatibility and maintainability is too biased to compatibility.

The Scientific Python Community has proposed SPEC-0000, a standard way to decide how to support core scientific packages. This is however very constrained with regards to Python support.

Questions/Ideas

• A proposal:
  • Keep our current Python support policy
  • Take the support policy of SPEC-0000 for core scientific packages.
    • If for a given Python version a SPEC-0000 is not possible for a package take the closest version to that of SPEC-0000
  • This gives for Python 3.8
    • scikit-learn >= 1.3.0
    • scipy >= 1.10.0
    • numpy >= 1.24.0 (not SPEC-000 compliant)
    • pandas >= 2.0.0
  • And for Python 3.9 (fully SPEC-000 compliant)
    • scikit-learn >= 1.3.0
    • scipy >= 1.10.0
    • numpy >= 1.26.0
    • pandas >= 2.0.0
  • For a given release we should check the SPEC-0000 to check which versions to drop.