Expose pinned_package_versions for task collection

[tcompa]

Quoting from fractal-analytics-platform/fractal-tasks-core#355 (comment)

We explored multiple options with @mfranzon, and we don't see any which makes sense to us via conditional dependencies or something similar. We then propose that:

• fractal-tasks-core depends on a more flexible torch version (e.g. <=2.0.0)
• The sysadmin keeps installing the "correct" version (of torch, for instance) after the task collection is complete.

---

Since this is very tedious, we also propose the following workaround for doing it automatically (to be included in fractal-server - we can then open issue over there).
The /api/v1/task/collect/pip/ currently takes this request body:

{
  "package": "string",
  "package_version": "string",
  "package_extras": "string",
  "python_version": "string"
}

We could add an additional attribute, like custom_package_versions. This would be empty by default, and only at UZH we would set it to custom_package_versions={"torch": "1.12.1"}. The behavior of the task collection would then be:

1. Perform the whole installation in the standard way (NOTE: this must not fail!)
2. After the installation is complete, run pip install torch==1.12.1 (where pip is replaced by the actual venv pip that is being used).

CAVEAT: this is messing with the package, and thus creating a not-so-clean log of the installation (although we would still include also the additional torch-installation logs). Such an operation is meant to be restricted to very specific cases, where there is an important dependency on hardware or system libraries - things that a regular user should not be using.

IMPORTANT NOTE 1
This workaround cannot bring us out of the versions supported by fractal-tasks-core (for instance). Say that we now require torch>=1.13.0, and then we set custom_package_versions={"torch": "1.12.1"}. This task-collection operation will fail, because the installation of the custom package goes conflicts with fractal-tasks-core.

IMPORTANT NOTE 2
We should never use this feature to install an additional package. For instance if the fractal-tasks-core does not depend on polars, and we specify custom_package_versions={"polars": "1.0"}, then task collection will fail.

MINOR NOTE:
This also fits perfectly with #686, where we would only need to add the same pip install line in the script.

[tcompa]

To do:

• Add pinned_package_versions: Optional[dict[str, str]] = None attribute to TaskCollectPip in fractal-common.
  Side note about fractal-web: having a dict will make it a bit more complex to create the UI for setting it, but we can start with a string in the fractal-web prompt and use JSON.parse while constructing the request body.
• In fractal_server.tasks.collection._pip_install function, add a new block after await execute_command(cwd=venv_path, command=cmd_install, logger_name=logger_name), which is only executed if pinned_package_versions is not None and which iterates over for (package_name, package_version) in pinned_package_versions.items().
  • The first operation (for each item) should be calling f"{pip} show {package_name}" via execute_command. If this command fails, we should raise an appropriate error. Note that pip is actually pip = venv_path / "venv/bin/pip" (as already defined in the same function). The reason why this error cannot be ignored is that it means you are trying to add a dependency which was not originally included in the main package.
  • The second operation (for each item) should be calling f"{pip} install {package_name}=={package_version}".
  • Appropriate logs should be written for each operation.
• Tests, of course ;)

[tcompa]

The simplest test could be a copy of test_pip_install from test_task_background_collection.

• Install devtools 0.8.0, with the extra pygments. Check that the original version is not 2.0.
• Pin pygments version to 2.0.
• Verify with pip show pygments that installed version is 2.0

[tcompa]

I just tested this in a real example, and it worked as expected:

[...]
2023-06-28 12:42:53,887 - fractal-tasks-core - DEBUG - 
2023-06-28 12:42:53,887 - fractal-tasks-core - DEBUG - Specific version required: torch==1.12
2023-06-28 12:42:53,887 - fractal-tasks-core - DEBUG - Preliminary check: verify that 1.12 is already installed
2023-06-28 12:42:56,172 - fractal-tasks-core - DEBUG - Subprocess call to: /data/homes/fractal/fractal-demos/examples/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.10.0/venv/bin/pip show torch
2023-06-28 12:42:56,173 - fractal-tasks-core - DEBUG - Name: torch
Version: 2.0.0
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: /net/nfs4/pelkmanslab-fileserver-fractal/data/homes/fractal/fractal-demos/examples/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.10.0/venv/lib/python3.9/site-packages
Requires: filelock, jinja2, networkx, nvidia-cublas-cu11, nvidia-cuda-cupti-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-runtime-cu11, nvidia-cudnn-cu11, nvidia-cufft-cu11, nvidia-curand-cu11, nvidia-cusolver-cu11, nvidia-cusparse-cu11, nvidia-nccl-cu11, nvidia-nvtx-cu11, sympy, triton, typing-extensions
Required-by: cellpose, triton

2023-06-28 12:42:56,173 - fractal-tasks-core - DEBUG - 
2023-06-28 12:42:56,173 - fractal-tasks-core - DEBUG - Currently installed version of torch(2.0.0) differs from pinned version (1.12); install version 1.12.
2023-06-28 12:43:35,311 - fractal-tasks-core - DEBUG - Subprocess call to: /data/homes/fractal/fractal-demos/examples/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.10.0/venv/bin/pip install torch==1.12
2023-06-28 12:43:35,311 - fractal-tasks-core - DEBUG - Collecting torch==1.12
  Using cached torch-1.12.0-cp39-cp39-manylinux1_x86_64.whl (776.3 MB)
Requirement already satisfied: typing-extensions in ./venv/lib/python3.9/site-packages (from torch==1.12) (4.6.3)
Installing collected packages: torch
  Attempting uninstall: torch
    Found existing installation: torch 2.0.0
    Uninstalling torch-2.0.0:
      Successfully uninstalled torch-2.0.0
Successfully installed torch-1.12.0

2023-06-28 12:43:35,311 - fractal-tasks-core - DEBUG - 
2023-06-28 12:43:37,129 - fractal-tasks-core - DEBUG - Subprocess call to: /data/homes/fractal/fractal-demos/examples/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.10.0/venv/bin/pip show fractal-tasks-core
2023-06-28 12:43:37,130 - fractal-tasks-core - DEBUG - Name: fractal-tasks-core
Version: 0.10.0
Summary: 
Home-page: https://github.com/fractal-analytics-platform/fractal-tasks-core
Author: Joel Lüthi 
Author-email: joel.luethi@fmi.ch
License: BSD-3-Clause
Location: /net/nfs4/pelkmanslab-fileserver-fractal/data/homes/fractal/fractal-demos/examples/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.10.0/venv/lib/python3.9/site-packages
Requires: anndata, dask, defusedxml, docstring-parser, lxml, numpy, pandas, pydantic, zarr
Required-by: 

2023-06-28 12:43:37,130 - fractal-tasks-core - DEBUG - 
2023-06-28 12:43:37,130 - fractal-tasks-core - DEBUG - Venv creation and package installation ended correctly.
2023-06-28 12:43:37,130 - fractal-tasks-core - DEBUG - Creating task list from manifest
2023-06-28 12:43:37,133 - fractal-tasks-core - DEBUG - Task list created correctly
2023-06-28 12:43:37,133 - fractal-tasks-core - DEBUG - Task-collection status: collecting
2023-06-28 12:43:37,158 - fractal-tasks-core - DEBUG - Task-collection status: finalising
2023-06-28 12:43:37,169 - fractal-tasks-core - DEBUG - Task-collection status: OK
2023-06-28 12:43:37,169 - fractal-tasks-core - INFO - Background task collection completed successfully