diff --git a/dev-docs/source/CondaPackageManager.rst b/dev-docs/source/CondaPackageManager.rst new file mode 100644 index 000000000000..2ea984e4bbe5 --- /dev/null +++ b/dev-docs/source/CondaPackageManager.rst @@ -0,0 +1,108 @@ +.. _CondaPackageManager: + +===================== +Conda Package Manager +===================== + +.. contents:: + :local: + +Mantid uses `Conda `_ as its package management system. This document gives a +developer overview on how we use the Conda package manager, including tips on how to debug dependency issues, and +our policy towards using ``pip`` packages (it is strongly discouraged). + +Creating Environments +--------------------- + +Creating an empty environment called ``myenv``: + +.. code-block:: sh + + conda create --name myenv + +Creating a Python environment: + +.. code-block:: sh + + conda create --name myenv python=3.10 + +Creating an environment from file: + +.. code-block:: sh + + conda create --name myenv -f mantid-developer-.yml + +Creating an environment from package: + +.. code-block:: sh + + conda create --name myenv -c mantid mantidworkbench + +Creating an in-place environment: + +.. code-block:: sh + + conda create --prefix $PWD/myenv python=3.10 + +Finding a broken dependency +--------------------------- + +The nightly pipelines can sometimes fail for obscure reasons, seemingly unrelated to the changes made within +Mantid. In this case, it is probable that a Conda dependency has updated, and the new update is "Broken" +(if it is a minor or patch update) or no longer compatible with Mantid (if it is a major update). + +To find the dependency which has changed, you can run the ``tools/Jenkins/dependency_spotter.py`` script. This +script takes two Jenkins build numbers, and optionally the OS label, the pipeline name, and the name of the file to +compare. It will then output the changes in Conda package versions used in the two builds, if there are any. A +few examples on how to use it: + +.. code-block:: sh + + python dependency_spotter.py -f 593 -s 598 + python dependency_spotter.py -f 593 -s 598 -os win-64 + python dependency_spotter.py -f 593 -s 598 --pipeline main_nightly_deployment_prototype + +Another useful command for investigating the dependencies of specific packages is `conda search `_. To find the dependencies of a package: + +.. code-block:: sh + + conda search -i + +Fixing a broken dependency +-------------------------- + +After identifying the Conda dependency and version which is causing the unwanted behaviour, there are several +options we can take to fix the issue. The following options are in order of preference: + +1. Raise an issue in the dependencies `feedstock `_ + repository with a minimum reproducible example. If appropriate, request that they mark the package version as + "Broken". See `Removing broken packages `_ to understand this procedure. + +2. If we need a fix urgently, you can consider pinning the package in question. This is not an ideal solution, + and so you should also open an issue to un-pin the package in future. When pinning a package, consider + using the not-equals-to operator ``!=x.y.z`` because this allows the package to upgrade automatically when + a new version arrives (which is hopefully a working version). + +Pip package policy +------------------ + +We have a strict policy with regards to using PyPi packages within Mantid. This policy can be summarised as +follows: + +.. code-block:: none + + We strongly encourage PyPi dependencies be built into Conda packages and uploaded to conda-forge. PyPi packages + will not be automatically installed into our Mantid Conda environments, and should instead be installed by + users of the software, if required. + +We do not want to include pip packages as dependencies in our Conda recipes because there is no guarantee of +compatibility between the two package managers. In the past, attempting to resolve compatibile package versions +when two package managers are involved have caused broken Mantid installations. Furthermore, the original +motivation for moving towards Conda was so that we had a unified package manager rather than using several +different systems or mechanisms. Including pip packages in our dependencies would be a backwards step. + +The other solution we considered was installing our pip dependencies downstream within our DAaaS workspace +configuration repository. We decided against this because it feels like bad practise to have a formalised +way of installing dependencies of a software in a way which is completely detached. The prevailing message is +this: please only use Conda packages. We provide :ref:`pip install instructions ` for users if +they would like to take the risk. diff --git a/dev-docs/source/index.rst b/dev-docs/source/index.rst index b364a0c091c0..3ffda4fb0481 100644 --- a/dev-docs/source/index.rst +++ b/dev-docs/source/index.rst @@ -127,6 +127,7 @@ Tools Eclipse WindowsSubsystemForLinux ObtainingABenchmarkForMantidFitting + CondaPackageManager :doc:`ToolsOverview` Describes ``class_maker``, ``valgrind`` and related tools. @@ -158,6 +159,9 @@ Tools :doc:`ObtainingABenchmarkForMantidFitting` Guide for setting up an environment to perform a benchmark of Mantid fitting minimizers. +:doc:`CondaPackageManager` + Guide on how to use the Conda package manager in Mantid, including tips and a ``pip`` policy. + ======= Testing =======