Python packages specified via PYTHONPATH (such as ROS 2 Python packages) are considered in pixi environment even if they are related to the apt python installation

[traversaro]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

You can test this on a Ubuntu 22.04 system, just follow the following steps:

# Install ROS2's ros-humble-urdfdom-py package
sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | sudo tee /etc/apt/sources.list.d/ros2.list > /dev/null
sudo apt install ros-humble-urdfdom-py
source /opt/ros/humble/setup.sh
# Show problem on clean pixi project
mkdir testpixiproj
cd testpixiproj
pixi init .
pixi add pip python catkin_pkg
# Install the same package with pip (the urdf_parser_py repo installs the urdfdom-py pip package)
pixi run pip install git+https://github.com/ros/urdf_parser_py.git@0.4.6

The last command fails with:

traversaro@IITBMP014LW012:~/testpixiproj$ pixi run pip install git+https://github.com/ros/urdf_parser_py.git@0.4.6
Collecting git+https://github.com/ros/urdf_parser_py.git@0.4.6
  Cloning https://github.com/ros/urdf_parser_py.git (to revision 0.4.6) to /tmp/pip-req-build-8ylw94ei
  Running command git clone --filter=blob:none --quiet https://github.com/ros/urdf_parser_py.git /tmp/pip-req-build-8ylw94ei
  Resolved https://github.com/ros/urdf_parser_py.git to commit 3bcb9051e3bc6ebb8bff0bf8dd2c2281522b05d9
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: urdfdom_py
  Building wheel for urdfdom_py (setup.py) ... done
  Created wheel for urdfdom_py: filename=urdfdom_py-0.4.6-py3-none-any.whl size=14476 sha256=b529ffc3bb39b5c3b94a9d7289b6b070e12dd1bbac5e7fd886b0b1a50f6b84a3
  Stored in directory: /tmp/pip-ephem-wheel-cache-mmdczfmr/wheels/6a/66/36/3ce8bb8ca5b756427ea225d2a959f2f1028145e8f12d3a4c0b
Successfully built urdfdom_py
Installing collected packages: urdfdom_py
  Attempting uninstall: urdfdom_py
    Found existing installation: urdfdom-py 1.2.1
    Uninstalling urdfdom-py-1.2.1:
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'urdf.py'
Consider using the `--user` option or check the permissions.

Issue description

In a nutshell, if you have a pixi environment that uses python, it could pick some apt python packages if the users sets the PYTHONPATH environment variable in their setup.sh. This is quite a common case for ROS 2 users, that typically just add source /opt/ros/humble/setup.sh in their .bashrc (at least this is my experience. Note that this does not happen with regular Python apt packages, as those are installed in /usr and are found automatically by the apt Python interprer, without the need to set any PYTHONPATH environment variable.

To be clear, this is not a pixi-specific issue, as it the exact same behavior you have with conda/mamba. However, it is still an highly non intuitive behaviour for users, see for example robotology/icub-models-generator#262 . In particular, I could imagine that many ROS 2 users that want to experiment with pixi may encounter this problem, so I preferred to clearly report it here.
For the time being, the workaround I used in robotology/icub-models-generator#263 is to prepend every task with export PYTHONPATH= &&, however this is a bit cumbersome and it is not helpful if users execute commands via pixi shell or non-task commands via pixi run.

Expected behavior

To be honest, I am not sure about this. For sure, this is related to #289 . However, I think that while for PATH it make sense to keep the system executables in the PATH (as users may want to use them in tasks and in pixi shell invocation), for PYTHONPATH I am not sure if it could make sense to unset it by default, and just have an option (by default OFF) to propagate in the pixi environment the PYTHONPATH system environment variable.

[traversaro]

This is probably also related to #288, as the workaround of setting PYTHONPATH could be simpler if it was possible to set env variables for all tasks and shells.

[ruben-arts]

He @traversaro, thanks for the write up. We remember this issue as we also started with conda to run ROS.

The discussion on what is should be default in isolation is indeed a hard one, we still have to figure that out.

To help your users directly setting these variables can be set with the activation.scripts

You could just generate a clean_env.sh and clean_env.bat which exports these variables:

export PYTHONUSERBASE=intentionally-disabled
export PYTHONPATH=

[traversaro]

We remember this issue as we also started with conda to run ROS.

The discussion on what is should be default in isolation is indeed a hard one, we still have to figure that out.

Indeed, I was aware of this for conda for a long time. The main difference I see between pixi and conda is about user expectation (or at least, how I started to introduce conda and pixi to our users). When I started to introduce conda, I always warned users that conda and apt are different things, it is important not to mix them, etc, etc (see https://github.com/robotology/robotology-superbuild/blob/master/doc/conda-forge.md and https://github.com/robotology/robotology-superbuild/blob/master/doc/install-mambaforge.md for example documents I pointed users to). So users are at least aware of possible conflicts.

The pixi story is a bit different. Quoting from https://prefix.dev/blog/uv_in_pixi#our-goals-for-pixi, the goal of pixi presented to the users is to be able to start working on a project with just:

Hence why I think it could make sense to think about what should be isolated by default in the pixi case.

To help your users directly setting these variables can be set with the activation.scripts

You could just generate a clean_env.sh and clean_env.bat which exports these variables:

export PYTHONUSERBASE=intentionally-disabled
export PYTHONPATH=

Cool, that is indeed much better then my solution! I will probably start adding it by default to any pixi package using Python packages. Just to understand, are these pixi activation scripts expected to run before any conda package activation script? Otherwise I am afraid of corrupting the value of PYTHONPATH set by some conda package. I could check directly the code, but I would like to understand if this is an implementation detail, or part of the documented and intended behavior of the functionality, thanks!

[ruben-arts]

Yeah I totally agree with you!

Just to understand, are these pixi activation scripts expected to run before any conda package activation script? Otherwise I am afraid of corrupting the value of PYTHONPATH set by some conda package.

This is what the activation script looks like that we generate. As you can see we put the custom activation in last.

➜ pixi shell-hook
export PATH="/home/rarts/development/pixi/examples/ros2-nav2/.pixi/envs/default/bin:/home/rarts/.local/bin:/home/rarts/bin:/usr/lib64/ccache:/home/rarts/.cargo/bin:/home/rarts/.npm-global/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/var/lib/snapd/snap/bin:/home/rarts/.pixi/bin"
export CONDA_PREFIX="/home/rarts/development/pixi/examples/ros2-nav2/.pixi/envs/default"
export PIXI_ENVIRONMENT_PLATFORMS="osx-arm64,linux-64,osx-64"
export CONDA_DEFAULT_ENV="ros2-nav2"
export PIXI_PROJECT_ROOT="/home/rarts/development/pixi/examples/ros2-nav2"
export PIXI_ENVIRONMENT_NAME="default"
export PIXI_PROJECT_VERSION="0.1.0"
export PIXI_PROMPT="(ros2-nav2) "
export PIXI_PROJECT_MANIFEST="/home/rarts/development/pixi/examples/ros2-nav2/pixi.toml"
export PIXI_PROJECT_NAME="ros2-nav2"
. "/home/rarts/development/pixi/examples/ros2-nav2/.pixi/envs/default/etc/conda/activate.d/gazebo_activate.sh"
. "/home/rarts/development/pixi/examples/ros2-nav2/.pixi/envs/default/etc/conda/activate.d/gdal-activate.sh"
. "/home/rarts/development/pixi/examples/ros2-nav2/.pixi/envs/default/etc/conda/activate.d/geotiff-activate.sh"
. "/home/rarts/development/pixi/examples/ros2-nav2/.pixi/envs/default/etc/conda/activate.d/libglib_activate.sh"
. "/home/rarts/development/pixi/examples/ros2-nav2/.pixi/envs/default/etc/conda/activate.d/libxml2_activate.sh"
. "/home/rarts/development/pixi/examples/ros2-nav2/.pixi/envs/default/etc/conda/activate.d/proj4-activate.sh"
. "/home/rarts/development/pixi/examples/ros2-nav2/.pixi/envs/default/etc/conda/activate.d/ros-humble-ros-workspace_activate.sh"
. "/home/rarts/development/pixi/examples/ros2-nav2/.pixi/envs/default/etc/conda/activate.d/ruby_activate.sh"
. "/home/rarts/development/pixi/examples/ros2-nav2/clean_env.sh"

[traversaro]

Ah, the problem is that for example in your use case /home/rarts/development/pixi/examples/ros2-nav2/.pixi/envs/default/etc/conda/activate.d/ros-humble-ros-workspace_activate.sh also sets the PYTHONPATH, so we would interfere with it if we unset the PYTHONPATH in /home/rarts/development/pixi/examples/ros2-nav2/clean_env.sh. Note that perhaps PYTHONPATH set by ros-humble-ros-workspace_activate.sh is not necessary, but I would need to double check this. Anyhow, this for sure is an improvement over setting the env variable manually in every step, thanks!

[traversaro]

Thanks @ruben-arts, we used your workaround in robotology/icub-models-generator#263 . Not sure if you want to keep this issue open or not, feel free to close if you think the problems discussed in the issue is tracked already somewhere else.

[ruben-arts]

Closing this in favor of #289