PYTHONPATH is empty when derivation only contains one instance of python

[iFreilicht]

Describe the bug

So here's a tricky one.

The main issue is that building a dev environment with mkShell will either set the PYTHONPATH variable (if there are multiple pythons), or will leave it empty (if there is only one python).

This can lead to applications built with buildPythonApplication shadowing parts of PATH from the development environment, causing imports in python scripts to fail.

Steps To Reproduce

The underlying bug that should be fixed can be reproduced with a simple flake:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        pythonEnv = pkgs.python3.withPackages (ps:
          with ps; [
            lxml
          ]);
      in
      {
        devShell = pkgs.mkShell {
          packages = with pkgs; [
            pythonEnv
            gnumake
            # Uncomment next line to fix the pythonpath issue
            # grc
          ];
        };
      });
}

1. Clone https://github.com/iFreilicht/nixpkgs-pythonpath-repro
2. Run nix develop -c make pythonpath. It outputs PYTHONPATH=
3. Uncomment the grc line from flake.nix
4. nix develop -c make pythonpath now outputs

PYTHONPATH=/nix/store/vyh48kzgy70ksh593xkik6kzkz0q718b-python3-3.11.9-env/lib/python3.11/site-packages:/nix/store/327bf08j5b7l9cnzink3g4vp32y5352j-python3-3.11.9/lib/python3.11/site-packages

Expected behavior

I would expect the first invocation to output

PYTHONPATH=/nix/store/vyh48kzgy70ksh593xkik6kzkz0q718b-python3-3.11.9-env/lib/python3.11/site-packages

Additional context

While it isn't strictly necessary to have a PYTHONPATH if only a single python derivation is an input to mkShell, it can cause imports to fail when using python in certain environments.

For example, you can try the following:

1. Have grc and direnv installed via home-manager or configuration.nix
2. Clone https://github.com/iFreilicht/nixpkgs-pythonpath-repro
3. (Make sure # grc is commented out again)
4. Run direnv allow
5. Run grc make run. This will fail with an import error
6. Uncomment the grc line from flake.nix again
7. Run grc make run again. Now, the script will succeed.

The reason for this is that grc is also a python application, which of course doesn't know about the custom python3.withPackages in our development environment, and because there is no PYTHONPATH, the different instance of python completely shadows all pythonPackages.

This can be seen when running grc make path:

PATH=/nix/store/327bf08j5b7l9cnzink3g4vp32y5352j-python3-3.11.9/bin:/nix/store/57lvk7cs01j1kr982xwnam46cl7zyy58-grc-1.13/bin:/nix/store/vyh48kzgy70ksh593xkik6kzkz0q718b-python3-3.11.9-env/bin:/nix/store/qkkliqj6qhzlkchlkqc6n2iry22gi78q-gnumake-4.4.1/bin:/nix/store/9g8qxz6r8x7g01hkzy739m5j6bawk4d5-clang-wrapper-16.0.6/bin:[...]

The first path element is the python that grc depends on. So /usr/bin/env in script.py will find that version, which isn't aware of the lxml pythonPackage.

This is resolved when PYTHONPATH is set, as even grcs python can now find the lxml package.

Fixing this might also improve the UX in other ways. For example, if you have jupyter installed gobally, you could then interact with the entire repository as expected without any additional work.

Notify maintainers

Are there any people with particular knowledge of python in nixpkgs? Those would be good candidates, I assume.

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 23.5.0, macOS 14.5`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.22.0pre20240320_7d2ead5`
 - channels(root): `"nixpkgs"`
 - channels(feuh): `"stable-22.11-darwin, unstable"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

---

Add a 👍 reaction to issues you find important.

[adisbladis]

The main issue is that building a dev environment with mkShell will either set the PYTHONPATH variable (if there are multiple pythons), or will leave it empty (if there is only one python).

This is a bit of a misunderstanding of what's happening here.
mkShell isn't aware of whether an input is a Python interpreter or not.
What you're hitting is caused by how dependency propagation & setup hooks work.

This:

       pythonEnv = pkgs.python3.withPackages (ps:
         with ps; [
           lxml
         ]);

returns a "closed" environment. No propagation of Python inputs happen from this derivation on.

However, when you add grc, that is the bare return of buildPythonPackage, a Python build with propagated dependencies, meaning you get the every propagated build inputs setup hook, and therefore a $PYTHONPATH.

This dependency propagation is a long standing source of issues.
I'm trying to fix the root cause in #272178.