Fix Python venv creation #326094

mcdonc · 2024-07-10T10:42:52Z

This is a reapplication of commits by @cwp from #297628, which aimed to fix Python virtualhost creation on top of a newer master. The original PR was closed after it caused regressions. The related Discourse discussion is at https://discourse.nixos.org/t/how-to-build-python-virtualenv-with-packages-provided-by-python3-withpackages/24766

https://hydra.nixos.org/jobset/nixpkgs/python-env-venv#tabs-evaluations

nixos-discourse · 2024-07-10T12:35:26Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-build-python-virtualenv-with-packages-provided-by-python3-withpackages/24766/27

SuperSandro2000 · 2024-07-12T09:32:51Z

pkgs/development/interpreters/python/wrapper.nix

-                makeWrapper "$path/bin/$prg" "$out/bin/$prg" --set NIX_PYTHONPREFIX "$out" --set NIX_PYTHONEXECUTABLE ${pythonExecutable} --set NIX_PYTHONPATH ${pythonPath} ${lib.optionalString (!permitUserSite) ''--set PYTHONNOUSERSITE "true"''} ${lib.concatStringsSep " " makeWrapperArgs}
+                if [ -f ".$prg-wrapped" ]; then
+                  echo "#!${pythonExecutable}" > "$out/bin/$prg"
+                  sed -e '1d' -e '3d' ".$prg-wrapped" >> "$out/bin/$prg"


The main reason this original PR got reverted, was the assumption that you can undo every wrapper by deleting those lines which sometimes deleted legit python code like for pretalx.

And it still deletes part of pretalx-manage:

#!/nix/store/pc8n829fmq2494s5vadw7cppiwfn68a7-python3-3.12.4-env/bin/python3.12 import sys;import site;import functools;sys.argv[0] = '/nix/store/07rcj9vcgvmsq7aqgi39csw7ypsyv990-pretalx-2024.1.0/bin/pretalx-manage';functools.reduce(lambda k, p: site.addsitedir(p, k), ['; import sys if __name__ == "__main__": os.environ.setdefault("DJANGO_SETTINGS_MODULE", "pretalx.settings") from django.core.management import execute_from_command_line execute_from_command_line(sys.argv)

vs https://github.com/pretalx/pretalx/blob/v2024.1.0/src/manage.py

missing the os import.

@RuRo suggests a somewhat hardened sed expression in https://discourse.nixos.org/t/how-to-build-python-virtualenv-with-packages-provided-by-python3-withpackages/24766/20

sed \ -e '/^#!\/nix\/store\//d' \ -e '/^import sys;import site;import functools;sys\.argv\[0\] = /d' \ ".$prg-wrapped" >> "$out/bin/$prg"

wegank · 2024-07-18T21:49:31Z

pkgs/development/interpreters/python/sitecustomize.py

-The module recursively adds paths that are on `NIX_PYTHONPATH` to `sys.path`. In
-order to process possible `.pth` files `site.addsitedir` is used.


Does dropping sitecustomize.py mean that packages whose correct behavior relies on .pth files (like setuptools as discovered in #326321 (comment)) would be broken? Or have I overthought this?

My understanding is that the logic in sitecustomize.py was essentially manually emulating what normally happens with the correct base_prefix set, including the .pth logic.

Assuming this PR works as intended, then the base_prefix would be set to the output path of the -env python derivation and so the ${sys.base_prefix}/lib/python3.X/site-packages directory (that includes all the required package files) would automatically be considered a site directory (and thus all the .pth files inside it would be executed when they are supposed to).

phaer · 2024-07-18T09:47:12Z

pkgs/development/interpreters/python/cpython/default.nix

@@ -566,8 +566,7 @@ in with passthru; stdenv.mkDerivation (finalAttrs: {
    # Strip tests
    rm -R $out/lib/python*/test $out/lib/python*/**/test{,s}
    '' + optionalString includeSiteCustomize ''
-    # Include a sitecustomize.py file
-    cp ${../sitecustomize.py} $out/${sitePackages}/sitecustomize.py
+


There's an empty string left here, I think that's not needed?

phaer · 2024-07-18T09:49:48Z

pkgs/development/interpreters/python/tests.nix

+        is_nixenv = "False";
+        is_virtualenv = "False";
+      };
+    } // lib.optionalAttrs (python.pythonAtLeast "3.8" && (!python.isPyPy)) {


Is it intentional that you repeat line 41 here? Could be one block

Also a comment why it needs to be at least python 3.8 would be nice.
The current comments around pypy executables is already quite confusing when it's in a block that doesn't support pypy anyway.

The reason for python 3.8 is probably because python 2 doesn't support venv. Quoting from virtualenvs website:

Since Python 3.3, a subset of it has been integrated into the standard library under the venv module.

phaer · 2024-07-18T09:50:47Z

pkgs/development/interpreters/python/tests.nix

+        is_nixenv = "True";
+        is_virtualenv = "True";
+      };
+    } // lib.optionalAttrs (!python.isPyPy) {


Is it intentional that you repeat line 41 here? Could be one block

phaer · 2024-07-18T10:01:45Z

pkgs/development/interpreters/python/wrapper.nix

@@ -35,14 +35,22 @@ let
      fi
      mkdir -p "$out/bin"

+      rm -f $out/bin/.*-wrapped


Are we sure that every wrapped executable in all of paths are python scripts? Otherwise this seems risky to me, but i might be missing something?

A short comment explaining why this is needed and safe would be appreciated. Is it just for the check below?

yajo · 2024-08-02T12:49:07Z

You might want to consider this one-line alternative: #261801

RuRo · 2024-08-03T02:01:00Z

You might want to consider this one-line alternative: #261801

FYI, as far as I can tell, that PR doesn't actually fix the problem, but just adds a nix-specific way to work around the problem. With that PR, nested venvs would still be broken, but you could achieve a similar result by manually adding stuff to NIX_PYTHONPATH.

So properly fixing venvs is still probably desirable.

yajo · 2024-08-08T11:54:19Z

Yes, indeed that's what it does. And of course a proper fix is better.

But I think #261801 is still valuable because:

it works
it's 1 line of ~~fix~~ workaround
it's available today (actually, since October 2023!)
doesn't harm
it could even be backported to stable, as a mostly-backwards-compatible fix.

This one PR is still draft after 1 month. I'm afraid it could become stale, or reach a dead end just like #302385 did.

And if it doesn't... perfect! Just remove my workaround once it works. Still no reason to ignore #261801 IMHO 😊

SuperSandro2000 · 2024-08-08T12:04:26Z

doesn't harm

well, I am not sure. Right now we are always overwriting the variable and nesting could surely trigger some edge cases.

phaer · 2024-08-08T12:08:47Z

Sorry, missclicked on mobile

vlaci · 2024-09-27T15:35:54Z

I also tried to take a stab at this.

It looks like it is a hell of a job to develop, as experimenting with the simplest Python derivation results in mass rebuilds. My idea to circumvent this while testing different approaches is to copy the related wrapping code, and wire up a new python derivation to stop cascading builds and also have some output that I can experiment with.

IDK if it is a viable approach.

RuRo · 2024-09-27T16:24:13Z

@vlaci my understanding is that the hardened sed expression I suggested is still a suboptimal solution, and the proper solution should avoid adding those magic lines in the first place. I'd recommend you to try contacting @cwp directly about this, since they previously mentioned that they have an idea about how to best implement this.

But I am totally in agreement with you, that fixing this issue is a pain due to the need to rebuild python itself and everything python-related from source. Also, the fact that the previous PR was reverted due to regressions indicates that python packages might not have sufficient integration tests, so it's borderline impossible to get this right without breaking things.

The way we build python environments is subtly broken. A python environment should be semantically identical to a vanilla Python installation in, say, /usr/local. The current implementation, however, differs in two important ways. The first is that it's impossible to use python packages from the environment in python virtual environments. The second is that the nix-generated environment appears to be a venv, but it's not. This commit changes the way python environments are built: * When generating wrappers for python executables, we inherit argv[0] from the wrapper. This causes python to initialize its configuration in the environment with all the correct paths. * We remove the sitecustomize.py file from the base python package. This file was used tweak the python configuration after it was incorrectly initialized. That's no longer necessary. The end result is that python environments no longer appear to be venvs, and behave more like a vanilla python installation. In addition it's possible to create a venv using an environment and use packages from both the environment and the venv. (cherry picked from commit 234bb31)

When building a python environment's bin directory, we now detect wrapped python scripts from installed packages, and generate unwrapped copies with the environment's python executable as the interpreter. (cherry picked from commit 9611885)

MagicRB · 2024-11-23T22:19:57Z

I've taken a stab at this, it seems like the core issue is that the preamble is executed when it shouldn't be. I therefore propose this slightly altered preamble:

''
  import sys
  import site
  import functools
  _nix_wrapped = '"'$f'"'.split('"'/'"')
  _nix_direct = sys.argv[0] == '"'/'"'.join([*_nix_wrapped[:-1], '"'''"'.join(['"'.'"', _nix_wrapped[-1], '"'-wrapped'"']) ])
  sys.argv[0] = '"'$(readlink -f "$f")'"' if _nix_direct else sys.argv[0]
  functools.reduce(lambda k, p: site.addsitedir(p, k), ['"$([ -n "$program_PYTHONPATH" ] && (echo "'$program_PYTHONPATH'" | sed "s|:|','|g") || true)"'], site._init_pathinfo()) if _nix_direct else None
''

This prevents the execution of the preamble when it shouldn't execute, which means we don't need to edit the script at all.

MagicRB · 2024-11-24T11:14:33Z

Okay different idea, what if wrap-python.nix creates a nix-support/site-packages/<python-version>/... directory with symlinks to what would be hardcoded in the executable. There would still be a preamble but it would just add that directory using addsitedir which would simplify things a lot. Then if you move the executable you unlink it from its dependencies, which once again solves the problem of having to delete the preamble when building a venv. I think this approach is cleaner in general but would add a directory with symlinks to every single store path which is wrapped by wrap-python.nix. It would also need changes in python3.withPackages to accommodate the need to merge that. Or we could rely on nix-support/propagated-build-inputs. What do people think? I'll implement some of these this evening to actually test whether they work.

MagicRB · 2024-11-24T22:09:28Z

I ended up implementing a completely different approach, which works, its not pretty, but hey it works.

github-actions bot added the 6.topic: python label Jul 10, 2024

domenkozar mentioned this pull request Jul 10, 2024

Automate nixpkgs-unstable bumps cachix/devenv-nixpkgs#2

Open

ofborg bot added the 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild label Jul 10, 2024

ofborg bot requested a review from andersk July 10, 2024 12:40

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Jul 10, 2024

SuperSandro2000 reviewed Jul 12, 2024

View reviewed changes

wegank reviewed Jul 18, 2024

View reviewed changes

phaer reviewed Jul 22, 2024

View reviewed changes

yajo mentioned this pull request Aug 2, 2024

python3.buildEnv: support extending NIX_PYTHONPATH #261801

Open

12 tasks

phaer closed this Aug 8, 2024

phaer reopened this Aug 8, 2024

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Sep 27, 2024

cwp added 2 commits October 19, 2024 15:12

mcdonc force-pushed the python-env-venv branch from b9b1a15 to 74fd490 Compare October 19, 2024 19:13

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Oct 19, 2024

ofborg bot requested review from mweinelt and natsukium October 19, 2024 22:49

MagicRB mentioned this pull request Nov 24, 2024

Python fix venv #358823

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Python venv creation #326094

Fix Python venv creation #326094

mcdonc commented Jul 10, 2024 •

edited by mweinelt

Loading

nixos-discourse commented Jul 10, 2024

SuperSandro2000 Jul 12, 2024

mweinelt Jul 12, 2024

vlaci Sep 27, 2024

wegank Jul 18, 2024

RuRo Jul 22, 2024

phaer Jul 18, 2024

phaer Jul 18, 2024

MagicRB Nov 23, 2024

phaer Jul 18, 2024

phaer Jul 18, 2024

yajo commented Aug 2, 2024

RuRo commented Aug 3, 2024

yajo commented Aug 8, 2024

SuperSandro2000 commented Aug 8, 2024

phaer commented Aug 8, 2024 •

edited

Loading

vlaci commented Sep 27, 2024

RuRo commented Sep 27, 2024

MagicRB commented Nov 23, 2024 •

edited

Loading

MagicRB commented Nov 24, 2024

MagicRB commented Nov 24, 2024

		The module recursively adds paths that are on `NIX_PYTHONPATH` to `sys.path`. In
		order to process possible `.pth` files `site.addsitedir` is used.

Fix Python venv creation #326094

Are you sure you want to change the base?

Fix Python venv creation #326094

Conversation

mcdonc commented Jul 10, 2024 • edited by mweinelt Loading

nixos-discourse commented Jul 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yajo commented Aug 2, 2024

RuRo commented Aug 3, 2024

yajo commented Aug 8, 2024

SuperSandro2000 commented Aug 8, 2024

phaer commented Aug 8, 2024 • edited Loading

vlaci commented Sep 27, 2024

RuRo commented Sep 27, 2024

MagicRB commented Nov 23, 2024 • edited Loading

MagicRB commented Nov 24, 2024

MagicRB commented Nov 24, 2024

mcdonc commented Jul 10, 2024 •

edited by mweinelt

Loading

phaer commented Aug 8, 2024 •

edited

Loading

MagicRB commented Nov 23, 2024 •

edited

Loading