-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Flakes UX: make it easy to evaluate an arbitrary expression in a context of a flake #5567
Comments
This comment has been minimized.
This comment has been minimized.
I’m a big fan of the overall idea, that’s something I want to have available reasonably frequently, and it’s always a pain. I’m not sure what the UX should be though:
A fourth way could be to have a |
My
I think the installables syntax is already different from usual Nix syntax. E.g. |
As far as I understand it, this evaluates to the same thing in the existing and proposed cases. It's like the difference between |
Not exactly. Current |
For the new UI, we have to careful about adding features that are not discoverable or have a steep learning curve. For instance, in the example
it's not obvious how a user would discover the existence of There's also the issue of cacheability. Currently we can cache flake output attributes like |
If we can translate CLI arguments to function calls, it's possible to build functions that work for common cases. This assumes some cooperation between nixpkgs and the nix CLI. Eg: Could map to It makes me think of httpie since they also try to convert CLI to JSON data structures: https://httpie.io/docs#non-string-json-fields |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/tweag-nix-dev-update-22/16251/1 |
I recently wanted to do the exact same thing as @balsoft, and I did the exact same workaround. I didn't know whether it was supported natively or not so here's what I tried (What I intuitively thought would work): |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/nix-shell-for-python-packages/16575/4 |
I marked this as stale due to inactivity. → More info |
Bad bot. |
I wanted this to work too, so I made a patch for it: https://github.com/Minion3665/nix/compare/master...Minion3665:nix:5567-make-installables-expr-context.patch Right now it only works on nixos-unstable (I believe it needs nix 2.9? I've only properly tested it on 2.8 (failing) and 2.10 (working fine) though) You can apply it by doing something like this in an overlay:
where The patch works by adding |
@Minion3665 please upstream this patch, it's awesome 🤩 |
I wanted this as well. I created this shell function until it's resolved: # Python with packages
pwp () {
local python=$1;
shift;
local pkgs=$@;
nix build \
--impure \
--expr "with builtins; with getFlake \"nixpkgs\"; (getAttr currentSystem legacyPackages).$python.withPackages (ps: with ps; [ $pkgs ])"
} Example:
It's even better than the |
If Nix was to allow specifying functions in place of "installables" and passing positional arguments to them, we could use it like: |
|
I came here by searching "nix shell flake expr" and my use-case was My reasoning was that since the following works with the repl:
Then why doesn't this work for nix shell?:
That doesn't make sense to me. The patch of @Minion3665 seem to align with this reasoning. I couldn't find a PR of your branch, is there a PR already? If not, do you mind to create one? Or if it helps, is it ok for me to create one from your patch? |
There is no pr yet, I've considered making one however:
For these reasons I haven't PRed, however I'm more than willing to have someone else make a PR or to work with someone on improving the patch and PR myself. I've planned to do these and PR for a while but I'm sure you know how life can be... |
Thanks to posters above for giving me a clue how to access e.g. $ nix eval --impure \
--apply 'pkgs: pkgs.callPackage pkgs/development/libraries/science/math/libtorch/test { cudaSupport = false; }' \
.#legacyPackages.aarch64-darwin Something like this for |
Adding on to examples like @viperML from above, I hadn't realized that For example, working on a local checkout of nixpkgs: $ nix shell "$(
nix eval --raw --apply '
py: (py.withPackages (pp: [ pp.torch ]))
' .#python310
)" \
-c python -c 'import torch; print(torch.__version__)'
2.0.1 Interestingly, I was initially getting segfaults and stack overflows trying to get this to work (maybe this issue?): $ nix eval --apply 'py: (py.withPackages (pp: [ pp.flask ]))' nixpkgs#python310
Segmentation fault: 11
$
$ # on another machine:
$ nix eval --apply 'py: (py.withPackages (pp: [ pp.requests ]))' nixpkgs#python310
...
trace: warning: Use `stdenv.tests` instead. `passthru` is a `mkDerivation` detail.
trace: warning: Use `stdenv.tests` instead. `passthru` is a `mkDerivation` detail.
trace: warning: Use `stdenv.tests` instead. `passthru` is a `mkDerivation` detail.
error: stack overflow (possible infinite recursion) But simply adding
So does
Oddly, when I add a second package it fails: $ nix build "$(nix eval --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ]))' nixpkgs#python3)"
error: path '/nix/store/h2i9izgfc5fh5c4b4nrg1pjn5y12axzw-python3-3.10.12-env' is required, but there is no substituter that can build it
$ nix-shell -I nixpkgs=~/git/nixpkgs -p 'python310.withPackages (ps: with ps; [ flask requests ])'
[nix-shell:/tmp/test]$ echo $?
0 |
@n8henrie, here's a fix for your last cmd: nix shell "$(nix eval nixpkgs#python3 --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ])).drvPath')" In general (and as you mentioned) you can run nix on arbitrary drv expressions that have single flake output as their input like so: nix <cmd> "$(nix eval nixpkgs#pkgs --raw --apply 'pkgs:
(<arbitrary expression that yields a derivation>).drvPath
')" |
Thanks! This also seems to work pretty well and doesn't require a separate call to "$(nix eval --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ])).out' .#python310)"/bin/python -c 'import flask; print(flask.__version__)' |
A bash function similar to what @Uthar uses above: $ nixShellWithPythonPackages() {
nix shell "$(nix eval --raw --apply "py: (py.withPackages (pp: with pp; [ $* ])).drvPath" nixpkgs#python310)"
}
$ nixShellWithPythonPackages torch requests
$ type -p python
/nix/store/p306n49i437di59p6lwn33qcksqrjk3z-python3-3.10.12-env/bin/python
$ python -c 'import torch; import requests; print(torch.__version__, requests.__version__)'
2.0.1 2.29.0 |
This only works because the drv has already been built previously. You want this: $(nix build --no-link --print-out-paths "$(nix eval nixpkgs#python3 --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ])).drvPath')")/bin/python -c 'import flask; print(flask.__version__)' |
you probably want to use the |
I second any sort of higher order implementation. Infact I think we should match flake.nix output syntax: $ nix build nixpkgs# github:somerepo#pkg --with '{nixpkgs, pkg}: nixpkgs.pkg2.override {inherit pkg;}' |
I'm interested in implementing this. It looks like a high-demand feature that could bridge the gap between the simple
|
Thanks for the writeup @iFreilicht , LGTM . Personally I think
I would be concerned about the
|
The manifest format already supports store paths with barely any additional info. For example, I compiled nix locally and installed it with
So while an element like this can't be upgraded, it can absolutely be installed without any changes to the manifest format. And as all the installables produced by either I don't want to advocate for "extensive support" for this kind of thing though, I'd much rather people write a flake with everything they want to have in their profile and install that. |
Thanks for the write-up and the offer to implement @iFreilicht ! I would weight in favour of the simplest |
According to the docs of This isn't quite as general as I think it should be. What if each installable creates a positional argument? That way you can really use anything as an input.
Not sure whether getting the default package is a good idea, when a fragment is missing. I guess |
Hmm, looking at the The same has been done for explicitly stating outputs: If the suffix of
Same for python:
It should then also work for packages that do not use
Because it is part of the URL, it'll work for All of these will work using Edit: thinking about this some more, it even makes sense to drop the 'multiple output'-syntax for URLs (
|
@roberth Yes, I discussed this solution under the "Split options" heading.
I would like this to not have as little special behavior as possible. I'll have to look at a few examples to see potential drawbacks.
@bobvanderlinden Very interesting idea as well! I guess in terms of parsing the flake name this isn't an issue, but about the output I'm not sure. It should be doable.
That's true and makes this a very elegant solution, though it does make parsing a flake URL much more difficult and in some cases potentially impossible. I'm thinking especially about the work in #8678. Also, as I understand, this would not solve the multiple-inputs usecase. So
Hmm, I'm not so sure about that. Having the output syntax as syntactic sugar is nice. This shields new and consumer-only users from Nix syntax on the command line, something that the old CLI required for some basic use-cases IIRC. But these should be equivalent, I agree with that. |
Triaged in Nix meeting:
|
Yeah that decision is fair. Especially as all proposals we had are non-breaking additions, stabilizing the current state is more important. |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/2023-10-27-nix-team-meeting-minutes-98/34695/1 |
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/decision-making-2-0-1-0-1/12851/24 |
I was under the impression that would be easily parsable. Both NodeJS and Python are able to parse them. However, according to RFC2396 it is indeed not a valid URI. Nix seems to follow that standard more strictly. Allowing more than RFC2396 defines URIs makes it a harder decision.
👍 |
I should've been clearer. What I meant was that it makes it impossible to find a human-readable name for flake from the URL by just parsing. You have to actually evaluate the Nix code inside of it to get all the output paths and can then potentially create a compound name. |
This is a design discussion issue, so feel free to give as much feedback and criticism as you like.
Preamble
In ye olde days of
nix-shell
, it was possible to easily evaluate expressions in the context ofnixpkgs
. Something like this:Drops you into a shell in which there's a
python3
executable with numpy and scipy available. This is really useful for ad-hoc scripting. You could also do something likeIt was also possible to add
alias p='nix-shell -p'
to save some typing.The flexibility is endless -- and all just a few keystrokes away!
Current situation
It is possible to do something similar with new, flaky interface:
However, this is a chore to do, and I'd much rather just run
pip install numpy scipy
and deal with all the usual stateful package management issues than type this monstrosity every time I need to try a python package.Proposal
Add a way to easily evaluate expressions "in the context of" a flake, meaning with the top-level attributes and
{packages,legacyPackages}.${currentPlatform}
of that flake available in scope.Possible implementations
I came up with three different ideas, all of which have some advantages and drawbacks.
(my current favourite) Allow flake fragments to be arbitrary expressions
Example:
nix shell 'nixpkgs#(python3.withPackages (ps: with ps; [ numpy scipy ]))'
This feels like a natural extension of the already-existing flake-fragment-as-attrpath interpretation. It shouldn't be too difficult to implement, and doesn't break compatibility. It does however introduce some potential complexity to newcomers.
With
alias p='nix shell'
it is possible to dop 'nixpkgs#python3.withPackages (...)'
, and some more keystrokes could be saved with a registry entryn -> nixpkgs
:p 'n#python3.withPackages (...)'
Add a new flag, like
--with
Example:
nix shell --with nixpkgs 'python3.withPackages (ps: with ps; [ numpy scipy ])'
This plays nicely with Nix'
with
keyword. It implies that some expression will be evaluated with some additional context.Aliasing
p='nix shell --with nixpkgs'
can give experience identical to the status quo --p 'python3.withPackages (...)'
Change the behaviour of
--expr
Example:
nix shell --expr nixpkgs 'python3.withPackages (ps: with ps; [ numpy scipy ])'
This breaks compatibility, but may be nicer than
--with
since--expr
was not very usable with flakes anyways and it's not nice to have a lot of useless flags around.The text was updated successfully, but these errors were encountered: