Make pixi create envs somewhere else than $PWD/.pixi

[pavelzw]

Problem description

It would be nice if pixi could have the option of creating envs somewhere else than $PWD.
This can be useful when your project is on a network drive and you don't want to have your conda envs there because it's too slow.

---

Possible solution

Pixi could have a global config option custom_env_location = "/path/to/envs" and pixi install would create a symlink .pixi/envs/default -> /path/to/envs/<sha-of-$PWD/envs/default>. (This is already possible if you create the symlink and the target directory beforehand; but quite tedious if you need to remember to do this all the time)
This way it would still look like the env is in $PWD but it actually is on a different drive that's faster (and in some cases you can then even use hardlinks for your packages instead of copies).
Also, since the directory is an arbitrary hash that people can't remember, people would not get wrong ideas to just use this directory as a named conda environment; they would still use the one in $PWD/.pixi/envs/default.
The env prefix still being at .pixi/envs/default should not be a problem in this scenario, right?

[ruben-arts]

That is an very interesting idea! We should discuss this. I love the idea for the faster disks.

[chawyehsu]

This may also help to resolve #777.

in some cases you can then even use hardlinks for your packages instead of copies

Would be so much helpful if this could be implemented.

[pavelzw]

You get the ncurses issue (#234) when trying to run pixi in a VSCode devcontainer on macOS. When this is solved, we could also properly using devcontainers. My current workaround:

.devcontainer/devcontainer.json

    // https://github.com/prefix-dev/pixi/issues/997
    "postCreateCommand": "mkdir -p /tmp/pixienv && ln -s /tmp/pixienv .pixi && pixi run postinstall && git config --global --add safe.directory ${containerWorkspaceFolder} && pixi run pre-commit-install",

This is not optimal because then the macOS installation wouldn't work (unless /tmp/pixienv exists locally too, otherwise i get os error 17)

EDIT:
This seems to be a better solution

    "mounts": ["source=${localWorkspaceFolderBasename}-pixi,target=${containerWorkspaceFolder}/.pixi,type=volume"],
    "postCreateCommand": "sudo chown vscode .pixi && pixi run postinstall && git config --global --add safe.directory ${containerWorkspaceFolder} && pixi run pre-commit-install",

[jaimergp]

I remember reading some concerns about activating symlinked environments. Something about some old compiled libraries resolving their own location in the wrong way or something like that. Pixi would have to resolve the symlink before activating/running/opening a shell, and that should be enough, I think?

Edit, here they are:

• Feature request: always_softlink for conda create --clone conda/conda#7867
• option to use soft links instead of hard links conda/conda#3308

[kxmh42]

This would also help with two other issues I have:

1. I want to use Pixi envs in VS Code, the same way I use Conda envs. To make a Pixi env show up among Python environments that I can use as notebook kernels, I need to manually symlink it to the directory with my Conda environments. It would be nice if the Pixi env was automatically put there instead and symlinked from my project directory.
   Just one note: It would be nice to have readable env names, so please don't use sha-of-$PWD suggested above. My suggestion would be to symlink .pixi/envs/default -> /path/to/envs/<pixi-project-name> and symlink non-default envs, e.g. .pixi/envs/dev -> /path/to/envs/dev@<pixi-project-name>. Of course the problem is that there might already be an existing environment with the name equal to current Pixi project name, but that could simply result in an error indicating that either the exisitng env or the Pixi project name should be renamed, or perhaps it could automatically add a short suffix to the env name.

2. On the HPC platform I use, there is a scratch partition with no backup and much larger quota, and we are asked to put our environments there. I've tried moving the rattler cache to the scratch partition, but the files from there are then copied to my pixi envs, taking up the quota on the default partition and getting unnecessarily backed up. If Pixi envs were also on the scratch partition, the files would be hardlinked from the rattler cache, saving disk space.

[moritzwilksch]

to make a Pixi env show up among Python environments that I can use as notebook kernels, I need to manually symlink it to the directory with my Conda environments.

this should not be necessary. You can chose .pixi/envs/default/bin/python as your interpreter with the "select interpreter" menu. Then - if you have ipykernel installed - the VSC jupyter extension can pick that up and work without symlinking.

I agree that this would be a great feature for the other reasons mentioned above, though.

[ruben-arts]

The plan we currently have:

• Support the global config entry `target_pixi_environment_directory = "/home/user/path/to/dir"
• Then installing an environment will directly install it in to /home/user/path/to/dir/package_name-{pwd hash}/env/default
• Symlink the dir to the .pixi in the project. But only when possible, e.g. this doesn't work on windows.
• Add it to the pixi info output.
• Add clear documentation when to use this feature and when this is bad practice.

[pavelzw]

Sounds great!

[0xbe7a]

Then installing an environment will directly install it in to /home/user/path/to/dir/package_name-{pwd hash}/env/default

/home/user/path/to/dir/package_name is this supposed to be project.name from pixi.toml @ruben-arts ?

[ruben-arts]

@0xbe7a Yep, that is the idea. I would love to support some kind of full path things but that would make the path to long.

[kxmh42]

to make a Pixi env show up among Python environments that I can use as notebook kernels, I need to manually symlink it to the directory with my Conda environments.

this should not be necessary. You can chose .pixi/envs/default/bin/python as your interpreter with the "select interpreter" menu. Then - if you have ipykernel installed - the VSC jupyter extension can pick that up and work without symlinking.

I agree that this would be a great feature for the other reasons mentioned above, though.

Thanks for the tip! It works as a workaround, but it's a bit cumbersome when I have more than one pixi env in a project and want to switch between them. E.g. with .pixi/envs containing prod and dev, I don't see prod and dev in the list of Python environments, and I have to type/paste the interpreter path (.pixi/envs/prod/bin/python or .pixi/envs/dev/bin/python), and only then select the environment.

Am I correct in thinking that microsoft/vscode-python#22978 will make it easier?

[pavelzw]

Am I correct in thinking that microsoft/vscode-python#22978 will make it easier?

yes, that should make it easier. i also thought that it might be interesting to implement some pixi detection in the jupyter plugin, i.e., that vscode can pixi add ipykernel if it's not available yet

[baichuns]

The plan we currently have:

• Support the global config entry `target_pixi_environment_directory = "/home/user/path/to/dir"
• Then installing an environment will directly install it in to /home/user/path/to/dir/package_name-{pwd hash}/env/default
• Symlink the dir to the .pixi in the project. But only when possible, e.g. this doesn't work on windows.
• Add it to the pixi info output.
• Add clear documentation when to use this feature and when this is bad practice.

when I set this global config entry at $HOME/.config/pixi/config.toml, I got error
WARN Ignoring 'target_pixi_environment_directory' in at /root/.config/pixi/config.toml
WARN Ignoring 'target_pixi_environment_directory' in at /root/.config/pixi/config.toml

and it still fails, can we be more specific how to fix this error? please be patient with me