Set VIRTUAL_ENV in Jupyter kernels

[charliermarsh]

Summary

It turns out activating the kernel does not change VIRTUAL_ENV, so we still install into the environment the Jupyter environment, rather than the project environment.

Unfortunately, after this change, we do still show a warning on uv add:

warning: `VIRTUAL_ENV=/Users/crmarsh/.cache/uv/archive-v0/3bddKDdYXuX2w57Fu6itL` does not match the project environment path `.venv` and will be ignored

uv pip install works without warning.

Closes #11154.

[charliermarsh]

@zanieb -- Do you think it's fair game to suppress that warning if UV_PYTHON is set?

[indrayam]

Thank you for this, @charliermarsh Will try it once I am home.

[charliermarsh]

Thanks so much for pointing it out. Not sure if I was confused when I wrote the docs or if something changed in Jupyter or uv.

[zanieb]

@zanieb -- Do you think it's fair game to suppress that warning if UV_PYTHON is set?

Not broadly. Like, UV_PYTHON just says which interpreter to use for creating the project environment. It doesn't change that the active environment is inequal to the project environment path.

I'll need to look more closely at what's going on here though.

[charliermarsh]

Jupyter runs in an ephemeral environment, since it's installed via --with. The project is located at .venv. The user creates a kernel for the project. We want users to be able to run !uv add or !uv pip install. VIRTUAL_ENV is set to the ephemeral environment, despite the kernel. !uv pip install fails because it installs into the ephemeral environment (so this change sets UV_PYTHON explicitly to solve that), and !uv add succeeds but erroneously warns since VIRTUAL_ENV does not equal the project path.

[charliermarsh]

The other option is setting VIRTUAL_ENV in the kernel spec, as opposed to UV_PYTHON. I don't know what the implications of that are.

[zanieb]

I see. It seems okay to silence the VIRTUAL_ENV warning if UV_PYTHON is set to the project environment path specifically / or (as a broader solution) if the requested interpreter matches the project environment, e.g., -p .venv.

It does seem more correct to set VIRTUAL_ENV if we can.

[charliermarsh]

We can, yeah.

[indrayam]

Thanks so much for pointing it out. Not sure if I was confused when I wrote the docs or if something changed in Jupyter or uv.

I absolutely adore "uv". And since NOTHING I do these days does not use uv, Jupyter related challenges had been bugging me. Anyways, about to try your changes. Will keep you all posted.

[danielhollas]

Btw: the warning situation is somewhat similar to what happens when uv is run from pre-commit

astral-sh/uv-pre-commit#36

[indrayam]

Thanks so much for pointing it out. Not sure if I was confused when I wrote the docs or if something changed in Jupyter or uv.

I absolutely adore "uv". And since NOTHING I do these days does not use uv, Jupyter related challenges had been bugging me. Anyways, about to try your changes. Will keep you all posted.

I recreated a new project and created a new ipython kernel using the updated uv run ipython kernel (as per @charliermarsh's commit diff). Everything worked like a charm! And by setting the VIRTUAL_ENV environment variable at the kernel level, there were no warnings that I saw, not even when I ran the !uv add command. Not sure if that was supposed to be the case.

[charliermarsh]

Awesome, thank you for testing! It's good to see that the warning is gone too.

[indrayam]

Thanks so much for pointing it out. Not sure if I was confused when I wrote the docs or if something changed in Jupyter or uv.

I absolutely adore "uv". And since NOTHING I do these days does not use uv, Jupyter related challenges had been bugging me. Anyways, about to try your changes. Will keep you all posted.

I recreated a new project and created a new ipython kernel using the updated uv run ipython kernel (as per @charliermarsh's commit diff). Everything worked like a charm! And by setting the VIRTUAL_ENV environment variable at the kernel level, there were no warnings that I saw, not even when I ran the !uv add command. Not sure if that was supposed to be the case.
...

Btw, on a somewhat related note, I am not entirely clear how uv run --with jupyter jupyter notebook invocations at times creates a new folder inside $HOME/.cache/uv/archive-v0/.. with all the jupyter related dependencies and at other times reuses the existing one.

Last evening, as I was dabbling with this, I saw uv run --with jupyter... create a folder under .../archive-v0/. This morning, when I decided to rerun the same command as I continued my work, I noticed it created a brand new folder and redownloaded Jupyter. Moreover, the older folder was not there anymore.

I understand that since it is inside $HOME/.cache/uv/.., it is ephemeral. I was just curious, especially about the logic you are using to clear the existing Jupyter-related dependencies folder from the cache.