[ENH] - Improve experience of developing packages on Nebari

[kcpevey]

Feature description

Now that we have disabled local pip installs, software devs who are developing packages do not have an easy path for interactive testing.

Resposted summary by @dharhas :

In the new release of Nebari we are disabling direct pip installs. There are a few reasons for this:

• Nebari is designed to be difficult to break. Python environments are read-only for this reason.
• The previous behavior of pip was to attempt to install into the read-only environment and then when that failed it installed packages in a .local folder in your home directory. Since this folder is at the beginning of your python path this means a bad or conflicting packages in .local will supercede all other versions and can potentially break every python environment you use on the platform. Sometimes it can also cause you to not be able to launch a server at all.

In the new release pip will ONLY work within a pip virtual env (i.e. a venv)

Recommended Approach for pip packages in general:

• Build an environment.yaml for your environment via the built in environment manager (conda-store)
• Pip packages can be included in the pip section of the yaml.
• Longer term we are working on a tool that can translate between pip & conda more automatically.

Workaround if you really must use pip:

• Use conda-store to build a basic environment with just python. say it is called myenv
• Open a terminal:
• list your envs to find the exact spelling: conda env list
• activate the environment: conda activate dharhas-myenv
• create a venv in a folder of your choice: python -m venv .venv_myenv
• activate the venv: source .venv_myenv/bin/activate
• install your packages via pip: pip install -r requirements.txt

Usage from terminal:

• activate the env: conda activate dharhas@quansight.com-myenv
• activate the venv: python -m venv .venv_myenv

Usage from Jupyter:

• Open a notebook with the dharhas@quansight.com-myenv kernel
• In the first notebook cell run (note that the python version might be different depending on which python you installed)
  • import sys
  • sys.path.append('/home/dharhas@quansight.com/venv_myenv/lib/python3.10/site-packages/')

I've tested this and it seems to work but I do not know how robust using this mechanism is from Jupyter.

---

@pavithraes - While I haven't tested this, I'd assume you can have a pip editable install with an active venv

@dharhas You can. --system-site-packages

Value and/or benefit

The workflow of software devs working on Nebari becomes easy again.

Anything else?

No response

[pavithraes]

Short term: Document the existing workflows for developers

Long term: Discuss improvements to workflow, for example improving conda-store CLI

[Adam-D-Lewis]

PIP_REQUIRE_VIRTUALENV=false pip install -e mypkg --no-deps seems to work reasonably well for development. Assuming you've installed mypkg's needed dependencies in your conda env.

Update: Seems like mypkg could only be installed this way in 1 environment though. If you try to do this in a 2nd environment, then it'll "uninstall" the first one.

  Attempting uninstall: mypkg
    Found existing installation: mypkg 0.0.1
    Uninstalling mypkg-0.0.1:
      Successfully uninstalled mypkg-0.0.1

[kcpevey]

PIP_REQUIRE_VIRTUALENV=false pip install -e mypkg --no-deps

This is effectively bypassing the guardrails on .local that we have in place in nebari. In other words, this is an inadvisable bypass.

Installing a dev package in this way will place it in your .local which will mean that its installed for ALL the environments. We disabled this because you may install a package for env1 but that package may be incompatible with env2. This can cause issues when Nebari is loading as well and we repeatedly saw users unable to access their instance because their .local had conflicting things in it (not sure the specifics there, but it required an admin with k9s access to go in and wipe out their .local folder).

The virual environment outlined above is a much safer solution.

[kcpevey]

I have tested the venv workflow that @dharhas described above and it works well in the terminal and Jupyter.

One thing that wasn't mention there was usage inside of VSCode. From VSCode I'm able to see the virtual environment itself as an environment option via the python plugin. Going this route you can just use the venv directly and dont have to worry with any sys.path changes.