How to Install local packages in Nebari

[empollon9]

Hi all,

I hope someone can help me. I am creating an environment on conda-store, but there is a proprietary package I need to install with pip. I have seen the documentation on here, but it does not cover my scenario. I have my proprietary package in Azure Devops Artifacts which is normally installed with:

pip install mypackage==x.y.z --index-url https://mypackage:pat_asdf@pkgs.dev.azure.com/{organization}/{project}/_packaging/{mypackage}/pypi/simple/

where:

• mypackage is the package name
• pat_asdf is a PAT key
• {organization} is your organization code in Azure
• {project} is you projectname in Azure DevOps

I have tried the following environment .yaml file

name: testenvironment
channels:
- conda-forge
- defaults

dependencies:
  - python=3.11.9
  - pip
  - pip:
      - --extra-index-url https://mypackage:pat_asdf@pkgs.dev.azure.com/{organization}/{project}/_packaging/{mypackage}/pypi/simple/
      - mypackage==x.y.z

and this works locally fine on a PC with conda installed, using
conda env create -f environment.yml

However when I try this on conda-store on nebari I get the following error:

  File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/packaging/requirements.py", line 37, in __init__
    raise InvalidRequirement(str(e)) from e
packaging.requirements.InvalidRequirement: Expected package name at the start of dependency specifier
    --extra-index-url https://mypackage:pat_asdf@pkgs.dev.azure.com/{organization}/{project}/_packaging/{mypackage}/pypi/simple/
    ^

I have tried other workarounds, like download the package and upload to nebari to install pip locally using -e flag but this seems not to be supported.

Any ideas? I am open to workarounds, as with this constraint we will not be able to roll out nebari.

Many thanks

[dharhas]

I think this is a case where the conda-store linter that checks whether an environment.yaml is valid needs to be extended to allow the 'extra-index-url' syntax.

In the meantime, a potential workaround is to specify an environment variable in the variables section on the yaml. From the conda docs, I think you can set PIP_EXTRA_INDEX_URL.

[kcpevey]

Unfortunately, the only env var that conda-store currently uses is CONDA_OVERRIDE_CUDA (conda-incubator/conda-store#826) which is for enabling GPU environment builds.

[empollon9]

Thanks for your reply. I looked at that conda-incubator/conda-store#826 and there is a completed Pull Request, so is it not live already?

I tried adding the variable, and yes this did not work, it just get ignore and looks for the package in pypi

Traceback (most recent call last):
  File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/solver.py", line 233, in _solve
    result = resolve_version(
  File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/__init__.py", line 7, in resolve_version
    return solver.solve()
  File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 83, in solve
    self._propagate(next)
  File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 123, in _propagate
    root_cause = self._resolve_conflict(incompatibility)
  File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 321, in _resolve_conflict
    raise SolveFailure(incompatibility)
conda_lock._vendor.poetry.mixology.failure.SolveFailure: Because -dummy-package- depends on mypackage (=x.y.z) which doesn't match any versions, version solving failed.

looks like mypackage name already exists in pypi, although the version I have does not exists for this package, and hence the error

[kcpevey]

There is no PR linked to resolve conda-incubator/conda-store#826 - the cross referenced PR that you see there is to the PR that added the CONDA_OVERRIDE_CUDA variable (only).

[kcpevey]

I could be wrong here, but just wanted to also mention that it looks like this error may be coming from conda-lock (which is the first step in a conda-store environment solve), based on this issue: conda/conda-lock#474 .

The only workarounds at this point would be:

• setting the flag to re-enable pip and installing it manually, with all the caveats that involves (see warnings here)
  • You can set this in the terminal by doing export PIP_REQUIRE_VIRTUALENV=false and then pip install -e . --no-deps. I can't stress how important --no-deps is here. Because ALL environments will pick up whatever you pip install, you can VERY EASILY break everything. So much so that you're user won't be able to spin up a server. This happened enough times that we felt we needed to disable it as it requires a k8s admin to go in a clean out your .local to allow you get on Nebari again. Consider yourself warned :)
• file issues on conda-store/nebari and we will put it in the backlog to fix (no estimate on timeline but we will get to it)
• reach out to Quansight for a support engagement so we can prioritize a fix.

[empollon9]

Thanks, I have use the export PIP_REQUIRE_VIRTUALENV=false and then pip install --no-deps. I did not want to be an editable package, just stand alone. This copied the file and seems to be working fine, although I would prefer this to be managed by conda-store. I will put that as in issue, and I can volunteer to help if I got some guidance

[kcpevey]

Contributions are certainly welcome! The fix for this will be to resolve the conda-store issue: conda-incubator/conda-store#826